They have smallweb and news indexing, but other than that AFAICT they rely completely on other providers. Which is a shame, Google allows submitting sites for indexing and notifies if they can’t.
Running a scraper doesn’t need to cover everything since they have access to other indexes, but they really should be developing that ability instead of relying on Bing and other providers to provide good results, or results at all.
Small web always returns 0 results for anything that isn’t extremely broad, unfortunately.
I’ve been using Kagi for the last year+.
Personally, I wish they’d tone down the AI stuff that ruined Google, but at least you can turn most of it off.
Their results are okay, a little better than Bing, but obviously they’re limited by their existing index providers, I wish they’d run their own spiders and crawl for their own data, since I think Bing fails on a lot of coverage of obscure websites.
In general I find the weighting of modern indexes to be subpar, though the SEO industry has made it a hard problem to tackle, I wish more small websites and forums were higher ranked, and AI slop significantly de rated.
Also not a huge fan of the company and a lot of it’s ardent customers, who heavily protested a suicide prevention popup if you used it to searched for how to kill yourself.
Matrix is probably something worth looking at, at least from an intellectual standpoint, for you. It uses shared message state and a DAG, plus some fancy perfect forward secrecy (using Signal’s Double Ratchet algorithm), which is at least interesting. There’s also Tox (chat/protocol) if you want totally distributed chat.
Personally, I really like distributed models from a theoretical standpoint; but for end-user applications they pose very difficult constraints, we live in a world with ⪅50% publicly routed IP for one, they fundamentally require immense data replication, latency in peer-finding, bandwidth constraints, and ultimately sub-par UX. I thought IPFS with a way to pay nodes to pin content was a really neat idea, but hasn’t caught on, for example. Not to discourage you, if you think it’s workable then have at it, but I think it at least explains the current state of things.
I do think the other home server implementations gaining parity (production-ready) with the reference home server would go a long way. I haven’t run a home server but I’ve heard from those that have that it really has a hard time scaling. (Though this serves as impetus to give it a try over spring break)
Which brings me to the caveats of the protocol, I personally don’t think the design is ideal, it’s more described as a distributed message bus, what I’ve read of the spec it’s over engineered, it made good decisions wrt using modern web technologies (JSON, WebRTC), but it didn’t scope itself to the particular task.
That said, I haven’t written a federated protocol, and they have. But if I was going to, I’d really want to look at Discord and see how to copy a lot of that model, but break parts of it out to facilitate federation:
I originally wrote a huge hypothetical design here that I speculated would fare better, but honestly the specifics become less relevant, point is that the shared state of rooms is a real challenge, and one out of scope for just a federated instant messaging system, and I’m no longer certain it’s viable.
I really wish Matrix had been more successful, but it has some pretty core problems that prevented it from gaining more traction.
It fell into the same trap as XMPP, though perhaps even worse, with a focus more on its protocol and specification than a single unified product vision. The reference server implementation is slow, and using a language not optimal for its purpose, with alternative server implementations left incomplete and unsupported. It took a long time for them to figure out voice and video and for it to work well, and the “user flow” still isn’t at Discord levels.
I’ve rooted for Matrix for a long time, but as a former XMPP evangelist, to me the writing on the wall says it isn’t suited for success either. I’d love to be wrong, but I don’t see a way through.
No thanks, people hop from centralized platform to centralized platform thinking things will be different.
Consolidated power, especially when fully captured by market forces, will always enshitify, the only out is federation.
If they were a small or free service I wouldn’t have much issue, but they do charge, I don’t think it’s too much to ask that they at least attempt to scrape the wider web.
Building their own database seems the prudent thing long-term, I don’t doubt they could shore up coverage over Bing. They don’t have to replace the other indexes wholesale, just supplement it.