Vlad from Kagi here, thanks for posting. The RSS feed unexpectedly broke (edit: the feed is back up! [3]) just as we published the blog post, in the true spirt of "small web" :) Should be up in 30 minutes which will enable the site to function too (it uses the same feed).
This has been a personal pet project of mine and I spent considerable time getting my hands dirty with the code, as the team was busy with other initiatives. When I said the "feed broke" for the launch I meant I broke it. Software is messy especially for an old school dev. I learned in the process I am not a very good coder anymore (if I ever was one?), constantly going back and fixing stuff I previously thought was solid. Check it out in the linked repo [1].
Most importantly - I found the site replace the need for discovery for me, and getting to know various different humans and their writing felt good! A lot of unexpected stuff surfaced and the web felt close again. I think there is a glimpse of hope in the concept and I hope you see it too. And the improvements to search quality and diversity this brings are real.
You can check the list of included websites here [2]. And all the recent posts already surface in Kagi results (for relevant queries).
Why do you let YouTube channels have advertisements/sponsors but require sites to be ad-free?
I understand the spirit of it and don't have any counter examples but seems like a bummer if someone has a nice indie blog but can't be added because they have a few ads or a sponsored post.
Most of the focus was on the website curation. The YT channels were curated from the initial list of sources [1] and then automatically filtered by recency and subscriber count criteria.
That is to say that very little other effort was made to curate the YT channels and we expect the user community will contribute to edit the list.
Also I was under impression that all YouTube channels had ads, so that is why this was not considered as a seperate criterion.
Thanks for clarifying. Yeah we should establish a criterion around that. Curating YT is much more work it feels like. Let's see how "Small YouTube" [1] feels like to begin with and we can take it from there.
By the way, you can perhaps check https://nebula.tv for a list of good YouTube channels (not all of them are great IMO, but a lot of people who make content on Nebula fit the "small web" criteria)
Now that I think of it, perhaps you can call Nebula "the Kagi of video streaming". Maybe you guys could collaborate somehow in the future ;D
I kinda wish there was a sort of proper blog-DMOZ. Would make this sort of stuff a lot easier. Right now everyone and their grandma is curating their own blog list. Lots of wasted effort that could be saved by collaborating somehow.
Because curation to keep spam out would be work. And how do you prevent trojan horses, where an initially promising blog is turned over to SEO once included in the official dmoz?
There’s also the issue of quality. Do you just include everything that is a blog, no matter how low quality it is?
And if the answer to that is no, then where do you draw the line when it comes to quality?
Curating anything is complicated and different people will have different opinions which is why you end up with different lists curated by different groups of people with different ideas about what should and shouldn’t be included.
I agree but it’s incredibly hard to collaborate on this kind of projects because everyone has their own take on what should and shouldn’t be included.
Curation is still hard. Also, I personally don’t mind having different curated directories because those will reflect the taste and interests of the curator.
I figure collaborate on answering the easy questions, is the website online, is it chock full of spam, is it a blog about horses or about keyboards, etc. Some of that could be automated (like onlineness, large change detection, etc.), but some of it needs manual supervision.
Right now everyone who is running any sort of curated discovery service needs to answer these same questions about roughly the same site.
ooh.directory gained some traction a few month ago [0] - and they are steadily adding blogs although they do it slowly. Manually reviewing and trying to have some diversity is no easy task.
Isn't this more likely to have been inspired by marginalia* than a personal blogs thread? Doesn't seem to have the same results from the Apple Watch example, but it's what immediately came to mind for me when I read this post's title.
Vlad was dabbling with similar things to Marginalia with Teclis around the same time Marginalia first made headlines.
And to be fair, I was simultaneously inspired by the blog thread to build a curated blog filter for my search engine, which led to a series of changes that overall tends to promote more of this types of results in general.
The fact that we're several to have similar ideas sort of validates the ideas I think.
I'm not complaining, glad there's more tools for searching outside the scope of SEO spam or content mills, and competition can only make the niche better (presumably, unless it gets big enough to incentivize disguising commercial content as niche blogs). Just felt like it might be a bit disingenuous to quote a blog list as inspiration for a search engine focused on "the non-commercial part of the web" which is exactly your engine's aim and wording, didn't know they've been working on small web initiatives since a similar timeline.
Hopefully you both find success and with it uplift hobbyist websites.
I really enjoy that this feature got released. Adding more visibility to the indie/small web is a great thing. I've complained in the past on HN that any useful personal blog is buried under spammy content in search engines.
It would be nice to also be able to just search within the small web, maybe using a lens in the future?
Regarding the topic of self promotion, I would disagree with the current rules and I would ask you to allow people to self promote. As long they have an old enough blog, maybe even cut that down to a year, would be helpful. Most users on the small/indie web lack visibility and this would do them service. My blog is already within the index because I think it might have been picked up during the "HN share your blog" post that happened a while ago, but others might not have been that lucky.
There is already a lens called the non commercial web or something that I suspect overlaps to some degree,but I don't think they are equal and I don't know if I am right.
Vlad helped me troubleshoot why my Orion browser wasn't syncing bookmarks (turned out to be a config issue on my device) and we got the fix added into the official documentation.
Imagine getting that level of support for any other web browser!
The more I uses Kagi (paid) the more I am feeling sad for the possibility that it might go away if it doesn't get more traction.
For me you represent an incredible accomplishment: the first search engine that gives better results than Google, respects privacy, offers customization and so much more.
Just to say: the search engine frog has long been boilt and it is amazing what crap people have to put up with when searching for anything. How jarring the experience of seeing TV commercials is after years of Kodi, is how the experience is of moving back to an ad-supported search engine after now months of Kagi.
Youtube doesn’t show ads in my country. Anytime I use a vpn or I travel, I can’t believe how bad the free tier is. My twitch consumption has gone way down because of the ads. Watching the last NBA season was so annoying. I felt more than half of the broadcast was ads.
Semi-unrelated, but is there any reason you don't make your search API as turn key as the other APIs?
It's expensive enough that I can't imagine anyone repackaging it profitably (2x Bing search prices for me) but having to email someone adds just enough friction to discourage a lot of tinkerers from even trying it.
I have been developing a "small web" static site, forked and customized a templating engine into a static site generator just for my site. Even wrote a new post recently, last week or so. How can I add it to the list? The GitHub says to make a pull request, is that really all there is to it?
Edit: just saw this:
> Do not submit your own website.
I see. I'm okay with that. Maybe it will show up there one day.
Are English "Small Web" results included even if I use the "Sweden" search region?
I noticed the example result for useyourloaf wasn't included if I switched it to "Sweden" and not sure if this is just an oddity or if the entire feature is nerfed because I just leave my locale on all the time.
Kagi is worth every penny. Been using it for 6 months as my primary search engine. I think I have used the Google !g 10 or so times in that period maybe?
The ability to block SEO garbage sites like GeeksForGeeks and not filling the first 3-5 results with ads is worth it alone. Not to mention the ability to boost certain sites results over others on a personalized level. Unfortunately for me, I end up regularly going over the "Pro" tier of searches a month (partially because I accidentally search all the time, but also I'm heavy search-engine user) is a bit of a letdown. At least they have an option to purchase additional searches instead of going up to the rather expensive unmetered tier.
> I end up regularly going over the "Pro" tier of searches a month (partially because I accidentally search all the time, but also I'm heavy search-engine user) is a bit of a letdown.
I hadn't considered this aspect of Kagi yet. I'm not a subscriber at this point, but I am strongly considering it. But I use search instead of typing domain names directly to avoid the typo phishing style attacks. I wonder how much "artificial" search that would generate based on my typical usage.
Bookmarks were my obvious answer as well, but bookmarks are annoying to me. I don't like managing them and I don't like having the bar visible and I don't like having to remember to use the bookmark for things that have a bookmark, and the addressbar for things that don't have a bookmark.
It's a work-around, not an actual resolution, unless you already happen to like using bookmarks.
This is my approach: address-bar autocomplete only uses bookmarks as suggestions (well, and open tabs). I almost never open my bookmarks, nor feel the need to organize them. I'm really pleased with the way Firefox has implemented that autocomplete: by default, Firefox autocompletes the domain names (with typeahead) while you're typing, while full bookmarked url's are just a down-arrow away, sorted by most recently used. It means I never have to bookmark bare domain names: as soon as I have a page bookmarked on a domain, Firefox will also allow me to quickly navigate to the bare domain site. And for individual pages, I can just type two or three letters (not necessarily the first few) to directly identify the page that I want to go to.
Obviously that means I'm using split url/search fields in the address bar, or that wouldn't work as easily. I also heavily use Firefox' keyword search to avoid doing a round-trip through a search engine if I already know which site I want to search; mostly using the same prefixes that DDG uses, so I don't need to adjust too much if I find myself using a different computer.
I have actually already split my address and search fields too, for a different reason. I got aggrevated with having the local machine names on my lan perform a public search, even though my local dns is configured to use only my opnsense router, which does resolve the local names correctly. Splitting off the search box makes the local names always do only what DNS says, and so I get my 3d printer instead of search results for it's name.
For that particular problem, going to "truenas" or "unifi" etc on my lan, the bookmarks are actually more convenient in nmost cases I have to admit! Especially with half of the services needing some special port number like unifi and jellyfin etc.
I use Kagi all the time for this kind of thing, and always wondered if it might be an issue. But in reality I don't actually end up using as many searches as I thought I would, like at most it's 30 a day (according to Kagi's stats), and even that only adds up to ~900 searches in a month. Always assumed it would be higher since I'm constantly searching stuff for my job, but I guess 30 is kind of a lot when you think about it.
So this may not be as big of an issue as you think. Should also keep in mind that higher search quality means you don't have to search as much in the first place, which leaves a larger buffer.
Alternatively you could just use "hashbangs" to search domains via google (!g), since I don't believe those cost you anything.
I haven't used bookmarks or even had the toolbar visible in 20 years, but only because of kagi I now had to add a bookmarks toolbar back, populate it, and consciously remember to use it for anything that it has, and only use the address bar for things it doesn't have.
This is not the worst thing in the world, but it is annoying to me and not a life upgrade.
Maybe the search results are better enough to make it worth it (on top of also paying money), or maybe not. I'm still determining that, but it's question and not a slam dunk.
> I end up regularly going over the "Pro" tier of searches a month
I'm now halfway through my billing period (22 aug - 21 sep) and I'm on 488 searches. Would be tight on the pro plan (1000 searches), but I'm still on the early adopter pro plan (1500 searches).
I don't understand the pricing. (Well, I do understand, I understand that it's the same anti-consumer gimmick as gym memberships.)
On the $5 tier it's 1.7 cents per search for the first 300, then 1.5 cents after that. As expected I blew past the 300 before the month was out and am currently sitting at a total of almost $9 for this month and there are still 3 more days to go, and this includes being away for labor day weekend and only using my phone (without kagi) for a couple days, and this is after populating and using a bunch of bookmarks specifically to cater to the fact that I now pay for searches. And this is only my laptop. I have not used kagi on my phone or anything else yet.
I haven't used bookmarks in 20 years and don't particularly want to. I normally don't even have the toolbar visible but now I had to un-hide it and add that clutter back to my browser.
So I'm both paying money and contorting my usage pattern.
I guess now that the first month is about done, I can say it looks like I should go up to the $10 plan, where the searches are only 1 cent, but only for the first 1000, and only if I actually use all 1000! If I pay $10 and only do 250 searches that month, then they weren't 1 cent were they?
As much as I like it, I don't know if I'm going to keep it.
I will not pay $10 or $25 just to have it sitting there available "in case", and I apparently will at least some times (who knows how often? every month? 3 out of 12?) will blow past the $5 an end up paying $10 anyway.
If I complicate my usage to cater to kagi so that my default is ddg and just invoke kagi sometimes when I feel like it would help, then I'll probably forget it exists most of the time and do about 10 per month and pay 50 cents each. Probably only one or two months of that and I'll just decide it's not worth $5 for a handful of searches and just cancel it.
The only way it will be useful for me is if it can just be the default search that I don't have to worry about.
They should just figure out whatever the fair per search price is and bill that.
The stupid tiers are probably going to drive me off.
You shouldn't need to be counting them. It was a necesseity to make sure we stay in business. We figured out ways to reduce costs in the meantime and we will be going back to $10/mo for unmetered searches this month.
I appreciate that. As long as you're still hashing things out I'm willing to give it more time. I really like the small web concept. What I would like most is not eb=ven $10 unmetered, or even $2 unmetered, but simply n$ per transaction, as long as you are obviously doing the work of counting them anyway.
$10 unmetered must always be a fallacy anyway. Surely I can't start up "bkwsearch" and sell searches for 3 cents each, and pay you only $10 for my million$ skin.
So don't even pretend.
Except really there probably needs to be both options, because some people value predictability more, so $10 or $20 or any number that never changes is better than paying only $2 on average but unpredictable each month.
I neither want to pay $10 for $2 worth of usage, nor soak you for $30 worth of usage while only paying $10, and even more than that I don't want to have to think about it to make sure if I choose the N tier that I'm making more or less full use of the N tier.
I appreciate that you took the time to read an individual users complaint, whatever you end up doing. Thanks.
one nice feature I found recently: you can actually search special characters: R "%<>%" actually explains operator, rather than linking to random websites about R.
Another very satisfied Kagi user here. It's totally worth the money, and a great example of the kinds of services that are possible if you're just willing to pay a little for 'em
...because I wasn't sure if this was just nostalgia or fake memories so I went down the rabbit hole of trying to refresh my memories about why I switched from AltaVista to Google
I had some memory of AltaVista being full of SEO (as I recall it was mostly a keyword search engine) spam.
1. AltaVista was slow and you needed to know your syntax to find things
2. AltaVista was cluttered. Google had very clean results.
I think on 1 google is still fast, but 2. not so much. Google's results are far from "clean", instead it feels like the main goal is showing you as much ads as possible and preferably getting you to accidentally click on one.
This in and the fact that Kagi results are as good or slightly better (and much more customisable) than Google is probably the main reason I'm so happy with Kagi.
> instead it feels like the main goal is showing you as much ads as possible and preferably getting you to accidentally click on one.
This is an interesting observation given Google's roots. Although it is a different era, where we have ad blockers now, it would be really weird and neat if Google were displaced by a new search engine that did literally the exact same thing Google did in 1998: clean, no-bullshit page that just gives you search results + page-rank.
Unfortunately that's pretty improbable due to Google's dominance and the changes in the tech landscape. When non-computer-literate people are making Google searches every day on their phones because it's pre-installed as the default, that's a difficult fight to win.
They will stop using Google because of the disastrous quality. The question is if they'll pay for another search engine or just search within Instagram, or install an AI agent.
I like it but a month after paying for it the subscription model was changed significantly to be way more expensive. I just can’t let myself become depending on something with such capricious business practices. Maybe it is a positive indication that they are better at product than they are business and customer relations.
My only worry is random searches, where I default to a "free" bang (!g, !b, etc.) to not burn through one of my paid searches. I know it's a mental thing, but it's still a thing.
I love it, though my only complaint is that I usually want sites in English from the US, but if I want to switch to finding sites in French in France, I have to switch regions. Google is much better at localizing me, and so I'll use !g for those.
It is such a weird experience to do a Google search on someone else's computer after being used to Kagi. I recently requested a small usability enhancement, and it was implemented within a few weeks. Zero chance of that happening with any other search engine.
The search results are consistently better than anyone else's, including DuckDuckGo, so I am, and will remain, a happy paying customer.
It’s hard to get specific examples, but I was happier with the results back when I tried it in beta (after several years of using DDG exclusively and only very rarely needing the !g escape). It’s not always (pure default, things change with personalization like lowering/blocking/raising/pinning) a giant difference, but simply the result you want being further up, and fewer useless filler results between hits.
To give a slightly less positive perspective: I trialed Kagi for a few weeks, and while I liked the features and their business model, its search results are no better than from my own Searx instance. Kagi essentially does the same thing as Searx: anonymized API calls to 3rd party search engines. It's just packaged in a friendlier UI, but the experience is not far off.
Plus, Searx supports many more search engines, and I can customize it exactly to my liking.
I wish them well, as they clearly have good intentions and a good product, but I prefer using an equivalent OSS and self-hosted solution over a proprietary SaaS I have to create an account for, even if it's not as polished or featureful.
EDIT: Actually, I'm wrong. Kagi apparently also has their own crawlers and indexes[1]. Still, I'm not finding Searx results to be deficient, so I'm not missing out on much.
One thing that does concern me with Searx, and partially with Kagi, is that those 3rd parties could decide to block these API requests at any point, leaving Searx unusable, and Kagi's results less relevant. I'm not sure this is a sustainable way to build a search engine, but I do appreciate both Kagi's and Searx's stance on ads. Using any mainstream search engine via their own frontends is a frustrating experience at best.
I’m mostly happy with ddg but I wish they had a feature allowing me to disable certain sites by default like kagi, even if that information was stored in local storage only and I’d have to add the exclude list manually.
There’s not much to reinvent. They could have setup & hosted an XMPP server with MUCs & microblogged posts on whatever ActivityPub option they chose (or similar softwares).
Instead of there being one central authority where you create your "Mastodon account" that is responsible for authentication and deciding who gets to view and post what, there are thousands of independent 'authorities' each deciding who they want to let on and how they want to do things and what subset of Mastodon they want their users to see. Much like Usenet and IRC
Rooting for small web does not mean one needs to be against other technology.
For example Kagi uses Google and even hosts on GCP - I think Google's technology and people are great, it is just the business model that is rotten and contributes to the deterioration of the web.
And interestingly enough, at least Discord (and to some extent Twitter) are trying to have a business model that does not put more ads down your throat (although admittedly I did cancel my Twitter subscription as unexplainably they still showed ads even when subscribing - you can't sit on two chairs).
Are you on Mastodon or on the fediverse btw? Any plan for Kagi to have an account there? (not really required, I don’t think that companies should necessarly be on social networks, that’s mostly a waste of time)
They also mentioned their Microsoft GitHub forge to file issues. They really want you to create proprietary accounts to interact with them. You’d think a small web initiative would be led with small, decentralized/federated, libre software choices.
This is quite interesting. I feel you should include content from blogs that publish less often than once a week - some of the best blogs post once in a blue moon and are still quite interesting.
We do as a part of our own search index, just not as a part of this particular initiative where we wanted a focus on the fresh stuff from the small web "oven".
This is really good. With more and more of the large web filled with SEO crap and gargantuan articles written because that's what google wants, it is great to read from the people who actually write the interesting things the AIs train on.
Things like this are why I keep rooting for Kagi. I haven’t purchased a subscription yet because I’m still on my free 100 queries, but I will as soon as they run out. I really love this project!
It sounds like it’s integrated into Kagi’s main results and you can use the API to reference only the small web results, but I also don’t see a way for a Kagi search user to toggle “small web” results as a lens or whatever. I assume and hope that may be added in the future.
#1 What's the rationale behind favoring recent blog updates? In my experience, recent updates make the weakest search results, more prone to updates and link breakage, and overall tend to be of lower quality. I also wonder if promoting recent content might incentivize pumping out low-quality entries to increase the odds of being listed.
#2 In dabbling in the domain I've always ended up with an almost absurd skew toward technical programmer:y blogs. While there is a strong overlap between the cohort with a blog, and the cohort with programmer interests, I feel it would be more inviting to other groups if other interests were better represented. Is this something you've thought about, and if so, what do you think might be done?
- More recent content tends to be more relevant given the same search query, or at least its freshnes will contribute to it not being completely irrelevant (which is the worse thing you want in a search engine). The quality of it is already guaranteed to some extent by this being a curated list to begin with.
- It was relatively easy to assemble and maintain the list because it relies on RSS feed tech, and there were a lot of sources to seed it.
- Focus on recent writing can encourage some people to write (more) as we had an example highlighted in the blog post. In general, the web needs more high quality, non-commercial content, and this is ultimately what we want to contribute towards with this. By providing a platform (even if very small) to encourage this behavior we get a step closer to the web we like.
#2 In general I agree. Although I should say we did spend effort to create a diverse pool of websites for this initiative (for example I am seeing a lot of economy or photography). Again, we can only encourage the creation of more content in various areas through platforms like Kagi (and Marginalia) and hope that it will work out at the end.
Unless I've missed something, they aren't doing that - the mention of a three year time frame is that you must have made a post on the blog within the last 3 years, this excluding old and dormant blogs
That is a requirement for it to be featured in the KSW website (as it uses iframes to embed). We will still crawl it and surface in search results regardless..
Vlad, you're awesome! It's hard to tell to what extent it is actually beneficial for users or your company, but I feel warm and fuzzy inside knowing these kinds of things still happen in modern Internet.
Hah, thanks. It was one of those things that I just wanted to do and I actually wrote 100% of code for this end to end which took me a good part of a month. Not the best use of CEOs time you might say, but it was fun and felt right.
This sort of stuff makes me really happy to be a Kagi subscriber. Not only do i get value out of Kagi, but this shows me that the money is being used to develop Kagi in a way i agree with. By comparison, Spotify (just picking one of my subs) feels hostile to me. I pay them, but would cancel in a heart beat if i felt i had options.
I really appreciate Kagi's development matching what i feel like i'm buying. Thanks Kagi Team <3
Music streaming is such a mess. I've tried a ton of services in an attempt to find one that's decent, and have found embarrassing problems for all of them.
Spotify: Really hostile, manipulative, and terrible to artists. On my end I hate how commercial the homescreen is, and the CarPlay interface is just a user-hostile to a degree that is frankly unsafe disaster.
TIDAL: Pretty good in a lot of ways. They pay their artists well. The recommendations are decent. The apps have some really stupid bugs that have persisted for years though, the most annoying of which is that if you shuffle a playlist it only shuffles the dozen or so tracks that the interface had pre-cached from the top. So if you try to shuffle your full library you wind up just hearing the same dozen songs only, over and over again.
Deezer: Wanted to like it, but the apps aren't great, and I ran into more missing tracks than I'd like.
So I finally settled on Apple Music. I've got an iPhone, so it's a natural fit there, and the CarPlay interface is great. They also pay artists almost as well as TIDAL. The recommendations are super good enough, and I don't really feel like the home page is constantly trying to push me to whatever the huge labels are paying them to promote (damn you Spotify). The Windows apps are terrible (like flat out embarrassing), and the Linux apps are non-existent, but luckily there's a pretty great open source app called Cider that solves that.
> The Windows apps are terrible (like flat out embarrassing), and the Linux apps are non-existent, but luckily there's a pretty great open source app called Cider that solves that.
This is what blocked me from trying Apple Music as well. Maybe i'll give Cider a look, thanks!
spotify's homescreen does my head in as well. im paying them but they still shove ads in my face. ive been meaning to try spicetify for a while now, it let's you customise the desktop ui so you can probably hide the homescreen completely
> These notes will vanish in about a week as we cycle in new content - emphasizing the fleeting, imperfect nature of the small web.
Kagi could just admit they don't want to moderate notes or store them permanently. No need to push down the small web, because a lot of small sites preserve their content.
I get that Kagi probably has data indicating the reality of how often sites down, but it seems from my experience that content in big platforms disappears often as well, even in the cases where the creator hadn't forgotten about it. The "Small web" websites made by a creator that cares have the room to be much more permanent.
It would be nice if Kagi Small Web would have an ActivityPub interface so that the most appreciated sites of a day could be added to a timeline on mastodon or lemmy.
Great initiative, but it feels like they're not quite eating their own dogfood. This post contains a link titled "highlighting blog posts from HN users" which goes to Twitter — a massive centralized social media app that's pretty much the opposite of "small web".
This sort of centralized, hand-curated list makes me nervous. The beauty of the web is that it’s open, you don’t have to “apply” or meet “criteria” for your website to viewable by other people. Obviously the status-quo (of Google Search) isn’t great, and I’m glad Kagi is trying to fix this problem. But this isn’t going to scale.
I can't see how it will scale, but in the immediate present, it seems useful and good to me. I might even prefer if this small web remained small (though that's likely... unappealing to Kagi's investors).
It looks like you're stripping MathML out of the RSS feed -- is that intentional, or are you using an older sanitizer that doesn't recognize it? For example, my RSS feed [1] for my most recent post [2] has:
I find this kind of domain name reuse unsettling: for years, Kagi.com was the address of a payment service for shareware authors.
Man those were the days. Lots of people were perfectly willing to pay me $20 for a useful little utility app and now they won't pay $2 in an App Store.
> Lots of people were perfectly willing to pay me $20 for a useful little utility app and now they won't pay $2 in an App Store.
what kind of utilities?
I am happy to pay $2 (or more) for tools I use if it is a one time payment or a payment for tokens (e.g. $n for $m ocr scans, generated images or whatever).
I loathe apps that demand monthly payments unless it is really understandable why they have to have it that way (service that require permanent storage comes to mind, although I think I only use iCloud for storage now).
Kagi appearently had a project "expertGPT" (contrast to FastGPT). Does anybody know what happened to that one?
On a side note, there is now a - to my best knowledge - completely unrelated product "ExpertGPT" from some totally different company. I am not talking about that one.
Huh I wonder that too. Seems it hasn't been mentioned on the forums for a few months. So maybe they decided it wasn't worth it to keep that experiment running.
190 comments
[ 5.2 ms ] story [ 233 ms ] threadThis has been a personal pet project of mine and I spent considerable time getting my hands dirty with the code, as the team was busy with other initiatives. When I said the "feed broke" for the launch I meant I broke it. Software is messy especially for an old school dev. I learned in the process I am not a very good coder anymore (if I ever was one?), constantly going back and fixing stuff I previously thought was solid. Check it out in the linked repo [1].
Most importantly - I found the site replace the need for discovery for me, and getting to know various different humans and their writing felt good! A lot of unexpected stuff surfaced and the web felt close again. I think there is a glimpse of hope in the concept and I hope you see it too. And the improvements to search quality and diversity this brings are real.
You can check the list of included websites here [2]. And all the recent posts already surface in Kagi results (for relevant queries).
[1] https://github.com/kagisearch/smallweb
[2] https://github.com/kagisearch/smallweb/blob/main/smallyt.txt
[3] https://kagi.com/smallweb
I understand the spirit of it and don't have any counter examples but seems like a bummer if someone has a nice indie blog but can't be added because they have a few ads or a sponsored post.
That is to say that very little other effort was made to curate the YT channels and we expect the user community will contribute to edit the list.
Also I was under impression that all YouTube channels had ads, so that is why this was not considered as a seperate criterion.
[1] https://github.com/kagisearch/smallweb#youtube-channels
[1] https://kagi.com/smallweb/?yt
Now that I think of it, perhaps you can call Nebula "the Kagi of video streaming". Maybe you guys could collaborate somehow in the future ;D
So happy so see the small, independent and more humane web being highlighted.
I’m trying to do my part[0] but I have no doubt that a search engine—even if still a niche one— can have a much bigger impact.
Really well done Vlad!
[0] https://peopleandblogs.com
Could be done, but it'd take time and effort.
And if the answer to that is no, then where do you draw the line when it comes to quality?
Curating anything is complicated and different people will have different opinions which is why you end up with different lists curated by different groups of people with different ideas about what should and shouldn’t be included.
Curation is still hard. Also, I personally don’t mind having different curated directories because those will reflect the taste and interests of the curator.
Right now everyone who is running any sort of curated discovery service needs to answer these same questions about roughly the same site.
[0] https://news.ycombinator.com/item?id=33719983
*https://search.marginalia.nu, commonly mentioned on HN https://hn.algolia.com/?query=marginalia
And to be fair, I was simultaneously inspired by the blog thread to build a curated blog filter for my search engine, which led to a series of changes that overall tends to promote more of this types of results in general.
The fact that we're several to have similar ideas sort of validates the ideas I think.
I'm not complaining, glad there's more tools for searching outside the scope of SEO spam or content mills, and competition can only make the niche better (presumably, unless it gets big enough to incentivize disguising commercial content as niche blogs). Just felt like it might be a bit disingenuous to quote a blog list as inspiration for a search engine focused on "the non-commercial part of the web" which is exactly your engine's aim and wording, didn't know they've been working on small web initiatives since a similar timeline.
Hopefully you both find success and with it uplift hobbyist websites.
why not Matrix? I think that's a more realistic alternative. :)
It would be nice to also be able to just search within the small web, maybe using a lens in the future?
Regarding the topic of self promotion, I would disagree with the current rules and I would ask you to allow people to self promote. As long they have an old enough blog, maybe even cut that down to a year, would be helpful. Most users on the small/indie web lack visibility and this would do them service. My blog is already within the index because I think it might have been picked up during the "HN share your blog" post that happened a while ago, but others might not have been that lucky.
I'm a Kagi customer, and a very happy one. The search engine is amazingly good. This only makes me happier with my search engine choice.
If I notice something is wrong I can report it and expect to have a developer look into it within reasonable time.
They are grateful for reports from users and if there is an actual problem (and not a misunderstanding) it is also fixed, again with a reasonable.
(The first major reason for me is premium search quality, no tracking probably comes at a distant third or something.)
Imagine getting that level of support for any other web browser!
For me you represent an incredible accomplishment: the first search engine that gives better results than Google, respects privacy, offers customization and so much more.
Thank you.
I notice google/bing and quickly kick in my muscle memory built up over the years to type duckduckgo.com and do another search.
After this I still am confused why the results are so crap. Then I notice the duck laughing at me and think... shouldnt this be a dog?
And then I start feeling bad for the original user of the device and tell them to do the search themselves and let me know when they get a result
https://github.com/kagisearch/smallweb/blob/main/smallweb.tx...
It's expensive enough that I can't imagine anyone repackaging it profitably (2x Bing search prices for me) but having to email someone adds just enough friction to discourage a lot of tinkerers from even trying it.
Edit: just saw this:
> Do not submit your own website.
I see. I'm okay with that. Maybe it will show up there one day.
I noticed the example result for useyourloaf wasn't included if I switched it to "Sweden" and not sure if this is just an oddity or if the entire feature is nerfed because I just leave my locale on all the time.
I hadn't considered this aspect of Kagi yet. I'm not a subscriber at this point, but I am strongly considering it. But I use search instead of typing domain names directly to avoid the typo phishing style attacks. I wonder how much "artificial" search that would generate based on my typical usage.
Just a suggestion. Search engines do supply their results with a level of confidence, and you're not wrong to use them like that.
On Chrome I think it is optimized so it always askes Google first.
Otherwise how would Google know all the pages you visit, and especially the ones you visit most?
It's a work-around, not an actual resolution, unless you already happen to like using bookmarks.
Obviously that means I'm using split url/search fields in the address bar, or that wouldn't work as easily. I also heavily use Firefox' keyword search to avoid doing a round-trip through a search engine if I already know which site I want to search; mostly using the same prefixes that DDG uses, so I don't need to adjust too much if I find myself using a different computer.
I have actually already split my address and search fields too, for a different reason. I got aggrevated with having the local machine names on my lan perform a public search, even though my local dns is configured to use only my opnsense router, which does resolve the local names correctly. Splitting off the search box makes the local names always do only what DNS says, and so I get my 3d printer instead of search results for it's name.
For that particular problem, going to "truenas" or "unifi" etc on my lan, the bookmarks are actually more convenient in nmost cases I have to admit! Especially with half of the services needing some special port number like unifi and jellyfin etc.
So this may not be as big of an issue as you think. Should also keep in mind that higher search quality means you don't have to search as much in the first place, which leaves a larger buffer.
Alternatively you could just use "hashbangs" to search domains via google (!g), since I don't believe those cost you anything.
This is not the worst thing in the world, but it is annoying to me and not a life upgrade.
Maybe the search results are better enough to make it worth it (on top of also paying money), or maybe not. I'm still determining that, but it's question and not a slam dunk.
I'm now halfway through my billing period (22 aug - 21 sep) and I'm on 488 searches. Would be tight on the pro plan (1000 searches), but I'm still on the early adopter pro plan (1500 searches).
On the $5 tier it's 1.7 cents per search for the first 300, then 1.5 cents after that. As expected I blew past the 300 before the month was out and am currently sitting at a total of almost $9 for this month and there are still 3 more days to go, and this includes being away for labor day weekend and only using my phone (without kagi) for a couple days, and this is after populating and using a bunch of bookmarks specifically to cater to the fact that I now pay for searches. And this is only my laptop. I have not used kagi on my phone or anything else yet.
I haven't used bookmarks in 20 years and don't particularly want to. I normally don't even have the toolbar visible but now I had to un-hide it and add that clutter back to my browser.
So I'm both paying money and contorting my usage pattern.
I guess now that the first month is about done, I can say it looks like I should go up to the $10 plan, where the searches are only 1 cent, but only for the first 1000, and only if I actually use all 1000! If I pay $10 and only do 250 searches that month, then they weren't 1 cent were they?
As much as I like it, I don't know if I'm going to keep it.
I will not pay $10 or $25 just to have it sitting there available "in case", and I apparently will at least some times (who knows how often? every month? 3 out of 12?) will blow past the $5 an end up paying $10 anyway.
If I complicate my usage to cater to kagi so that my default is ddg and just invoke kagi sometimes when I feel like it would help, then I'll probably forget it exists most of the time and do about 10 per month and pay 50 cents each. Probably only one or two months of that and I'll just decide it's not worth $5 for a handful of searches and just cancel it.
The only way it will be useful for me is if it can just be the default search that I don't have to worry about.
They should just figure out whatever the fair per search price is and bill that. The stupid tiers are probably going to drive me off.
$10 unmetered must always be a fallacy anyway. Surely I can't start up "bkwsearch" and sell searches for 3 cents each, and pay you only $10 for my million$ skin.
So don't even pretend.
Except really there probably needs to be both options, because some people value predictability more, so $10 or $20 or any number that never changes is better than paying only $2 on average but unpredictable each month.
I neither want to pay $10 for $2 worth of usage, nor soak you for $30 worth of usage while only paying $10, and even more than that I don't want to have to think about it to make sure if I choose the N tier that I'm making more or less full use of the N tier.
I appreciate that you took the time to read an individual users complaint, whatever you end up doing. Thanks.
It feels a bit like how it felt discovering Google back when AltaVista was still a thing.
I had some memory of AltaVista being full of SEO (as I recall it was mostly a keyword search engine) spam.
And that was probably true, but I also came across this Quora page https://www.quora.com/Why-did-Altavista-search-engine-lose-g... and it reminded me about two other things stand out to me as true
1. AltaVista was slow and you needed to know your syntax to find things 2. AltaVista was cluttered. Google had very clean results.
I think on 1 google is still fast, but 2. not so much. Google's results are far from "clean", instead it feels like the main goal is showing you as much ads as possible and preferably getting you to accidentally click on one.
This in and the fact that Kagi results are as good or slightly better (and much more customisable) than Google is probably the main reason I'm so happy with Kagi.
After all, Kagi sources results from Google so it's no surprise the quality is similar https://help.kagi.com/kagi/search-details/search-sources.htm...
This is an interesting observation given Google's roots. Although it is a different era, where we have ad blockers now, it would be really weird and neat if Google were displaced by a new search engine that did literally the exact same thing Google did in 1998: clean, no-bullshit page that just gives you search results + page-rank.
The search results are consistently better than anyone else's, including DuckDuckGo, so I am, and will remain, a happy paying customer.
I tried Kagi for about a week, and it felt more or less identical in quality to DDG, just infinitely more expensive as DDG costs nothing.
Plus, Searx supports many more search engines, and I can customize it exactly to my liking.
I wish them well, as they clearly have good intentions and a good product, but I prefer using an equivalent OSS and self-hosted solution over a proprietary SaaS I have to create an account for, even if it's not as polished or featureful.
EDIT: Actually, I'm wrong. Kagi apparently also has their own crawlers and indexes[1]. Still, I'm not finding Searx results to be deficient, so I'm not missing out on much.
One thing that does concern me with Searx, and partially with Kagi, is that those 3rd parties could decide to block these API requests at any point, leaving Searx unusable, and Kagi's results less relevant. I'm not sure this is a sustainable way to build a search engine, but I do appreciate both Kagi's and Searx's stance on ads. Using any mainstream search engine via their own frontends is a frustrating experience at best.
[1]: https://help.kagi.com/kagi/search-details/search-sources.htm...
https://github.com/iorate/ublacklist
For example Kagi uses Google and even hosts on GCP - I think Google's technology and people are great, it is just the business model that is rotten and contributes to the deterioration of the web.
And interestingly enough, at least Discord (and to some extent Twitter) are trying to have a business model that does not put more ads down your throat (although admittedly I did cancel my Twitter subscription as unexplainably they still showed ads even when subscribing - you can't sit on two chairs).
Twitter isn’t really trying to do that. It’s just something the owner likes to claim, but what he’s actually done is the opposite:
- Hired a CEO from the ad world
- Increased personal data collection (with updated Terms of Service to reflect that)
- Limited tweet access without login (those views would be harder to target with ads)
As a Tesla customer I’m fed up with this person’s endless lies. Seems like you got a taste of it with your Twitter subscription too.
I think the only condition is that the article is less than one week old.
https://kagi.com/smallweb displays the homepage for the Kagi Blog at the moment, though.
In the meantime you can toggle the 'non-commercial web' lens which will include KSW (and some other things).
#1 What's the rationale behind favoring recent blog updates? In my experience, recent updates make the weakest search results, more prone to updates and link breakage, and overall tend to be of lower quality. I also wonder if promoting recent content might incentivize pumping out low-quality entries to increase the odds of being listed.
#2 In dabbling in the domain I've always ended up with an almost absurd skew toward technical programmer:y blogs. While there is a strong overlap between the cohort with a blog, and the cohort with programmer interests, I feel it would be more inviting to other groups if other interests were better represented. Is this something you've thought about, and if so, what do you think might be done?
#1 There are a couple of factors.
- More recent content tends to be more relevant given the same search query, or at least its freshnes will contribute to it not being completely irrelevant (which is the worse thing you want in a search engine). The quality of it is already guaranteed to some extent by this being a curated list to begin with.
- It was relatively easy to assemble and maintain the list because it relies on RSS feed tech, and there were a lot of sources to seed it.
- Focus on recent writing can encourage some people to write (more) as we had an example highlighted in the blog post. In general, the web needs more high quality, non-commercial content, and this is ultimately what we want to contribute towards with this. By providing a platform (even if very small) to encourage this behavior we get a step closer to the web we like.
#2 In general I agree. Although I should say we did spend effort to create a diverse pool of websites for this initiative (for example I am seeing a lot of economy or photography). Again, we can only encourage the creation of more content in various areas through platforms like Kagi (and Marginalia) and hope that it will work out at the end.
I really appreciate Kagi's development matching what i feel like i'm buying. Thanks Kagi Team <3
Spotify: Really hostile, manipulative, and terrible to artists. On my end I hate how commercial the homescreen is, and the CarPlay interface is just a user-hostile to a degree that is frankly unsafe disaster.
TIDAL: Pretty good in a lot of ways. They pay their artists well. The recommendations are decent. The apps have some really stupid bugs that have persisted for years though, the most annoying of which is that if you shuffle a playlist it only shuffles the dozen or so tracks that the interface had pre-cached from the top. So if you try to shuffle your full library you wind up just hearing the same dozen songs only, over and over again.
Deezer: Wanted to like it, but the apps aren't great, and I ran into more missing tracks than I'd like.
So I finally settled on Apple Music. I've got an iPhone, so it's a natural fit there, and the CarPlay interface is great. They also pay artists almost as well as TIDAL. The recommendations are super good enough, and I don't really feel like the home page is constantly trying to push me to whatever the huge labels are paying them to promote (damn you Spotify). The Windows apps are terrible (like flat out embarrassing), and the Linux apps are non-existent, but luckily there's a pretty great open source app called Cider that solves that.
This is what blocked me from trying Apple Music as well. Maybe i'll give Cider a look, thanks!
It's a great desktop client.
And of course it showed me Questionable Content, which I first got to via stumbleupon.
Kagi could just admit they don't want to moderate notes or store them permanently. No need to push down the small web, because a lot of small sites preserve their content.
I get that Kagi probably has data indicating the reality of how often sites down, but it seems from my experience that content in big platforms disappears often as well, even in the cases where the creator hadn't forgotten about it. The "Small web" websites made by a creator that cares have the room to be much more permanent.
It would be nice if Kagi Small Web would have an ActivityPub interface so that the most appreciated sites of a day could be added to a timeline on mastodon or lemmy.
[1] https://www.jefftk.com/news.rss
[2] https://www.jefftk.com/p/weekly-incidence-vs-cumulative-infe...
[3] https://kagi.com/api/v1/smallweb/feed
what kind of utilities?
I am happy to pay $2 (or more) for tools I use if it is a one time payment or a payment for tokens (e.g. $n for $m ocr scans, generated images or whatever).
I loathe apps that demand monthly payments unless it is really understandable why they have to have it that way (service that require permanent storage comes to mind, although I think I only use iCloud for storage now).
https://small-tech.org/research-and-development/
On a side note, there is now a - to my best knowledge - completely unrelated product "ExpertGPT" from some totally different company. I am not talking about that one.
https://kagifeedback.org/?sort=latest&q=expertgpt