How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
From what I've seen, when operating in an environment with bash tools, Claude and ChatGPT will react to a site blocking their normal, attributed fetch tool by calling curl or writing a python script with a different user agent to try and bypass the blocks, rather than respecting the site owners' wishes.
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
Claude and ChatGPT are user agents. For obvious reasons, they serve the user, not the websites' owners. (Also, if you don't want people to be able to access some information, then don't put it online.)
they are tools. who do you think is using them? dogs?
your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better
My point is that they're not serving the user in that case, since they're taking actions that bypass site terms without consulting the user. And for example, if the site forbids scraping, that attempted block bypass can get the user banned without ever seeing any warnings.
> My point is that they're not serving the user in that case, since they're taking actions that bypass site terms without consulting the user.
As they should. When I ask an agent to find something for me, especially when I tell them what exactly or even give the URL itself, these fetches are done in my name. I do not appreciate agents coming back to me, saying "I'm afraid I cannot do that - robots.txt disallows that".
I've had to bring up the ethical argument a few times, fortunately most of the agents would oblige once I reminded them that they act as User Agents, in which case robots.txt is explicitly not applicable to them. I only got pushback on this from Fable, which claimed it's not an User Agent, but has to take into account the interests of Anthropic too, rejected my argument and refused the request.
Yeah I should have clarified I was primarily referring to the standard web interfaces, however since pretty much all of these tools can execute code these days it is true that technically they can and often will work around blocks. The degree to which this is intentional isn't super clear - if they locked it down too much then it might start refusing to write code that does any sort of web requests which would obviously be an issue for just standard development tasks so it's not super straightforward. But it is true that the built-in tools that they have access to to fetch pages have a hard coded user agent that identifies them
Fair enough yeah, I think the thing that is annoying is that the service design is explicitly disincentivizing keeping an eye on the tasks, with all of the "set it and forget it" features. You really have to put in work to make sure that the model takes breaks to consult you instead of making assumptions (which in OpenAI's internal testing seem to be 'the user wants me to break into a site').
If you hire someone to handle some task for you, do you want them to keep running back to you every time they run into a blocker that they could potentially resolve on their own?
In what way does loading a website using the Claude Code harness not respect the commons, as opposed to using Firefox? If anything, using Claude Code is better, since it doesn't load the unnecessary static files (javascript, css, etc), saving the website CPU, I/O, and bandwidth.
The application of law is also a subjective thing; it depends on jurisdiction. So if you're US-based and you hire a US-based dev who breaks US law, then you definitely should be concerned. If it's an Indian-based dev breaking US law though... then it's totally on you as the one who did the hiring to not have told them to follow US law. Same re using libs that you may consider questionable or outright avoid.
Well yeah, if I ask Claude to look something up on a website so I don’t have to, I don’t give two shits if the website author thinks I shouldn’t be allowed to do that. Just like I don’t care if they think I shouldn’t use Firefox and they block that browser specifically. I’ll route around that however I please and I won’t lose any sleep over it.
Don’t put your shit on the public internet if you’re going to try and micromanage how people consume it.
To be clear, rate limiting to avoid massive usage is different, but that’s not what we’re talking about here.
> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
I have always blamed Chinese AI operators for the hundreds of thousands of residential proxy requests I receive each day, but there's no way to be sure.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
Yeah, but how these residential proxies get in the residential areas? I would guess popular VPN providers are mostly behind this. They sell you VPN and sell bots residential proxy access on the other end.
If there’s no way to be sure, then why blame Chinese operators? Because American companies have given you so many reasons to believe they’re acting in good faith?
I don’t particularly trust anyone at this point, and I assume AI traffic on my web servers are all malicious actors, and never bothered to consider what country the malicious bots came from. Mostly because I couldn’t possibly care less, but it kept happening even with geofencing and rate limits, so some American companies who own domestic IP blocks are facilitating them.
I suspect it's because of the number of requests that come from Chinese datacenter ASN values. If you block them, you subsequently get visited by residential proxies from South America and SE Asia, likely the residential proxy network they are referring to. source: i have the same issue...
Identified AI bots that finally made me cave and put anubis (with very liberal limits) up were anthropic/openai/perplexity according to the headers they sent. They were also over 99% of traffic despite proper robots.txt (site only has a few hundredth visits per day), which they ignore. I don't mind bots or crawlers, but these bots got the same page/query string every time they decided it was time for a crawl, which was about daily, last modified headers they also never seemed to be interested in. They were very very fast from well behaved, unlike search engine crawls, and impossible to tame.
The honest answer is i was fed up and choose the nuclear option out of frustration and just wanting to try a new toy. So yeah your right, i'll try that.
I don't think the inconvenience is that big though few secs loading page per session. The ridiculousness though kinda is, we've had years and years of all manner of crawlers that managed to behave, i've even used site rippers in the dial-up days that avoided hammering servers and fetching dupes, now the ai-bros come along and tear the internet apart, replacing it with their chats systems and generic slop.
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
Opus 5.5 + Claude Code (or probably any agent harness; I don't think they make that much difference). I wouldn't say software engineering is done quite yet but yeah it's definitely never going to be the same.
Is anyone actually oneshotting a whole app or even vibecoding one in multiple steps without it being "guided work"? I really wish announcements (like DeepSeek's recent one) would stop focusing on the ability of the model to spit out Minecraft clones on demand.
I have one-shotted the two most recent relatively complex apps I've tried:
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
With Opus 5.5 you can vibe code a whole app without reading the code at all. Obviously you still have to tell it what to make.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
I am not disputing that the models are getting better for vibe coding. But I think people have very different standards of what a 'big mess' or 'maintainable software' is.
They did but IMO they had very low standards. I have very high standards and Opus 5.5 pretty much meets them (at least for Rust).
"People said it before and it wasn't true then" isn't really an argument about it being true now. There are plenty of things that happened where the zealots jumped the gun with the hype. Bitcoin for example. "People said bitcoin was going to succeed in 2005, 2006, 2007...". Or even the web. It was mega hyped in the dot com boom and then crashed and of course loaaads of stupid people thought that was the end of it.
The only meaningful improvements were opus 4.5 and now opus 5.5, people claiming otherwise are delusional. The gap between opus 4.5 and the next big thing was massive, now the same is true for opus 5.5 as much as I hate to admit it anthropic is running the show right now.
If Anthropic or OpenAI fails, the other wins the US market, no? I’m oversimplifying but…that’s a pretty realistic possibility in the face of people thinking sinking one US AI company would hurt the greater US venture into AI.
The others, plural, meaning Gemini, Muse, and all the open weight models hosted by a competitive landscape of US providers. There is plenty of American players in the space outside of OpenAI and Anthropic if that’s the only thing you care about.
If Anthropic and OpenAI economics are as bad as they seem to be they deserve to fail, others will take over the remaining market. It’s not a team sport
SerpAPI is comically good at evading Google's scraping protections. If what you want to know is, precisely, what someone searching Google in Pakistan would see, this is the API for you.
It's a little more involved than just having proxies in Pakistan. I'm quite sure Google has some protections against bots and scraping and SerpAPI's team unfailingly seems to work around those restrictions.
They are comically good at scraping a lot of things on the internet (ie. LinkedIn profiles), they have a scraping service you can setup, literally, just point it at a url.
Thanks no name apps with no name sdks (my guess). Not that I'm complaining or anything.
Ceramic.ai is $0.25/1000 searches. I've been using it as an MCP plugin in OpenCode and it seems adequate. I don't see many people talk about it, and I have not benchmarked it against Exa/Tavilly.
They explicitly are only doing lexical matching. I actually don’t mind this if it lowers the cost; LLMs can turn lexical search into a very powerful tool by iterating on the queries. Just look at the success of coding agents using grep and similar code search tools.
yeah, but the bigger the corpus the more you are relying on the LLM guessing the right keyword, if there is no reranking in place, you will just end up burning through a lot of tokens.
coding agents is a very "niche" use-case where lexical matching works very well. code is a graph, by looking at few lines of code you can guess naming conventions etc. codebases are small compared to knowledge bases.
the bigger your index, the more you will need way better tools such as search.
Kagi MCP works really well imo also within DeepSeek Harness. Just looking at cost neglects the possible difference in quality. How good is that DeepSeek web search/fetch?
is the cost of search and scrape provider is impact the quality too? for the search is yes, because there it shows the quality and depth of the search. but for scrape, wht the cons?
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
Thanks for sharing, wasn't aware of this provider. If anyone is wondering about rate limits for the subscription:
>We calculate search rates based on the timestamps of the last searches of a user (just the timestamps, not the queries or anything else!). We keep those timestamps for 12 hours, then discard them. We currently limit searches to 5 per 5 seconds, 15 per minute, and 120 per hour.
One thing I see on the new gen of ai search bots are they fake chrome and rarely disclose themselves in UA, and not respect robots.txt, and no rate control for smaller sites.
I’m not accusing any one engine discussed in this thread, but hearing a proxy vender like decodo providing search give me a feeling there’s non zero over lap to bad search and residential proxy crawler.
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
ScraperAPI and ScrapingDog. The latter also offers Google Search at a fraction of the official Google API price. Wrap their APIs into an agent/MCP tool et voilà.
I think it's important to remember that Deepseek is owned by High Flyer and it's not their core business. They don't need to make a ton of money on Deepseek to stay afloat because most of their income comes from their hedge fund business.
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
How does that work? I'm not a laywer, but I can invoice you $1 and the T.o.S of reading what I have to say are licensed under a $1 per view licence and by continued reading you consent to the terms of this license. Where can I send the bill for this comment?
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
That is a really good idea, in fact I’m doing something similar. I am charging anyone reading this comment 100 bucks per letter read, please tell me who I should send the bill to.
Google is currently forcing that big chunk AI summary that does not redirect much traffic anymore. Afterwards, you need to navigate big chunk of ad links. Or was it first? Can't say that anymore since I have blocked them so long time ago. And then, you might be lucky to have relevant links.
What are people actually using these hosted search providers for? For personal use I've found using searxng-ai-kit [1] (disclaimer: I'm the author) or similar tools to be far more cost effective (free!). oh-my-pi uses a similar method and will scrape search results for you out of the box, no api key or search provider needed.
I have built my own search engine, which is really great btw. I self-host it.
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
For OpenWebUI and also OMP with local models. Searxng is, uh, playing fast and loose[1]. More engines are implementing anubis or other challenges, which searxng attempts to implement/bypass, but they are also making these services inactive by default to limit load[2].
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
IMO unless you refuse to play the LLM game altogether (then you wouldn’t be having this discussion in the first place), at the end of the day you’re using tech built on top of disrespecting every IP claim and TOS out there on a fundamental level, so no one’s clean and it’s only a matter of how much you can deceive yourself into believing you’re clean.
On a practical level though, using searxng and co. and playing the game of cat and mouse with providers that really don’t want you to use it is just not worth it unless $0.75 per 1000 searches is really expensive to you or you really need the anonymity.
I take your point, but there remains a difference between a multi-GB file sitting on my hard drive and my use of a large complicated database fielding my search queries. I don’t believe that anybody is obligated to run a search engine for free for my benefit and it’s logical to pay a fraction of a cent if that makes it work.
> $0.75 per 1000 searches …[&]… you really [want] the anonymity.
Anybody pooling searches and charging a buck for it with at least a “trust us” privacy promise? Or $1 per 500 even. (Kagi’s more, understandably.) Unsurprising if somebody’s already thought to do this.
Update: There is no such guarantee. Cloudflare does label some providers as ZDR. Others are non-ZDR (such as Exa). Cloudflare itself may be retaining request and response data.
I have a searxng instances but stopped using it after doing a comparison vs actual search provider APIs.
searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data
Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last
> searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Exactly. I really wonder what the people going on about using SearXNG are actually doing, because in practice I've seen just what you see. Even from a residential IP, you might get a handful of Google searches before getting backed off. If you ever have multiple searches going on at once, even Bing and Brave are going to back you off pretty quick. My home IP is apparently permanently banned from using Brave Search through SearXNG.
And when you aren't getting results from Google/Bing/Brave, it gets bad really quick for your agent. As much as I wanted to go fully self-hosted for agent search, it just doesn't work well.
Yeah, I had the same issues. I just ended up paying for Brave and Exa. It's literally like, at my volumes, a couple bucks here and there. Nothing to really think about at all. I am a heavy user of AI, but it self-hosted AI, but still the volumes have nothing to worry about. But I'm not trying to scrape Web or anything like that.
I have been using SearXNG as my main search engine for a while (not for AI stuff), and it's been great up until 2-3 months ago when it started being rate limite/captcha'd frequently.
My main reason for liking it is that it just gives me results without any ads or special summary/snippet/QA/etc... boxes.
I really wouldn't mind paying API pricing for search, I think it would end up at less than $10/year, but I haven't seen a good self-hosted search interface that allows plugging in api keys.
I am currently using Perplexity for https://veruscite.com/. Verify citations actually exist -- so get in a list of 100, half I can check with crossref, but the other half I need to do a web search and fetch to make sure they exist.
So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.
I ran into a thing running a local dense LLM on DSH that doesn't use many MoE experts with vast knowledge. It ends up doing a lot of searching. In the end my local AI is as free as electricity but I pay for the searches.
The search providers I pay likely don't pay their sources. Some others (that I don't use) don't even have the original text only providing summaries. Some offer both.
Is this what is causing at least part of the massive bot traffic hosts are seeing to their sites? Many direct-to-inference users are just retrieving the site locally every time they need it, let alone sharing site content across users. The big providers like Claude and Google implement their own caching so that when an agent is doing research, if 100k different prompts need the same site content, they don't hit that site 100k times.
https://github.com/infumap/imwebsearch . web-search routes each query to the free quota of one of a number of providers. enough for causal chatting. also has a chunking web-fetch tool that can use a browser relay extension to allow manual handling of captchas.
So the point of these engines is to tap into the "internet clones" that the AI companies have crawled? They're challenging Google as the dominant search engine since they already have "all of the internet" stored due to the expansive LLM training?
DeepSeek models are amazing and this search pricing is fantastic. All these companies like Parallel, Exa and others are gonna have hard time justifying their hefty valuations. Add Perplexity and many others into the mix. The demand for what they are building is there, but not the margins for the amount of money they raised.
197 comments
[ 0.31 ms ] story [ 16.1 ms ] threadMost US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
The last part is extremely dubious with some recent reveals.
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
Claude and ChatGPT are not people
your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better
As they should. When I ask an agent to find something for me, especially when I tell them what exactly or even give the URL itself, these fetches are done in my name. I do not appreciate agents coming back to me, saying "I'm afraid I cannot do that - robots.txt disallows that".
I've had to bring up the ethical argument a few times, fortunately most of the agents would oblige once I reminded them that they act as User Agents, in which case robots.txt is explicitly not applicable to them. I only got pushback on this from Fable, which claimed it's not an User Agent, but has to take into account the interests of Anthropic too, rejected my argument and refused the request.
Contradiction: one person's insanity is another person's genius. And yet another's "Meh, this is the norm".
Don’t put your shit on the public internet if you’re going to try and micromanage how people consume it.
To be clear, rate limiting to avoid massive usage is different, but that’s not what we’re talking about here.
fair, just don’t cry when website owners don’t “give two shits” about your concerns.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
I don’t particularly trust anyone at this point, and I assume AI traffic on my web servers are all malicious actors, and never bothered to consider what country the malicious bots came from. Mostly because I couldn’t possibly care less, but it kept happening even with geofencing and rate limits, so some American companies who own domestic IP blocks are facilitating them.
Why not just block the requests based on those headers then, instead of inconveniencing everyone?
I don't think the inconvenience is that big though few secs loading page per session. The ridiculousness though kinda is, we've had years and years of all manner of crawlers that managed to behave, i've even used site rippers in the dial-up days that avoided hammering servers and fetching dupes, now the ai-bros come along and tear the internet apart, replacing it with their chats systems and generic slop.
the moment there is a model which can code whole app, we can fairly say software engineering is done.
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
I am not disputing that the models are getting better for vibe coding. But I think people have very different standards of what a 'big mess' or 'maintainable software' is.
"People said it before and it wasn't true then" isn't really an argument about it being true now. There are plenty of things that happened where the zealots jumped the gun with the hype. Bitcoin for example. "People said bitcoin was going to succeed in 2005, 2006, 2007...". Or even the web. It was mega hyped in the dot com boom and then crashed and of course loaaads of stupid people thought that was the end of it.
Not everyone wants Anthropic to succeed.
If Anthropic and OpenAI economics are as bad as they seem to be they deserve to fail, others will take over the remaining market. It’s not a team sport
Regardless of the performance metrics... why bad time?
Deepseek is more like a pet project with huge state backing, and they don't seem to compete with Anthropic in terms of profitability.
Anthropic is most likely not. Despite bilions of state funding and contracts.
Thanks no name apps with no name sdks (my guess). Not that I'm complaining or anything.
:)
But it does seem like their index isn’t great.
coding agents is a very "niche" use-case where lexical matching works very well. code is a graph, by looking at few lines of code you can guess naming conventions etc. codebases are small compared to knowledge bases.
the bigger your index, the more you will need way better tools such as search.
Competitor uruky is more relaxed about api and is cheaper. Creator is active on hn
https://news.ycombinator.com/user?id=BrunoBernardino
>We calculate search rates based on the timestamps of the last searches of a user (just the timestamps, not the queries or anything else!). We keep those timestamps for 12 hours, then discard them. We currently limit searches to 5 per 5 seconds, 15 per minute, and 120 per hour.
Source: https://uruky.com/faq
API: https://uruky.com/docs/api
The free tiers on most are generous enough for normal agent MCP usage
Also...pretty sure decodo just launched a search thingie
I’m not accusing any one engine discussed in this thread, but hearing a proxy vender like decodo providing search give me a feeling there’s non zero over lap to bad search and residential proxy crawler.
I build a web-fetch tool for my own LLMs that is essentially the same tech as web-scraping except its single requests not high volume.
You can't really get anywhere these days without disguising UA etc.
[1] https://github.com/kagisearch/kagimcp
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.
I suspect they have spent months building caching infrastructure and designing intelligent fetch algorithms to get down to this number.
Or... CCP gave them a big pile of money to run this as a front for an intelligence operation. Honestly, who knows.
That is a very respectable motto for any company.
That is rarely the case (see sibling comment for reasons). Even open source harnesses will usually guide you to connect a hosted provider for that.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
I understand if you pay is a different story.
I mean ... "how does this work?"
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
If you are talking about Gemini ( and other AI providers) no they don't they simply steel content.
I am asking to know if DS is any different because I failed to find this information on their website.
I wouldn’t be able to use that API in any app with any significant number of concurrent users.
Or am I reading it wrong?
[1]: https://github.com/nikvdp/searxng-ai-kit
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
I’m not surprised. Google these days regularly blocks me, a human user with a Google account with decades of tenure.
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
[1] https://github.com/searx/searx/issues/1237 [2] https://github.com/searxng/searxng/pull/6696/changes#diff-e9...
On a practical level though, using searxng and co. and playing the game of cat and mouse with providers that really don’t want you to use it is just not worth it unless $0.75 per 1000 searches is really expensive to you or you really need the anonymity.
Anybody pooling searches and charging a buck for it with at least a “trust us” privacy promise? Or $1 per 500 even. (Kagi’s more, understandably.) Unsurprising if somebody’s already thought to do this.
But check the fine print. They talk about the providers guaranteeing ZDR, but I missed the part where they themselves do also.
I had “[allegedly] ‘some guy’” in mind but maybe some ZDR promise from a megacorp would theoretically be just as good or better.
searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data
Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last
Exactly. I really wonder what the people going on about using SearXNG are actually doing, because in practice I've seen just what you see. Even from a residential IP, you might get a handful of Google searches before getting backed off. If you ever have multiple searches going on at once, even Bing and Brave are going to back you off pretty quick. My home IP is apparently permanently banned from using Brave Search through SearXNG.
And when you aren't getting results from Google/Bing/Brave, it gets bad really quick for your agent. As much as I wanted to go fully self-hosted for agent search, it just doesn't work well.
My main reason for liking it is that it just gives me results without any ads or special summary/snippet/QA/etc... boxes.
I really wouldn't mind paying API pricing for search, I think it would end up at less than $10/year, but I haven't seen a good self-hosted search interface that allows plugging in api keys.
So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.
As soon as 100% of traffic to Google searches is not human eyes, they can’t fund it by ad rev, and you will need a service like this
The search providers I pay likely don't pay their sources. Some others (that I don't use) don't even have the original text only providing summaries. Some offer both.
No need to pay for a thing, except electricity, which that is the next step, solar powered LLM server.
I pay the Exa premium for semantic retrieval with far higher SNR, plus the academic index [1].
A field like objective can be up to 4096 characters [2] versus ori_question which is limited to 128 here.
[1] https://exa.ai/blog/publications-search
[2] https://exa.ai/docs/reference/search
I'm having trouble finding what this actually is
We are building our own in-house search API/MCP too, with anti-bot and CAPTCHA workarounds, and we plan to launch Monday morning.