If you actually click through Google's result pages, you will find that there are actually only about 87 results. Even if you "repeat the search with the omitted results included" Google returns only about 226 results. Google's estimated result count is routinely off by so much as to make it virtually useless.
Google like all other web search engines never scan their whole index when searching, it's way too expensive. All sorts of tricks are used to aggressively prune matching results at every opportunity during retrieval until maybe a few hundred best scoring docs are left and that's what you're paging through.
But yeah, estimated result count is still a big lie either way
I found that going to the last page showed a more realistic number for results. This would only be shown on that "last page" of results though. However, it does seem strange that there are only 447 results for a google search of "apple"[0] down from a whopping 4,070,000,000 results shown before that page. And that is with show omitted results. Seems like max result limiting for the 0.1% of users that want to look past page 10 of search results
It is actually plausible that there could be billions of pages with apple on the internet.
The 447 results you paged through are just the most relevant docs for your particular query ("apple" + country) and Google doesn't bother retrieving any further. Your use case of finding all the pages with the term "apple" is simply too rare and too expensive to support and they don't optimize for it.
If you try different query variants apple + something with different locales (&hl=) you'd get very different top NNN results.
As others are pointing out, this is actually a popular site not just for russian content. Owing to the jurisdiction, it's actually a popular place for hosting various ripped/pirated PDFs.
Well that's weird. Searching vkontakte gives a lot of results (in the realm of what you would expect). Maybe it's a command or something else is happening with that query?
The single result it gives is a link to VKontakte. Given that it's largely a walled garden, I think that's a reasonable result to give, with no other input.
That's not correct. You can show the proper results without setting the region. Do this by adding an empty token that doesn't get stripped, "https://duckduckgo.com/?q=vk+%22%22". That also shows the results list.
yeah there is some absolutely questionable stuff there, I wonder why searching for VK brings them all up, some of these seem like random direct-to-youtube uploads but there's so many of kids that it almost seems like someone's collection, but why does a search for VK return them all? seems like a strange coincidence that there'd be troubling image results and empty normal results
The images returned on DuckDuckGo are, in general, really dodgy. While debugging a web-server of mine, I recently found that various constant strings of file servers return similarly `dodgy' results, like for example "Index of /" [0].
So this is a complete tangent, but one of the images was some photograph off a geocities archive site. The woman had red eyes. I realized I haven't seen that in years. Cameras sure have improved.
It is (used to be?) a pretty common google "hack" to put an intitle:"index.of" on a search to find sites serving files, usually mp3s. Of course once this was widespread enough, SEO spam sites starting including it in the generated clickbait sites.
There's definitely some suggestive underage content in there. Some of it looks like random photos gathered from the web but there's also some pedo / ephebo shit that was purposely made.
> Might be a partnership with Yandex to not show russian results outside Russia?
Quite the opposite.
My search results from DDG over the past year or so have been infested with Russian sites, articles and even twitter account titles of English language accounts, despite not living there and not having any preference for Russian in any browsers settings or regions and never issuing a single query in Russian.
"infested" is a little much, don't you think? I can sympathise with frustration on having results that don't match your language preferences, but "infested"? Come on.
They're only against Communism. Russians with money are just fine, apparently. The logical conclusion of "money is speech" is that foreign intelligence agencies can buy as much speech as they like.
Protip: its not just Russians playing around with sentiment in other countries for their own gains, and it is extremely easy to deflect all attention towards Russia. It can be long time allies that you harbor the least feelings about.
I mean the sentiment lacks nuance and it gives context to why someone would use pejorative adjectives about Russian-anything even when referring to search engine results
And you can pinpoint which generations of people are likely speaking by their use of said pejoratives on otherwise benign subjects
> "Manually changing region to Russia gives many more results."
This probably explains it: DuckDuckGo indexes different languages together in one big index internally(?). "VK" returns many results in Russian, filling some maximum number of results. And then a filter just removes all the Russian ones if you're searching from a non-Russian locale?
Searching for ”vk” should return a lot of things beside the Russian social network. Vodka Kick, Vulkan, Voight-Kampff, Västerbottens-kuriren, etc. etc. I don’t think this is intentional.
I've been trying to use duckduckgo for almost a year, but I can't really justify it. I've even gotten used to just adding !g at the end of each search. It has terrible results for anything non english and pretty bad latency where in asia it seems.
Does anyone have a recommendation for another search engine that does decently well in terms of privacy and also supports chinese results?
Another search engine I recommend is Ecosia. They’re slightly better than average with privacy, and focused on the environment. I think they just use straight bing results so I’m not sure how well it handles Chinese searches.
Startpage.com is the one I use day to day (I work at the company that owns it). They use straight Google results through a proxy in their own data centers, it's anonymized before it hits anything
I'm also very interested in the details. According to their website they are paying Google to provide them with the results. But the revenue streams to startpage are not provided on their homepage (or at least I haven't found them yet).
Edit: https://support.startpage.com/index.php?/Knowledgebase/Artic... This gives additional information. Apparently they are showing ads that are only related to the search terms. I guess they are showing more ads than Google? Otherwise this wouldn't really make sense. Or maybe individualized advertisements in the search results aren't that much of a big money-maker for Google (!)
It's the same amount of ads. The hyper targeting Google does is incredibly profitable because of the scale they operate at. If you make the click through rate of an ad even 0.1% better over billions of searches it more than pays for the R&D of an engineering team.
Really the only way to compete against Google from a feature perspective is the privacy angle. You're sacrificing better click through rates to target a market that cares about privacy. Making something private requires a lot less engineering and product power than hyper optimizing tracking. By definition if you're not doing any tracking you don't need to employee people to set up tracking systems.
The deal is 15+ years old. To my knowledge it was made back when Google was still a very young company. There's a surprising amount of legacy contracts from Google/Bing/Yahoo floating around that power all of these search engines (Ecosia, DDG, etc).
>We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).
That crawler is only used for Instant Answers for specific queries, not the standard 10 search result links. Yahoo has also been powered by Bing for a long time now as well.
Ive been using DDG as a default search for a while now. My conclusion is that the results are just NOT as good as Google's. I find myself using Google searches for time sensitive or mission critical searches, which totally defeats the purpose of a search engine.
As an example, I was playing around with Phaser and found that DDG just does not surface the entire Phaser documentation or forums correctly, and had to switch my default browser search for when I was putting time there.
I want to want to support DDG, but the lack of quality around searches makes it difficult.
Actually, by searching with !g, you’re leaking your info to both DuckDuckGo and Google, instead of just Google.
If you want to do Google searches without leaking your data to Google, you could use Startpage.com, but personally I don’t trust Startpage more than Google.
If it runs the query through DDG's server side, then Google would just see at as an anonymous query along with a million other anonymous queries coming from DuckDuckGo's servers.
Unfortunately, that's NOT what they do. Instead, they just forward you to the google search result, which is pointless and stupid.
Not so. The point is that you can set DDG to be your default search engine, and put !g anywhere in the search string to search google in case you need it.
I got in the habit of just appending this to many of my queries in time sensitive situations, as if by reflex, negating the purpose (I default my search engine to DDG so !g is the fastest way to Google for me). I am having a hard time breaking the habit.
I'm not sure about the "without leaking your info to Google" part, there. All it does is forward you search query to Google as if you'd entered it on google.com. There's not really anything DuckDuckGo can do about that.
I think you’re implying that Reddit’s search is subpar, which I agree, although it seems it has improved slightly, recently. When I want firsthand perspective on something I usually append “reddit” to my google searches instead of searching reddit itself, even if I have a reddit tab open already.
On relatively sparse, self-post (fow or no link) subreddits, Reddit search is fairly useful for searchin posts only.
Comments are not indexed.
At volume and with links, search breaks down from both insufficient context (esppecially with editorialised, sensationalised, or vague titles) and excess low-yield content.
Plenty of sites have terrible search and discovery stories. I routinely search in a general purpose search engine and filter to the site whose results I want because it's faster and more reliable than trying to navigate the site itself.
It's been a decade since the majority of sites had a usable search of their own. Many that still do are just hosting a site-specific Google search at this point.
I agree, sadly, and in (literally) every discussion on DDG that I've seen, people cite as one of DDG's plus points that you can use !g to search Google instead.
Just out of curiosity, what do you (anyone reading) see when you search "curry" on DDG images?
I actually like the results on DDG better than Google most of the time:
1. It's more likely to respond to the actual query, rather than guessing what I want based on the query.
2. The results are not stuffed with ads and Google service substitutions.
However it's not as good at searching every last corner of the web, so occasionally I need to g!. It's pretty rare that I use that these days, and rarer that it actually gives me a better result.
Same. DDG often requires 1-2 additional search terms to give it
enough context, but in return I get results that are actually
relevant instead of half a page full of pinterest links and
other spam content.
In other weird search results if you search for 4chan with tor you get wikipedia and news results. If you search without tor you get the real website and in general more relevant results.
DDG's raison d'etre is to position itself as the Gutmensch of search engines but when it comes to providing peace of mind that I'm getting the best and most relevant results, there's a reason why my grandmother hasn't heard of it.
DuckDuckGo isn't the best search engine. I really want it to be. I use it a by default and I use it often. However, especially when I'm searching for products/things to buy I end up on google search. DuckDuckGo is more like old school search, think Altavista - tons of results but not much useful.
Maybe it's because Google has very little data on me, but I find Google to be horrible in 2020. It's degraded considerably over the years to the point that I don't even try Google if DDG doesn't return anything useful. It's especially useless if I'm trying to buy something. If I search for a service business using my hometown, it'll return results from businesses everywhere other than my hometown.
Heh, I don't even know how google feels, much like I don't know how ransomware and heroin feels. We don't really have to try everything in life, do we?
Tbh I don’t really generically search that much. I search within specific sites (e.g. Wikipedia, YouTube) but my first reach out for tool is rarely if ever generic web search. Lots of bangs on ddg
I have the same experience. I started to notice about 6 months ago. It's happening a lot more often that i'm frustrated with results even when looking for trivial stuff. My last search frustration was: 'bitmitigate'. I had to make a benchmark of CDN services and their company site isn't on the first page like other products i'm searching for. I suspect it's because this company had business with alt-right imageboards, but it scares the shit of me if this kind of political controversies are incorporated now in google's algorythm.
By contrast, searching 'bitmitigate' on DDG produces the expected outcome: the first result is the official .com site of the company.
(Edit: added the last paragraph comparing to duckduckgo)
>My last search frustration was: 'bitmitigate'. I had to make a benchmark of CDN services and their company site isn't on the first page like other products i'm searching for.
Seems to be working for me? SERP for me (not private browsing, but I'm not signed in and regularly clear cookies)
1. BitMitigate - CDN, WAF, DDoS Protection, Load Balancing, VPN <--- this is the main site, as expected.
2. BitMitigate: What we know about 8chan's new web protection ...
3. BitMitigate (@bitmitigate) | Twitter
4. Epik reverses course, says BitMitigate will not support 8chan ...
5. BitMitigate becomes new security provider for 8chan ...
Google search has gotten a lot worse for me now that I use it as a backup for DDG. There's some categories it still does well, but other things are like DDG doesn't find it, but neither will Google. I'm not sure how much of that is Google going downhill (which was a trend before I switched), and how much is the loss of personalization data.
I've also noticed google has developed an extreme recency bias in search results. If you're searching for a term that is somehow related even tangentially to any hot or trending topic over the past year, it is virtually impossible to find relevant, yet dated sources on that topic. I know I can do the timeline filter and all that, but sometimes it's not obvious what window of time you should be searching, etc.
I agree, somewhat. While the results are clearly worse my feeling is that they are covering more of my use cases as time passes. The really cool thing (and a trick Google themselves used in the beginning) is that a Google search is only a !g away, so I can easily have it as my default search engine.
I've set it as my default and I have to say I didn't realize hw often I search for silly stuff it can nail. Like the url of a blog I know the exact name of. If I start to get into detailed programming issues or deeper searches I revert to google. It only seems to be 2-3x a week for me.
In my experience google is pretty good for fresh and current data. If you try to find things from some years back, results are overwhelmed by irrelevant current results —even setting custom dates doesn’t help much. Seems it’s biased for the present as well as content from big players (obviously), but to the detriment of smaller outfits.
Also in NZ and as you say lots of results but almost entirely related to the Russian social network 'vk'. Using mojeek, as suggested upthread, gives a much more diverse set of results. Which leaves the question why does NZ get a different set of results than some other places ?
Google forgets so much, and has a tendency to bias results, while DuckDuckGo is so incompetent at times. Add the walled gardens that can’t be searched unless you are part of them, and we have a worst search experience today than over a decade ago.
I am confident that the amount of data that needs to be indexed plays a huge factor too. It just isn’t as easy as it used to be.
One of my issues with instagram is it cant be searched other then using hashtags. Those are just really crappy key words. So many artists put their art on there and its just sucked into a blackhole of pictures to never be seen again. It's impossible to search and find anything specific or anything that occurred more then a few hours ago
There is no sorting. Nothing. The comments are ranked by the AI, the posts are ranked by the AI, the explore section is ranked by the AI.You can't sort by time or likes.
All of those things, and don't forget the steady arms race: the contemporary Web is stunningly adversarial to good search results.
Ten years ago, when Google screwed up, you'd land on a really obvious linkfarm page with a bunch of SEO spam: it was literally just a collection of keywords repeated over and over.
Now, when $search-engine screws up, which it often does, you land on a plausible-looking page, with bad but not terrible English, that just... doesn't tell you anything worth knowing. I know the smell of them, and it can still take me a section or two of precious time to figure out what's going on.
There's often not that much link juice, either: as far as I can tell, the business model is purely to put an ad or two in front of my eyeballs.
If it is a blacklist, it's not a very good one. "pedobear." and "pedophile." work fine. I don't understand why they would want to block these anyway. One's just a meme, and the other is a bunch of mugshots of strange looking men.
They have been censoring any results vaguely related to pedophilia for the past year. If you search for "lolis" or "lolicon" which are fictional cartoons, that will be censored.
It's unfortunately not uncommon for such images to appear on the Russian internet. The Youtube thumbnails are possibly things that were linked to from VK communities. Or maybe there are links in the content somewhere.
If I go on Yandex and search "img " and then a bunch of random numbers, I get almost all pictures of young women, and many pictures of underage girls in provocative poses.
This sort of thing does happen. A few months ago, people discovered similar results on Google Images(!) by searching "TV television and film". The results were being pulled from a 4chan archive of the board with that name.
I don't think it's related to the "VK" web results problem though. If you search "VK," or "VK!" or VK with any punctuation, all the web results come up as expected.
We hired a baby sitter once that was from Ukraine. I made the mistake of Googling her unique name. Her VK profile was a whole different person from the sitter at my house.
Haha her name is probably far less unique than you think. My Slavic name is fairly unique in the Western world but there are literally thousands of people with it on VK.
When you want to add a friend on VK, you have to ask them for things like city, school, and age just to narrow down the results!
Is it picking up VK profile pictures? Sort of reminds me of myspace but with, err, Russian sensibilities. I also tried it with Bing and got literally the exact same results.
Google is definitely different, it shows mostly logos instead of photos.
I never used VK but I got the impression it is a real social network, for better or for worse. Should Bing drop them from its index? Now I'm morbidly curios what kind of weird crap searches of social networks from other cultures might dredge up. Like a Chinese VK.
huh... looks like the default search is now showing multiple things for me (showed just one result first time I checked)
However I'm seeing lots of slightly uncomfortable pictures of young girls (not porn, but didn't want to scroll too far because it felt like there could be some mixed in) and tanks in the image search. This doesn’t change noticeably using the three strictness settings.
I'm a bit conflicted on this. I don't want content like this appearing in search results. But I don't think a clumsy keyword filter will make it go away. And it might push pedophiles looking for a quick fix to places with harder content or echo chambers.
214 comments
[ 2.8 ms ] story [ 201 ms ] threadBut yeah, estimated result count is still a big lie either way
[0] https://www.google.com/search?q=apple&start=440&sa=N&filter=...
The 447 results you paged through are just the most relevant docs for your particular query ("apple" + country) and Google doesn't bother retrieving any further. Your use case of finding all the pages with the term "apple" is simply too rare and too expensive to support and they don't optimize for it.
If you try different query variants apple + something with different locales (&hl=) you'd get very different top NNN results.
100*10 results per page= 1000
Google only shows 1000 results for each query. Even if it says 100 million found
Atleast vk.com?
As others are pointing out, this is actually a popular site not just for russian content. Owing to the jurisdiction, it's actually a popular place for hosting various ripped/pirated PDFs.
Surely “vk” appears on more than 1 page on the web, so… show those pages? Like, y’know, a search engine.
This feels worse than trying to explain a joke.
Searching by putting region=Russia gives more results
So you get russian results.
Something strange is there with the query 'VK'. Its filled with pics of children.
Don't search
A lot of pirated content exists there. Mainly songs.
https://www.urbandictionary.com/define.php?term=VK
No strange meanings either.
[0]: https://duckduckgo.com/?q=%22Index+of+%2F%22&iax=images&ia=i...
There's definitely some suggestive underage content in there. Some of it looks like random photos gathered from the web but there's also some pedo / ephebo shit that was purposely made.
https://duckduckgo.com/?t=ffab&q=VK&kl=ru-ru&ia=web
Might be a partnership with Yandex to not show russian results outside Russia? Because they use Yandex in Russia.
https://www.reddit.com/r/duckduckgo/comments/a7nl1v/duckduck...
It might be a Yandex partnership to not show outside Russia
Quite the opposite.
My search results from DDG over the past year or so have been infested with Russian sites, articles and even twitter account titles of English language accounts, despite not living there and not having any preference for Russian in any browsers settings or regions and never issuing a single query in Russian.
And you can pinpoint which generations of people are likely speaking by their use of said pejoratives on otherwise benign subjects
Perhaps it was intended as a slight against Russians, but I'd ask you to assume good faith.
This probably explains it: DuckDuckGo indexes different languages together in one big index internally(?). "VK" returns many results in Russian, filling some maximum number of results. And then a filter just removes all the Russian ones if you're searching from a non-Russian locale?
Does anyone have a recommendation for another search engine that does decently well in terms of privacy and also supports chinese results?
Edit: https://support.startpage.com/index.php?/Knowledgebase/Artic... This gives additional information. Apparently they are showing ads that are only related to the search terms. I guess they are showing more ads than Google? Otherwise this wouldn't really make sense. Or maybe individualized advertisements in the search results aren't that much of a big money-maker for Google (!)
Really the only way to compete against Google from a feature perspective is the privacy angle. You're sacrificing better click through rates to target a market that cares about privacy. Making something private requires a lot less engineering and product power than hyper optimizing tracking. By definition if you're not doing any tracking you don't need to employee people to set up tracking systems.
https://github.com/benbusby/whoogle-search
Compare with Bing (where DDG gets its search results from) https://www.bing.com/search?q=vk
It's mostly just Bing.
Although yes, its usually Bing.
[0] https://web.archive.org/web/20180203071252/https://duck.co/h...
As an example, I was playing around with Phaser and found that DDG just does not surface the entire Phaser documentation or forums correctly, and had to switch my default browser search for when I was putting time there.
I want to want to support DDG, but the lack of quality around searches makes it difficult.
If you want to do Google searches without leaking your data to Google, you could use Startpage.com, but personally I don’t trust Startpage more than Google.
Can you elaborate? It just does the same query on Google, how is it preventing leakage of information to Google?
Unfortunately, that's NOT what they do. Instead, they just forward you to the google search result, which is pointless and stupid.
Not so. The point is that you can set DDG to be your default search engine, and put !g anywhere in the search string to search google in case you need it.
DDG and google both know what you searched for. It is not a proxy. It just opens google search with the query
Search for '5vh9ld'. It is reddit post's URL.
https://startpage.com/do/metasearch.pl?query=5vh9ld
https://www.google.com/search?hl=en&q=5vh9ld
Better yet, if I already know I want a particular doc or site, I'll go directly to that site instead of hopping through a search engine first.
Comments are not indexed.
At volume and with links, search breaks down from both insufficient context (esppecially with editorialised, sensationalised, or vague titles) and excess low-yield content.
Just out of curiosity, what do you (anyone reading) see when you search "curry" on DDG images?
And yeah it's weird that the non-image search is fine, results about the food and about the player, but the image results really stand out.
I have to say I didn't know who Steph Curry was before, but I'm impressed that in the US at least he's bigger than the food.
1. It's more likely to respond to the actual query, rather than guessing what I want based on the query.
2. The results are not stuffed with ads and Google service substitutions.
However it's not as good at searching every last corner of the web, so occasionally I need to g!. It's pretty rare that I use that these days, and rarer that it actually gives me a better result.
https://duckduckgo.com/bang?q=vk
Searching YT gives results https://duckduckgo.com/?t=ffab&q=yt
They voluntary banned to be available in Ukraine?
Sometimes Google is better though, especially when I'm looking for something specific (an article I know I read or a video I know I watched, etc)
By contrast, searching 'bitmitigate' on DDG produces the expected outcome: the first result is the official .com site of the company.
(Edit: added the last paragraph comparing to duckduckgo)
And articles pointing out that they serve Daily Stormer and 8chan.
Seems to be working for me? SERP for me (not private browsing, but I'm not signed in and regularly clear cookies)
1. BitMitigate - CDN, WAF, DDoS Protection, Load Balancing, VPN <--- this is the main site, as expected.
2. BitMitigate: What we know about 8chan's new web protection ...
3. BitMitigate (@bitmitigate) | Twitter
4. Epik reverses course, says BitMitigate will not support 8chan ...
5. BitMitigate becomes new security provider for 8chan ...
Now I revert to Google for local and programming. It seems to me tho that the worse Google gets the better ddg gets.
But still beats having AMP, that was last straw for me.
I appreciate it's a smaller index and smaller player but it's providing the true results for that query.
Disclaimer: I work for Mojeek
I am confident that the amount of data that needs to be indexed plays a huge factor too. It just isn’t as easy as it used to be.
There is no sorting. Nothing. The comments are ranked by the AI, the posts are ranked by the AI, the explore section is ranked by the AI.You can't sort by time or likes.
Ten years ago, when Google screwed up, you'd land on a really obvious linkfarm page with a bunch of SEO spam: it was literally just a collection of keywords repeated over and over.
Now, when $search-engine screws up, which it often does, you land on a plausible-looking page, with bad but not terrible English, that just... doesn't tell you anything worth knowing. I know the smell of them, and it can still take me a section or two of precious time to figure out what's going on.
There's often not that much link juice, either: as far as I can tell, the business model is purely to put an ad or two in front of my eyeballs.
BTW, NSFW warning!
If I go on Yandex and search "img " and then a bunch of random numbers, I get almost all pictures of young women, and many pictures of underage girls in provocative poses.
This sort of thing does happen. A few months ago, people discovered similar results on Google Images(!) by searching "TV television and film". The results were being pulled from a 4chan archive of the board with that name.
I don't think it's related to the "VK" web results problem though. If you search "VK," or "VK!" or VK with any punctuation, all the web results come up as expected.
When you want to add a friend on VK, you have to ask them for things like city, school, and age just to narrow down the results!
Is it picking up VK profile pictures? Sort of reminds me of myspace but with, err, Russian sensibilities. I also tried it with Bing and got literally the exact same results.
Google is definitely different, it shows mostly logos instead of photos.
I never used VK but I got the impression it is a real social network, for better or for worse. Should Bing drop them from its index? Now I'm morbidly curios what kind of weird crap searches of social networks from other cultures might dredge up. Like a Chinese VK.
However I'm seeing lots of slightly uncomfortable pictures of young girls (not porn, but didn't want to scroll too far because it felt like there could be some mixed in) and tanks in the image search. This doesn’t change noticeably using the three strictness settings.
[0]: https://www.nytimes.com/interactive/2019/11/09/us/internet-c...