With everything happening in the tech world today, I think there are opportunities for players that have users as their priority in their culture and don't consider them as a product.
I like DuckDuckGo, but I'm getting to the point where I just automatically do !g similar to adding Reddit after a search, which defeats the entire point of DDG.
The problem is I often search weird esoteric things that just don't appear well in DDG. Things about a game from 20 years ago. Specific, niche mods for said games. Indie music artists. Bits and pieces from old television or movies. Lesser known product brands. Almost anything that isn't very well named.
For those things, DDG's first couple (or several) results are often quite poor as they reference something that almost sounds related and is very popular/common, but actually has nothing to do with what I needed.
I like privacy and take easy steps to get it where I can, but I may end up ditching DDG.
I use DDG as my default and rarely use !g. My only wish is that DDG had better “immediate results” like Google, where if I am looking for a conversion or a time in a different city, it just displayed the result rather than me having to go to one of the first results and type in the desired conversion there. I love how Google does that, but hate everything else Google does so it isn’t enough to cause me to switch back.
I've been on ddg for years, have it defaulted in all my browsers/machines, and I can't remember the last time I used !g.
Not sure what you mean about immediate results though... if I search for, say, "10C in F" I get "10 Celsius = 50 Fahrenheit" at the top of the page. And "time in mumbai" gives me "7:37 AM" plus the offset from UTC and the date. Also "weather in chicago".
Same for conversions of lb/kg, mph/kmh, and some other units I just tried, as well as currencies. But either it doesn't do mpg<->L/100km or I'm formatting the search wrong.
It also doesn't answer "hello in french" with an immediate translation like !g does. But I'm ok with missing translations because languages that I want to look up are in wordreference, are easy to make shortcuts for, and generally have better translations and more thorough info.
When I do have to use google directly I sometimes find it _harder_ to get results because it doesn't just show some links and summaries, it seems to want to try to give me "solutions" which aren't always quite what I want. I've been told the results get better the more you use it (ie. the more info you feed to the machine, it learns what you want) but I almost always get what I need on the first attempt searching with ddg and usually within the first few results. If it's not there, my experience has been that I'm not going to find it with !g either. (Usually something to do with really obscure programming problems or error messages.)
An example of where it's lacking in immediate results is searching for Tottenham. DDG provides relevant news articles and a box for the city itself. Google provides the result of the most recent match for Spurs as well as upcoming matches.
I kind of wish somebody would do a command line app for these things. Currency conversions, time in x place, weather, unit conversions, date diffing, etc.
KDE KRunner has some of this, though not on the command line. I'd prefer to have a search engine running locally that handled this kind of thing, just because the browser can display richer results.
You have: 37817.55 per second
You want: per day
* 3.2674363e+09
/ 3.0605034e-10
Or 3,267,436,320 requests per day.
I was profiling redis performance, and I got approximately 37817 requests per second for MSET on 8x terminal windows when running redis-benchmark -P 2000 -c 6 -s /var/run/redis/redis-server.sock
They lost me when they decided to filter out disinformation from search results. I’m good on needing a search engine to “protect” me from wrong ideas.
Short of them completely reversing course and throwing out an apology, I’ve moved on from DDG. Companies that curate content for political reasons are a dime a dozen and we don’t need another one.
Why would you want a search engine to provide you results from websites/companies that are known liars, that you very likely would never know was propaganda etc?
I'll try to answer your question in earnest. "Why would you want a search engine to provide you results from [websites] that are known liars?" Claiming someone is a liar is almost always a gross oversimplification and a form of propaganda in and of itself. Propaganda is partially defined as selectively presenting facts to influence people, which is exactly what you're telling people to do by shutting out opposing viewpoints.
If you are going to then say that people are too stupid to discern real from fake information, and therefore need the loving hand of Big Tech to tell them how to think, that's a pretty negative take. Asking Google or DDG if something is misinformation is like asking Monsanto if it's safe to drink Roundup.
> If you are going to then say that people are too stupid to discern real from fake information, and therefore need the loving hand of Big Tech to tell them how to think, that's a pretty negative take.
The first part of it is pretty busy proving itself true though. Don't get me wrong, I don't think big american tech companies are good curators of this, but previously extinct diseases are showing up again because some people are literally too stupid to vaccinate. There is far too many people who now believe the earth is flat, and so on.
So yeah, people are too stupid to discern real from fake information, and why wouldn't they be? If the school system fails to teach you how science works, then you're going to read sources that say vaccines are dangerous and you're going to see doctors who tell you vaccines are dangerous, because vaccines are actually dangerous. The problem with anti-vaxxers (not all of them, some of them think Bill Gates is inside the vaccine and that's just crazy) is that they read the information, but they don't seem to understand the information. Because vaccines are dangerous, but not vaccinating is more dangerous, which is why we vaccinate. This is completely anecdotal, but some of the anti-vaxxers I've talked with even seem to understand the whole vaccine > not-vaccine, but then don't fully seem to grasp the scale of it. They'll go "yeah, I know it's a little selfish but why would I risk my kids to help others?", which is very understandable, but what they don't consider is what happens when everyone thinks like that. Because vaccines aren't just > not-vaccine. That little > is actually on the scale of society breaking bad, but that part of the information doesn't make it through the noise.
I don't really think the world wide filter is the solution. I think we should get systems that are better at prosecuting people who wilfully spread harmful information instead. So that it doesn't take 10+ years for people who claim things like the Sandy Hooks parents are actors to face the consequences of their evil. On the flipside, I don't really want worldleadersarelizards.com to be the first results on my search engine either.
> Claiming someone is a liar is almost always a gross oversimplification and a form of propaganda in and of itself.
I don't buy this. Seems like fake information to me.
Now, to be clear, I do agree that tech companies must not be trusted to make decisions about politics. But the reason for this isn't because there isn't such a thing as a liar. I know you said "almost always", and didn't say "always a gross simplification", but still.
It's simple. Whenever I query for "X", under the hood duck duck go queries "X reddit".
Jokes aside I think fixing the SEO hacking as the author mentions would go the longest way. It's really bad for stack overflow posts. When I Google a programming error the first few results are copy cat sites with the exact Stack Overflow questions and replies. I don't understand how SEO has evolved to favor that over the actual, more popular, stack overflow page.
Both Google and DDG already do that, you just format the query correctly. Once you are an expert in a field you will begin to understand how bad reddit is for information.
Everyone on HN is catching up to this, but what we are all realizing is we want actual user curation, trusted user curation.
Reddit is currently a good proxy because the best recommendations on Reddit for a question are upvoted reasonably democratically by the type of people you’d trust. For the time being the gamification of Reddit isn’t bad enough to be an issue here.
But that’s just a matter of time. As more and more people start using Reddit as the most authentic source of dependable information, folks are gonna start gaming that to the core as well, basically this is going to be a game like the Red Queen.
One possibility is that the search engine itself starts becoming like Reddit maybe? They clearly know what result you’re clicking, they could start making user similarities (or ask you for an explicit list of your peers) to find similar users and rank results based on what people like you click. Obviously google can do it easier since it knows YOu. But even DDG maybe able to do something like user curation or site blacklisting (and have popular lists like exist for tracking avoidance that you can subscribe to). This is the only solution IMO.
For example for any coding related search I’d be happy to rid of every top 10 result in google except SO. All the rest is garbage.
Maybe there’s also an upvote button. But the score is not universal but personalized to you based on votes from people you actually might trust.
Agreed. There are certain topics which for which one cannot find anything resembling unbiased and factual, and it's getting worse as the owners continue to turn Reddit into a sanitised, ad-friendly social platform. If one is using Reddit to find relevant information, they first need knowledge about which subreddits can be trusted, and this is in constant flux.
> Reddit is already a cesspool for many issues including politics, news and race/culture based questions.
So? The blast radius of those issues doesn't fully spillover into the non-political forums.
You want to know which of two different products in a particular class is better? /r/<productclass> won't be filled with political or culture-based proselytising.
Looking to buy a new laptop/car/power-tool/chair? Good advice is to be found on reddit and you may never even see a single off-topic discussion in your search.
I just gave Neeva a shot and it's really great. I love that I can up and down rank domains, and it doesn't appear to filter domains using some shadowy ideological algorithm. I don't seem to have access to that Onebox feature yet but I imagine it's being rolled out by region.
One question: why are there no dates under results? Even for articles where the date is near the top. This is something I rely on heavily to filter out the older results. To provide an example, I searched for "keychron k2v2" and found this article from Sep 26, 2020 (https://appleinsider.com/articles/20/09/26/review-the-keychr...). I then filtered to show me only results from the last year, and this result still popped up. This suggests the Neeva engine doesn't detect dates at all.
> So, that’s another opportunity for DuckDuckGo, by ranking the results based on the content’s quality and relevance, as Google used to do before.
This makes it sound like that's easy. Yes, SEO twattery is an issue, and with the growth of the web it's also much more prevalent and the stakes are higher. Google from 2022 has a much harder problem to solve than Google from 2000. DDG can't "just" solve that just as Google can't "just" solve that.
Let users just rate that search result, so if someone else searched for the same thing and didn't find it relevant, they can thumbs down. If they found it relevant, they can thumb up.
There should be a relevance score, and it might be cross-referenced to other similar search queries that mean the same thing, but that's a machine learning project all on itself
That only works if people are using the same search term AND looking for exactly the same thing... If they're looking for different search things but using the same search terms (i.e. some people are using sub-optimal search terms), that's just going to add effective random noise to the ratings.
Given how location-aware some things sometimes need to be (including spellings / alternative words, which in my experience, Google is a lot better at than DDG), I don't see how that works at all.
Nobody said I can't use a SIMILAR search query to have a slightly lower bound for relevance. Machine learning can identify if two queries mean a similar thing in the context of search
You don't just average user ratings, you need to have a smart system.
First, each user needs to have a history or they won't be considered in the rating almost at all. Then, each user's history gets analyzed for average rating vs. the average userbase so you can determine whether this user upvotes everything or only rates the sites that are terrible.
So each user gets a score of "threshhold" of what rating site they will thumbs up. Some users will thumbs up a site that's 50% (mostly useless, but not outright spam), while some only upvote sites that are 80% (good content at the very least).
Then regressed for this variable, you can calculate their feelings on each site and see if it correlates well with other users. The more ratings, the more each additional rating is weighted.
This will reduce the number of ratings required for a statistically valid sample for each search query (since you need to tell users if it's relevant for the thing being searched, not just quality of the site). But you also need to fight bots as well.
This system may not be perfect, but it would add value to a search engine
Mate, I'm going to pay $1000 and get a bunch of Filipinos in a LAN centre using that SaaS company that routes your requests through other people's VPN software to click a bunch of thumbs up. Watch as my cialis video tops your search for EC2 instances.
Not OP, but allow users to follow others and create an aggregate filter. That leads to its own problems, but if results are done with a weighted preference depending on if have been noted as "good" by people I've followed, or to a lesser extent those they've followed, it may produce better results.
You need to do it with user accounts so the search engine team can remove fake clicks and ban the user. Users with a low amount of ratings won't have any say in the rating of the site, since they will have a low confidence.
They will first need to rate a few hundred sites (valid ratings, because if they don't have high validity with the rest of the sites in the database the confidence this is a valid user will be low) to give the spam site a high rating. So they will have already built a database of good ratings before they can give the last vote that they get paid for?
Also, you can manually remove a spam site that uses this tactic and all their money will have been wasted.
Even better, let users leave comments on on search results. That stops all the "search query +reddit" searches because now Google is reddit. The discussions are happening on the search engine response page.
I disagree; Google isn't even trying - after all, Pinterest is still polluting image searches despite not even trying to do anything malicious.
However even if we assume that detecting spam content is hard, detecting the funding of said content is much easier - most of this content is there to get you to buy something or look at ads, and they use analytics tools to track conversion.
Use the presence of ads, analytics, affiliate links and other marketing tools as a proxy for "spam" and downrank them, or offer the user a "non-commercial" option like Kagi does.
Now you move the problem from old-school SEO "make my content rank higher" to "conceal my ads/analytics/etc from the search crawler so I don't get downranked". That is a very effective strategy because advertising has to be served via third-party code to protect against ad-fraud, but third-party scripts are trivially detectable.
Now the spammers and their partners have a problem - the only way to fool the crawler is to self-host the ads, but doing that would expose the advertisers to ad fraud. You're pitting the spammers against each other in a game where the only winning move is not to play and the extra overhead will make the spam less profitable (or not profitable at all).
If search engines start penalizing this kind of content then the 0.1% might grow. Not to mention, most of my web search usage nowadays revolves around open-source library documentation, blogs, GitHub and occasionally StackOverflow, so being constrained to the 0.1% of the non-commercial web doesn't appear to be a downside at least when it comes to searching work-related issues as a programmer.
A lot of the high quality content that exists on the internet only exists because the advertising pays for the people who create it.
Be careful what you wish for. If you make sites with ads rank lower the void will just be filled with crypto scams or affiliate marketing. Remember it's a lot easier to make a spam site than a high quality one, changes that disincentive high quality sites from existing won't necessarily do the same for spam sites.
Quantity of tracking and advertising is a great proxy for spam content. If a page contains 10+ affiliate links, I don’t want to see it. If it contains just one it might be more legitimate content like an author talking about their book and link which gives them a commission on it.
I've been using crowd-sourced tools like SponsorBlock and Return YouTube Dislike, and I really like them. So far they've resisted attack using some kind of consensus algorithm. I've also been using the search engine Kagi, which lets me downrank and even eliminate certain domains and sites which I believe to be low quality. I wonder if it's possible to marry these. If the majority of users dislike a domain, it gets downranked. Of course bot detection and perhaps the use of logins might be required, but compromises must be made if the desire is a better search engine.
I follow a lot of YouTube creators who product genuinely excellent and valuable content. They typically limit their sponsorships to one short segment per long video. At the same time, their video descriptions often contain numerous affiliate links, preceded by a disclosure that the following links are to be used if you’re interested in purchasing one of the products and you want to support the creator.
Those links honestly do not bother me at all. I barely even notice them since video descriptions are hidden on YouTube, apart from a few sentences at the top. So how do we make search engines that distinguish legitimate creators like that from spammers who create fake blogs with fake reviews and tons of affiliate links, all using a fancy blog template that makes the site seem trustworthy?
YouTube descriptions started being useless since they became a copy/pasted generic blurb of text (irrelevant to the video at hand) and a dumping ground for various social media or similar links, so no surprise that you aren't bothered by spam links in something that nobody looks at anymore because it's already been spammed to hell. But there was a time when descriptions were all manually written and were an actual summary/overview of what's in the video.
> So how do we make search engines that distinguish legitimate creators like that from spammers who create fake blogs with fake reviews and tons of affiliate links, all using a fancy blog template that makes the site seem trustworthy?
It's a whole other problem for videos so I'm going to approach this from a website search results point of view: we simply use presence of commercial content as a ranking factor - if the commercial content is all that matches your query then you get it, but if something non-commercial matches your query equally well then it comes first. This encourages non-commercial content without banning commercial content entirely. Ads/marketing/etc are noise, so it would be normal for a search engine designed to serve the user to rank low-noise content above noisy content, right?
I don't know what content you consume, but most commercial content I find is garbage, while non-commercial content shines because it's produced by someone who actually cares.
First, what's wrong with having Pinterest images in the search results? For the types of queries where they show up, they usually seem to be pretty relevant.
Second, many high-quality sites use ads and analytics. What are you going to do? Downrank the New York Times because it has ads and uprank your local conspiracy theorist because he doesn't?
Pinterest hides the actual image from the search results in a sea of unrelated images, and blocks you from accessing the images without a login.
It’s impossible to find the actual image from the search results if the source is pinterest now that Google removed the direct image link from the search results.
> Downrank the New York Times because it has ads and uprank your local conspiracy theorist because he doesn't?
If the only difference in ranking between the NYT and the local conspiracy theorist is the presence of ads then yes absolutely. Realistically speaking there's no way the local conspiracy theorist will get the SEO "juice" of the NYT (including domain age/reputation, backlinks, etc), but if he does then he deserves his place. I don't want a search engine to be the one deciding what is considered the "truth".
Yes. Why hadn't I thought about it? Everyone talks about how easy would be to solve this problem. Well if it so easy do it and become the next Page and Brin, instead of suggesting other to do.
No need to say why [particular brand] of search engine can outperform Google. Really, any search engine could outperform Google by doing the following:
1. When I search for "X Y Z", ensure that the resulting documents contain X AND Y AND Z. The plus signs and quotes don't work. No exceptions to this rule, unless the user specifically requests "OR" searches, fuzzy searches, stemming, synonyms, etc. But no one will request those things because those things suck. (Recommending spelling corrections are okay, however. Just don't automatically perform them.)
2. Do not show me what "People also ask". Fire the product manager who is keeping this shit in the results.
3. Do not mix images and video and news with web results. I see the "images" etc links. I know how to click on things.
...
Has anybody had this experience with Google lately: Fighting and fighting to get it to NOT show 800 billion non-matching results, and then finally getting the actual result you want, only to have it be accompanied by the warning, "It looks like there aren't many good results for your search." Bad design award of the year, for that one.
Their mobile browser makes search faster. It has a nice autofill email feature to proxy your own with a @duck.com domain, stripping the trackers from them. The !bang feature has been useful for me socially, like in conversations, making search quick enough to be a part of it.
Now if they can pursue some tactful strategies like the article hopes for, they can eject the Microsoft ads that they have been grinding their teeth over.
For me, DDG just needs to add blacklists/whitelists to search results that users can customize and I'll stay. I'm looking at switching away because SEO is so bad.
Kagi has this and it's amazing. The domain ranking is a range so you can completely blacklist something or you can make it appear higher in the results
Yes, so much autogenerated nonsense SEO sites in every search for me on DDG for about a year now, also thinking of moving back to google. Sure, !g, but if I use it every time, why not just google?
Every time I try g! to bypass SEO or “no relevant results”, the google results are just as bad as the DDG ones. I’ve found myself trying g! more often this year, but it has worked zero times for me in the last > 5 years.
DDG is a dead end. The good news, if you already tolerate DDG mostly because of the bangs, you'll love Kagi (https://kagi.com/), which supports them and has far superior ad-free search results.
Disclaimer: No connection except as a satisfied paying user ($12/mo).
I appreciate their stated commitment to avoid censorship in general, but would prefer an explicit dismissal of ideological filtering. By the way, their pricing page is currently offering premium service for $10/mo.
Unrelated - I use kagi too! I was watching a supervised learning series on YouTube by Killian Weinberger here [0] and he used an email from kagi as an example of spam.
I’d like to try out Kagi but having every search I do tied to my real world identity (via payment information) with no ability to search privately really kills it for me.
Is it possible to take the google results and apply a domain score overlay on it to get rid of the spammy SEO sites? ie a community curated whitelist of sites that get preferential treatment
If a person had the knowledge/skills to get a spammy SEO site to rank high in Google, they would likely have the knowledge/skills to manipulate a community curated whitelist/blacklist.
You could have a group of humans vote on whether sites can be included in the whitelist and a main criteria is that sites nexus to an actual human involvement. Could that work?
For example, they may show a copy of stack overflow because it has more ads - for many users they will still click through and deem the copy site sufficient.
Same thing with other SEO sites that blatantly copy other websites
Do you have any evidence of this? It would be a huge scandal if Google was found to be boosting organic SERP rankings/placement based on the site's use of Google Ads and I have no reason to believe this is true.
The search algo is just a community curated whitelist, it's just more efficient than a human maintained one, using more metrics and optimising for relevance instead of ban/don't ban a site.
The problem Google has now is either: it's algorithm is broken and it isn't showing relevant or quality results, or, maybe, the majority of the web is just trash and there are lots of relevant but very few "quality" results
Brave state that it's 'built on top of an independent index' but once you compare a few queries, it's obvious that they're just using a slightly modified version of Google results.
At least for me google search results have degraded drastically over the past years, especially over maybe the last. I haven't used it more than a few times in the past few months. I find brave search to have the original google quality, allowing me to find things right away. Maybe it's because google is adapting towards natural language queries, and brave is still "old school", and I haven't adapted, I don't know...
Second this. I have been using Brave as my default for half a year now and I am quite happy with it. It is remarkably free of spam, probably because it is not actively targeted by spammers (yet). I use Google if I am looking for something to buy, because that is where all the ads and deals are.
It's not just how DDG works, but how search works and how difficult the search problem is. This was a complete waste of my time and people voting this up are clearly equally clueless or haven't even bothered to read it before voting.
A choice quote from the article:
> So, that’s another opportunity for DuckDuckGo, by ranking the results based on the content’s quality and relevance, as Google used to do before. I consider that there are some methods to accomplish this; one could be to rank the content through the community or actually use artificial intelligence to categorize the academic/relevance level.
Yeah but I'm accustomed to the HN ignorance of the form "why do they not simply rank the document I want at the top?" People may underestimate the difficulties of web search and I understand that. But how can they not understand that DDG is just a marketing outlet of Bing? That doesn't take any thought at all.
Fair. I guess different "ignorance" ticks off you and me differently.
I dismissed that aspect, since I assumed it's possible people are simply interpreting "DDG" as "any search engine that's not Google".
Moved from DDG to Brave Search about 6 months ago and haven't missed DDG at all. I can still use the !bangs I was accustomed to.
Given that Brave uses its own search index, and only a few times do I need to fall back to Google, it feels good to have a measure of independence from the Google/Bing dominance in search
Brave automatically changed my default search engine on private tabs to their search. I don't know if it was a bug or feature, but this kind of behaviour keeps me wary from committing.
Did not know that. I immediately grabbed some recent "poor result" Google searches from history and tried them at search.brave.com — so far ALL results are better. Reminds me a bit of Google from ten years ago.
Google's decline into its present inability to NOT show only "assumed popular content" created a huge time sink in my life that wasn't there before. Not a fan of the over- and inaccurate use of "disruption" but it was a legitimate description for what Google did to other search engines, and this feels like a similar level of improvement. I'm frankly amazed at how good the search results are in my initial test.
> I immediately grabbed some recent "poor result" Google searches from history
Did you revisit every query from your recent Google searches (or) Is it that you have a good memory that you were able to remember poor results from looking at the query?
I'm asking this because, I've been contemplating a 'Search Engine Wall of Shame'[1] where people can submit their poor search results for the engines to make actionable changes towards improving them.
I re-opened several Google SERPs from my history and compared to Brave.
I often find NO results relevant to my query on the first three pages of search Google results, often seemingly because there is a "more common/popular" aspect of the topic I am searching about, and everything is about the common and related aspects and none about my use case.
Would you be willing to submit your bad search results to such forum? I welcome you to post your bad search results with the queries to that needgap thread for the time being unless there's a dedicated forum for that.
Why hasn't DDG grown internal search competency? It's been years, and there's still no sign of search being a first class concern at DDG. That seems like a number one must have for any search product company.
Rebranded Bing with a new interface isn't hard tech. It risks so much on the business relationship. They should be staffing for this yesterday.
Hire more search folks, DDG! Also, ditch that awfully long name.
would also like to hear from DDG ppl. On another note, Really appreciate DDG's map functionality using apple map with iaxm query parameter set "maps" and q set to whatever you want to search for:
They are welcome too, but perhaps are restrained by their stakeholders.
Looking at the suggestions in the blog, and for transparency - Mojeek CEO:
> Stop trying to look like Google.
Agreed. We and our users still believe that "10 blue links" have great value. But drowning these out with too many of things like Ads, Videos and Answers on the page helps pretty much just Google.
> Arrange that algorithm to make it less vulnerable to SEO hacking.
Admittedly we don't yet have that problem yet. Still there are plenty of measures that we provide, and will expand upon to mitigate that. Without going into details these generally amount to giving users, and API customers, more control over searching and ranking.
> Discard AI-generated text.
A good idea. But can also be done on the SERPs. Do users benefit from AI answers? Mojeek is a search Engine not an answer Engine.
> Results in other languages.
Bing?
> New opportunities.
DuckDuckGo is doing a great job of providing new and improved privacy products. We stand with them on many things. I am sure they appreciate the opportunity also in providing more informational (search) diversity from and with a fully independent search stack (infrastructure, crawler, index, ranking).
This seems to be the gist of the response you've linked:
> mobile searches are the largest category of searches, and local searches are the largest category of searches within mobile. Instead our local search content is a combination of our own indexes in partnership with Apple, TripAdvisor, and others.
This suggests Bing isn't used for local searches, but it is still used for mobile and desktop searches that aren't local. If I am searching for a tech problem or how to tie a knot or recipes or the history of hair metal, I'm getting Bing results wrapped in DDG. Is that inaccurate? Is Bing still used for searches that aren't local searches?
We've indexed billions of pages already at you.com to avoid this situation.
The majority of our apps like Stackoverflow and Medium etc, we've indexed ourselves.
The absolute flattening of the "daily searches" chart is interesting and I'm curious how it has continued.
I know a handful of us immediately switched out of DDG when they decided they were going to artificially down-rank Russian web sites from results[1], but I imagine the overlap between DDG users and people who care about this is quite small.
It is likely due to the most recent period being a partial time span. These articles come out every few months, and the most recent period is always low.
As for the Russia thing, as far as I can tell, they still show Russian propaganda sites (and often as the first result). They just don’t let SEO’ed sock puppet sites drown out all the other results.
I imagine the number of people that switched away because of that is vanishingly small.
The killer feature would be to rate this search result. If I could see other users hate the site, I won't spend time looking at it. Thumbs up and down would work just fine.
I just want to see if it's a spam/scam page and avoid those results with less than 50% rating. The same way I used to see thumbs up/down on YouTube videos to see which videos are clickbait
If they upvote a lot of spam sites, their account will have a low confidence rating, and won't influence each site's confidence.
If they actually downvote a few hundred spam sites before casting a vote for a spam site to promote it, then they will have built a 99.5% good database already and one site can be manually removed. The only way to have a high confidence in a user if they both downvote bad sites and upvote good sites. If they did that work to verify their accounts to be valid, then they will have done more good than harm
DDG works great if you search for `happy synonyms`. It even has an equivalent list of synonyms embedded in the results page.
Actually even with "what is another word for happy", the first result is a wordhippo synonyms page.
Though I agree that question based queries are probably more popular these days, and DDG isn't handling them as well. I imagine this doesn't bother the tech / privacy crowd too much.
That’s a one box / instant answers query. A DDG search for “happy synonym” (no quotes) provides synonyms and antonyms.
The google result shows more antonyms than the DDG one, but the DDG result has a click through to rhymes and sounds-like words.
So, it is a draw in my book. I don’t try to ask english language questions to search engines though (and find it uniformly produces worse results than listing keywords, but YMMV).
Based on autocomplete suggestions, it looks like google auto rewrites your query to mine.
Arguably, it would be good to be able to search for pages containing phrases like the one you typed, instead of just ignoring user intent for people that have used search engines before.
Yeah, but if you want to compete with Google your going to have to serve the answer faster. No one wants to click on webpage to find an answer to a simple question.
This shows Google is way better at understanding what the user wants when searching.
"what is another word for happy" "happy synonyms", and "happy synonym" all give the user what they want right away on Google doesn't matter how you search it.
With DDG it does matter how you search it.
Another example is another simple query like "florida capital". Google in big letters tell your what the capital of Florida. It shows you a map of the city that you can click and shows basic information on the city.
DDG gives you information on the capitol building and the first link is too the Florida Capital Bank website.
Why would I ever use DDG over Google? Other than privacy which I don't think is that big of an issue as people make it out too be on Google.
I think any search engine with smallish market share has an interesting advantage that few people are going to do search engine optimization targeting minor search engines. So, if one uses a different ranking algorithm, even if it isn't technically better in any sense, it's still an advantage in that it's using signals that people aren't deliberately trying to subvert.
>"Total Internet Users Worldwide" *numbers are estimates.
According to the world map infograph in the article there are 4.5 million more people using the internet in Australia than there are people in Australia...?
So I'd say the estimates aren't great, or they're likely estimating number of internet connections rather than the people using them.
Recently the son of a radio personality died. I was curious to if the family had released a cause of death and I was stunned by how many spam/bot-written sites where out there churning content for a fairly obscure death.
If there are that many for the son of someone on the radio, I can't imagine how many junk sites there are for actual celebrities. It is a huge problem and I'm not surprised even google can't stop all the crap from rising to the top.
I never used DuckDuckGo because they used Yandex as one of their backends. I just did not want my queries, even anonymised, sent to Russia. They "paused" this practice only in 2022. Too little, too late.
We will never see this happen, but I wish there was a legally mandated separation of web indexing services from the result ranking services. So user-facing search engines would be forced to rely on independent back-end companies for indexing the pages and storing results (in a neutral and interchangeable way), and launching a competing search engine would not involve a multi-billion-dollar investment.
> So user-facing search engines would be forced to rely on independent back-end companies
Interesting notion.
So, if a state or business wanted to suppress certain information, they'd only have to kneel on the neck of one indexing company, rather than many. Having said that, if the index is certifiably clean, fair and uncensored, and access is free, I can see huge value in a public global web-index.
Who cares about “how DuckDuckGo can outperform Google”? For some users, it does. Their value proposition is privacy, which for me was appealing enough to move from Google. When I found out they were actively censoring search queries, it was enough for me to move elsewhere. Every solution does not need to be THE solution for all users. There is nothing wrong with being a niche platform focused on catering to a specific demographic. In fact, it probably allows the platform to better cater to that exact demographic.
If DuckDuckGo really wants to beat Google, they could start by fixing the way the minus sign works in their searches. For example, if I type "cats -dogs" I don't want the first five result I get back to be all about dogs. This is the single biggest reason I ditched DuckDuckGo for Brave Search.
I'm not literally talking about cats and dogs. Cats and dogs is DuckDuckGo's syntax example. DuckDuckGo's minus sign functionality is a well-known issue.
195 comments
[ 3.1 ms ] story [ 251 ms ] threadThe problem is I often search weird esoteric things that just don't appear well in DDG. Things about a game from 20 years ago. Specific, niche mods for said games. Indie music artists. Bits and pieces from old television or movies. Lesser known product brands. Almost anything that isn't very well named.
For those things, DDG's first couple (or several) results are often quite poor as they reference something that almost sounds related and is very popular/common, but actually has nothing to do with what I needed.
I like privacy and take easy steps to get it where I can, but I may end up ditching DDG.
Not sure what you mean about immediate results though... if I search for, say, "10C in F" I get "10 Celsius = 50 Fahrenheit" at the top of the page. And "time in mumbai" gives me "7:37 AM" plus the offset from UTC and the date. Also "weather in chicago".
Same for conversions of lb/kg, mph/kmh, and some other units I just tried, as well as currencies. But either it doesn't do mpg<->L/100km or I'm formatting the search wrong.
It also doesn't answer "hello in french" with an immediate translation like !g does. But I'm ok with missing translations because languages that I want to look up are in wordreference, are easy to make shortcuts for, and generally have better translations and more thorough info.
When I do have to use google directly I sometimes find it _harder_ to get results because it doesn't just show some links and summaries, it seems to want to try to give me "solutions" which aren't always quite what I want. I've been told the results get better the more you use it (ie. the more info you feed to the machine, it learns what you want) but I almost always get what I need on the first attempt searching with ddg and usually within the first few results. If it's not there, my experience has been that I'm not going to find it with !g either. (Usually something to do with really obscure programming problems or error messages.)
Most of it could even be done offline.
$ units
You have: 37817.55 per second You want: per day * 3.2674363e+09 / 3.0605034e-10
Or 3,267,436,320 requests per day.
I was profiling redis performance, and I got approximately 37817 requests per second for MSET on 8x terminal windows when running redis-benchmark -P 2000 -c 6 -s /var/run/redis/redis-server.sock
Short of them completely reversing course and throwing out an apology, I’ve moved on from DDG. Companies that curate content for political reasons are a dime a dozen and we don’t need another one.
If you are going to then say that people are too stupid to discern real from fake information, and therefore need the loving hand of Big Tech to tell them how to think, that's a pretty negative take. Asking Google or DDG if something is misinformation is like asking Monsanto if it's safe to drink Roundup.
The first part of it is pretty busy proving itself true though. Don't get me wrong, I don't think big american tech companies are good curators of this, but previously extinct diseases are showing up again because some people are literally too stupid to vaccinate. There is far too many people who now believe the earth is flat, and so on.
So yeah, people are too stupid to discern real from fake information, and why wouldn't they be? If the school system fails to teach you how science works, then you're going to read sources that say vaccines are dangerous and you're going to see doctors who tell you vaccines are dangerous, because vaccines are actually dangerous. The problem with anti-vaxxers (not all of them, some of them think Bill Gates is inside the vaccine and that's just crazy) is that they read the information, but they don't seem to understand the information. Because vaccines are dangerous, but not vaccinating is more dangerous, which is why we vaccinate. This is completely anecdotal, but some of the anti-vaxxers I've talked with even seem to understand the whole vaccine > not-vaccine, but then don't fully seem to grasp the scale of it. They'll go "yeah, I know it's a little selfish but why would I risk my kids to help others?", which is very understandable, but what they don't consider is what happens when everyone thinks like that. Because vaccines aren't just > not-vaccine. That little > is actually on the scale of society breaking bad, but that part of the information doesn't make it through the noise.
I don't really think the world wide filter is the solution. I think we should get systems that are better at prosecuting people who wilfully spread harmful information instead. So that it doesn't take 10+ years for people who claim things like the Sandy Hooks parents are actors to face the consequences of their evil. On the flipside, I don't really want worldleadersarelizards.com to be the first results on my search engine either.
I don't buy this. Seems like fake information to me.
Now, to be clear, I do agree that tech companies must not be trusted to make decisions about politics. But the reason for this isn't because there isn't such a thing as a liar. I know you said "almost always", and didn't say "always a gross simplification", but still.
There are liars. They are people who tell lies.
Jokes aside I think fixing the SEO hacking as the author mentions would go the longest way. It's really bad for stack overflow posts. When I Google a programming error the first few results are copy cat sites with the exact Stack Overflow questions and replies. I don't understand how SEO has evolved to favor that over the actual, more popular, stack overflow page.
Both Google and DDG already do that, you just format the query correctly. Once you are an expert in a field you will begin to understand how bad reddit is for information.
Reddit is currently a good proxy because the best recommendations on Reddit for a question are upvoted reasonably democratically by the type of people you’d trust. For the time being the gamification of Reddit isn’t bad enough to be an issue here.
But that’s just a matter of time. As more and more people start using Reddit as the most authentic source of dependable information, folks are gonna start gaming that to the core as well, basically this is going to be a game like the Red Queen.
One possibility is that the search engine itself starts becoming like Reddit maybe? They clearly know what result you’re clicking, they could start making user similarities (or ask you for an explicit list of your peers) to find similar users and rank results based on what people like you click. Obviously google can do it easier since it knows YOu. But even DDG maybe able to do something like user curation or site blacklisting (and have popular lists like exist for tracking avoidance that you can subscribe to). This is the only solution IMO.
For example for any coding related search I’d be happy to rid of every top 10 result in google except SO. All the rest is garbage.
Maybe there’s also an upvote button. But the score is not universal but personalized to you based on votes from people you actually might trust.
So? The blast radius of those issues doesn't fully spillover into the non-political forums.
You want to know which of two different products in a particular class is better? /r/<productclass> won't be filled with political or culture-based proselytising.
Looking to buy a new laptop/car/power-tool/chair? Good advice is to be found on reddit and you may never even see a single off-topic discussion in your search.
I couldn't agree more which is why we recently launched our Onebox product bringing you forum content directly inline with your search results.
https://neeva.com/blog/introducing-neeva-onebox-community-co...
One question: why are there no dates under results? Even for articles where the date is near the top. This is something I rely on heavily to filter out the older results. To provide an example, I searched for "keychron k2v2" and found this article from Sep 26, 2020 (https://appleinsider.com/articles/20/09/26/review-the-keychr...). I then filtered to show me only results from the last year, and this result still popped up. This suggests the Neeva engine doesn't detect dates at all.
We're actively working on this!
We should have significantly better coverage for dates in results in the next few weeks. Stay tuned!
Content written by humans has been moving to walled gardens like Facebook, Twitter, tiktok, Reddit.
Most of these don’t let google index them properly, as they would prefer uses create an account on their site and use their own search.
https://search.brave.com/goggles?goggles_id=https%3A%2F%2Fra...
This makes it sound like that's easy. Yes, SEO twattery is an issue, and with the growth of the web it's also much more prevalent and the stakes are higher. Google from 2022 has a much harder problem to solve than Google from 2000. DDG can't "just" solve that just as Google can't "just" solve that.
Given how location-aware some things sometimes need to be (including spellings / alternative words, which in my experience, Google is a lot better at than DDG), I don't see how that works at all.
First, each user needs to have a history or they won't be considered in the rating almost at all. Then, each user's history gets analyzed for average rating vs. the average userbase so you can determine whether this user upvotes everything or only rates the sites that are terrible.
So each user gets a score of "threshhold" of what rating site they will thumbs up. Some users will thumbs up a site that's 50% (mostly useless, but not outright spam), while some only upvote sites that are 80% (good content at the very least).
Then regressed for this variable, you can calculate their feelings on each site and see if it correlates well with other users. The more ratings, the more each additional rating is weighted.
This will reduce the number of ratings required for a statistically valid sample for each search query (since you need to tell users if it's relevant for the thing being searched, not just quality of the site). But you also need to fight bots as well.
This system may not be perfect, but it would add value to a search engine
They will first need to rate a few hundred sites (valid ratings, because if they don't have high validity with the rest of the sites in the database the confidence this is a valid user will be low) to give the spam site a high rating. So they will have already built a database of good ratings before they can give the last vote that they get paid for?
Also, you can manually remove a spam site that uses this tactic and all their money will have been wasted.
However even if we assume that detecting spam content is hard, detecting the funding of said content is much easier - most of this content is there to get you to buy something or look at ads, and they use analytics tools to track conversion.
Use the presence of ads, analytics, affiliate links and other marketing tools as a proxy for "spam" and downrank them, or offer the user a "non-commercial" option like Kagi does.
Now you move the problem from old-school SEO "make my content rank higher" to "conceal my ads/analytics/etc from the search crawler so I don't get downranked". That is a very effective strategy because advertising has to be served via third-party code to protect against ad-fraud, but third-party scripts are trivially detectable.
Now the spammers and their partners have a problem - the only way to fool the crawler is to self-host the ads, but doing that would expose the advertisers to ad fraud. You're pitting the spammers against each other in a game where the only winning move is not to play and the extra overhead will make the spam less profitable (or not profitable at all).
Be careful what you wish for. If you make sites with ads rank lower the void will just be filled with crypto scams or affiliate marketing. Remember it's a lot easier to make a spam site than a high quality one, changes that disincentive high quality sites from existing won't necessarily do the same for spam sites.
Those links honestly do not bother me at all. I barely even notice them since video descriptions are hidden on YouTube, apart from a few sentences at the top. So how do we make search engines that distinguish legitimate creators like that from spammers who create fake blogs with fake reviews and tons of affiliate links, all using a fancy blog template that makes the site seem trustworthy?
> So how do we make search engines that distinguish legitimate creators like that from spammers who create fake blogs with fake reviews and tons of affiliate links, all using a fancy blog template that makes the site seem trustworthy?
It's a whole other problem for videos so I'm going to approach this from a website search results point of view: we simply use presence of commercial content as a ranking factor - if the commercial content is all that matches your query then you get it, but if something non-commercial matches your query equally well then it comes first. This encourages non-commercial content without banning commercial content entirely. Ads/marketing/etc are noise, so it would be normal for a search engine designed to serve the user to rank low-noise content above noisy content, right?
Second, many high-quality sites use ads and analytics. What are you going to do? Downrank the New York Times because it has ads and uprank your local conspiracy theorist because he doesn't?
It’s impossible to find the actual image from the search results if the source is pinterest now that Google removed the direct image link from the search results.
If the only difference in ranking between the NYT and the local conspiracy theorist is the presence of ads then yes absolutely. Realistically speaking there's no way the local conspiracy theorist will get the SEO "juice" of the NYT (including domain age/reputation, backlinks, etc), but if he does then he deserves his place. I don't want a search engine to be the one deciding what is considered the "truth".
1. When I search for "X Y Z", ensure that the resulting documents contain X AND Y AND Z. The plus signs and quotes don't work. No exceptions to this rule, unless the user specifically requests "OR" searches, fuzzy searches, stemming, synonyms, etc. But no one will request those things because those things suck. (Recommending spelling corrections are okay, however. Just don't automatically perform them.)
2. Do not show me what "People also ask". Fire the product manager who is keeping this shit in the results.
3. Do not mix images and video and news with web results. I see the "images" etc links. I know how to click on things.
...
Has anybody had this experience with Google lately: Fighting and fighting to get it to NOT show 800 billion non-matching results, and then finally getting the actual result you want, only to have it be accompanied by the warning, "It looks like there aren't many good results for your search." Bad design award of the year, for that one.
https://blog.google/products/search/how-were-improving-searc...
Now if they can pursue some tactful strategies like the article hopes for, they can eject the Microsoft ads that they have been grinding their teeth over.
2022-06-22 https://news.ycombinator.com/item?id=31837986
Disclaimer: No connection except as a satisfied paying user ($12/mo).
[0] https://youtube.com/playlist?list=PLl8OlHZGYOQ7bkVbuRthEsaLr...
https://kagi.com/privacy
Is it possible to take the google results and apply a domain score overlay on it to get rid of the spammy SEO sites? ie a community curated whitelist of sites that get preferential treatment
Same thing with other SEO sites that blatantly copy other websites
The problem Google has now is either: it's algorithm is broken and it isn't showing relevant or quality results, or, maybe, the majority of the web is just trash and there are lots of relevant but very few "quality" results
https://search.brave.com/
A choice quote from the article:
> So, that’s another opportunity for DuckDuckGo, by ranking the results based on the content’s quality and relevance, as Google used to do before. I consider that there are some methods to accomplish this; one could be to rank the content through the community or actually use artificial intelligence to categorize the academic/relevance level.
LOL.
People on HN knows that Bing powers much of DDG, but how would the average user know that?
Unfortunately, as long as DDG depends on third-party crawlers, the suggestions to improve search results (& "the algorithm") seem far-fetched & naive.
(DDG does have its own crawler, DuckDuckBot, but apparently it's only used for very specific functionality.) https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
> For DuckDuckGo, it may be tricky to resolve the issue permanently as long as it relies on Bing. https://torrentfreak.com/duckduckgo-restores-pirate-sites-an...
> According to various online forums, the best way to ensure your site gets indexed by DuckDuckGo is to submit it to Bing and Yandex. https://www.jessesquires.com/blog/2022/03/25/my-website-disa...
Given that Brave uses its own search index, and only a few times do I need to fall back to Google, it feels good to have a measure of independence from the Google/Bing dominance in search
Love being able to customize the ranking order for specific domains.
Did not know that. I immediately grabbed some recent "poor result" Google searches from history and tried them at search.brave.com — so far ALL results are better. Reminds me a bit of Google from ten years ago.
Google's decline into its present inability to NOT show only "assumed popular content" created a huge time sink in my life that wasn't there before. Not a fan of the over- and inaccurate use of "disruption" but it was a legitimate description for what Google did to other search engines, and this feels like a similar level of improvement. I'm frankly amazed at how good the search results are in my initial test.
Did you revisit every query from your recent Google searches (or) Is it that you have a good memory that you were able to remember poor results from looking at the query?
I'm asking this because, I've been contemplating a 'Search Engine Wall of Shame'[1] where people can submit their poor search results for the engines to make actionable changes towards improving them.
[1] https://needgap.com/problems/207-search-engine-wall-of-shame... (Disclosure: It's my problem validation forum).
I re-opened several Google SERPs from my history and compared to Brave.
I often find NO results relevant to my query on the first three pages of search Google results, often seemingly because there is a "more common/popular" aspect of the topic I am searching about, and everything is about the common and related aspects and none about my use case.
Would you be willing to submit your bad search results to such forum? I welcome you to post your bad search results with the queries to that needgap thread for the time being unless there's a dedicated forum for that.
Rebranded Bing with a new interface isn't hard tech. It risks so much on the business relationship. They should be staffing for this yesterday.
Hire more search folks, DDG! Also, ditch that awfully long name.
https://duckduckgo.com/?q=Africa&iaxm=maps
It is refreshing to be able to use a usable map on tor without having to accept google big term of service.
Maybe because it's already a profitable business model as it is?
Looking at the suggestions in the blog, and for transparency - Mojeek CEO:
> Stop trying to look like Google.
Agreed. We and our users still believe that "10 blue links" have great value. But drowning these out with too many of things like Ads, Videos and Answers on the page helps pretty much just Google.
> Arrange that algorithm to make it less vulnerable to SEO hacking.
Admittedly we don't yet have that problem yet. Still there are plenty of measures that we provide, and will expand upon to mitigate that. Without going into details these generally amount to giving users, and API customers, more control over searching and ranking.
> Discard AI-generated text.
A good idea. But can also be done on the SERPs. Do users benefit from AI answers? Mojeek is a search Engine not an answer Engine.
> Results in other languages.
Bing?
> New opportunities.
DuckDuckGo is doing a great job of providing new and improved privacy products. We stand with them on many things. I am sure they appreciate the opportunity also in providing more informational (search) diversity from and with a fully independent search stack (infrastructure, crawler, index, ranking).
That isn't an accurate characterization of what we do now.
https://news.ycombinator.com/item?id=32360642
> mobile searches are the largest category of searches, and local searches are the largest category of searches within mobile. Instead our local search content is a combination of our own indexes in partnership with Apple, TripAdvisor, and others.
This suggests Bing isn't used for local searches, but it is still used for mobile and desktop searches that aren't local. If I am searching for a tech problem or how to tie a knot or recipes or the history of hair metal, I'm getting Bing results wrapped in DDG. Is that inaccurate? Is Bing still used for searches that aren't local searches?
I know a handful of us immediately switched out of DDG when they decided they were going to artificially down-rank Russian web sites from results[1], but I imagine the overlap between DDG users and people who care about this is quite small.
[1] https://slate.com/technology/2022/03/duckduckgo-russian-disi...
As for the Russia thing, as far as I can tell, they still show Russian propaganda sites (and often as the first result). They just don’t let SEO’ed sock puppet sites drown out all the other results.
I imagine the number of people that switched away because of that is vanishingly small.
If they actually downvote a few hundred spam sites before casting a vote for a spam site to promote it, then they will have built a 99.5% good database already and one site can be manually removed. The only way to have a high confidence in a user if they both downvote bad sites and upvote good sites. If they did that work to verify their accounts to be valid, then they will have done more good than harm
For example if I search for something super simple like:
"what is another word for happy"
Google lists synonyms at the top of the page from Oxford.
If I do the same exact search on DDG
I just get the definition of happy and no synonyms.
Today, I would bet that the majority of searches aren't to find websites but to find answers to questions.
DDG sucks at answering questions.
Actually even with "what is another word for happy", the first result is a wordhippo synonyms page.
Though I agree that question based queries are probably more popular these days, and DDG isn't handling them as well. I imagine this doesn't bother the tech / privacy crowd too much.
The google result shows more antonyms than the DDG one, but the DDG result has a click through to rhymes and sounds-like words.
So, it is a draw in my book. I don’t try to ask english language questions to search engines though (and find it uniformly produces worse results than listing keywords, but YMMV).
Based on autocomplete suggestions, it looks like google auto rewrites your query to mine.
Arguably, it would be good to be able to search for pages containing phrases like the one you typed, instead of just ignoring user intent for people that have used search engines before.
shrug
This shows Google is way better at understanding what the user wants when searching.
"what is another word for happy" "happy synonyms", and "happy synonym" all give the user what they want right away on Google doesn't matter how you search it.
With DDG it does matter how you search it.
Another example is another simple query like "florida capital". Google in big letters tell your what the capital of Florida. It shows you a map of the city that you can click and shows basic information on the city.
DDG gives you information on the capitol building and the first link is too the Florida Capital Bank website.
Why would I ever use DDG over Google? Other than privacy which I don't think is that big of an issue as people make it out too be on Google.
Because that, amusingly, gives me an info box with the answer. Either way, the Wikipedia page for Tallahassee is the third link on the page.
It is amusing that DDG seems to be assuming I spelled "capitol" wrong, and pushing the correct results down as a result. I know the difference.
According to the world map infograph in the article there are 4.5 million more people using the internet in Australia than there are people in Australia...?
So I'd say the estimates aren't great, or they're likely estimating number of internet connections rather than the people using them.
Interesting notion.
So, if a state or business wanted to suppress certain information, they'd only have to kneel on the neck of one indexing company, rather than many. Having said that, if the index is certifiably clean, fair and uncensored, and access is free, I can see huge value in a public global web-index.
OK, I'll bite. I tried the search: all articles on the first page are about cats, not dogs; most have the word "cat" at the beginning of the title.
https://www.reddit.com/r/duckduckgo/comments/w5dzol/minus_si...