BW already has all the URLs of services you use: you saved them yourself for them to keep safe. It's obvious you already trust them enough to know that sort of data.
So if favicons were gathered locally, then you would prefer that your own browser would reach out and make multiple requests to every (potentially dodgy) site listed on the search result page?
Note also that these are favicons for results that DDG has already given you. This isn't tracking your clicks. The list of sites that appear on the search result page is not new information to the search engine that just gave them to you.
EDIT: Like other commenters, I was not previously aware that DDG had a browser, and my comments were about this behaviour for the search engine results page.
Can you explain your edit a little more? I still only understand your original viewpoint. Just because there's an app doesn't mean they don't have to contact DDG servers at all, right?
We had already had created this anonymous favicon service for our private search engine. In addition, doing it this way avoids another request (and potentially multiple) to the end site.
The service is private as we do not collect any personal information (e.g. IP addresses) on any requests for this or any service and the requests are all end-to-end encrypted.
Why not do it locally? It is one extra request to the favicon link in the header and cache it. Or cache it when the user visits the website. Or explicit button in the settings UI to refresh favicons locally.
Your answer to all these criticisms is a lazy one. You are saying "this already exists for other things so we use it regardless of what the correct method is". Just because something exists in one form doesn't mean it's the correct choice.
I personally believe you, but this approach adds multiple unnecessary trust boundaries. Just thinking about how many separate logs are triggered outside of DDG is troublesome. And what happens when you guys think everything is secure, but one day find out it isn't?
I understand utilizing a single service/feature to accommodate multiple platforms is a big win from a programming standpoint. But if it carries the risk of losing the trust of your user-base while at the same time risks breaking the core mission statement of the business (privacy), it's probably not worth it.
What do you mean by "end-to-end encrypted"? Isn't this just a request to your service that will then go out and fetch the icon from the site, so it can't be e2e since you need to know which site to request the icon from.
Why use that term when it clearly can't be true or even seems applicable/relevant?
This is concerning because it indicates a lack of care in terms of privacy and understanding that the best privacy is achieved by knowing the least. Does this approach permeate their backend as well?
I'm not so sure. I think the privacy-functionality trade-off is understood and expected by the users, it woul be used by very few people if it were extremely spartan.
(But this instance, the favicon service, is not a good privacy-functionality trade-off)
What would a browser even need a backend for? The only valid use that I can think of is Google's Safe Browsing list, but if ad blocking can be implemented totally on-device, surely that can, too?
The use case I was going to suggest was a safe browsing list. Possibly also something like Have I been Pwned. So, yes, there are valid reasons to have a backend but they are very narrow and privacy is key when building them.
Essentially a way to collect where people are visiting. I believe them that it’s anonymous, this valuable info wouldn’t need to be identifiable to be of value.
They should probably change the behavior to how it’s suggested in the thread, but I’m still going to use DDG over alternatives for the bang feature.
I don't really trust DuckDuckGo, but I use their search service because I trust Google less... I still trust Firefox more for a browser although it won't take much at this point to make me switch.
What does Google have to gain from it? Google has pretty aggressive anti-scraping protections to protect against this exact behavior, so why would they allow Startpage to get away with it?
What does Startpage have to gain from it? Unlike DDG, they don't seem to have any core product, so they fully depend on Google's goodwill which is very shaky grounds when it comes to a long-term business.
"You can’t beat Google when it comes to online search. So we’re paying them to use their brilliant search results in order to remove all trackers and logs."
> Unlike DDG, they don't seem to have any core product, so they fully depend on Google's goodwill which is very shaky grounds when it comes to a long-term business.
Isn't that exactly like DDG but Google instead of Bing?
My understanding is that DDG is using Bing results as well as their own. Whether their in-house results can stand on their own is another matter, but at least they're trying to reduce their dependency on Bing, where as Startpage is not.
> We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).
So basically it's all Bing and I see no effort to reduce that dependency as you claim.
Runnaroo is also a search engine that is Google based (for organic results). IMO it has an advantage over Startpage because of its integration of vertical search sources, like Stack Overflow [0]. I'm the creator.
Mojeek [1] is also a good privacy option that doesn't get talked about as much as DDG. They distinguish themselves by having their own search index.
If you do that, there's no evidence that Google is more privacy invading than DuckDuckGo, since you're left with taking their word for it. And frankly I trust a big, bureaucratic company more than I trust a startup. The ideal would be to trust technology (e.g. end to end encryption, open source) but that's not the case here.
Note that I also use DuckDuckGo in my Private Mode, as Firefox allows setting different search engine for Private Mode. I do that because it's better to compartmentalize your online personas, plus I keep my "web and app activity" in Google on, with deletion after 3 months.
Nevermind privacy. How are favicons so complicated that they need a special service that understands edge cases. Just do it one standard way and if a minority of websites don't work, then exclude them. We've been through this mess before with all kinds of web standards devolving into mess.
One major issue is that the "standard" favicon size has historically been 16x16 pixels... which, in the age of high density displays will render either comically small or comically blurry. There are other meta tags like Apple's "apple-touch-icon" which has some higher resolution options. But already you can see the logic here isn't trivial.
It still seems trivial to me. Parse all meta icon tags, prefer one that matches exact client resolution of current display (don't even have to download the icons - the sizes are defined as part of the meta tag), else use the largest-resolution one, else in desperate bid try site/favicon.ico, otherwise give up.
Really shouldn't be complicated enough to need a special service to handle "edge cases".
Another comment mentions web manifest - I guess try those first before meta tags, or whatever order the standard says to use. I mean, we're talking a web browser here, it's designed to do these kind of tasks.
I guess the problem is when you want to quickly provide a favicon with the search result (beit in the omnibar or actual search page), as you're typing and results are being displayed you cannot send off a request to the site, wait for HTML html to finish up downloading, send off a request to the manifest if its defined for multiple sites at once.
On a technical level of course its doable but in reality it's a complete waste of data and processing, not to mention it could take a long time to show up. I imagine they have these favicons all cached on their side so they can quickly send the right file down and/or do this processing if needed.
That being said maybe they should just not use a favicon if it's that big of a deal.
Oh, this is for putting an icon next to search results? Yeah, that changes the calculus considerably. I thought this was about showing the favicon in the browser for a site the user visits (per the issue title).
In that case, yeah, I don't think the icons are necessary to show at all...
Even if you wanted to implement this, the logic of the service could be directly embedded in the browser as an extension or similar. There’s no reason to depend on a network service for this functionality.
I feel like the only user benefit is to see the favicons of sites you are familiar with, so caching those locally after you visit sites is probably good enough.
Agreed, creating work arounds for non-standards compliant websites just eliminates the motivation and incentives to be standards compliant in the first place.
This is a bad look for a company that is trying to build its brand on privacy and trust. Even though I don't use the DDG browser I hope they own up to this, rectify it quickly, and learn from it.
Privacy has to be scrutized constantly and be the top priority. If it isn't, then you're going to end up with another google.
A direct correlation exists between the revenue Google receives for selling data and the quality of its search. Google focuses completely on tracking and search, with privacy behind a far far away afterthought (if it's a thought at all).
I'm sure Google's thinking about privacy. They're thinking "will our latest privacy violation create enough of an uproar to affect our bottom line, or will the users just accept it like they usually do?".
In my view, anyone who trusts ddg is a bit silly - founder has a bad track record on user privacy. Founded Names Database[1], a social media website designed to collect user information as aggressively as possible, before selling all the information to classmates.com.
I don't really think is an ad hominem fallacy. An ad homihem fallacy is when you attack the person making an argument rather than the argument. But no one here is attacking the people making the argument.
The argument is "Trust DDG". That argument is being attacked as "DDG's founder has done bad stuff in the past, it's likely DDG will do bad stuff, so I won't trust it". That seems to be attacking the argument to me, thus not an ad hominem.
That’s not an ad hominem attack. The comment didn’t say that the person’s argument is wrong because that person is silly. Clearly the comment is saying that the argument itself is silly.
The intent of the comment was probably not ad hominem, as if it were rephrased like "trusting ddg is silly" because then silly is modifying the act of trusting. But as-written, silly is modifying anyone (a person) without distinguishing whether silliness is the cause or the effect; if silliness is the cause of the trust then it's an ad hominem attack. It would be more clear-cut if instead of "silly" they said "low intelligence."
[My opinion on trusting DDG] - [Reason for my opinion on trusting DDG].[More information about the reason]
In this case, because I don't use the "personal attack" to reach my conclusion that the person was wrong, I don't think it would be a case of argument ad hominem.
But if someone read it like this:
[One reason for my opinion on trusting DDG] - [Another reason for my opinion on trusting DDG].[More information about the second reason]
Then I am using the "personal attack" to reach my conclusion that the person was wrong, so I think it would be a case of argument ad hominem.
My comment wasn't written very well, and I'll try to write better comments in the future. Not that adding an ad hominem to a valid argument makes it invalid, I guess?. But it's still good to avoid fallacies, and to write one's comments so they're likely to be understood the way one meant them.
A common fallacy I see, though I don't know if there's a name for it, is assuming that just because what someone is doing can be described by the name of a fallacy that what the person is doing is fallacious.
> Argument from fallacy is the formal fallacy of analyzing an argument and inferring that, since it contains a fallacy, its conclusion must be false. It is also called argument to logic (argumentum ad logicam), the fallacy fallacy, the fallacist's fallacy, and the bad reasons fallacy.
I'm talking about situations where no fallacy has actually occurred, not situations where a fallacy has occurred but a correct conclusion has been arrived at anyway.
So what does any fallacy tell me then? Especially the fallacy fallacy? If it does invalidate the invalidation of fallacies it is in itself untrue...bz...recursion error..bz stack overflow
Also worth mentioning they're closed-source, US-based and for-profit. Why exactly do people trust them? Simply because they write a few articles/ads saying "privacy is important"?
If you're willing to sacrifice search quality for privacy, as in switching from Google to DuckDuckGo, then you might as well take a step further and switch from Google to Searx/Ask.Moe.
As a DDG user I don't feel like I'm sacrificing anything. Two sets of results are better than one (I can see Google results by adding !g, which I do less than once per day on average) and, ironically, DDG bangs are the easiest way to use even Google services like Translate and Scholar.
For me the fact that they’re open source. This thread is a prime example of why it’s so important. We really have no idea what DuckDuckGo is doing because they’re closed source. For all we know they could be forwarding users’ IP to Microsoft/Yandex/etc.
If you want to market yourself as a champion of privacy, then the absolute minimum criteria should in my opinion be that your codebase is open source.
Because they have a good privacy policy. They would face legal consequences if they were lying. Nation state actors can presumably override privacy polices but it's better than nothing.
The reason I use them is not because I trust them but because they do not put me in my own search bubble like Google does. I hate that part of using Google. Also as a bonus I do not get any AMP results.
I couldn't figure out which searx instance to use and found no good way of knowing who to trust, most of the engines i used were broken and telling me to find another searx engine.
It looks like it's pulling most of its info from duckduckgo anyway.
Personally I'd rather trust a known entity than an unknown entity anyday, especially when the unknown entity is slow, complicated, buggy and broken in many places.
They’re pulling info from all the other search engines, not just DuckDuckGo. FYI DuckDuckGo is also pulling their results from other search engines, the only difference is that DuckDuckGo aren’t being blocked/rate-limited by them, presumably because they’re paying for API access.
If you hosted your own instance then it would be a lot more reliable since the IP wouldn’t send a suspiciously high amount of requests.
As for your trust argument, I couldn’t disagree more. You choose to trust DuckDuckGo, who happens to be closed source, because of their branding. The same way people trust/trusted Google/Apple/etc. because of theirs. This thread is a perfect example why being open source is the most important thing for any privacy service (because otherwise this privacy leak likely wouldn’t have been discovered, and people wouldn’t have known that the company so carelessly violate people’s privacy and fail to correct it when people point it out.. it should really make you wonder what’s happening in the search engines codebase).
EDIT: I didn't notice that this topic was about the DDG browser (which I didn't know existed) and responded assuming this was about the site/extension. For a browser, yes, a client-side solution is possible and probably preferable. Please check and upvote other comment trees.
This makes sense to me and is not alarming. Getting favicons actually is difficult to do robustly; many applications and websites use Google's service to do so, which then leaks the request to Google: https://www.google.com/s2/favicons?domain=ycombinator.com
Putting this logic in the client is not feasible. You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
DuckDuckGo hosting this functionality themselves is also a positive. They have previously been burned when the Web of Trust service they were originally using was found to be farming data, and turned it off immediately once discovered. Processing, hosting, and serving the icon themselves prevents that from happening again.
I don't see how this is a privacy violation in any way. The headline is that DDG is tracking the domains you visit, but favicons are loaded for every search result you see unconditional on whether you click them, and DDG already knows what those results are when they serve them to you.
I don't need the favicon for a domain I never visited just because it adds some visual cool to your UI. That is not a thing. Cache it whenever I end up visiting the page and be done with it.
For the search results DDG can do whatever they way. As far as I understand from the title, this is for the pages that you actually visit. For those pages it totally make sense for the browser to figure out favicon locally on its own.
As for complexity, I'm sure users of privacy oriented software would prefer to not have favicon in small percentage of corner cases rather than leaking browsing history.
Which other browser doesn't fetch favicons locally?
And I have no clue how search results come into this? Of course the browser isn't fetching favicons on search result pages, that's the servers job, but that's not what this is about.
It's really odd that this is the top comment right now, given that even the headline makes it clear its about the browser, not the search results.
> Putting this logic in the client is not feasible. You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
I don't see why not. Browsers are constantly dealing with these very issues. It's one of their core competencies.
"This is not to say that DDG is perfect: links you click do seem to be redirected through a /l/ page on their domain ..."
I am surprised the user is not complaining about this instead of the favicons. Their privacy policy goes on about the privacy implications of Referer headers and instead of calling out browsers for sending Referer by default, they instead give themselves power to record all the user's clicked results themselves. The Referer problem is something that can be solved by the user at the browser level through, e.g., using a client that does not send Referer, browser extensions/plug-ins that can control headers sent, or perhaps with a local proxy to remove the Referer header.
Unless DDG has changed, these prefixed result URLs are the default. It is possible to get unprefixed result URLs using the "lite" version of DDG however that is not the default. "Privacy-focused" search engine chooses less private default. News at 11.
I recently noticed that DDG has started redirecting queries submitted via POST to /lite/. The redirect is to the same domain. No explanation. I have a custom client that does not follow redirects and I now have to submit two sets of HTTP headers instead of one.
These guys are trying to make money from advertising just like everyone else. They have to be very particular in the methods they use to do it -- check the exceptions in their privacy policy -- but it is the same game. Ads and affiliate links. That sort of business and privacy are always going to be at odds with each other.
To get prefixed result URLs, use GET not POST. Try duckduckgo.com/lite/ and lite.duckduckgo.com/lite/
If I recall correctly there is a way to turn off the prefixed result URLs without having to turn on Javascript, by adding/changing a URL parameter, e.g., kh. However that is not the default setting. Not very friendly toward the privacy-conscious user who has Javascript disabled and does not read HN to find out about otherwise undocumented URL parameter usage.
For certain queries, I can actually get a different first result based on the HTTP method I use.
I'm not sure what you mean by bringing up HTTP method. The two main ways I know to get to the results page are to enter a query into the website or the browser search bar, and both of those seem to issue GET requests. They also both produce real URLs without any intervention. What defaults are different for you than they are for me?
This is done only for browsers that don't support setting the referrer policy. https://caniuse.com/referrer-policy. It should not be happening with any modern browser, and (like the other reply) I don't see it in mine (Firefox).
Are you saying DDG is forcing you to send a User-Agent header?
DDG's privacy policy also goes on about the privacy implications of User-Agent headers combined with IP addresses.
So let's say I take what they have put in their privacy policy to heart and I stop sending a User-Agent header. In response DDG sends prefixed result URLs? WTF?
Using haproxy, for example, I can use a "modern browser" and send no User-Agent header. I still get prefixed result URLs.
You can disable this on the settings page. Frankly I'd infinitely rather send my request through a redirect served by DDG than send my referer to every site I visit. I think this is a perfectly sensible default for the average user, who both doesn't block / obscure their user agent and doesn't have referer masking software in place.
You're free to disagree, but in that case you can make use of the option to change the default that they provide for you.
> Putting this logic in the client is not feasible.
> You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
Looking at DuckDuckGo search results and visiting a page that you navigate to are two different things.
1. DuckDuckGo search results:
DDG already returns the search results so there's no privacy violation to return the favicon or the URL for the favicon in the list.
2. Any page that isn't a DDG search results page:
Use client side logic to locate the favicon. This means worse performance but better privacy - which aligns with DDG's goals.
If you want to optimise this then DDG could send the client some precalculated Bloom filters with info about known sites. The client could use these to try certain methods of favicon retrieval first.
Set duckduckgo.com as your default search engine with a blank home page. But you could also use @pkrumins home pages of https://techurls.com or https://finurls.com as nice home pages.
Use Mullvad VPN: https://mullvad.net/
(They are EVEN available on F-Droid now, which is AMAZING)
Stock Firefox doesn't actually have that much bloat, and it's not noticeably slower than Chrome on reasonably modern hardware (i.e.: most page loads are near-instant, same as in Chrome on a good connection).
... Until Google inexplicably restored it a few days later, but not before lots of accusations were thrown around.
---
IMO it's also worth noting that ungoogled-chromium is (obviously) an unofficial fork of Chromium. Google may at any time change Chromium so substantially as to either require Google integration at some fundamental level for even the most basic functionality, causing too much work for such a low-profile effort to continue, or just make Chromium closed-source. With Firefox, that risk doesn't exist because of the business motivations of the company that develops it.
As I said in other another comment, I'm a power user. And by power user I mean probably 10x the normal standard of what a 10x power user means. Tooling is everything to me; which is why I prefer this over anything else. I'm very biased.
When it comes to a browser, I'm curious as well. I often end up with 100+ tabs in firefox before I close them all. I don't think think I'd consider that being a power user, but would a 10x power user simply mean someone with over 1,000 tabs open?
I haven't used Chrome regularly in the last year or so, but it would get noticeably slow by the time I had 3-4 windows with 10-20 tabs each. Firefox hasn't really had that issue yet.
Years ago though, Chrome was noticeably faster than Firefox. That changed (IMO at least) at some point in the last few years.
I use Firefox day to day, but if I was a “power user” and picking apart web traffic everyday I’d probably use Chromium. The dev tools in it are way better than the ones built in Firefox.
That has fresh meme potential. How many 100x power-users do you need to make a 10x developer? And how many normal users for a 100xPU? Are 10 100xPU equal to a 1kPU? So many questions...
You are probably seriously overestimating yourself and/or underestimating a large subset of your audience here.
People here are:
- early developers from majir browsers
- extension creators
- creators of major web properties
- people who spend their work days on the web
- etc
seriously. Don't underestimate HN.
Edit: and as someone who's been very into customizing and extending browsers since early Firefox: your ideas about Firefox vs Chrome tells me that you are either
- deeply biased
- use a subset of browser features that is too small to realize the problems with Chromium (in which case I suspect your 100x superuser is wild hyperbole)
You can come back when Tree Style Tabs including related sub-extensions works as well on Chrome as on Firefox.
You can install uBlock Origin in 10 seconds from the GitHub release page (see previous link shared). This is vastly better than anything I've seen for ad-blocking.
Also that just makes me trust them more – they are the only browser vendor that are trying to find business models that don't involve selling my data or selling out to someone who does.
The issue is that nothing is as pure as ungoogled-chromium is right now. I've also felt that Brave was slow, and not best for developers. As a power user I'm very biased, so Brave may work better for folks that are not as technically skilled and have no idea how to install from GitHub/brew/etc.
Edit: I feel it is slow because it has extra bloat added. Like I said, take my comments here with a grain of salt because I'm a very biased power user. I respect the efforts of these developers regardless of what project it is, the focus on privacy and building something different is truly awesome.
Through better tech and with better developer usability, we can conquer these monopolistic false promises. Our work with the Lad framework is instrumental.
Thanks! I'm rolling out the following within probably the next 7 days:
- All new load balanced infrastructure
- Browser extension + API wrapper
- Support for pixel tracking blocker (opt-in checkbox or TXT setting)
- Smart alerting
- Globby/regex support
If you follow my Twitter or the GitHub releases you will get updates.
That's outdated information. Nowadays you can enable Encrypt All Sites Eligible and the extension will force HTTPS everywhere skipping the whitelist. It'll also prompt you to continue to the HTTP version if HTTPS fails to load.
Advertising your Twitter for the advice of "switch to somewhat well-known browser X, install these very common extensions and use a VPN" is also a bit ... odd.
An ex DuckDuckGo employee recommending people use an alternative browser over the DuckDuckGo browser, in a post about the DuckDuckGo browser spying on its users, is about as on topic as you can get; after the current employee giving an explanation.
I haven't checked all links but some things need to be updated, for example Skimmer Scanner is gone from the Play Store and Yalp Store is abandoned and doesn't work anymore, you should be using Aurora Store. I'd also recommend Aegis or FreeOTP+ over FreeOTP for 2FA. NewPipe is better installed from this[1] repository until this[2] issue is solved.
Yeah, I do use Aurora Store. No longer use Skimmer Scanner. I have to update that post. Working on a new blog/site and a book (eventually). Thanks for heads up.
Your "Security harden your mac" guide has a bunch of amazon affiliate links, it would be good to call that out when piggybacking on a post about privacy at least.
The favicon is acquired from DDG servers for the result you've just retrieved from DDG servers.
How is this leaking anything? What additional privacy would you gain from getting the favicons from the domains directly of search results delivered by DDG?
DuckDuckGo staff here. As mentioned in the linked page, the purpose of the request is to retrieve a website's favicon so that it can be displayed in certain places within the app or on the results page. We use an internal favicon service because it can be complicated to locate a favicon for a website. They can be stored in a variety of locations and in a variety of formats. The service understands these edge cases and simplifies retrieval within our apps and our search engine.
Like our search results, the favicon service adheres to our strict privacy policy[1] in that the requests are anonymous and we do not collect or share any personal information.
We had already had created this anonymous favicon service for our private search engine. In addition, doing it this way avoids another request (and potentially multiple) to the end site.
The service is private as we do not collect any personal information (e.g. IP addresses) on any requests for this or any service and the requests are all end-to-end encrypted.
It seems like DDG just doesn't want to do more work to make the browser do the sensible thing and instead piggy back on their existing favicon service, and then going to make the argument that it is safe with us. The domain of browser is totally different from reaching out to the website and rendering favicons server side. Do you not see this as a problem? I think it is acceptable to say "We agree this is a problem, and we are gonna look into fixing this" instead of pushing back on an obvious problem.
> In addition, doing it this way avoids another request (and potentially multiple) to the end site.
Potentially saving a few requests here and there is certainly not worth phoning home with that kind of data regardless of what records you keep how much you do to anonymize it. This is especially true for a company that has built its brand on promises of privacy!
Besides, favicon requests are small potatoes compared to the kind of tracking, ads, metrics, and other often-unnecessary page resources that bog down most of the modern web. And a well-designed website can mitigate the issue pretty easily.
Saving a request is not a good reason to phone home, especially given what the DDG brand is built on and the main reason I would use it over Google. Please reconsider.
Surely you could fetch the favicon asynchronously and update the address bar or tabs or wherever you display it only after the other assets are loaded? It’s not like the DOM has elements that depend on it. This is a really, really weird hill to die on.
I understand we’re suppose to take your word but it’s making me skeptical now about the actual intentions. If DDG truly cared about user privacy then you shouldn’t transmit user information for something as trivial as favicons.
Please ask your higher-ups to educate you about essentials of privacy in the modern age. It's not about what you may be doing wrong, it's about what you technically could do wrong.
> The service is private as we do not collect any personal information
This sentence is 100% meaningless. I understand you have good intentions, but these things must rely on proof, never on trust. Either you get this information or you don't; whether you say you "collect" it is inconsequential.
I'm sorry, but this is bullshit. If the received HTML contains a <link rel="shortcut icon" ...> element, you already only need one request, so this is not an improvement. If that is present, you should NEVER use your API.
If it's not present, then you have options, and yes, using your weird API is an option (which I still don't like, but ok).
But sending private information to your servers even when sites follow the standard show either that you're probably not trustworthy, or that your product team is so painfully incompetent that I'd be afraid to use their browser at all.
> We had already had created this anonymous favicon service for our private search engine.
I don't think here's a need for adjectives here. Why stress that it's anonymous (when that's hard to verify) or that the search engine is private, when that too is starting to come into question? Repeating these things won't will them into the reader's perception.
> In addition, doing it this way avoids another request (and potentially multiple) to the end site.
This isn't true, unless I'm missing something here? When I access a website, the HTML response I get from that website includes all the information my browser needs to, on its own, get and display the favicon. Can you clarify why you think/say this avoids one or more requests? What mechanism is this service a substitute for?
Are all these edge cases part of the html/w3c/whichever standard? If not, let the edge cases fail. I'm not going to lose sleep over an icon not showing for a site I'm visiting once.
Sadly, not anymore - used to be the case quite a while ago, but not today anymore as the ads come from the same domains the videos are streamed from.
Those simpler popover-ads which can be closed clicking an X in the upper right corner still are blocked tho...
Spilling my secret tho (and YouTube execs hate me for it!): i block YouTube in my mind and only rarely go to it if i really need to watch a video (which, for me, is rarer than i ever thought it'd be).
This is essentially the same, but pi-hole gives you a nice UI, stats etc, and of course you can point all your machine at a pi-hole instead of editing the hosts file on each.
This should be the top response. Just under it should be a DDG staffer telling us they will be turning this config option off by default starting with the the next release.
You really can't use "we promise we won't misuse the information" as an argument, that's what everyone says whether it's true or not, and the whole point of using a privacy-centric browser is that as a user you can't trust those kinds of promises.
They’re primarily a search engine, so yes, you definitely have to trust that they won’t misuse information about what search terms you are querying for.
I suppose that's valid. This is the first time I had heard that DDG has a browser too, and I was just assuming that anyone who would use the browser probably also uses the search engine, which they obviously have to trust when they send search queries to it.
By the way how DDG spend money on the ads like at bus stop, I already starting to NOT trust them. Spend the money on development of development of products.
I'll state this in no uncertain terms: this is not acceptable, and you need to stop doing it. It makes sense on your search engine, but adding it to your web browser is very much over the line.
I have read your explanations in good faith and they don't cut it. This behavior cannot continue. Good privacy promises are not based on trust - they're based on not ever handling private data in the first place. If you don't quickly admit your mistake and roll this back, it will jepoardize your entire brand - and rightfully so. If you believe this behavior is okay, then it demonstrates incompetence; if you don't believe this behavior is okay but do it anyway, it demonstrates malice.
I appreciate you answering, probably knowing you'd face some negative feedback. Saying "we should trust you, it's for a good reason" is what google and everyone else says. You'll be better off if you just end this. The loss of the fav icon is less important than keeping your credibility.
thirded. an inconsequential token is not a suitable reason to engage in this kind of data collection, especially for a search tool that prides itself on 'not tracking' users..
Fourthed, and honestly it's strange that they are willing their promise to privacy on saving 2-3 queries on something as trivial as favicons? Why does no other browser need to do this?
Chrome, according to my understanding, hardcodes a few favicon URLs for builtin search engines, and caches everything else on site visit. There's SQLite3 database named Favicons in your profile directory (e.g. on macOS it's ~/Library/Application Support/Google/Chrome/<Profile>/Favicons). Here's the schema:
CREATE TABLE meta(key LONGVARCHAR NOT NULL UNIQUE PRIMARY KEY, value LONGVARCHAR);
CREATE TABLE icon_mapping(id INTEGER PRIMARY KEY,page_url LONGVARCHAR NOT NULL,icon_id INTEGER);
CREATE TABLE favicons(id INTEGER PRIMARY KEY,url LONGVARCHAR NOT NULL,icon_type INTEGER DEFAULT 1);
CREATE TABLE favicon_bitmaps(id INTEGER PRIMARY KEY,icon_id INTEGER NOT NULL,last_updated INTEGER DEFAULT 0,image_data BLOB,width INTEGER DEFAULT 0,height INTEGER DEFAULT 0,last_requested INTEGER DEFAULT 0);
CREATE INDEX icon_mapping_page_url_idx ON icon_mapping(page_url);
CREATE INDEX icon_mapping_icon_id_idx ON icon_mapping(icon_id);
CREATE INDEX favicons_url ON favicons(url);
CREATE INDEX favicon_bitmaps_icon_id ON favicon_bitmaps(icon_id);
I would hope that it can also be made to cache 404 and 410 responses to favicon requests too (or at least 410 even if not 404), so that it won't keep trying to access it.
Also, in SQLite, note that LONGVARCHAR is the same as TEXT, and that you don't need to specify both UNIQUE and PRIMARY KEY (it is redundant), and that if it is not a INTEGER PRIMARY KEY and not WITHOUT ROWID, then it isn't the real primary key but just an index (same as UNIQUE); add WITHOUT ROWID if you want to make it a real primary key, but note that the way the data is stored differs then, and WITHOUT ROWID is inefficient with tables storing large blobs.
They've already started down that path, judging by all the DDG billboards I see driving my 18 wheeler around the country, and all the DDG ads I hear on NPR-related podcasts.
Not that I'm against making money. But there's a tipping point associated with some height value in a pile of cash, and once you cross that point then the pile controls you. DDG probably hasn't crossed that point yet, but self-justification is one of the steps on that path.
They call it the "DuckDuckGo Privacy Browser" yet it sends data back to their servers on every site you visit. Collecting all your search terms and saving them wasn't enough, now they want data on all the sites you visit outside of the DDG search page. That they refuse to turn it off tells you they are just as data-focused as every other company. They should update their app description: "By the way, every website you visit is being sent back to our servers, don't worry it's just for data collection purposes, no personal information, just every website you visit (LOL)"
For those of you who don't understand, Yes they are saving data on all your search terms, and Yes they are saving data on these favicon requests, don't be so naive, they admit it themselves. They are not going to turn this feature off because it is too valuable to have all your information on all the sites you visit
I think you're being downvoted because you chose to piggyback your comment on a seemingly unrelated one at the top, are being vitriolic, and didn't back up your claims with respect to their intent and refusal to change this.
Well it's related because Mozilla actually cares about your privacy vs DuckDuckGo which obviously could care less from their reaction to this issue. Their refusal to change this is all the proof you need to know. I dont even use DDG I use Google I just think its funny they have a "Privacy browser" that sends all the sites you visit back to their servers
I wonder how many lines of code from big open source applications are generic enough to be reused in other projects.
Firefox and Google Chrome probably have the equivalent of many small high quality libraries embedded in them, implementing 'business' logic or protocols, that could be reused in more places.
I guess a large scale study on github could be done, with a graph analysis to show potential "cut off" points in codebase.
Yeah... the gesture is nice, but good luck extracting any code from a massive project. Might as well say “Here’s some free oil; all you have to do is dig for it.” Unlike oil, this might not be worth the excavation.
It’s a bit telling that they linked to the GitHub repositories rather than specific lines of code they were talking about.
There is also a content script I think and storage for these icons and their meta data. It is probably less generic on iOS than on Android but should be fairly simply to take some big chunks for reuse.
Android components is a foundational technology for our browser products on Android but also for many other applications. We’ve designed things in such a way that you can pull in just those things that you need. If that is not the case, file an issue and we will take a look at how to improve things.
One issue is that if a bunch of code of a similar theme can be broken off, many times it will be (depending on the culture of the software team). Is looking at the source code of a big project with a hundred dependencies that were all libraries spun off to support the project different to looking at a project where people tried to keep everything monolithic (where you’d expect better re-use potential per line-of-code?)
Yes, apart from Chrome's implementation there is probably no better tested and more mature implementation. Still it seems to be a non-trivial problem because Firefox sometimes shows me a favicon from another site I used.
If you don't visit the site explicitly, just have the site somewhere in a list/bookmark/whatever, that site now has your IP address and basic header info when it needs to go retrieve it. By going through DDG, the site has a bot hit.
To me it looks to be trying to uphold your anonimity until you commit (click) through to the site/link. But certainly other ways they can approach this if it really bothers people.. I'd prefer DDG doing the lookup.. or having no fav icons.. over my computer going and downloading all my bookmark or other source icons
It's not immediately obvious whether it is more privacy preserving if the client automatically makes a request to each site in the search results while scrolling through the results, especially since you're already trusting DDG when performing the search.
Maybe this should be an opt-in rather than an opt-out feature?
Edit: as pointed out by warpspin in another comment, this is about the DDG Browser, not search results.
Scrolling through results is not a good place to ping websites for their icon. Not from a technical perspective but also not from a privacy perspective.
I’ve been an avid DDG user for years and it worries me that DDG staff don’t see why this is an issue. We shouldn’t have to trust your privacy policy if you minimize exposure.
That's not fair. There is a very clear reason: it simplifies development for them and avoids duplicating functionality in two different codebases. You can argue about whether or not that's a good trade-off given the privacy implications (and I would agree that it's not) but you can't claim that there is "literally no reason".
The reviews are in for this response and they are bad. It's concerning that given the react it got, there's no edit addressing the concerns. The HN audience has to be the power user, bread and butter of a product like this and when you see a company ignore the concerns of a key constituency like this, their future almost never looks bright.
What a bizarre potential privacy flaw to introduce for a tiny little icon nobody cares about. I understand it, usability and UX is important - but you guys are DDG! Come on! Your customers all have tinfoil hats!
What a wussy excuse. This should have been a no-brainer decision: do we quietly compromise privacy so that our users can have little icons on their browser tabs? How absurd.
Duckduckgo chose compliance with the inconsequential minutae of bigtech over its primary pain point. This is indicative of misalignment between stated values and the values demonstrated through actions. If you guys made this call, sacrificing privacy for something so banal, it doesn't bode well for what's going on in the rest of your operations. That means the problem isnt technical, its cultural, which are unfixable, therefore it's pretty much over. Good luck to regain trust once you've been outed as a put on.
What is this possibly meant to imply? Nonprofits run international ad campaigns, governments run international ad campaigns, the military runs international ad campaigns, most organizations could probably find a reason if they have international domain. What is your insinuation specifically?
I’m very disappointed in how you guys responded to this. As a privacy focused company I would not have expected an answer that sounds like it came from a data collecting company like Google.
I just switched to DDG browser a week ago and will now be looking for a new browser now. I hope you know this is not an appropriate response to the situation. Especially because all you guys do is preach about how much you protect your users’ privacy. Now you’re here asking us to trust you not to abuse our data and just linking us your privacy policy. I’m sad to say that my faith in the DuckDuckGo company and team is now lost.
Does Gabriel know about this? If not could you please clue him in and get some guidance because you are absolutely getting roasted here and are wrecking DDG's carefully built up reputation. I can easily see how this might seem to be a good idea to you and other DDG engineers but it goes 180 degrees against DDG's stated mission. In other words: you may be well outside your paygrade on this.
I don't think it's fair to say this comment is 'wrecking DDG's carefully built up reputation'. If what he describes is damaging their reputation to you, that's on the business, not on the commenter. If anything, the commenter seems to have been honest and straightforward with what's going on.
Sounds like you'd prefer him to have run a message past management/public relations first?
If you speak in a public form in response to something that seems damaging to the reputation of the company in a way that is probably even more damaging to the reputation of that company you are either an official spokesperson or way out of your depth. Pointing that out is a service, not a sleight. And yes, if this is not in an official capacity he should either keep quiet or run it by the PR dept. People have lost their jobs for far less. Anyway, I've pinged @yegg, hopefully he can shine some light on this.
Not as worse as publicly denouncing an honest engineer while referencing his paygrade. I hope there is no affiliation you have with DDG to be honest, because this is much, much worse.
What do you mean? This response is fine. It’s honest feedback, and it isn’t a personal swipe to say it’s above someone’s pay grade to be responding to a PR crisis as an honest engineer.
I was once an honest engineer too, publicly. Being honest in private is enough for me now. It’s a lesson worth learning.
That said, it’s not like a single HN comment will make or break a company, so if they’re really just a rank-and-file engineer, I hope the company won’t come down on them too hard. A simple “don’t do that” would suffice.
Yes, but chastising an employee is not in the interest of good PR, even if we sadly see that quite often in the days of social media. That is especially true if the employee just honestly laid out the facts, I wouldn't even call that a mistake.
I use DDG and the possibility of getting a statement directly from an engineer conveys much more trust than a carefully crafted PR statement ever could. I would think again about using it if the company does indeed come down on employees that live the values the company writes on its flags to have honest and transparent business practices.
That said, I am careful too when I state things about my company, even if I believe there is nothing to hide. Still, people that think it isn't the place for others with knowledge to comment are often not too impressive and would have difficulties in convincing me that privacy and transparency are real goals instead of just looking decent enough.
Furthermore the naming of management of DDG creates a stark contrast to the suggestion for more professional distance. I don't like PR very much as you might have guessed, but like a good design it needs some congruence.
If people find out that you just shut up for your company, it might give people the wrong impression about their business.
It’s your duty as an employee to shut up for your company. I didn’t learn this until later in my career. Fortunately it didn’t have lasting impact.
By commenting on an ongoing PR crisis without consulting management, you are both undermining their ability to respond in an effective way — imagine how strange it would look to see a “Hey, X from <company> here” after an existing one was already posted — and you’re acting on your own rather than in a team. You’re a part of a team; how could you think it’s a good idea to act alone?
Of course, I am talking to my former self with this comment, since that’s exactly what I did at S2 when working on HoN. It was a mistake, and I gave the community the wrong impression about the company’s priorities.
You have to understand, when you’re given money to do a job, you’re not given authority to become that job. Just because your job is getting beat up on social media doesn’t mean you should just jump in and go “Hey, that’s not true!” It doesn’t matter whether it’s true. Here, let me pretend to be DDG:
“Hi, Shawn from DDG here. You’re right; this was an oversight on our part. Obviously we dropped the ball on this. To clarify, we were unintentionally gathering the data as a side effect of our favicon service. <some technical details here>. We’ll be acting immediately to reverse this, and we’ll be enacting policy changes to ensure that user privacy — our core mission — is maintained going forward.”
But that’s not what they said. And if you’re gonna tell the community the opposite of what they want to hear, you’d better be in charge of the company’s Telling The Community Things division.
Thank you for explaining this all far more eloquently than I ever could. DDG is precious and worth preserving, it wouldn't take much for the press to have a field day with a couple of careless comments. And no, I don't have a stake in DDG other than as a user.
No, it is the response of someone who was once upon a time CEO of a startup that fortunately did not operate in the present day news cycle where reputations can be destroyed in a couple of hours. DDG is precious, saying things that are very much against what I know to be the founding principles of the company is something that should only be done by those that have been given the proper authority. The response here is candid but ultimately not the one that DDG would and should give. Gabriels' contribution upthread pretty much confirms that. My comment was simply to ensure things would not get worse than they already were until someone in a position of authority at DDG could step in.
All you have to do is to read this comment thread to see the kind of damage that a single statement by someone affiliated with the company can do.
For me it does the opposite. Hearing engineers speaking off-message inspires trust, while pr-evasion and weasel speak makes me think they have something to hide.
That's nice. But in this particular case the engineer says things that I know for a fact are not in line with DDG's mission statement so even if he speaks off-message he's actually destroying that trust. There is nothing about @yegg's response that strikes me as weasel speak, it is the content that is different and exactly where it matters: privacy first.
At this point we're all well aware why the app phones home. Continuing to spout that like it's some ward against the fact that this is a very real vulnerability is an insult. Trust me, your target audience doesn't give a crap if favicons work; they care that DDG acknowledges the risk of a glaringly obvious vulnerability. Who do you even think you're arguing with on HN and GitHub? My children can't multiply yet but they'd be able to understand why this is bad practice.
The repeated handwaving that no one in your company is ever going to do something bad or stupid when the browser phones home for what amounts to a cute sticker is extremely suspicious.
This is a bit concerning that for a company with a marketing so focus on privacy, you guys thought it was a good idea to have such a service.
Nobody in the company at any point thought that it could be a problem?
Your strict privacy policy mean nothing by the way, because you are a USA company and you must respect your local laws such as the patriot act, which are not very privacy friendly.
The road to hell is paved with good intentions. At the end of the day, your privacy policy is just a bunch of words with nothing to actually prevent you from abusing the data collected. Instead of us relying on DuckDuckGo to act ethically, just don’t collect the data in the first place.
Maybe I'm too old for this, but wasn't a favicon supposed to be located at "fancy.url/favicon.ico", or alternatively as a "<link rel="shortcut icon" \>"?
In germany we have the words "Datensparsamkeit" (data parsimony) and "Datenvermeidung" (data prevention) [1]. Which wikipedia merely translates as "Privacy by design" [2].
DDG is unneccessaryly producing (aggregating), transmitting (and collecting?) very sensitive user data here, which is just the opposite of data protection. I can't even understand why they try to justify their actions. It's like omitting the seat-belt in a car, then telling customers that this was required to make the in-car entertainment system more usable.
In fact I think what they do here is illegal by GDPR. It does not matter that they say they do not collect the information, it is enough it is unnecessarily sent to their servers to make the whole function illegal.
The transmission of ip address alone, which is necessary for the TCP request to happen, deanonymizes the request enough to not be considered anonymous within the GDPR framework.
GDPR Article 5 (1) c:
"Personal data shall be ...
adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (‘data minimisation’);" - this is the "Datensparsamkeit" you mentioned.
Exceptions from Article 7 do not apply: The user has to give wilfully give informed consent, which he cannot do as the privacy policy of the browser omits the information that all visited domains are transmitted to DDG servers.
GDPR Recital 30
"Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags."
Oh, and the fact I'm downvoted for a purely informational comment additionally does not shine a good light on DDG.
Yes it does not look very GDPR conformant. They may try to argue that the transmission of visited domain names serves a purpose (browser performance) and that the user agreed to that transmission by accepting the TOS etc. However, I think they may be in trouble as consent for data processing under GDPR needs to be given "freely" [1]:
"When assessing whether consent is freely given, utmost account shall be taken of whether, [..] the performance of a contract[..] is conditional on consent to the processing of personal data that is not necessary for the performance of that contract."
As DDG's favicon-hack is not strictly neccessary for operating the DDG-browser, DDG would need to give users the option to opt-out of the favicon-retrieval, otherwise they may have "forced" the users to consent to the data processing, thereby voiding that consent as far as the GDPR is concerned.
You are completely right about the not "freely given" aspect. But well, it already breaks down at the "informed" aspect.
Their appstore pages link to the generic privacy policy of the company instead of for the browser specifically. This alone will usually already NOT be accepted by the supervisory authorities at least in Germany - the Landesdatenschutzaufsichten usually require a product specific privacy policy being linked for it being valid (personal experience), or at least making clear in the general policy what applies to the product, what not. And as mentioned, the fact visited domains are sent to the DDG servers (also outside the European Union) was not mentioned at all in that policy when I looked some minutes ago.
If it is an unnecessary request to another service, yes.
IP-adddresses are considered personally identifying information. TCP requests transmit IP addresses.
Under the strict interpretation of the GDPR, a lot of things which are common outside the EU might be illegal, like e.g. embedding Google Fonts. To be on the safe side, people usually at least list these external dependencies in their privacy policies to construct some kind of "consent", but till we have more actual court rulings, this is a huge problem area.
For the problem at hand, it is pretty clearly illegal, as it's not only an ip address transmitted, it is a combination of ip address plus visited unrelated domain. This allows the creation of profiles. It does not matter for the GDPR, if the profile is ACTUALLY created, the pure possibility of creating it any time is enough to be a problem.
I don't think this is an accurate way to analyze GDPR compliance. As the staffer points out this favicon service follows their own privacy policy, if by this policy they keep (or analyze, sell, distribute, etc.) no data on your use of the service then there is nothing of interest for the GDPR.
They might have to prove that their privacy policy is indeed GDPR conformant and that their service works as advertised, but in practice this is likely more about public trust that legality.
Art. 4 GDPR (1) clearly makes the (ip-address, visited domain) tuple personal data
Art. 4 GDPR (2) defines "processing" data, and the pure "collecting" of data, even if immediately thrown away, is usually already considered "processing", therefore the GDPR applies.
If you are doubting this, just for a moment imagine, instead of the visited domain they would have sent all form data, including for example credit card data, you entered somewhere on a third party webpage to their central server and did not mention the fact in their privacy policy.
Do you really think then there is "nothing of interest for the GDPR" just because they do not actually permanently record that information? It would clearly be a violation. But to the GDPR, the importance of that data is equal. In fact, the domainnames might actually be more important to the law, as article 9 establishes event stricter rules for "sensitive" data about e.g. health or sex life of a person, and the domainnames might just leak that information.
> Wait what? a TCP request already breaks the GDPR rules?
If the TCP request carries personal data like the name of a visited website plus the user's IP address, then it "breaks the GDPR rules" in so far as you now have to fullfil your GDPR transparency/consent etc. duties /before/ sending that request.
Maybe not all website names look like sensitive data to you, but some website visits you surely want to be treated as sensitive, personal data (like names of hospitals, doctors, political parties, religion etc.).
I think you are being downvoted because your interpretation of GDPR is extremely broad and people disagree with that. I also don't think it's in line with the way it is being applied in practice. Nothing to do with DDG.
I guess I might be one of the few people here who actually have been regulated under the GDPR by a supervising authority.
When that happens, you have to do insanely stupid seeming stuff like explaining in your privacy policy even, why your application, which is clearly for accessing a webservice, actually needs the Android permission to access the internet at all, true story. So I am pretty sure, an app sending visited domains to a central server outside the EU without even a mention of that fact in the privacy policy will cause problems with them, should they ever check the app. Of course, other countries might handle this differently, as Ireland shows.
So whoever thinks my interpretation is overly broad should first have the decency to step forward and actually explain why, instead of hammering a button and second, talk to me again after he had a meeting with the responsible authority and listen to THEIR interpretation of the GDPR ;-)
People should not mistake my interpretation with endorsement of the overly broad text of the GDPR itself.
I'm surprised at how you're handling this. DDG is supposed to be friendly to privacy-aware users. You're dismissing people's valid points and asking them to trust you, just like any other privacy-non-friendly service would do.
Edit: I'm speculating here. But specifically because of the way you've replied here and on Github, my actual level of trust in DDG team went down.
Doubling down is fairly ridiculous when one has to imagine the original reason for doing this was to save on time and engineering for the app by leveraging what DDG had already built, but a mindful response would be that you are aware of the downsides of this approach and you'll be working to change it.
Besides, how do you handle Intranet, VPN sites, and auth-only sites where DDG's god-tier favicon parser in the cloud couldn't fetch the URL anyway?
Does it have an option to disable the internal favicon service and/or an option to disable favicons entirely? (I disable the favicons on my own computer, since I don't use them.)
It’s amazing how tone deaf technologists can be when it comes to privacy, even when they have nothing to gain by exploiting the user’s data. DDG’s response reminds me of Mark Shuttleworth’s argument that they “have root”, so we can trust them with our life.
Dear DDG, you are getting complaints on GitHub and Hacker News. This is not the general public, it’s people who understand the issue. You should definitely reconsider whether you’re doing something wrong.
... is completely irrelevant. Even if they were trying to save babies from a fire (which they really aren't) it wouldn't excuse the fact that they're doing something orthogonal to their stated policy and sole reason for existing.
Everyone makes mistakes, that's not the point. The point is to correct them when they're found, instead of digging one's heels in the ground and pretending it's nothing.
It actually happens very often. People monitor their networks often, and pay close attention to such problems. Just search for “Apple phone home” and you’ll find many cases where people complain about Apple’s various services making worrying requests.
Everyone is missing the point here. Let me break this down as simple as I can:
1. End user does a DDG search for "food"
2. The "food" query returns a list of search results, these results have each have a link, DDG wants to display the favicon for each link.
3. To be clear, DDG does not store or log the IP address of the user doing the query. They do, however, know what was queried, so they know "somebody" somewhere searched for "food". They have to know this, they are a search engine after all.
4. Since DDG wants to show the favicon "privately", and they dont want to put that logic/work on the client side (which could leak your IP), so instead DDG finds the favicon internally.
5. A DDG server, completely separate from anything search-related is then tasked with finding the favicon for your "food" query results, lets say the #1 result is www.allrecipes.com, so a DDG server goes to www.allrecipes.com and finds the exact favicon location.
6. The "found" favicons are then stored in a cache, and displayed from the cache like this: https://external-content.duckduckgo.com/ip3/www.allrecipes.c... (and if no favicon is found in the local cache, you get a grey arrow by default)
7. I'd like to note, even with all this action, DDG doesn't know if you actually "visited" www.allrecipes.com, they simply know that some anonymous user did a search for "food", www.allrecipes.com was a search result, and a favicon was displayed. They dont know who searched for it because the users IP is not stored anywhere, they dont know if you visited www.allrecipes.com, they prevented you from leaking your IP to allrecipes.com since they didn't force the end user to load the favicon.
So whats the issue? What am I missing here?
PS: You know this works because after doing all these searches for food and seeing allrecipes.com (and even clicking allrecipes.com result in the DDG Mobile App or browser extension), guess what? allrecipes.com doesn't follow you around with re-targeting ads! Why? Because DDG prevented that from happening!
I think that distinction needs to be made. I think DDG should treat this app as a web browser which means phoning home to this endpoint is unacceptable.
Seems a bit much, but k-anonymity could work here. Hash the domain, take the prefix, get a batch of favicons back. They won’t know which you visited, but still get the benefits of consistent favicon support.
When there's a miss, send the domain to the server so that it can fill the cache. You probably won't even have to do that as DDG could have already filled the cache with their search results.
The favicons on the duckduckgo browser are often worse than other browsers in my opinion. For example the BBC website where DDG interestingly enough just uses /favicon.ico and the other browsers use the apple touch icon. (Information I found from just looking at the pages headers)
Don't really understand why they do extra work to get worse results... This feels to me slightly worse than just a privacy concern, it's a misunderstanding of their domain which leads me to the question of what else do they not fully understand.
The good news is that you can have the DDG search engine as a default in other browsers.
(I understand that the DDG browser is probably not their main focus and any lack of knowledge can potentially be just on their mobile browser.)
Because it's the easiest (and sometimes only) way to get this functionality on major browsers and browser developers have no plans to implement this functionality natively.
I maintain a set of such keywords in my Firefox, that I use on iOS and Android too.
The easiest would be to just use Google btw, as it does a reasonable job to give you the website you're thinking of just by mentioning it in the search query, e.g "Frozen imdb" (though it does a perfect job in this case with just "Frozen").
If you use DDG, because I'm assuming you value your privacy, sending your search history to DDG when you could avoid it doesn't make much sense.
The main issue for me is that they only work in the url bar not the search bar. For the same reason people are upset about the favicon thing I don't want anything (mis)typed in the urlbar to be sent to a search engine. I've failed to find the correct setting to make this happen a few times before but just looked again and setting keyword.enabled to false prevents it from searching from the url bar (the setting is confusing and not related to the search keywords feature).
I did try firefox search keyword anyway once a while ago but it isn't that easy to configure and the setting were lost not too long after I started using it (I don't remember the details but I think my profile was corrupted by ecryptfs and the keywords didn't import to a new profile or something like that). Not importing them isn't a horrible choice (if that is what actually happened) considering there doesn't seem to be an easy way to even list them, but there should at least be a distinct import/export for keywords. The search page of preferences has a keyword field next to search engines but it doesn't seem to actually list them nor can you add a keyword from that page. All in all, it doesn't seem like a feature Mozilla cares about at all, although it would be a nice feature with a bit more work. Use from right click search on text selection would be another thing that would make it much more useful.
I tend to use ! in two circumstances, one as others mentioned when a DDG search didn't return what I wanted (and I am fine with letting DDG know that since it is the exact same search I just searched DDG) and secondly when I am looking for something on a particular site. Some of the second case would be best done via Firefox keyword search but the advantage of DDG is that they have a lot of obvious names going to the obvious place so rare specific searches are much more convenient than doing anything to set up a list. Since I very rarely use such searches, there isn't anything DDG could learn about me from avoiding just those searches going to DDG. OTOH, using site searches more frequently would be a win for privacy.
I was astonished by this at first, but I think you must mean "you're sending all searches performed with bangs to DDG". I worried that you meant somehow the browser search history was being sent to DDG, but that seems impossible.
well, this isn't something that I have given that much thought...
But yes, when I go to ddg and type in "!g <thing>" after having typed in "<thing>", I'm on some level hoping that this is somehow informing ddg that their search results for <thing> could be better.
Now, if they have somewhere stated that they are not going to do this, then yes, it would be bad.
530 comments
[ 1.4 ms ] story [ 330 ms ] threadAlso different expectations
Note also that these are favicons for results that DDG has already given you. This isn't tracking your clicks. The list of sites that appear on the search result page is not new information to the search engine that just gave them to you.
EDIT: Like other commenters, I was not previously aware that DDG had a browser, and my comments were about this behaviour for the search engine results page.
The service is private as we do not collect any personal information (e.g. IP addresses) on any requests for this or any service and the requests are all end-to-end encrypted.
There are so many options.
Even if you don't like the reply it's good that we're getting replies.
I understand utilizing a single service/feature to accommodate multiple platforms is a big win from a programming standpoint. But if it carries the risk of losing the trust of your user-base while at the same time risks breaking the core mission statement of the business (privacy), it's probably not worth it.
Why use that term when it clearly can't be true or even seems applicable/relevant?
(But this instance, the favicon service, is not a good privacy-functionality trade-off)
They should probably change the behavior to how it’s suggested in the thread, but I’m still going to use DDG over alternatives for the bang feature.
What does Google have to gain from it? Google has pretty aggressive anti-scraping protections to protect against this exact behavior, so why would they allow Startpage to get away with it?
What does Startpage have to gain from it? Unlike DDG, they don't seem to have any core product, so they fully depend on Google's goodwill which is very shaky grounds when it comes to a long-term business.
"You can’t beat Google when it comes to online search. So we’re paying them to use their brilliant search results in order to remove all trackers and logs."
Isn't that exactly like DDG but Google instead of Bing?
https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
> We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).
So basically it's all Bing and I see no effort to reduce that dependency as you claim.
Mojeek [1] is also a good privacy option that doesn't get talked about as much as DDG. They distinguish themselves by having their own search index.
[0] https://www.runnaroo.com/search?term=python+unit+testing
[1] https://www.mojeek.com/
Go here and turn your "web and app activity", your "location history" and "ad personalisation" off, if they aren't already:
https://myaccount.google.com/data-and-personalization
If you do that, there's no evidence that Google is more privacy invading than DuckDuckGo, since you're left with taking their word for it. And frankly I trust a big, bureaucratic company more than I trust a startup. The ideal would be to trust technology (e.g. end to end encryption, open source) but that's not the case here.
Note that I also use DuckDuckGo in my Private Mode, as Firefox allows setting different search engine for Private Mode. I do that because it's better to compartmentalize your online personas, plus I keep my "web and app activity" in Google on, with deletion after 3 months.
1. Check for <link rel="icon" ...> tag(s)
2. Check for /favicon.ico
3. ...give up?
Someone correct me if I'm wrong!
Really shouldn't be complicated enough to need a special service to handle "edge cases".
Another comment mentions web manifest - I guess try those first before meta tags, or whatever order the standard says to use. I mean, we're talking a web browser here, it's designed to do these kind of tasks.
On a technical level of course its doable but in reality it's a complete waste of data and processing, not to mention it could take a long time to show up. I imagine they have these favicons all cached on their side so they can quickly send the right file down and/or do this processing if needed.
That being said maybe they should just not use a favicon if it's that big of a deal.
In that case, yeah, I don't think the icons are necessary to show at all...
A direct correlation exists between the revenue Google receives for selling data and the quality of its search. Google focuses completely on tracking and search, with privacy behind a far far away afterthought (if it's a thought at all).
[1]https://en.wikipedia.org/wiki/Names_Database
The argument is "Trust DDG". That argument is being attacked as "DDG's founder has done bad stuff in the past, it's likely DDG will do bad stuff, so I won't trust it". That seems to be attacking the argument to me, thus not an ad hominem.
[My opinion on trusting DDG] - [Reason for my opinion on trusting DDG].[More information about the reason]
In this case, because I don't use the "personal attack" to reach my conclusion that the person was wrong, I don't think it would be a case of argument ad hominem.
But if someone read it like this:
[One reason for my opinion on trusting DDG] - [Another reason for my opinion on trusting DDG].[More information about the second reason]
Then I am using the "personal attack" to reach my conclusion that the person was wrong, so I think it would be a case of argument ad hominem.
My comment wasn't written very well, and I'll try to write better comments in the future. Not that adding an ad hominem to a valid argument makes it invalid, I guess?. But it's still good to avoid fallacies, and to write one's comments so they're likely to be understood the way one meant them.
https://en.wikipedia.org/wiki/Argument_from_fallacy
If you're willing to sacrifice search quality for privacy, as in switching from Google to DuckDuckGo, then you might as well take a step further and switch from Google to Searx/Ask.Moe.
If you want to market yourself as a champion of privacy, then the absolute minimum criteria should in my opinion be that your codebase is open source.
Because they have a good privacy policy. They would face legal consequences if they were lying. Nation state actors can presumably override privacy polices but it's better than nothing.
I couldn't figure out which searx instance to use and found no good way of knowing who to trust, most of the engines i used were broken and telling me to find another searx engine.
It looks like it's pulling most of its info from duckduckgo anyway.
Personally I'd rather trust a known entity than an unknown entity anyday, especially when the unknown entity is slow, complicated, buggy and broken in many places.
If you hosted your own instance then it would be a lot more reliable since the IP wouldn’t send a suspiciously high amount of requests.
As for your trust argument, I couldn’t disagree more. You choose to trust DuckDuckGo, who happens to be closed source, because of their branding. The same way people trust/trusted Google/Apple/etc. because of theirs. This thread is a perfect example why being open source is the most important thing for any privacy service (because otherwise this privacy leak likely wouldn’t have been discovered, and people wouldn’t have known that the company so carelessly violate people’s privacy and fail to correct it when people point it out.. it should really make you wonder what’s happening in the search engines codebase).
he made big money in the past selling user data to the highest bidder. you can google all about it. I never understood why people trusted duckduckgo
This makes sense to me and is not alarming. Getting favicons actually is difficult to do robustly; many applications and websites use Google's service to do so, which then leaks the request to Google: https://www.google.com/s2/favicons?domain=ycombinator.com
Putting this logic in the client is not feasible. You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
DuckDuckGo hosting this functionality themselves is also a positive. They have previously been burned when the Web of Trust service they were originally using was found to be farming data, and turned it off immediately once discovered. Processing, hosting, and serving the icon themselves prevents that from happening again.
This is not to say that DDG is perfect: links you click do seem to be redirected through a /l/ page on their domain, which can cause problems: https://lapcatsoftware.com/articles/duckduckgo.html
So if you're a privacy browser. Don't. Favicons are not essential. Or, at the very least make them "opt in" and explain it.
DuckDuckGo can't do that because they don't have access to your browser's history and favicon cache, precisely to protect your privacy.
EDIT: This reply is wrong if DDG is the browser.
As for complexity, I'm sure users of privacy oriented software would prefer to not have favicon in small percentage of corner cases rather than leaking browsing history.
Wait, what? Why?
And I have no clue how search results come into this? Of course the browser isn't fetching favicons on search result pages, that's the servers job, but that's not what this is about.
It's really odd that this is the top comment right now, given that even the headline makes it clear its about the browser, not the search results.
I don't see why not. Browsers are constantly dealing with these very issues. It's one of their core competencies.
I am surprised the user is not complaining about this instead of the favicons. Their privacy policy goes on about the privacy implications of Referer headers and instead of calling out browsers for sending Referer by default, they instead give themselves power to record all the user's clicked results themselves. The Referer problem is something that can be solved by the user at the browser level through, e.g., using a client that does not send Referer, browser extensions/plug-ins that can control headers sent, or perhaps with a local proxy to remove the Referer header.
Unless DDG has changed, these prefixed result URLs are the default. It is possible to get unprefixed result URLs using the "lite" version of DDG however that is not the default. "Privacy-focused" search engine chooses less private default. News at 11.
I recently noticed that DDG has started redirecting queries submitted via POST to /lite/. The redirect is to the same domain. No explanation. I have a custom client that does not follow redirects and I now have to submit two sets of HTTP headers instead of one.
These guys are trying to make money from advertising just like everyone else. They have to be very particular in the methods they use to do it -- check the exceptions in their privacy policy -- but it is the same game. Ads and affiliate links. That sort of business and privacy are always going to be at odds with each other.
If I recall correctly there is a way to turn off the prefixed result URLs without having to turn on Javascript, by adding/changing a URL parameter, e.g., kh. However that is not the default setting. Not very friendly toward the privacy-conscious user who has Javascript disabled and does not read HN to find out about otherwise undocumented URL parameter usage.
For certain queries, I can actually get a different first result based on the HTTP method I use.
See here: https://help.duckduckgo.com/results/rduckduckgocom/
DDG's privacy policy also goes on about the privacy implications of User-Agent headers combined with IP addresses.
So let's say I take what they have put in their privacy policy to heart and I stop sending a User-Agent header. In response DDG sends prefixed result URLs? WTF?
Using haproxy, for example, I can use a "modern browser" and send no User-Agent header. I still get prefixed result URLs.
You're free to disagree, but in that case you can make use of the option to change the default that they provide for you.
I would not call haproxy "referer masking software". In any event, a proxy is not even needed.
Modern browsers are open source, right? Users can edit the source and remove the code that sends Referer header.
Even easier, I wrote my own http client. I can send any header I want, or none at all. According to DDG's privacy policy this is a good thing.
> You want to send requests directly to every shady site that shows up in your search results, load their pages in the background, work through network delays and HTTP errors, and parse out the location/format of the favicon files?
Looking at DuckDuckGo search results and visiting a page that you navigate to are two different things.
1. DuckDuckGo search results:
DDG already returns the search results so there's no privacy violation to return the favicon or the URL for the favicon in the list.
2. Any page that isn't a DDG search results page:
Use client side logic to locate the favicon. This means worse performance but better privacy - which aligns with DDG's goals.
If you want to optimise this then DDG could send the client some precalculated Bloom filters with info about known sites. The client could use these to try certain methods of favicon retrieval first.
https://github.com/duckduckgo/Android/blob/b2131d7d2f47fb09d...
My advice:
Install ungoogled-chromium: https://github.com/Eloston/ungoogled-chromium
Install these extensions: https://github.com/gorhill/uBlock https://github.com/ilGur1132/Smart-HTTPS
There is also a Chromium extension that lets you install from Chrome Web Store: https://github.com/NeverDecaf/chromium-web-store
Set duckduckgo.com as your default search engine with a blank home page. But you could also use @pkrumins home pages of https://techurls.com or https://finurls.com as nice home pages.
Use Mullvad VPN: https://mullvad.net/ (They are EVEN available on F-Droid now, which is AMAZING)
Security harden your Android device: https://niftylettuce.com/posts/google-free-android-setup/
Security harden your Mac: https://gist.github.com/niftylettuce/39597a7b3bc0660ffe1e09d...
P.S. If you need email forwarding for your domain name, you can use something I made. https://forwardemail.net - it is 100% open source.
Follow me @niftylettuce on GitHub and Twitter for more
Disc: Googler
>Faster, less bloat
Stock Firefox doesn't actually have that much bloat, and it's not noticeably slower than Chrome on reasonably modern hardware (i.e.: most page loads are near-instant, same as in Chrome on a good connection).
>Extension support for Chromium is way better too
Until Google decides that your extension is unworthy of being in their store. This notably happened with Pushbullet quite recently: https://news.ycombinator.com/item?id=23168874
... Until Google inexplicably restored it a few days later, but not before lots of accusations were thrown around.
---
IMO it's also worth noting that ungoogled-chromium is (obviously) an unofficial fork of Chromium. Google may at any time change Chromium so substantially as to either require Google integration at some fundamental level for even the most basic functionality, causing too much work for such a low-profile effort to continue, or just make Chromium closed-source. With Firefox, that risk doesn't exist because of the business motivations of the company that develops it.
I haven't used Chrome regularly in the last year or so, but it would get noticeably slow by the time I had 3-4 windows with 10-20 tabs each. Firefox hasn't really had that issue yet.
Years ago though, Chrome was noticeably faster than Firefox. That changed (IMO at least) at some point in the last few years.
I got used to Firefox's dev tools and can't find my way around in Chrome's.
People here are:
- early developers from majir browsers
- extension creators
- creators of major web properties
- people who spend their work days on the web
- etc
seriously. Don't underestimate HN.
Edit: and as someone who's been very into customizing and extending browsers since early Firefox: your ideas about Firefox vs Chrome tells me that you are either
- deeply biased
- use a subset of browser features that is too small to realize the problems with Chromium (in which case I suspect your 100x superuser is wild hyperbole)
You can come back when Tree Style Tabs including related sub-extensions works as well on Chrome as on Firefox.
https://en.wikipedia.org/wiki/Impossible_Is_Nothing_(video_r...
https://www.youtube.com/watch?v=UGMaVC1YVlQ
Brave is the only browser that includes strong ad and tracker blocking by default, which is the best way to ensure your privacy on the internet.
In fact, every decision I've seen from them makes me think they care the most about user privacy, period.
Previous discussion: https://news.ycombinator.com/item?id=23474842
You can install uBlock Origin in 10 seconds from the GitHub release page (see previous link shared). This is vastly better than anything I've seen for ad-blocking.
Also that just makes me trust them more – they are the only browser vendor that are trying to find business models that don't involve selling my data or selling out to someone who does.
Edit: I feel it is slow because it has extra bloat added. Like I said, take my comments here with a grain of salt because I'm a very biased power user. I respect the efforts of these developers regardless of what project it is, the focus on privacy and building something different is truly awesome.
Ditto for developer / power-user experience – it's still chromium. Just with an ad-blocker written in Rust built-in and some other features.
Data or stop spreading FUD. There is nothing "slow" about Brave.
A Brave fork meant to address this issue. Time will tell.
- All new load balanced infrastructure - Browser extension + API wrapper - Support for pixel tracking blocker (opt-in checkbox or TXT setting) - Smart alerting - Globby/regex support
If you follow my Twitter or the GitHub releases you will get updates.
Advertising your Twitter for the advice of "switch to somewhat well-known browser X, install these very common extensions and use a VPN" is also a bit ... odd.
I haven't checked all links but some things need to be updated, for example Skimmer Scanner is gone from the Play Store and Yalp Store is abandoned and doesn't work anymore, you should be using Aurora Store. I'd also recommend Aegis or FreeOTP+ over FreeOTP for 2FA. NewPipe is better installed from this[1] repository until this[2] issue is solved.
[1] https://archive.newpipe.net/fdroid/repo/
[2] https://github.com/TeamNewPipe/NewPipe/issues/1981
The favicon is acquired from DDG servers for the result you've just retrieved from DDG servers.
How is this leaking anything? What additional privacy would you gain from getting the favicons from the domains directly of search results delivered by DDG?
Like our search results, the favicon service adheres to our strict privacy policy[1] in that the requests are anonymous and we do not collect or share any personal information.
[1] https://duckduckgo.com/privacy
The service is private as we do not collect any personal information (e.g. IP addresses) on any requests for this or any service and the requests are all end-to-end encrypted.
Potentially saving a few requests here and there is certainly not worth phoning home with that kind of data regardless of what records you keep how much you do to anonymize it. This is especially true for a company that has built its brand on promises of privacy!
Besides, favicon requests are small potatoes compared to the kind of tracking, ads, metrics, and other often-unnecessary page resources that bog down most of the modern web. And a well-designed website can mitigate the issue pretty easily.
It's a really bad look and you should ditch it.
This is troubling.
If you say the service is anonymous and does not leak data, prove it.
This sentence is 100% meaningless. I understand you have good intentions, but these things must rely on proof, never on trust. Either you get this information or you don't; whether you say you "collect" it is inconsequential.
If it's not present, then you have options, and yes, using your weird API is an option (which I still don't like, but ok). But sending private information to your servers even when sites follow the standard show either that you're probably not trustworthy, or that your product team is so painfully incompetent that I'd be afraid to use their browser at all.
Which presumably means you've already created the logic for determining favicons. I'm not sure why this couldn't be implemented in the browser.
I don't think here's a need for adjectives here. Why stress that it's anonymous (when that's hard to verify) or that the search engine is private, when that too is starting to come into question? Repeating these things won't will them into the reader's perception.
> In addition, doing it this way avoids another request (and potentially multiple) to the end site.
This isn't true, unless I'm missing something here? When I access a website, the HTML response I get from that website includes all the information my browser needs to, on its own, get and display the favicon. Can you clarify why you think/say this avoids one or more requests? What mechanism is this service a substitute for?
Sidenote: The more I use pi-hole the more I realise how essential it is!
Those simpler popover-ads which can be closed clicking an X in the upper right corner still are blocked tho...
Spilling my secret tho (and YouTube execs hate me for it!): i block YouTube in my mind and only rarely go to it if i really need to watch a video (which, for me, is rarer than i ever thought it'd be).
Or mpv (for single videos, or local playlists), or mps-youtube, or youtube-dl.
Uncheck the very last option "Site Icons"
The issue, as I understand it, is that the Android app loads the favicon service for search results you actually open in the app.
Sure they can. Doesn’t mean you have to believe them.
Seems like time to get SearX a try now: https://searx.me
I have read your explanations in good faith and they don't cut it. This behavior cannot continue. Good privacy promises are not based on trust - they're based on not ever handling private data in the first place. If you don't quickly admit your mistake and roll this back, it will jepoardize your entire brand - and rightfully so. If you believe this behavior is okay, then it demonstrates incompetence; if you don't believe this behavior is okay but do it anyway, it demonstrates malice.
This is the one thing you Should Not Have Done.
Do Chrome, Firefox, or Safari do this? I would assume they do it on-device.
I haven't looked at Firefox and Safari but I assume they do something similar.
Also, in SQLite, note that LONGVARCHAR is the same as TEXT, and that you don't need to specify both UNIQUE and PRIMARY KEY (it is redundant), and that if it is not a INTEGER PRIMARY KEY and not WITHOUT ROWID, then it isn't the real primary key but just an index (same as UNIQUE); add WITHOUT ROWID if you want to make it a real primary key, but note that the way the data is stored differs then, and WITHOUT ROWID is inefficient with tables storing large blobs.
Not that I'm against making money. But there's a tipping point associated with some height value in a pile of cash, and once you cross that point then the pile controls you. DDG probably hasn't crossed that point yet, but self-justification is one of the steps on that path.
https://github.com/mozilla-mobile/android-components
https://github.com/mozilla-mobile/Firefox-iOS
For those of you who don't understand, Yes they are saving data on all your search terms, and Yes they are saving data on these favicon requests, don't be so naive, they admit it themselves. They are not going to turn this feature off because it is too valuable to have all your information on all the sites you visit
So let’s call them out on that, briefly, then get on with the argument.
We all, including Mozilla!, have made design decisions that in hindsight could have been better.
The important thing here is dialogue and change. And both are happening.
Firefox and Google Chrome probably have the equivalent of many small high quality libraries embedded in them, implementing 'business' logic or protocols, that could be reused in more places.
I guess a large scale study on github could be done, with a graph analysis to show potential "cut off" points in codebase.
It’s a bit telling that they linked to the GitHub repositories rather than specific lines of code they were talking about.
To me it looks to be trying to uphold your anonimity until you commit (click) through to the site/link. But certainly other ways they can approach this if it really bothers people.. I'd prefer DDG doing the lookup.. or having no fav icons.. over my computer going and downloading all my bookmark or other source icons
Maybe this should be an opt-in rather than an opt-out feature?
Edit: as pointed out by warpspin in another comment, this is about the DDG Browser, not search results.
This is mostly a UX issue IMO.
https://duckduckgo.com/settings#appearance
Uncheck the very last option "Site Icons"
How if you type a url into the browser how do you stop the browser from sending that url to ddg to get the favicon?
I’ve been an avid DDG user for years and it worries me that DDG staff don’t see why this is an issue. We shouldn’t have to trust your privacy policy if you minimize exposure.
Can you tell how many visited site A and also site B?
Generally speaking. Mine is shielded with lead.
Duckduckgo chose compliance with the inconsequential minutae of bigtech over its primary pain point. This is indicative of misalignment between stated values and the values demonstrated through actions. If you guys made this call, sacrificing privacy for something so banal, it doesn't bode well for what's going on in the rest of your operations. That means the problem isnt technical, its cultural, which are unfixable, therefore it's pretty much over. Good luck to regain trust once you've been outed as a put on.
I just switched to DDG browser a week ago and will now be looking for a new browser now. I hope you know this is not an appropriate response to the situation. Especially because all you guys do is preach about how much you protect your users’ privacy. Now you’re here asking us to trust you not to abuse our data and just linking us your privacy policy. I’m sad to say that my faith in the DuckDuckGo company and team is now lost.
I'ts not like we couldn't have predicted this disaster.
One more reason to never trust a companies "word". Show me the code.
If that's true, then I am so glad they ghosted me when I applied there.
Sounds like you'd prefer him to have run a message past management/public relations first?
Not as worse as publicly denouncing an honest engineer while referencing his paygrade. I hope there is no affiliation you have with DDG to be honest, because this is much, much worse.
I was once an honest engineer too, publicly. Being honest in private is enough for me now. It’s a lesson worth learning.
That said, it’s not like a single HN comment will make or break a company, so if they’re really just a rank-and-file engineer, I hope the company won’t come down on them too hard. A simple “don’t do that” would suffice.
I use DDG and the possibility of getting a statement directly from an engineer conveys much more trust than a carefully crafted PR statement ever could. I would think again about using it if the company does indeed come down on employees that live the values the company writes on its flags to have honest and transparent business practices.
That said, I am careful too when I state things about my company, even if I believe there is nothing to hide. Still, people that think it isn't the place for others with knowledge to comment are often not too impressive and would have difficulties in convincing me that privacy and transparency are real goals instead of just looking decent enough.
Furthermore the naming of management of DDG creates a stark contrast to the suggestion for more professional distance. I don't like PR very much as you might have guessed, but like a good design it needs some congruence.
If people find out that you just shut up for your company, it might give people the wrong impression about their business.
By commenting on an ongoing PR crisis without consulting management, you are both undermining their ability to respond in an effective way — imagine how strange it would look to see a “Hey, X from <company> here” after an existing one was already posted — and you’re acting on your own rather than in a team. You’re a part of a team; how could you think it’s a good idea to act alone?
Of course, I am talking to my former self with this comment, since that’s exactly what I did at S2 when working on HoN. It was a mistake, and I gave the community the wrong impression about the company’s priorities.
You have to understand, when you’re given money to do a job, you’re not given authority to become that job. Just because your job is getting beat up on social media doesn’t mean you should just jump in and go “Hey, that’s not true!” It doesn’t matter whether it’s true. Here, let me pretend to be DDG:
“Hi, Shawn from DDG here. You’re right; this was an oversight on our part. Obviously we dropped the ball on this. To clarify, we were unintentionally gathering the data as a side effect of our favicon service. <some technical details here>. We’ll be acting immediately to reverse this, and we’ll be enacting policy changes to ensure that user privacy — our core mission — is maintained going forward.”
But that’s not what they said. And if you’re gonna tell the community the opposite of what they want to hear, you’d better be in charge of the company’s Telling The Community Things division.
All you have to do is to read this comment thread to see the kind of damage that a single statement by someone affiliated with the company can do.
Agree, didn't mean to imply that.
The repeated handwaving that no one in your company is ever going to do something bad or stupid when the browser phones home for what amounts to a cute sticker is extremely suspicious.
Nobody in the company at any point thought that it could be a problem?
Your strict privacy policy mean nothing by the way, because you are a USA company and you must respect your local laws such as the patriot act, which are not very privacy friendly.
Curious to know why this is an issue.
An online favicon generator will create these variations
<link rel="apple-touch-icon" sizes="57x57" href="/ico/apple-icon-57x57.png">
<link rel="apple-touch-icon" sizes="60x60" href="/ico/apple-icon-60x60.png">
<link rel="apple-touch-icon" sizes="72x72" href="/ico/apple-icon-72x72.png">
<link rel="apple-touch-icon" sizes="76x76" href="/ico/apple-icon-76x76.png">
<link rel="apple-touch-icon" sizes="114x114" href="/ico/apple-icon-114x114.png">
<link rel="apple-touch-icon" sizes="120x120" href="/ico/apple-icon-120x120.png">
<link rel="apple-touch-icon" sizes="144x144" href="/ico/apple-icon-144x144.png">
<link rel="apple-touch-icon" sizes="152x152" href="/ico/apple-icon-152x152.png">
<link rel="apple-touch-icon" sizes="180x180" href="/ico/apple-icon-180x180.png">
<link rel="icon" type="image/png" sizes="192x192" href="/ico/android-icon-192x192.png">
<link rel="icon" type="image/png" sizes="32x32" href="/ico/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="96x96" href="/ico/favicon-96x96.png">
<link rel="icon" type="image/png" sizes="16x16" href="/ico/favicon-16x16.png">
<link rel="manifest" href="/ico/manifest.json">
<meta name="msapplication-TileColor" content="#ffffff">
<meta name="msapplication-TileImage" content="/ico/ms-icon-144x144.png">
Nonetheless, the browser can see this when parsing the page and choose the appropriate path.
DDG is unneccessaryly producing (aggregating), transmitting (and collecting?) very sensitive user data here, which is just the opposite of data protection. I can't even understand why they try to justify their actions. It's like omitting the seat-belt in a car, then telling customers that this was required to make the in-car entertainment system more usable.
[1] https://de.wikipedia.org/wiki/Datenvermeidung_und_Datenspars...
[2] https://en.wikipedia.org/wiki/Privacy_by_design
The transmission of ip address alone, which is necessary for the TCP request to happen, deanonymizes the request enough to not be considered anonymous within the GDPR framework.
GDPR Article 5 (1) c: "Personal data shall be ... adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (‘data minimisation’);" - this is the "Datensparsamkeit" you mentioned.
Exceptions from Article 7 do not apply: The user has to give wilfully give informed consent, which he cannot do as the privacy policy of the browser omits the information that all visited domains are transmitted to DDG servers.
GDPR Recital 30 "Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags."
Oh, and the fact I'm downvoted for a purely informational comment additionally does not shine a good light on DDG.
"When assessing whether consent is freely given, utmost account shall be taken of whether, [..] the performance of a contract[..] is conditional on consent to the processing of personal data that is not necessary for the performance of that contract."
As DDG's favicon-hack is not strictly neccessary for operating the DDG-browser, DDG would need to give users the option to opt-out of the favicon-retrieval, otherwise they may have "forced" the users to consent to the data processing, thereby voiding that consent as far as the GDPR is concerned.
[1] https://gdpr.eu/gdpr-consent-requirements/
Their appstore pages link to the generic privacy policy of the company instead of for the browser specifically. This alone will usually already NOT be accepted by the supervisory authorities at least in Germany - the Landesdatenschutzaufsichten usually require a product specific privacy policy being linked for it being valid (personal experience), or at least making clear in the general policy what applies to the product, what not. And as mentioned, the fact visited domains are sent to the DDG servers (also outside the European Union) was not mentioned at all in that policy when I looked some minutes ago.
Wouldn't they need to give users the option to opt-in, under GDPR?
Any human readable ways of dealing with that?
IP-adddresses are considered personally identifying information. TCP requests transmit IP addresses.
Under the strict interpretation of the GDPR, a lot of things which are common outside the EU might be illegal, like e.g. embedding Google Fonts. To be on the safe side, people usually at least list these external dependencies in their privacy policies to construct some kind of "consent", but till we have more actual court rulings, this is a huge problem area.
For the problem at hand, it is pretty clearly illegal, as it's not only an ip address transmitted, it is a combination of ip address plus visited unrelated domain. This allows the creation of profiles. It does not matter for the GDPR, if the profile is ACTUALLY created, the pure possibility of creating it any time is enough to be a problem.
They might have to prove that their privacy policy is indeed GDPR conformant and that their service works as advertised, but in practice this is likely more about public trust that legality.
Art. 4 GDPR (1) clearly makes the (ip-address, visited domain) tuple personal data Art. 4 GDPR (2) defines "processing" data, and the pure "collecting" of data, even if immediately thrown away, is usually already considered "processing", therefore the GDPR applies.
If you are doubting this, just for a moment imagine, instead of the visited domain they would have sent all form data, including for example credit card data, you entered somewhere on a third party webpage to their central server and did not mention the fact in their privacy policy.
Do you really think then there is "nothing of interest for the GDPR" just because they do not actually permanently record that information? It would clearly be a violation. But to the GDPR, the importance of that data is equal. In fact, the domainnames might actually be more important to the law, as article 9 establishes event stricter rules for "sensitive" data about e.g. health or sex life of a person, and the domainnames might just leak that information.
If the TCP request carries personal data like the name of a visited website plus the user's IP address, then it "breaks the GDPR rules" in so far as you now have to fullfil your GDPR transparency/consent etc. duties /before/ sending that request.
Maybe not all website names look like sensitive data to you, but some website visits you surely want to be treated as sensitive, personal data (like names of hospitals, doctors, political parties, religion etc.).
Or put another way, a TCP request sent by your app from my computer can not be considered anonymous.
When that happens, you have to do insanely stupid seeming stuff like explaining in your privacy policy even, why your application, which is clearly for accessing a webservice, actually needs the Android permission to access the internet at all, true story. So I am pretty sure, an app sending visited domains to a central server outside the EU without even a mention of that fact in the privacy policy will cause problems with them, should they ever check the app. Of course, other countries might handle this differently, as Ireland shows.
So whoever thinks my interpretation is overly broad should first have the decency to step forward and actually explain why, instead of hammering a button and second, talk to me again after he had a meeting with the responsible authority and listen to THEIR interpretation of the GDPR ;-)
People should not mistake my interpretation with endorsement of the overly broad text of the GDPR itself.
Edit: I'm speculating here. But specifically because of the way you've replied here and on Github, my actual level of trust in DDG team went down.
Besides, how do you handle Intranet, VPN sites, and auth-only sites where DDG's god-tier favicon parser in the cloud couldn't fetch the URL anyway?
- Would you be ok to use a third party for this with same privacy policy?
That must be the worst justification for this possible. Favicons. Complicated to locate? Who are you trying to fool, 5 year olds?
Dear DDG, you are getting complaints on GitHub and Hacker News. This is not the general public, it’s people who understand the issue. You should definitely reconsider whether you’re doing something wrong.
why?
If you think the next time I hit the shitter I'm not going to be looking for a new browser, you're dead wrong.
Just do the basic checks and then fall back to a DDG logo, no one cares that much about the favicon.
... is completely irrelevant. Even if they were trying to save babies from a fire (which they really aren't) it wouldn't excuse the fact that they're doing something orthogonal to their stated policy and sole reason for existing.
Everyone makes mistakes, that's not the point. The point is to correct them when they're found, instead of digging one's heels in the ground and pretending it's nothing.
this would never happen with a consumer-facing product from apple or google; someone would have to MITM their whole OS to discover phone-home
1. End user does a DDG search for "food" 2. The "food" query returns a list of search results, these results have each have a link, DDG wants to display the favicon for each link. 3. To be clear, DDG does not store or log the IP address of the user doing the query. They do, however, know what was queried, so they know "somebody" somewhere searched for "food". They have to know this, they are a search engine after all. 4. Since DDG wants to show the favicon "privately", and they dont want to put that logic/work on the client side (which could leak your IP), so instead DDG finds the favicon internally. 5. A DDG server, completely separate from anything search-related is then tasked with finding the favicon for your "food" query results, lets say the #1 result is www.allrecipes.com, so a DDG server goes to www.allrecipes.com and finds the exact favicon location. 6. The "found" favicons are then stored in a cache, and displayed from the cache like this: https://external-content.duckduckgo.com/ip3/www.allrecipes.c... (and if no favicon is found in the local cache, you get a grey arrow by default) 7. I'd like to note, even with all this action, DDG doesn't know if you actually "visited" www.allrecipes.com, they simply know that some anonymous user did a search for "food", www.allrecipes.com was a search result, and a favicon was displayed. They dont know who searched for it because the users IP is not stored anywhere, they dont know if you visited www.allrecipes.com, they prevented you from leaking your IP to allrecipes.com since they didn't force the end user to load the favicon.
So whats the issue? What am I missing here?
PS: You know this works because after doing all these searches for food and seeing allrecipes.com (and even clicking allrecipes.com result in the DDG Mobile App or browser extension), guess what? allrecipes.com doesn't follow you around with re-targeting ads! Why? Because DDG prevented that from happening!
Except that you do, exactly in the way that the reporter of the issue explained to you.
But you choose to patronize them and ignore the issue.
I think that distinction needs to be made. I think DDG should treat this app as a web browser which means phoning home to this endpoint is unacceptable.
Don't really understand why they do extra work to get worse results... This feels to me slightly worse than just a privacy concern, it's a misunderstanding of their domain which leads me to the question of what else do they not fully understand.
The good news is that you can have the DDG search engine as a default in other browsers.
(I understand that the DDG browser is probably not their main focus and any lack of knowledge can potentially be just on their mobile browser.)
By using bangs you're sending your search history to DDG even when using search engines that aren't DDG.
https://support.mozilla.org/en-US/kb/how-search-from-address...
I maintain a set of such keywords in my Firefox, that I use on iOS and Android too.
The easiest would be to just use Google btw, as it does a reasonable job to give you the website you're thinking of just by mentioning it in the search query, e.g "Frozen imdb" (though it does a perfect job in this case with just "Frozen").
If you use DDG, because I'm assuming you value your privacy, sending your search history to DDG when you could avoid it doesn't make much sense.
I did try firefox search keyword anyway once a while ago but it isn't that easy to configure and the setting were lost not too long after I started using it (I don't remember the details but I think my profile was corrupted by ecryptfs and the keywords didn't import to a new profile or something like that). Not importing them isn't a horrible choice (if that is what actually happened) considering there doesn't seem to be an easy way to even list them, but there should at least be a distinct import/export for keywords. The search page of preferences has a keyword field next to search engines but it doesn't seem to actually list them nor can you add a keyword from that page. All in all, it doesn't seem like a feature Mozilla cares about at all, although it would be a nice feature with a bit more work. Use from right click search on text selection would be another thing that would make it much more useful.
I tend to use ! in two circumstances, one as others mentioned when a DDG search didn't return what I wanted (and I am fine with letting DDG know that since it is the exact same search I just searched DDG) and secondly when I am looking for something on a particular site. Some of the second case would be best done via Firefox keyword search but the advantage of DDG is that they have a lot of obvious names going to the obvious place so rare specific searches are much more convenient than doing anything to set up a list. Since I very rarely use such searches, there isn't anything DDG could learn about me from avoiding just those searches going to DDG. OTOH, using site searches more frequently would be a win for privacy.
I was astonished by this at first, but I think you must mean "you're sending all searches performed with bangs to DDG". I worried that you meant somehow the browser search history was being sent to DDG, but that seems impossible.
If I don't want ddg to know what I google, ofc I won't do it by typing in !g in ddg...
I actually use bang functions because I want to help ddg out by informing them when I'm unhappy with the results I got from them.
That's what they promise to not do and the whole point behind many people's decision to use them.
And it might be obvious for some, but it still makes no sense :-) given the browser is capable of doing it.
But yes, when I go to ddg and type in "!g <thing>" after having typed in "<thing>", I'm on some level hoping that this is somehow informing ddg that their search results for <thing> could be better.
Now, if they have somewhere stated that they are not going to do this, then yes, it would be bad.