As noted in the article, apparently Google is presenting the top four search engines used in a given country. So presumably this means they're seeing a lot more DuckDuckGo searches in the data they're collecting from Chrome users.
It's also a solid choice for them to hedge against antitrust claims, if they can point to having just added them to their browser, regardless of the fact that Google is the default and they do not present a choice screen like Microsoft had to in the EU.
> So presumably this means they're seeing a lot more DuckDuckGo searches in the data they're collecting from Chrome users
There's a lot to unpack in that statement... Is there any recent analysis on the usage stats that chrome is reporting back that someone could point to?
No clue at all why politician was downvoted for this.
Isn't it well known that Google scoops up web history from the browser or have they stopped doing/never done this? In the latter case any pointers would be appreciated.
Isn't Chrome Sync opt-out at this point? I seem to recall some small controversy about that a while ago, and setting a passphrase seems like something that it's unlikely most people will do.
A default open browser history synced across devices seems like exactly the sort of thing that would show that DDG has increased its market share.
== most of the time. The majority of people don't care enough to change the defaults. Most of the time, they don't even think about whether there even is something to change. The overwhelming majority thinks roughly like this: "Ooo, Computer just knows all of this about me? Neat!".
Source: I work in education - even in a highly educated area in a developed EU country, young and old alike think like this.
In the default config the sync data is encrypted end-to-end with the user's Google account password. However, there is also an option to share browser history with Google for telemetric reasons, and it's on by default (regardless of sync encryption settings)
Dunno about currently, but about three years ago, if you opened your own site in Chrome, a couple minutes later you'd get a visit from Google bot on the same url.
Dunno about currently, but about three years ago, if you opened your own site in Chrome, a couple minutes later you'd get a visit from Google bot on the same url.
Not only is Google's indexing infrastructure not that fast, but they deliberately don't do that because some poorly designed sites have passwords or unique keys in the URL that should not be used to retrieve content for the public search index.
At its simplest, in the West we have a thing for threes. Three bits of God, three little pigs, three branches of government (in the USA at least), "things come in threes", three books/movies in a series (a trilogy), stories that have a "beginning/middle/end".
Bottom line, the West tends toward organizing and thinking of things in threes. Some might even be superstitious about threes (perhaps a Pythagorean influence).
In China the number 4 plays a similar role. I don't know much about 'numerology' in China, save to say that recently the number 4 (which apparently sounds like 'death' in Chinese) has been considered bad luck. Here's a better explanation than I could give: https://www.quora.com/In-Chinese-culture-why-is-the-number-4...
In Hong Kong, "Chinese" buildings skip floor 4, 14, ..., while "Western" buildings skip floor 13. More recent culturally inclusive buildings skip floors 4, 13, 14, ... The most prestigious floors are 8 and 88.
To add a complication, Chinese tends to use the US convention (with the ground floor being floor 1), while the English convention is the British one (with the ground floor being floor 0).
By happy coincidence, then, the 13th floor is also 十四樓, ie the 14th floor, so you only need to skip one floor, rather than two. That explains why HK skyscrapers are so high.
4 seasons, the 4 corners of the world, the 4 cardinal directions, the 4 bodily humors (https://en.wikipedia.org/wiki/Humorism), you can find a ton of 4's in the West.
3 is important in decision making because it makes it easier to form a consensus. If I disagree with your idea, you have another entity to act as an arbiter.
Similarly, how many different options are available for similar classes of items found at Costco?
Say, frozen chicken, napkins, instant noodles, paper cups, etc. In some cases there is only 1 option offered, sometimes 2, rarely are there ever more than 4 options offered at Costco for a single type of item. When you trust that you are being offered the best choice or a top choice, well, we know what happens at Costco. People buy pallets in that warehouse.
Yes, but (1) Costco has nowhere near a monopoly on grocery, clothing, and electronics and (2) not even Costco customers do 100% of their grocery, clothing, and electronics shopping at Costco.
It was an odd thing to me but Chrome would not list DuckDuckGo until after you had visited DuckDuckGo.com manually. Once on DDG it became an option. That's been around for a while, as I've had DDG as my Chrome default for a couple years. I presume it's now an option even if you've never visited.
Actually consented, as in understood the implications and freely decided that Google should have this data, probably none. That would take a lot of generosity, especially to pay that team of lawyers and technical experts, so that you have any chance of actually understanding the implications.
Unwillingly consented, that's the vast majority of Chrome Sync users. Unless you enable the end-to-end-encryption (for which they require a second passphrase, so probably less than 0.1% actually use that), they will use your data for ad profiling etc.. Yes, that is on page 1312 of the Chrome Sync privacy statement. (They're only required to write it into there, if they do it, so it is quite certain that they didn't just want the bad PR for nothing.)
Is consent required? Assuming they actually do collect this data from their Chrome Sync data or through similar personally identifiable ways, consent would be required in many jurisdictions, especially the EU.
However, if they cared enough, it would be possible for them to collect this particular data point without personal identification.
You could for example create a UUID per installation that's only associated with this one data point.
Or you could have a time-based solution where each Chrome instance goes out to "vote" for their default search engine e.g. every 4 weeks. If you then look at the statistics on a weekly basis, you can just take these values times 4 to even roughly correct numbers. It's certainly going to be representative enough, you don't need every browser instance to have their vote in every week's statistic.
These metrics are from UMA stats. They are collected from everyone who ticks the box to report stats when installing Chrome.
They only get histograms of counts of visits to search engines, not the entire URL, and not search engines or other sites not in the list of things they track (which is at the bottom of the file).
Well, the first question is why are the pages rendering slow to begin with?
One way to make the pages I visit load faster is to disable Javascript. Another is to remove (or block) advertising. Another is to put DNS data for these sites into local hosts or zone files.
Those actions are how I prefer to approach the problem.
However as far as I can tell, those are not actions Google wants to take. They have their own preferred approach.
It is possible there are users who are aligned with Google in terms of how they want to approach the problems created by misuse/overuse of Javascript and advertising.
It is also possible there are some users who have no idea why pages are slow to load.
Those groups might want to send usage data to Google.
However I am not in either group. I dislike the web advertising business that Google depends on and therefore must nourish and support.
As such, there is no reason I can think of why I would want to send data to Google.
Also, I have not checked but I wonder if Google is restricted in how they can use the collected diagnostic data. Are they prohibited from using it for the purposes of selling advertising?
Usage data helps us make UI changes. For example, if not a ton of people are using some functionality, we might prioritize modifying or removing it. When we make a change, seeing how it affected usage is an important part of verifying we did the right thing.
So if Chrome's ever made a UI change you disagreed with, then you're in a group that would have benefitted from sending Google usage data.
Having grown tired of graphical software back in the 90's I have little interest in graphical user interfaces and interactive use. Chrome has never made a UI change I disagreed with because I do not care about the popular graphical browsers.
I care about command line programs, less-interactive and non-interactive use. Truly, the best interface is no interface.
The whitepaper.html appears to explain how usage data is utilised in ways that help Chrome improve but does not appear to contain any restrictions on use of the data to help further Google's ad sales business, whether directly or indirectly.
It is the business model that I do not wish to support.
Producing software such as Chrome is just something the company is doing in the course of selling advertising and collecting maximal amounts of data from users, whether the data is anonymised or not.
Regarding "data they're collecting": The list here is based on popularity of search engines in different locales, determined using publicly available data.
For those that don't know 'quietly' is newspeak for something that happened that no press release was issued. [1]
[1] Because in the world of the press everything should be announced so they can broadcast it and sell advertising by running stories. And not have to find it out by other more laborious methods.
It is also often followed with a statement that the real authority on the issue didn't even bother responding with any comments, when in all likelihood the journalists also didn't try very hard to reach anyone. As it is in this case.
For those that don't know 'quietly' is newspeak for something that happened that no press release was issued.
This is a deliberate Orwell reference? Vernor Vinge speculated in Rainbows End that everything which couldn't be searched for in a search engine would effectively become invisible. In 2019, that manifests as, "everything which can't be searched for in a search engine, which is backed up by crosslinked mainstream news sites and which isn't warded by words meant to scare casual readers away."
Imagine that the only thing that is reported to the Chrome backend is a client ID and a URL, which would be the only thing needed to render this app: https://myactivity.google.com
Would you think of Google as trustworthy because they only gave their backend two pieces of data? I myself would not, because I'm pretty sure the actual request and response messages are looked up by client ID (in their Google Analytics data store).
Give Google some privacy of what is being sent home :)
Chrome (and Chromium) creates at start and maintains SSL connections to Google. It is not easy to sniff what is being sent. Even if you MITM it, like in enterprise transparent proxies, Chrome will throw an error because of cert pinning. google domains should be whitelisted: "we recommend that you avoid the use of transparent proxies." https://support.google.com/chrome/a/answer/3504942?hl=en
Chrome has the all powerful "omnibox" that still sends stuff to Google. Since searches or URLs go through the omnibox there's a good chance Google gets (some of) the data.
> They need it in order to properly train their ad-network.
Given that they scoop up all this data I'd appreciate if their ad-network actually improved. Just the other day the dating site scams where back.
"We'll try not to show it again" they say. Well for vacuuming the market for the best and brightest they either don't try very hard or they are very dysfunctional because they fail as a group.
They have designed the whole android interface to scatter the settings around so it is very difficult to turn off many of the privacy-invasive features. And even if you manage to turn them off, somehow they always end up turning themselves on again.
Everybody working for this company should be ashamed.
So if you're web based (like me) then activities such as sending an email, checking out YT, reading HN, watching Twitch, and jerking off, all end up as entries in that log file.
So if you're web based (like me) then activities such as sending an email, checking out YT, reading HN, watching Twitch, and jerking off, all end up as entries in that log file.
Do you really think Google would have trained an AI to determine that last activity? How would they have trained the AI?
Presumably you aren't visiting porn sites for the articles. Voila, no AI needed.
Sloppy. How do they know I'm not preparing for actual intercourse? How do they know I'm not downloading porn for later? I could also be watching gun videos, since they've been hosted there.
Since I've already outed myself as a porn consumer I might as well get this off my chest: the recommendation engines of the video sites I visit (all the big ones) have poor, very poor AI. It's like they don't know me as a customer. I'm out here busting my balls to find "The Perfect Clip" but it's a jungle and they are not making it easy on me.
How graphic do you want me to be? But isn't the real question: is there utility for an ad network to know about your preferences in porn? If there is such utility, you're best to believe Google implemented a way to get them.
If I was an ad network I would love to hear about your porn habits. I would absolutely love it.
If you know, you are obviously one of the people who has that data. If you don't know, you aren't.
If I was an ad network I would love to hear about your porn habits. I would absolutely love it.
Such networks don't have to know whether or not and the exact moment a given user jerks it. Though, it would actually be better if you were actively browsing around and not jerking it currently. I guess that's why discovery and AI are so bad on porn sites. It's actually better for their ad revenue!
Yes. As you have pointed out, the ISP can only log a host name [0]. Well, if the user story is porn, then as it happens, host names are pretty darn telling. Also, looking at a list of host names in chronological order might help them to classify me as "closet this" or "closet that", as I find myself less and less inhibited by society's rules the closer I come to crescendo.
Chrome Sync is so sweet though. It's probably pretty close to impossible to live w/o Chrome Sync. It's my favorite tech feature. The USP for me is "As a user I want switch devices while I'm browsing, so whatever I was reading prior to having to go to the bath room I can seamlessly continue reading."
It's probably not as sinister as I make it out to be but I can think of a few items of data that differ between using DDG's native UI and that of the Chrome search bar: the headers Chrome sends to DDG are different, and the autocomplete results that come back from DDG can now be monitored
Good point, I hadn't thought of the auto complete. I figured you were more referring to the actual query itself which Chrome already sends home to Google via it's cloud synced web history. In which case it does not matter whether you use the omnibar or visit the website directly as long as you have history sync enabled.
I seem to recall reading it can be a mix of both, though generally the way you mentioned. A Bloom filter that filters locally, and if it's a hit then it sends over the URL to double-check. Would be nice if someone could confirm though.
Older versions were Bloom filters, but newer versions have moved away from that (and to a list of hash prefixes) because Bloom filters are hard to update.
A hash prefix list gets downloaded locally; Chrome checks locally against the prefix list. If a URL hits, Chrome will send the hash prefix (not the full hash and not the URL) to the server, the server will send back all full hashes that match that prefix, and then the client will complete the check locally.
In theory, if the server had a small number of matching full hashes, it could guess about what URL a client might be hitting, but in practice the system is designed as much as possible to avoid ever leaking data about what you're visiting to Google servers.
Clients download a database of partial hashes of malware URLs. If they get a hit on one of those partial hashes, they make a request for the full list of hashes with that prefix.
Google knows when a client makes one of those requests, but the exact URLs (or hashes) they're looking up are never revealed. The partial hash is 32 bits long, so there's enough collisions that making a request isn't especially revealing.
You can. In fact, Chrome automatically creates search engines for any site you search on. I can search Amazon by typing "Am<tab><query><enter>" in the address bar for example, and Chrome learned how to do that automatically despite not having any knowledge of how Amazon's search system works when I first installed it.
I guess the only difference is that with this change, DDG is available as a search engine by default with a blank install, even before you've actually used it.
I would expect to see a small bump in the stats [1] which given this is DDG's main source of revenue is absolutely a good thing.
"Duckduckgo is one of our main rivals." Is a bit of a self fulfilling prophecy for Google. They need to amp up DDG's legitimacy to ward off accusations of antitrust. Credibility, legitimacy and awareness are really the only things DDG needs to reach a wider audience and gain greater adoption.
Is this the first of a wave of anti-anti-trust moves by big tech? It's a play that I certainly would advise. It makes sense to trade marginal revenue with low hanging fruit gestures like these to take the air out of folks like Warren and the European Competition Committee.
On a side note: the author of this post is an EU focused reporter who echoes every anti google sentimente coming from there -> she is linking to the twitter of the guy who runs the PR for the media lobbying cartel in DC that lobbies against google -> and he in turn is posting a clip of the former state AG now senator that regularly attacks/sues google.
I started doing the same. Bitwarden is the best! I urge HN users who haven't checked it out to give it a try and please support the project so it lives on.
I have started doing this as well, except I'm using KeePassXC and using Dropbox to distribute the file everywhere. Would Bitwarden work behind a company firewall at a company that doesn't allow Dropbox?
I'm the same with KeePassXC but use Resilio Sync for redundancy between devices. There's also SyncThing for a free solution to Resilio Sync but haven't dived too much.
Bitwarden is pretty nice, using it at work. I do still like my control that KeePass gives, though.
@wintorez, I started using Brave browser[1] and DuckDuckGo for work and personal. It's based on Chrome but with privacy in mind.
Also, I currently use 1Password, but have been thinking about using Enpass[2] because you can sync with any cloud drive. I like the idea of syncing to a third party cloud drive in case my password service is compromised.
My concern with Brave is them being beholden to the direction of Chromium - I am wrong about that?
I used Brave for awhile then switched to Firefox+uBlock Origin, hoping to do my teensy part in decreasing the market share of Chromium-based browsers while still being privacy-focused.
At the same time, having the same base as Chrome means you won't be left behind when people start only developing for Chrome (which is a problem right now).
Heard good things about brave. I like to experience the web though more than one browser, just to see if there are any discrepancies that I'm not aware of.
Often vulnerabilities go unpatched for days, which is pretty bad when the exact vulnerability exploit code is already visible in the chromium bug tracker!
Initially I was doing the same, but then switched to using Firefox profiles - 1 for work + google search; and 2 for personal with DDG and ublock origin.
Try the Multi-Account Containers extension on Firefox. It helps isolate sites across tabs and helps avoid the normal need to create multiple profiles. There are several other container extensions (the first and most famous one being Facebook container).
Curious — why not Firefox and Startpage for work stuff or Firefox and DDG for work stuff? You can always resort to bang commands if DDG results aren’t great for particular searches. You can use the Multi-Account Containers extension (and related container extensions) to have Firefox work for multiple “profiles” of usage.
Or you could even use Chrome and DDG or Chrome and Startpage for work.
Anything where Chrome and/or Google are avoided is a good thing, IMO.
In that case you can probably at least use Chrome and Startpage at work. Startpage.com pays Google for the right to use their search results, so you'll still find your pages about obscure error messages, and Startpage doesn't track you.
I have Chrome for one purpose, and one purpose only: vSphere installations that have yet to be upgraded to include the HTML5 version. Chrome is the "Flash browser".
Over the last few years DuckDuckGo have become so good at handling my queries that I only occasionally use Google. That typically happens when DuckDuckGo doesn't find what I expect, but it always turns out that neither does Google.
wow, this is totally different than my experience. i'm a privacy advocate who defaults to DDG, but I find myself forced to use Google for many technical queries because DDG rarely handles anything except obviously answered searches. side by side comparisons with Google have been devastating, so i switch back and forth depending on technical workload and privacy-interestedness
My impression Google helps DDG to become a popular alternative to its own search engine. Why? Just funny fact: In 2018 Google transferred ownership of the domain name Duck.com to DuckDuckGo.
Look, you know that Google in an act of benevolence gave them duck.com last year, that’s PR.
Antitrust angle is obvious. They want to appear they aren’t the only game in town. Esp when you have people like Warren making (hollow) antitrust campaign noise.
With some of the huge anti-trust fines levied against Google by the EU, this seems to me like Google trying to support that they are not a monopoly in search.
Weinberg explained the beginnings of the name with respect to the children's game duck, duck, goose. He said of the origin of the name: "Really it just popped in my head one day and I just liked it. It is certainly influenced/derived from duck duck goose, but other than that there is no relation, e.g., a metaphor."
Nice try G. I still won't have a byte of your code on anything I use. No data for you. I wish websites stop using your fonts and analytics and captcha too.
> DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from a variety of partners, including Oath (formerly Yahoo) and Bing.
How does that work with their privacy stance? Do Yahoo/Bing get to keep and use that search data and it's just anonymized, or does DDG pay to keep it untracked?
Kind of disheartening regardless. I assumed they had their own scrappy, independent tech stack.
Headless Chrome is available and widely-used, and commonly you can get around the JS thing by simply waiting a few seconds before scraping. I'd assume the crawling itself isn't the hard part (aside from maybe just the raw compute time it takes).
For the most part yes. They could be getting search results from other paid search engine APIs but you have to balance cost of providing results with ad/affiliate revenue.
You can get paid API search results from Google and Yandex for example just like with Bing (similar prices, different limits). And you can even use Wolfram Alpha API for certain types of queries ("what is apple's average revenue per employee?").
Doing all these would allow you to surface better results than using just one. But it comes at a cost.
Google and Bing are the only ones that matter and you can’t compete with Google by paying them for their search results.
Yandex is Russian and has pretty poor results. And you’d be naive to think that those of us concerned about privacy would ever touch something built on top of it.
In other words there aren’t paid search engines that DuckDuckGo could turn to. Unless they build their own crawler, the only game in town is Bing.
There is no difference in terms of building something on top of Bing or Yandex as your private data never touches their servers. All they get is anonymized stream of queries, in this case from DuckDuckGo.
And yes you can compete with Google (for a certain target group) by paying them for their search results. Results are just a distribution channel, it is what you do with them that matters. For example Google and DuckDuckGo both choose to show you ads and affiliate links but that is hardly the only option.
They're desperate to collect user click data because they know that's the only way they'll have any chance of success. Even anonymized, that's very valuable data.
Disclaimer: @yegg, if you're reading this, I'm posting this rant with love.
I am so disappointed with DDG recently, it has adopted Google's strategy of returning searches that have nothing to do with your query if not enough results were found [0], and dialed it up to 11. If "I" "don't" "put" "each" "word" "in" "quotes," the results I get have nothing to do with my search... but if I do that (apart from the inconvenience of it all) it means (presumably?) that stemming isn't done on the search terms.
Maybe I'm old school, but I expect search results to match the search terms. Fuzzy matching (stemming, synonyms) is an added bonus, but silently dropping words which don't appear is decidedly not. Moreover, a search result returning "only" two results should be taken as a good thing for someone with confidence in their dataset (DDG naturally doesn't have that, because their coverage is far from 100% of the web) - it means the search terms were extremely precise and the results are highly relevant, with irrelevant results filtered out. Decreasing the signal-to-noise ratio by willfully ignoring my search terms may increase the quantity of search results but - and I don't know about you - for me I don't care about quantity and would choose relevance as the more appropriate metric to benchmark against.
(All that said, I still use DDG as my main search engine even if I am turning to appending !g far more than I ever used to because I firmly prefer DDG's respect for my privacy and person over Google's treatment of the same. But I'm disgruntled and, frankly, very disappointed. Sorry, @yegg!)
Edit: actually the situation is even worse. DDG doesn't seem to even always respect "quoted" terms. Here's literally the first search I did after posting this [1]. The quoted term "CFF2" doesn't even appear in the majority of the results DDG pulls in - not just not in the page summary displayed, but literally not on the result page at all. For comparison, here's the Google equivalent:
>Moreover, a search result returning "only" two results should be taken as a good thing for someone with confidence in their dataset
I completely agree with you here but in my experience it's not anything new with DDG, that's always been a problem as far as I'm concerned.
As a hobby I sometimes have to reverse engineer electronic circuits, when I'm not sure what a chip does I try to search the inscriptions on the package to see if I can find a datasheet online. Sometimes you end up with very cryptic strings like "xardc10-egh" or whatever. If you input this string on Google it gives you no results:
That being said DDG improved slightly, when I did searches like those a couple of years ago I'd often end up with results containing completely broken encodings, binary dumps as ascii and other obviously erroneous content that got indexed by mistake. Here the results at least appear to link towards proper pages.
Seriously: Where do you get the idea that DuckDuckGo is just Bing? I can't find a single source for that claim. What I can find is a post from Gabriel Weinberg that says that DuckDuckGo is not Bing.
It comes up VERY times DuckDuckGo is mentioned, yet there's not a single source that suggests that DuckDuckGo is just a frontend for Bing.
So again, please provide a source! I can't find one.
It's clear that DuckDuckGo used Bing for some result, but not to what extend. Are all result Bing? Does Bing only provide results when DuckDuckGos own crawler fails? Are the results mixed? I very much get the impression that results are mixed, but that's not completely clear either.
I've seen the ddg bot in my home webserver (with a .com) logs in the past month. I even bothered to check to make sure it's IP matched the ones on the bot about page.
Their own crawler is only used for fluff like widgets. All organic search results are from Bing and Oath:
> In fact, DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from a variety of partners, including Oath (formerly Yahoo) and Bing.
One argument to be made is that Google Search can only go downwards from here, as it is currently a clear market leader, and the remaining segments are not easy for them to break into.
For example, Baidu has a stranglehold on search in China, and that's not likely to change drastically, with Google facing internal opposition to entering China.
This is true, but Google is also adding more search surfaces (e.g., google home, assistant etc.). So, it's possible that they might attract proportionately more users using these surfaces.
I don’t have any Google home devices, but I’d be curious to know how many of those queries are simply commands “Google play my music” and how many are actual internet searches “Google what is the capital of Alabama”.
I think you're misinterpreting the chart you quoted.
Google search growth rate is always positive in that page. It just decelerated. Growth rate being negative means you're actively losing more users than you gain.
Also, in 2000, the entire Internet had about 10% the number of users it has today (~300M vs around 4B). Still a very impressive number, but the comparison isn't very meaningful.
How important to Google is their web search product?
I know it was their first product, but I would imagine they get much of their revenue from other avenues, such as Android's built-in totally-not-antitrust web search app, and YouTube and Gmail and web ads...
I haven’t looked at their annual report recently, but back in 2016, advertisement made up a majority of their revenues and profits — around 90% if I recall correctly. I’d be willing to bet that keyword advertisements on search make up a larger portion of that traffic than that through YouTube videos.
Web Search is still Google's unicorn but it is not as much profitable as few years ago mostly because there now better advertising channels like social nets and online videos. Many people today also use price comparison apps instead of web search.
I started to use Chrome only for Google services (gmail, youtube, maps, etc) and Firefox with DDG for everything else. With this setup Goggle can send home only the data they already know.
As someone who supports Firefox, I would say that it’s important to signal Google that there are Firefox users using its services. People have been reporting about issues with some of Google’s services on Firefox. Skype from Microsoft was recently discovered as not supporting Firefox. Every signal users send to these companies matters.
If you don't send back crash reports and statistics to a company, you shouldn't get upset when they stop supporting your usecase or won't fix bugs you encounter.
312 comments
[ 2.8 ms ] story [ 281 ms ] threadIt's also a solid choice for them to hedge against antitrust claims, if they can point to having just added them to their browser, regardless of the fact that Google is the default and they do not present a choice screen like Microsoft had to in the EU.
There's a lot to unpack in that statement... Is there any recent analysis on the usage stats that chrome is reporting back that someone could point to?
Isn't it well known that Google scoops up web history from the browser or have they stopped doing/never done this? In the latter case any pointers would be appreciated.
A default open browser history synced across devices seems like exactly the sort of thing that would show that DDG has increased its market share.
Source: I work in education - even in a highly educated area in a developed EU country, young and old alike think like this.
Not only is Google's indexing infrastructure not that fast, but they deliberately don't do that because some poorly designed sites have passwords or unique keys in the URL that should not be used to retrieve content for the public search index.
https://www.google.com/chrome/privacy/whitepaper.html#usages...
https://developers.google.com/web/tools/chrome-user-experien...
Good. 4 is a good number. It's on the low end of the number range people think of as "enough choice."
At its simplest, in the West we have a thing for threes. Three bits of God, three little pigs, three branches of government (in the USA at least), "things come in threes", three books/movies in a series (a trilogy), stories that have a "beginning/middle/end".
Bottom line, the West tends toward organizing and thinking of things in threes. Some might even be superstitious about threes (perhaps a Pythagorean influence).
In China the number 4 plays a similar role. I don't know much about 'numerology' in China, save to say that recently the number 4 (which apparently sounds like 'death' in Chinese) has been considered bad luck. Here's a better explanation than I could give: https://www.quora.com/In-Chinese-culture-why-is-the-number-4...
To add a complication, Chinese tends to use the US convention (with the ground floor being floor 1), while the English convention is the British one (with the ground floor being floor 0).
By happy coincidence, then, the 13th floor is also 十四樓, ie the 14th floor, so you only need to skip one floor, rather than two. That explains why HK skyscrapers are so high.
Say, frozen chicken, napkins, instant noodles, paper cups, etc. In some cases there is only 1 option offered, sometimes 2, rarely are there ever more than 4 options offered at Costco for a single type of item. When you trust that you are being offered the best choice or a top choice, well, we know what happens at Costco. People buy pallets in that warehouse.
Effectively, Costco shoppers are people who already have chosen, "the cheapest fairly good quality option."
Costco is the "I'm feeling lucky" button.
What percentage of Chrome users consented to the data collection? (Is consent even required?)
Does the data represent all Chrome users or only those who have consented?
They have slimmed it down to only a few pages and now have very simplified statements.
Obviously every statement is now very carefully worded...
Unwillingly consented, that's the vast majority of Chrome Sync users. Unless you enable the end-to-end-encryption (for which they require a second passphrase, so probably less than 0.1% actually use that), they will use your data for ad profiling etc.. Yes, that is on page 1312 of the Chrome Sync privacy statement. (They're only required to write it into there, if they do it, so it is quite certain that they didn't just want the bad PR for nothing.)
Is consent required? Assuming they actually do collect this data from their Chrome Sync data or through similar personally identifiable ways, consent would be required in many jurisdictions, especially the EU.
However, if they cared enough, it would be possible for them to collect this particular data point without personal identification. You could for example create a UUID per installation that's only associated with this one data point. Or you could have a time-based solution where each Chrome instance goes out to "vote" for their default search engine e.g. every 4 weeks. If you then look at the statistics on a weekly basis, you can just take these values times 4 to even roughly correct numbers. It's certainly going to be representative enough, you don't need every browser instance to have their vote in every week's statistic.
These metrics are from UMA stats. They are collected from everyone who ticks the box to report stats when installing Chrome.
They only get histograms of counts of visits to search engines, not the entire URL, and not search engines or other sites not in the list of things they track (which is at the bottom of the file).
chrome://chrome-urls/site-engagement
and other chrome-urls
These can provide useful data for me but not sure why I would want send the data to Google.
One way to make the pages I visit load faster is to disable Javascript. Another is to remove (or block) advertising. Another is to put DNS data for these sites into local hosts or zone files.
Those actions are how I prefer to approach the problem.
However as far as I can tell, those are not actions Google wants to take. They have their own preferred approach.
It is possible there are users who are aligned with Google in terms of how they want to approach the problems created by misuse/overuse of Javascript and advertising.
It is also possible there are some users who have no idea why pages are slow to load.
Those groups might want to send usage data to Google.
However I am not in either group. I dislike the web advertising business that Google depends on and therefore must nourish and support.
As such, there is no reason I can think of why I would want to send data to Google.
Also, I have not checked but I wonder if Google is restricted in how they can use the collected diagnostic data. Are they prohibited from using it for the purposes of selling advertising?
So if Chrome's ever made a UI change you disagreed with, then you're in a group that would have benefitted from sending Google usage data.
In terms of the restrictions on usage data, see https://www.google.com/chrome/privacy/whitepaper.html#usages... .
I care about command line programs, less-interactive and non-interactive use. Truly, the best interface is no interface.
The whitepaper.html appears to explain how usage data is utilised in ways that help Chrome improve but does not appear to contain any restrictions on use of the data to help further Google's ad sales business, whether directly or indirectly.
It is the business model that I do not wish to support.
Producing software such as Chrome is just something the company is doing in the course of selling advertising and collecting maximal amounts of data from users, whether the data is anonymised or not.
Reminds me of the sort of advantage Facebook had from its VPN app to identify competitors early to kill/acquire them.
Also quite interesting that they changed the default search engine for China to be Baidu.
It seems to be a logical move since I believe Google is blocked in China.
https://en.wikipedia.org/wiki/Google_China
[1] Because in the world of the press everything should be announced so they can broadcast it and sell advertising by running stories. And not have to find it out by other more laborious methods.
This is a deliberate Orwell reference? Vernor Vinge speculated in Rainbows End that everything which couldn't be searched for in a search engine would effectively become invisible. In 2019, that manifests as, "everything which can't be searched for in a search engine, which is backed up by crosslinked mainstream news sites and which isn't warded by words meant to scare casual readers away."
Would you think of Google as trustworthy because they only gave their backend two pieces of data? I myself would not, because I'm pretty sure the actual request and response messages are looked up by client ID (in their Google Analytics data store).
Chrome (and Chromium) creates at start and maintains SSL connections to Google. It is not easy to sniff what is being sent. Even if you MITM it, like in enterprise transparent proxies, Chrome will throw an error because of cert pinning. google domains should be whitelisted: "we recommend that you avoid the use of transparent proxies." https://support.google.com/chrome/a/answer/3504942?hl=en
But I doubt the Chrome extension that would do that patching could stay at Chrome store.
Yes the data. The whole request and the whole response. They need it in order to properly train their ad-network.
</speculation>
Given that they scoop up all this data I'd appreciate if their ad-network actually improved. Just the other day the dating site scams where back.
"We'll try not to show it again" they say. Well for vacuuming the market for the best and brightest they either don't try very hard or they are very dysfunctional because they fail as a group.
Everybody working for this company should be ashamed.
And anyway, it's “to improve your experience”, so you can't opt out. It's for your own good. Remember, Big Brother loves you.
Because Chrome keeps a log of all your activity, your DDG searches are easy to find here: https://myactivity.google.com/myactivity
So if you're web based (like me) then activities such as sending an email, checking out YT, reading HN, watching Twitch, and jerking off, all end up as entries in that log file.
Do you really think Google would have trained an AI to determine that last activity? How would they have trained the AI?
Sloppy. How do they know I'm not preparing for actual intercourse? How do they know I'm not downloading porn for later? I could also be watching gun videos, since they've been hosted there.
Granted, it's probably 95% accurate.
I think the causal order is flipped around here.
Yes!
>> How
How graphic do you want me to be? But isn't the real question: is there utility for an ad network to know about your preferences in porn? If there is such utility, you're best to believe Google implemented a way to get them.
If I was an ad network I would love to hear about your porn habits. I would absolutely love it.
If you know, you are obviously one of the people who has that data. If you don't know, you aren't.
If I was an ad network I would love to hear about your porn habits. I would absolutely love it.
Such networks don't have to know whether or not and the exact moment a given user jerks it. Though, it would actually be better if you were actively browsing around and not jerking it currently. I guess that's why discovery and AI are so bad on porn sites. It's actually better for their ad revenue!
Edit: And presumably you're using Incognito for... some of those activities, which wouldn't be captured regardless.
They would be captured by the ISP.
Did you know that Google happens to partner up with a lot of ISPs? Hmm, I wonder what for. What could they possibly have that Google needs?
;)
Ever heard of https?
https://www.upturn.org/reports/2016/what-isps-can-see/
EDIT: spelling
* Enabling Chrome Sync, which is opt-in
* Syncing history, which is on by default if you enable sync
* Not using a custom passphrase for sync data (not using one is the default)
* Having "My Activity" save "Web & App activity", which is opt-out
* Having synced Chrome history data sent to "Web & App activity", which is opt-out
For the last two bullets, the opt-outs are at https://myaccount.google.com/activitycontrols .
EDIT: Added italicized text for clarity.
If you're thinking of Google Safe Browsing (used by both Chrome and Firefox), you're wrong.
It works the other way around: Google sends you the list of undesired domains, and your client prevents you from visiting domains found on that list.
Nothing needs to be shared with a third party for that functionality.
A hash prefix list gets downloaded locally; Chrome checks locally against the prefix list. If a URL hits, Chrome will send the hash prefix (not the full hash and not the URL) to the server, the server will send back all full hashes that match that prefix, and then the client will complete the check locally.
In theory, if the server had a small number of matching full hashes, it could guess about what URL a client might be hitting, but in practice the system is designed as much as possible to avoid ever leaking data about what you're visiting to Google servers.
Clients download a database of partial hashes of malware URLs. If they get a hit on one of those partial hashes, they make a request for the full list of hashes with that prefix.
Google knows when a client makes one of those requests, but the exact URLs (or hashes) they're looking up are never revealed. The partial hash is 32 bits long, so there's enough collisions that making a request isn't especially revealing.
https://developers.google.com/safe-browsing/v4/update-api
For Safe Browsing protection, here's how it works (in progress): https://chromium.googlesource.com/chromium/src/+/refs/change...
[Disclosure: I'm the Software Engineer on Chrome who wrote parts of this Safe Browsing code, and that incomplete documentation linked above.]
I might be wrong, but this is obe of the reasons I don't use Chrome so if anyone has links that proves something else I'm interested.
[1]: https://en.m.wikipedia.org/wiki/Safari_(web_browser)
(I know ... for business reasons ... but isn't Chrome open source? How is this in practice prevented?)
I guess the only difference is that with this change, DDG is available as a search engine by default with a blank install, even before you've actually used it.
https://en.wikipedia.org/wiki/OpenSearch
You go to Settings > Manage search engines > Add or pick one of the auto discovered ones. This also works in the mobile versions of Chrome.
You can find more instructions here: https://support.google.com/chrome/answer/95426
"Duckduckgo is one of our main rivals." Is a bit of a self fulfilling prophecy for Google. They need to amp up DDG's legitimacy to ward off accusations of antitrust. Credibility, legitimacy and awareness are really the only things DDG needs to reach a wider audience and gain greater adoption.
[1] https:/duckduckgo.com/traffic
Nothing new to see here.
It's quite the ecosystem.
1 - I use Chrome + Google for work stuff 2 - I use Firefox + DuckDuckGo for personal stuff
I sync my passwords with Bitwarden.
I sync my passwords with Bitwarden.
I have started doing this as well, except I'm using KeePassXC and using Dropbox to distribute the file everywhere. Would Bitwarden work behind a company firewall at a company that doesn't allow Dropbox?
Bitwarden is pretty nice, using it at work. I do still like my control that KeePass gives, though.
Also, I currently use 1Password, but have been thinking about using Enpass[2] because you can sync with any cloud drive. I like the idea of syncing to a third party cloud drive in case my password service is compromised.
[1]: https://brave.com [2]: https://www.enpass.io/
I used Brave for awhile then switched to Firefox+uBlock Origin, hoping to do my teensy part in decreasing the market share of Chromium-based browsers while still being privacy-focused.
Often vulnerabilities go unpatched for days, which is pretty bad when the exact vulnerability exploit code is already visible in the chromium bug tracker!
Also slowly migrating to Bitwarden.
Or you could even use Chrome and DDG or Chrome and Startpage for work.
Anything where Chrome and/or Google are avoided is a good thing, IMO.
Over the last few years DuckDuckGo have become so good at handling my queries that I only occasionally use Google. That typically happens when DuckDuckGo doesn't find what I expect, but it always turns out that neither does Google.
Look, you know that Google in an act of benevolence gave them duck.com last year, that’s PR.
Antitrust angle is obvious. They want to appear they aren’t the only game in town. Esp when you have people like Warren making (hollow) antitrust campaign noise.
Microsoft did this with Apple.
https://duck.co/help/company/name
Maybe the creator really enjoyed that game as kid!
Weinberg explained the beginnings of the name with respect to the children's game duck, duck, goose. He said of the origin of the name: "Really it just popped in my head one day and I just liked it. It is certainly influenced/derived from duck duck goose, but other than that there is no relation, e.g., a metaphor."
> Another pro-privacy search rivals, French search engine Qwant, has also been added as a new option — though only in its home market, France.
> DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from a variety of partners, including Oath (formerly Yahoo) and Bing.
https://duck.co/help/results/sources
AFAIK Oath / Yahoo has switched to using Bing under the hood since 2009: https://www.nytimes.com/2009/07/30/technology/companies/30so...
Kind of disheartening regardless. I assumed they had their own scrappy, independent tech stack.
For one thing, loads of sites load all their content via Ajax, so at a minimum you're gonna need a browser engine as the base of your crawler...
Doing all these would allow you to surface better results than using just one. But it comes at a cost.
Yandex is Russian and has pretty poor results. And you’d be naive to think that those of us concerned about privacy would ever touch something built on top of it.
In other words there aren’t paid search engines that DuckDuckGo could turn to. Unless they build their own crawler, the only game in town is Bing.
And yes you can compete with Google (for a certain target group) by paying them for their search results. Results are just a distribution channel, it is what you do with them that matters. For example Google and DuckDuckGo both choose to show you ads and affiliate links but that is hardly the only option.
Sorry, but I don't trust DuckDuckGo that much and when technology fails, I'd rather use a search engine based in a country that has rule of law ;-)
They're desperate to collect user click data because they know that's the only way they'll have any chance of success. Even anonymized, that's very valuable data.
I am so disappointed with DDG recently, it has adopted Google's strategy of returning searches that have nothing to do with your query if not enough results were found [0], and dialed it up to 11. If "I" "don't" "put" "each" "word" "in" "quotes," the results I get have nothing to do with my search... but if I do that (apart from the inconvenience of it all) it means (presumably?) that stemming isn't done on the search terms.
Maybe I'm old school, but I expect search results to match the search terms. Fuzzy matching (stemming, synonyms) is an added bonus, but silently dropping words which don't appear is decidedly not. Moreover, a search result returning "only" two results should be taken as a good thing for someone with confidence in their dataset (DDG naturally doesn't have that, because their coverage is far from 100% of the web) - it means the search terms were extremely precise and the results are highly relevant, with irrelevant results filtered out. Decreasing the signal-to-noise ratio by willfully ignoring my search terms may increase the quantity of search results but - and I don't know about you - for me I don't care about quantity and would choose relevance as the more appropriate metric to benchmark against.
(All that said, I still use DDG as my main search engine even if I am turning to appending !g far more than I ever used to because I firmly prefer DDG's respect for my privacy and person over Google's treatment of the same. But I'm disgruntled and, frankly, very disappointed. Sorry, @yegg!)
[0]: https://neosmart.net/blog/2016/on-the-growing-intentional-us...
Edit: actually the situation is even worse. DDG doesn't seem to even always respect "quoted" terms. Here's literally the first search I did after posting this [1]. The quoted term "CFF2" doesn't even appear in the majority of the results DDG pulls in - not just not in the page summary displayed, but literally not on the result page at all. For comparison, here's the Google equivalent:
[1]: https://duckduckgo.com/?q=windows+10+%22cff2%22&t=ffab&ia=we... [2]: https://www.google.com/search?hl=en&q=windows%2010%20%22cff2...
I completely agree with you here but in my experience it's not anything new with DDG, that's always been a problem as far as I'm concerned.
As a hobby I sometimes have to reverse engineer electronic circuits, when I'm not sure what a chip does I try to search the inscriptions on the package to see if I can find a datasheet online. Sometimes you end up with very cryptic strings like "xardc10-egh" or whatever. If you input this string on Google it gives you no results:
https://www.google.com/search?hl=en&q=xardc10%2Degh
If I do it on DDG I get pages of irrelevant results:
https://duckduckgo.com/?q=xardc10-egh&t=ffab&ia=web
That being said DDG improved slightly, when I did searches like those a couple of years ago I'd often end up with results containing completely broken encodings, binary dumps as ascii and other obviously erroneous content that got indexed by mistake. Here the results at least appear to link towards proper pages.
It comes up VERY times DuckDuckGo is mentioned, yet there's not a single source that suggests that DuckDuckGo is just a frontend for Bing.
It's clear that DuckDuckGo used Bing for some result, but not to what extend. Are all result Bing? Does Bing only provide results when DuckDuckGos own crawler fails? Are the results mixed? I very much get the impression that results are mixed, but that's not completely clear either.
I should check server logs at work, I don't run my own web servers, so I can't claim to have seen it.
> In fact, DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from a variety of partners, including Oath (formerly Yahoo) and Bing.
Why should I care about where DDG gets its search results from? They promise a good web search and don't deliver.
* DDG in 2018 has served similar amount of search queries as Google in 2000.
* DDG growth rate is accelerating
* Google search growth rate is negative
* Google's share of global search is shrinking
DDG stats: https://duckduckgo.com/traffic
Google stats: http://www.internetlivestats.com/google-search-statistics/
Google is at 93% so not a ton more share to take.
http://gs.statcounter.com/search-engine-market-share
Google search growth rate is always positive in that page. It just decelerated. Growth rate being negative means you're actively losing more users than you gain.
I know it was their first product, but I would imagine they get much of their revenue from other avenues, such as Android's built-in totally-not-antitrust web search app, and YouTube and Gmail and web ads...
just like Apple lets Pandora on Apple Watch and not Spotify.