> On top of those privacy issues, User-Agent sniffing is an abundant source of compatibility issues, in particular for minority browsers, resulting in browsers lying about themselves (generally or to specific sites), and sites (including Google properties) being broken in some browsers for no good reason.
> The above abuse makes it desirable to freeze the UA string and replace it with a better mechanism.
UA sniffing should have died out a long time ago. It's frustrating that it's 2020 and I'm still having my browsing experience broken for no other reason than the site doesn't like my UA.
It is a subjective version of “too long not fixed”. Like gp remembers this exact problem in 1980/1990/2000 and still there it is. A specific year has nothing to do with it.
So come on, it’s 2020 and we still have to explain idioms to each other :)
User agent sniffing is still sometimes required. Take the SameSite bugs that Safari had. On the server you needed to act on that based on User Agent sniffing.
I also remember a CSS bug Safari 4 or 5 had with tables and touch scrollbars that we had to resort to UA detection for - it's not always possible to feature detect bugs.
Agreed. For media streaming UA sniffing is required because of quirks each browser has in decoding media. Safari has a few quirks about the MP4 packaging it likes via MediaSourceExtensions. Firefox's network stack includes a delay not related to the actual request because of its implementation[0] which makes timing of small downloads very inaccurate. None of these differences are discernible except by UA.
[1] I believe FF includes in the download length the time a request spends sitting in the request queue, but I can't exactly remember.
I think it's distinctly possible that the current status quo may cause an overreaction. Feature detection is good. But we may also need the ability to do specific version checking sometimes.
What I think would be best is two things: A feature advertising string (preferably with some versioning in itself), and a new User Agent string that is just that: The user agent. As in, "This is Firefox 72.2.813". Maybe the OS in there too. I think a non-trivial part of the problem with the UA today is "Hey... I'm Firefox. Also, I'm Mozilla. Also, I'm edge 3, and Internet Explorer 6, and today I think I'm also Chrome. Also, I may be a monkey with a banana plugged in to an ethernet cable." Part of the problem is that people abuse the user agent, but part of the problem is that the user agent string is also a pack of outright, outrageous, self-contradictory lies. A truthful field may still have significant utility.
One of the things I've learned over the years in software engineering is that you should do you best to make sure that your system doesn't contain lies, or, failing that, treat them like any other issues that you should contain at the edges rather than letting it run riot through your system. For instance, if you decide to let customers have 10GB of bandwidth free before you start charging them, you should never accomplish that goal by tweaking the bandwidth counting system to report 7.5GB of usage as 0GB. That is a lie. Tell the truth about the usage and let the billing system apply the discount, which is itself a truth as well. You always pay for lies in the system in the long run. It is a very common pattern in coding as well where some function author figures out how to assemble the correct lie to some other function to make it do what the author things they want, but in the long run you're better off telling the truth and fixing the code to work with the truth. Otherwise, well, you really do end up with a tangled web of code. Which is... exactly what we've gotten with UA-detection-based code. We won't necessarily get the same mess if there's a new UA that isn't a lie.
I'm not claiming this will lead to utopia. Minority forks of popular browsers will fail to pick up the necessary ad-hoc bug fixes this way, for instance. My claim is more like in the long run, the best thing for everybody is just to have a user agent that tells the truth, even if on occasion in the short term you experience occasional problems.
(One way to at least partially achieve this is to standardize the new truthful agent string to something like (\w+) ((\d+.)+\d) (\w+), and specify it as "non-conforming true user agent strings MUST be entirely rejected and treated as being absent", so anyone who tries to be Firefox and Chrome and also a monkey ends up being nobody in particular.)
But that's what this proposal is about. Instead of using the user agent string (which lies and needs to be parsed) you have some well specified variables to check.
Also I think you're missing why the user agent lies now. It wasn't done on a lark. It's because it's been abused so much by web devs. Both by sloppy regular expressions and deliberately. For example, even big names like Microsoft and Google have at times used it to unnecessarily deliver poorer browsing experiences to people not using their browser.
Sure browsers could have stuck to their guns and been "honest" about their UA. But honesty is no comfort to users of the browser when they find websites break for no reason. The average user is more likely to blame the browser than the website.
I am completely aware of why the user agent lies now. Didn't I explain exactly how it is broken in my post? I was basically there for when it broke; I remember when IE came out claiming to be "Mozilla" because otherwise a surprising number of sites wouldn't serve them the latest whizbang Netscape 3 HTML. (I thought it was a bad idea then, but with much less understanding of why.) This is why I kept calling what I'm asking for a new field; the User Agent itself can't be rehabilitated.
The parent of my post is correct; in practice we're still going to need the occasional ability to shim in browser-specific fixes, because even if the browsers do their best, they're going to inadvertently lie in the future and claim to support WebVR1.0 in Firefox 92, but, whoops, actually it crashes the entire browser if you try to do anything serious in it. Or, whoops, Firefox 92 does do a pretty decent job of WebVR1.0 but I need some attribute they overlooked. Or any number of similar little fixups. We know from experience from the field in the real world that we're talking about crashing bugs here at times; this is real thing that has happened. Whatever proposal gets implemented should deal with this case too.
If we standardized on the format like I suggested at the end of my post, it would go a long ways towards preventing future browsers from mucking up the field. If you just get "$BROWSER $VERSION $OS" in a rigid specification, and if the major browsers are sure to conform to that, and the major frameworks enforce it, it'll be enough to prevent it from becoming a problem in the future. It won't stop Joe Bob's Bait Shack & Cloud Services from giving their client a custom browser and/or server that abuses it, but there's no stopping them from doing things like that no matter what you do, so shrug.
Then I'm not sure I understand you. The proposal is clearly proposing new fields that are less susceptible to abuse (whether intentional or not). Your idea of parsing a "$BROWSER $VERSION $OS" string seems inferior to client hints that use structured headers.
I'm saying we still need a browser version field, in addition to a feature field. Features would do on their own, if they were perfect, but we shouldn't plan on them always being perfect. We have a demonstrated, in-the-field history of browsers claiming to support features when in fact they don't quite support them, and can even have crashing bugs. In the real world, supporting WebVR1.0 is more than just putting "web-vr/1.0.0" in the feature string.
Culturally, you should prefer to use feature detection. Most developers would never need to use anything else. But when Amazon makes its new whizbang WebVR1.0 front-end in 2024, they may need the ability to blacklist a particular browser. Lacking that ability may actually prevent them from being able to ship, if shipping will result in some non-trivial fraction of the browsers claiming "web-vr/1.0.0" will in fact crash, and they have nothing they can do about it.
Besides... they will find a way to blacklist the browser. Honestly "prevent anyone from ever knowing what version of the browser is accessing your site" is not something you can accomplish. If you don't give them some type of user agent in the header, it doesn't mean the Amazon engineers are just going to throw their hands up and fail to ship. They will do something even more inadvisable than user agent sniffing, because you "cleverly" backed them into a corner. If necessary, they will examine the order of headers, details of the TLS negotiation, all sorts of things. See "server fingerprinting" in the security area. You can't really stop it. Might as well just give it to them as a header. But this time, a clean, specified, strict one based on decades of experience, instead of the bashed-together mess that is User-Agent.
Or, to put it really shortly, the fact that a bashed-together User-Agent header has been a disaster is not sufficient proof that the entire idea of sending a User Agent is fundamentally flawed. You can't separate from the current facts whether the problem is that User Agent sniffing is always 100% guarantee totally black&white no shades of grey mega-bad, or if it's the bashed-together nature of the field that is the problem.
Working around browser bugs only helps to ensure that those bugs will never get fixed. Better that things like the UA are removed to make that impossible. In that world, sites would be best served be coding to the standard and putting the work of fixing incompatibilities/bugs back onto the browser makers where it belongs.
Of course, when the User Agent sniffing messes up, you're perfectly free to change it, and see what else you can get the server to send you. Can you still do that with whatever crazy "feature detection" replaces it?
Thanks, I assumed so but it's useful to have this confirmation before I looked it up. Honestly this is a little surprising to me. Sites pulling shit like matching UA usually works out in Chrome's favour so it's nice to see them throwing the minor web browsers an olive branch.
Developers will still mostly target the most popular browser(s), particularly Chrome. It would be great if browsers like Firefox managed to implement a "feature detection spoof" that you can enable to still present yourself as Chrome (or other browsers) regardless of the actual features requested. More or less like changing the UA does now.
This looks like it's intended to fix specific issues, not for fingerprint blocking. I was thinking more like being able to present yourself as generic Chrome on Windows 10 or Firefox on Android if you chose to, even with the risk of breaking the site.
Firefox resists fingerprinting by default, and go faster addon provides mitigations for sites that break in this mode, like providing a specific UA string.
> And finally, starting fresh will enable us to drop a lot of the legacy baggage that the UA string carries (“Mozilla/5.0”, “like Gecko”, “like KHTML”, etc) going forward.
Why not drop just that and leave nothing but the exact HTML and JavaScript engine names and versions like "Blink a.b, V8 x.y"?
And what about robots? Will the Googlebot give the UA up as well?
There are plenty of unmaintained sites out there that do stupid things in reaction to UA strings, and one of the stupid things is using a regex that expects specific strings to exist.
Using a completely new string format in the same field (or removing it entirely) breaks a lot of sites that'll never be fixed.
Freezing it prevents this. And if we're freezing and creating a new system then why not go for something queryable without all the baggage?
Users still want to see those pages. If browser X removes the UA string, that site brakes for them, and they change to browser Y that still has the old useragent.
I tried using no UA header at all for a period of a few weeks, many years ago when "appsites" weren't as common, and yet a lot of sites failed to load mysteriously, showed odd server errors, or even banned my IP for being a bot.
I expect no UA header to be even less usable now that sites are more paranoid and app-ified, so instead I use a random one. That still confuses some sites...
Maybe after people stop using the user agent for stupid reasons it can be made useful again. User agents should be stuff like "Firefox 71.0" or "Chrome 79.0.3945", not "Mozilla but really AppleWebKit disguised as KHTML, but no really I'm Chrome but maybe I'm Safari"
My user agent is "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15". Clearly I'm running a repackaged Firefox browser?
It is still possible to determine what browser caused an exception. The "sec-ch-ua" header is used by Chrome (but only to HTTPS websites) to send the "real" user agent, which for me is "Google Chrome 79" (and I am indeed running Google Chrome 79).
There are plenty of legitimate purposes. I use it to disable keepalive for POST requests, redirect to https:// for browsers known to be able to establish a secure connection, whitelist tor browser bundle, detect behavior of Accept: header, which differs even between browser versions, and some other things.
The trigger for this was most likely the Brave/Vivaldi browsers. Brave used to 'Brave/X.y' at the end of user agent. Whatsapp didn't work with that useragent. Now Brave uses Google Chrome's useragent.
In my app Dictanote (https://dictanote.co) - which uses Chrome's speech-to-text API, I have no way to distinguish Brave/Vivaldi and user doesn't understand why its not working :/
Using user agents to detect features is the wrong way to do it because you’ll have these exact problems.
To determine if a browser has the text to speech API just check if the webkitSpeechRecognition object exists (if that’s the one you’re using). It will exist in Chrome and will not exist in other browsers that lack the feature.
Shouldn't you be using feature detection[1] here anyway? Making browsers unsupported in a blanket fashion via user-agent is the exact thing people should stop doing.
The webkitSpeechRecognition object shows up on both browsers. When you start recognition, it acts like you are not connected to the network. Not connected to the network is a common error , so you cannot fingerprint it to Brave.
This specific API only works in Google Chrome unfortunately. So we need to stop people from trying in another browser and getting frustrated as to why its not working.
We should of course default to feature detection whenever possible, but non-standard behavior like this in certain browsers is exactly why feature detection alone is never going to cover 100% of current use cases for UA detection.
To add to OP's point, this thing in Safari also comes to mind as an example of something that isn't easy to detect and address outside of UA detection: https://github.com/vitr/safari-cookie-in-iframe
Deprecating UA Strings and moving towards UA Client Hints seems like a move in the right direction though.
Some pages are intended to be embedded in iframes (which have their own security context isolated from the embedding page), and happen to use cookies for authentication. Safari not allowing cookies from third party domains in iframes is the issue here.
Assuming these iframes will never work means degrading the experience for all users, when only users on Safari are actually affected. Detecting the UA and branching based on that is a much more pragmatic solution.
Absolutely they should. But they haven't. Checking the UA string allows a developer to get around the issue, but once it's gone an individual developer will be powerless to fix the issue for their users.
Just test whether the API works or whether the properties are missing or methods throw exceptions?
The kind of testing you are complaining you can't do is exactly why user agent is broken, what if you test for Chrome and then Brave/Vivaldi start supporting the speech-to-text API and your broken website still says "sorry, you need Chrome" for no reason?
The webkitSpeechRecognition object shows up on both browsers. When you start recognition, it acts like you are not connected to the network. Not connected to the network is a common error , so you cannot fingerprint it to Brave.
"Somethings wrong! You might be using Chrome and having connectivity issues, or you might be using Brave who have a dispute with Google about usage of their speech recognition API. In the former case, get better WiFi, and in the latter case there's nothing you can do, switch browsers."
While the intent is good, didn't we recently see from the roll-out of secure/strict same-site cookies that feature detection isn't mature enough? Or does that not apply here.
Getting rid of user agent strings is great, as long as we get a better way to determine browser capabilities that doesn't require some kind of special feature checking library...
For older browsers, the UA string remains - so that's still viable for compatibility issues. https://wicg.github.io/ua-client-hints/ will provide the cleaner, opt-in approach in the future.
So, while UA hints could potentially supply more information than the current UA string - each item needs to be explicitly requested by the site meaning the browser can make a choice on what to return. This may depend on user's preferences, level of trust in a site, the amount of identifying information already provided to the site, etc.
> It changes the passive fingerprinting vector to an active one
You say this as though the ad industry cares.
> So, while UA hints could potentially supply more information than the current UA string - each item needs to be explicitly requested by the site meaning the browser can make a choice on what to return.
Let me introduce you to useragent switchers.
The replacement is strictly worse. Simply freezing the user agent solves things well.
They don't have a choice? The point about passive vs active is that it places control with the browser/user where they didn't have it before. You'll be able to respond to some hints and ignore others.
> Let me introduce you to useragent switchers.
And what's the adoption rate of those, I wonder... less than 1% of users? This client hints standard will make it a lot more reasonable for non-power users to control what information is being disclosed, should they wish.
> They don't have a choice? The point about passive vs active is that it places control with the browser/user where they didn't have it before. You'll be able to respond to some hints and ignore others.
So, you are saying that every time someone wants to test browser compatibility, the browser will prompt the user?
No, they're not doing that. Which means that the information is in the hands of anyone that cares. It just isn't in Apache server logs by default.
> And what's the adoption rate of those, I wonder... less than 1% of users?
About as high as the dynamic equivalent will be.
Which is why not replacing the useragent string is the only option that makes things better.
What's cleaner about this new approach? I can't see the point of it.
It's exactly the same as the User-Agent header we had, but worse.
UA was used for tracking? With this new standard, just ask the user agent to include all details in its Accept-CH header.
UA was used for feature detection? People will use this new standard to do feature detection.
And it's worse because there's legitimate uses of UA sniffing, and JS won't have access to it anymore - TFA wants to deprecate navigator.userAgent, so only the webserver would have access to user agent details? Why?
> With this new standard, just ask the user agent to include all details in its Accept-CH header.
That becomes an explicit choice by the site to request more information, it's up to the client/browser how it responds to that. Fewer bits of information are exposed by default.
> JS won't have access to it anymore - TFA wants to deprecate navigator.userAgent, so only the webserver would have access to user agent details? Why?
I should have linked to the top-level repo with the explainer (https://github.com/WICG/ua-client-hints) as it's not immediately clear from the spec, but access to the hint values is provided via getUserAgent()
User-agent was never feature detection, only a proxy for it, and frequently resulted in preventing users from using things that the browser was capable of supporting.
Why does edge say "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"? Because people kept abusing it for feature detection.
You're not wrong, but there are still various functionalities that are not easily feature detected in browser especially in older versions and sometimes you need to err on the side of caution and tell the user their browser is unsupported (even if it may work)
Google recently introduced a new “secure” login page which forces my chromium based browser (qutebrowser) to fake a Mozilla firefox user agent in order to work.
As such, I think if detection comes in some other form, it might be harder to trick some sites into working properly.
Despite all the PR about "privacy", they don't seem to intend to prevent detection at all and are introducing even more surface for detection with client hints. Basically the opposite of what the title implies.
Client Hints (https://wicg.github.io/ua-client-hints/) move a passive fingerprinting vector to an active one, i.e. information must be explicitly requested by the site and then the browser can choose how to respond.
The default level of information exposed drops to just the browser name and major version, which is only sent to sites on HTTPS and with JavaScript enabled.
Additional hints are only sent on subsequent requests by the browser if the site sends the matching header in its initial response and the browser chooses to send a value. The current set of proposed hints define the same amount of information exposed Chrome's User-Agent string.
Yes, but the fear remains that all sites will just always request everything from ua-client-hints. It is also totally unclear how browsers will handle this. I think making this permissioned will just add to the PrivacyNagOverload. Also, browsers will continue to lie in the hints because sites will always make broken assumptions or even try to do mischief with the info.
I think the only winning move is not to play: Freeze the User-Agent and do not provide a replacement. Or at the very least, make the replacement based on actual feature bits, not version numbers and device models.
This is part of what the Privacy Budget (https://github.com/bslassey/privacy-budget) proposal aims to tackle. Freezing the User-Agent string reduces the amount of information exposed by default. UA Client Hints means the site has to explicitly request the additional information. The browser makes a choice about how to allocate / enforce budget. You're right though about how that works and how it would be exposed to the user in their browser still being open questions! More permission pop-ups certainly aren't the answer.
That would still cause pages to do evil things if users set their privacy budget to "0/paranoid" or anything below "11/just gimme all".
Just as with adblockers users will be nagged about "please turn that dial to 11". On average nothing will improve except for users who are able enough to get around those shenanigans even now.
I think there's a fundamental principle/security in privacy that we don't really understand broadly enough across the industry -- that if you allow someone to know whether or not you're hiding/disabling something, they can often just force you to change the setting.
Just as one example, active-permissions that can be revoked after being granted aren't perfect, but are a big step up over manifests, because they're more work to exploit and often allow users to retroactively change permissions after an app checks if they're allowed.
Not to pick on the Privacy Budget specifically, but I worry that proposals like this don't really get that larger principle yet -- that it's still something we haven't quite internalized in the privacy community. If a site exceeds the privacy budget, it shouldn't get told. It should just get misinformation.
It's like autoplay permissions. Autoplay permissions on web audio are awful, because you can just keep trying things until you get around the restriction. What would be better is to auto-mute the tab, because that would be completely invisible to code running on the page.
Agreed, for things like autoplay. But dual-use features like feature detection that also enables fingerprinting cannot be replaced by randomized misinformation because that would really randomly break legitimate stuff.
The only privacy-conscious way would be no feature-detection at all or a very coarse-grained approach like "I support HTML2021".
You can't lie that you do support something, but you can lie in the opposite direction. And for sites that legitimately need that feature to function, you don't get much benefit -- if a site genuinely needs Chrome's Native File access, saying that you don't have it just means the site won't work.
But there's a grey area past that, which is that sites that don't need a feature, but are just using it to fingerprint, can have that feature broken without repercussion. If a news site requests Native File Access, and I say "I don't support that", then whatever.
This puts engineers in an interesting position. You can't just break your site whenever the full range of features that every browser supports aren't available, because:
A) You want to support a wide range of browsers, and if your news site doesn't work with multiple browsers you're just losing potential market.
B) A fingerprinting metric that just rejects every browser that doesn't support everything is not an effective fingerprinter. At that point, we basically have the corse-grained approach you're talking about.
The problem with this approach is that when a site requests capabilities, you need some way to figure out whether or not they're actually required, and whether or not you can lie about them. Permission prompts are... well, there are probably UXs that work, but most of them are also probably too annoying to use. In practice, I suspect that manually tagging sites is not an insurmountable task -- adblockers already kind of do this today.
One thing to push for with Client Hints is that it really, really needs to be an API that's interceptable and controllable by web extensions.
The same thing is true of fonts today -- if you lie and say you already have a font that you don't, congratulations, your text rendering is broken. But you can still lie about not having fonts, and you can still standardize your installed fonts to a smaller subset to make your browser less unique.
I've written about this in the past, but we do really need at least one user-accessible, general computing environment that protects against these kinds of privacy attacks. It doesn't need to be the web, but I don't know of a better, currently-usable platform.
I often hear proposals that the web should just be for static documents, and I'm fine with that, but very rarely are those proposals followed up with alternative ways for ordinary people to run untrusted code. The assumption seems to be that if the web didn't exist, users would instead be responsibly vetting every binary on their computer, rather than downloading them en-mass from dozens of sources. And just looking at the smartphone app market, I don't think that assumption is true.
Again, not to say that a better alternative platform couldn't exist, but who's working on it? The native desktop platforms I see almost all do a worse job than the web at protecting against fingerprinting. It's almost universally better for privacy to use Facebook in a browser instead of downloading their native phone app.
I don't think sandboxing is the solution. It gets in the way of functionality and you still have to trust the developer: They can abuse the permissions that their program legitimately needs and they can use dark patterns. You should never run untrusted code so there is no need for a platform that facilitates it.
What we need are standardized protocols, strict customer protection laws and trustworthy software repositories so users can get software they can trust without having to vet it.
> Again, not to say that a better alternative platform couldn't exist, but who's working on it?
~Nobody is working on such a platform because the app web exists. Remove it and there is a lot more incentive to create a replacement. Regression to the mean alone practically guarantees that it will be superior.
> You should never run untrusted code so there is no need for a platform that facilitates it.
I don't believe this is a practical philosophy given the way that ordinary people use both the web and apps today. If you can convince me that you have a plan to make everyone en-mass stop installing the Facebook App, I might be persuaded to change my mind. But I regard the advice, "don't run untrusted code" to be a bit like saying, "stick to abstinence to avoid pregnancy". The advice isn't technically wrong, we just have good evidence that it doesn't work for society in general.
I also think that "untrusted" is being used as a really broad catch-all here. Trust isn't binary. I trust a calculator app to give me the correct answer to a multiplication problem, I don't trust it to store my banking information.
What sandboxes do are they allow us to set up boundaries for apps that we partially trust. Of course, sandboxes don't remove the need for consumers to be taught not to blindly trust everything. But they're not designed to solve that problem, just to make it easier to manage. There is no way to remove the need to educate consumers; trust is too broad of a topic to divide every app into a single "trusted" or "untrusted" bucket. So what sandboxes do is embrace that grey area rather than ignore it.
In the physical world, if I'm securing a bank or an organization, there are going to be people I distrust so much that they can't even enter the premises. There are going to be people I trust enough to walk around while supervised. There are going to be people I trust enough to be unsupervised, but not enough to give them keys to my vault or server room. And finally there will be a minuscule number of people I trust to have full access to everything.
Imagine if instead, our policy was, "you shouldn't let untrustworthy people into your building in the first place, so security measures past that point are useless." Would we be able to build a database of 'trustworthy' people who could have access to the vaults of every bank they walked into?
When we get rid of sandboxes, we're still using a permissions system -- all we've done is made that permissions system less granular and harder to moderate, because we've removed our ability to say nuanced things like, "I trust Facebook to connect to the Internet, but not to read my contacts."
> Client Hints (https://wicg.github.io/ua-client-hints/) move a passive fingerprinting vector to an active one, i.e. information must be explicitly requested by the site and then the browser can choose how to respond.
"However, one form of phishing, known as “man in the middle” (MITM), is hard to detect when an embedded browser framework (e.g., Chromium Embedded Framework - CEF) or another automation platform is being used for authentication. MITM intercepts the communications between a user and Google in real-time to gather the user’s credentials (including the second factor in some cases) and sign in."
In other words, the OAuth sign in is intended for applications which want to access the Google APIs in some way.
qutebrowser (as an application) doesn't need access to your Google Account in any way. It doesn't care about your Google authentication data, and it doesn't use Google APIs.
How about doing this but also NOT providing the client hints API and NOT providing any way for JavaScript to explicitly ask for OS and browser product name and version?
While browser and JavaScript engines are likely to continue being detectable due to behavior and performance for the foreseeable future, it's probably possible to make the OS, browser frontend and exact engine version undetectable.
Browser vendor/version is hugely important information for triaging issues and implementing progressive enhancement approaches on large scale websites.
Client hints are not any more useful than traditional UA for privacy or progressive enhancement. Its the same content, just split up. Any browser not sending everything will have atrocious permission UX, nagging websites or just get the broken IE5 version.
UA hints solve nothing, even after reading the spec
Triaging issues, sure. Progressive enhancement? If you’re parsing User-Agent to implement that, you’re doing it wrong. (Feature detection is the correct approach, when necessary.)
Edit: OK, this isn't progressive enhancement. But it's still a major problem you have to sniff the User-Agent for. I don't want to sniff UAs either but when Chrome is wont to change how fundamental parts of the web work like cookies it's sometimes necessary.
I know UA hints will still allow you to do this, but requiring a second request is going to make things like redirect pages difficult to implement.
You're right, I should have replied elsewhere. By the way, if anyone knows what this _is_ called I would be interested to know. As far as I can see it's basically feature detection with no other way of detecting it besides the UA.
Can’t feature detect during SSR, so making an educated guess and falling back gracefully (when possible—many times it just isn’t, as with ES6 syntax) is important to get initial page load right by default.
Because so many people abused the user agent header for things that could gracefully fall back (or they just made invalid assumptions from it) it was made an unreliable indicator for times you actually want to send shimmed content on first load.
How do you propose, without some sort of browser and os detection and version to address specific bugs for those environments? Just leave a site/app broken for a few months until/if/when the browser fixes the bug?
> It provides the required information only when the server requests it
In the interest of fingerprinting a server would request this every time.
> It provides the information in small increments, so servers are only exposed to the information they need and request
Then the server would need and request the most comprehensive list possible in order to fingerprint someone with better granularity than a UA string could.
I'm not against this as I appreciate the value of this kind on information for the developers. It would be done one way or another. But why is this billed as mainly a privacy move? Nothing suggests it intrinsically offers better privacy when facing a website configured to fingerprint you. It actually looks like it gives even more granular info over which the user has less control than they used to with the UA.
Feature detection isn't the only use case User Agent strings service, two I've seen frequently are:
* Exception Tracking
- The User Agent string is usually attached to exceptions to aid in reproduction.
* Outdated Client Detection
- Primarily in internal dashboards in BYOD environments, I've seen the server display an error when a known outdated/insecure browser connects.
When Safari froze their UA, another big one that came up was browser bugs. Specific versions of browsers on specific platforms have bugs, and you need to be able to tell your user to upgrade.
No user is going to see that sentence. They're going to read right past it, click some button, it doesn't work, "this site sucks" and go somewhere else.
I'll take that over a feature that just gets abused for all kinds of things and will create breakage which makes users just go elsewhere because it's broken.
It is ridiculous for every website to be babysitting their users software, when we already have operating systems that keep software up to date, package managers that keep software up to date, and software that keeps itself up to date.
If I need to make a really important website that can't lose users like that, I just need to make damn sure it works with any relevant browser without any UA sniffing hacks that will bite back down the line.
If I'm making some bleeding edge crap that really needs a bleeding edge browser, I'm sure I can find a place for that sentence where most users will find it, and if I lose a few users, it probably isn't that big of a deal. Chances are I'd lose more users anyway due to many not wanting or being able to update for any number of reasons.
I'd still like a way to feature detect, rather than make a round-trip to the browser. This would let me embed webasm rather than js+branch to begin a second download if feature found... etc
I suggest UA string be a bitmask of features. Then feature detection should stop being broken
Extra bits could be used for js-on/js-off, and is-bot/is-human
--
Ah I see they're kind of doing the bitmask, but keeping a round-trip, and making things complicated (though I realize latest http standards can probably remove those round-trips in the average case)
I'd still suggest the bitmask for non-sensitive information, and have everything else simply js-tested as it currently is
Maybe is-user-blind might be a nice bit too, since canvas based websites could switch to the dom, or whatever
Please could we also have a couple more privacy setting bits for i-accept-your-cookies and i-want-to-be-told-about-cookies-on-every-single-website-because-i-forget-what-they-are-and-really-want-to-click-through-to-your-privacy-settings
If we have those bits, then the user can make a set of choices once, for every site, and we get rid of cookie pop-ups
-- Websites could still ask if they want/need to do something that violates those choices
Or we can just assume like reasonable adults that websites are going to put cookies in your browser and promote privacy-oriented tech to users rather than trying to pretend that having every website ask for permission in order to enable basic functionality solves anything.
> every website ask for permission in order to enable basic functionality
I don't believe that purely functional cookies require GDPR permission - that's covered by "provide services to the user". It's the ones which are functionality to third parties not the user which are the problem.
> I don't believe that purely functional cookies require GDPR permission - that's covered by "provide services to the user". It's the ones which are functionality to third parties not the user which are the problem.
Ah, I didn't realize that. Well, that does sound much more reasonable.
> Necessary cookies enable core functionality such as security, network management, and accessibility. You may disable these by changing your browser settings, but this may affect how the website functions.
> Analytics cookies [toggle On/Off]
> We'd like to set Google Analytics cookies to help us to improve our website by collecting and reporting information on how you use it. The cookies collect information in a way that does not directly identify anyone. For more information on how these cookies work, please see our 'Cookies page'.
The implication is that a consent dialog would not be required if they weren't using Google Analytics or any other third-party.
That would be nice. I feel like the "cookie warnings" basically read as "this site doesn't actually need cookies to work, but we want to track you". We should just have some sort of "do not track" header that indicates we don't accept those terms, and then websites can badger us if they really need cookies, like for logins.
The eprivacy regulation, taking care of that, was supposed to be finished by the time the GDPR went into force. But Austria's pro-business government managed to delay it until there wasn't enough time before the last European Elections.
Quite happy that the market leader is taking this step! Hopefully this will decrease the problems with lacking feature detection.
I was surprised that the post doesn't contain any example user agent strings for a sample of how they are expected to look starting from the different milestones.
Safari tried to do this but had to walk back some of their changes: https://bugs.webkit.org/show_bug.cgi?id=180365. Currently, it reports the marketing OS and version of Safari truthfully, while lying about the bundled version of WebKit. For example, my user agent is "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15", but I am running Safari 13.1, WebKit 15609.1.13.4 on macOS Catalina 10.15.3.
Being a monopoly on iPhones means the rest of the community has to work around their bugs. Why would they put any effort into fixing those bugs - web devs are working around them just fine?
Safari could say "Sorry - <div> tags aren't working in the current release", and all big websites would switch to other html tags in a matter of weeks.
Well, Chrome was based on Webkit for a while and then they forked it and as a result stopped contributing to it. For Google, Chrome is a major product (as it forces their Search engine and products). For Apple, Safari is pretty minor. Apple simply doesn't have a reason to throw so much manpower at it.
Chrome published a draft of User-Agent Client Hints only a month ago. They are already moving ahead to deprecate user Agent string and saying "Where feature detection fails developers, UA Client Hints are the right path forward" (a thing that's not even a standard yet).
Safari has been very wary of this break-neck speed of how Chrome imposes Google's view of standards onto the world. WebKit team has even walked away from proposals when concerns were not addressed (sadly, I can't find the link in GitHub issues right now. Edit: [1]).
And yes, lack of manpower is another pressing concern.
Oh boy. Right now, I am implementing a workaround for websockets in Safari on iOS 13 — apparently, they've somehow broken their already non-standard "x-webkit-deflate-frame" compression scheme and can't properly send fragmented messages anymore when it's enabled: they mess up the zlib stream at the start of the second fragment (or maybe at the end of the first fragment).
This is a backwards step, I get the user agent is revealing things it has no right to like os, but not all browsers are made equal I need to know what its capable of.
What reasonable default? You don't know where people want to install your binaries and you do want people to know what other OSes you support. In UX if the number of choices is very small, like those couple of OSes you support, it is always best to present them all.
Every major player provides the reasonable default when downloading binaries. Every single one. There’s a reason for that.
Regular users are not technical and if you ask them to make a choice based on technical matters (Windows, MAC, Linux, Apple AppStore, Android AppStore, Chrone WebStore, whatever), where you instead could have had -1 clear choice- already presented by default in a big nice button, you will lose conversions or increase tech-support costs.
The evidence for this is so overwhelming that knowingly trying to ignore that makes it look like you have an agenda or horse to grind.
No, these are not UX metrics, these are commercial metrics. There is a reason conversions and tech-support costs and almost all a/b tests and all that are not even remotely good UX metrics, as they do not exist to measure anything users care about in the product, only what companies care about.
My "WebOptions" idea would permit the user to customize these and other settings (both globally and locally; there could be some set of "common" keys as well as supporting keys specific for the server), in a similar format to cookies, although the server and document scripts would not be allowed to set them, and the user only sets them explicitly. For email, there is a "From" header, so that can be used if it is available.
But anyways, the service should not make it difficult to download if it detects a wrong computer; it should allow it always, and should not be so difficult by trying to hide it.
A good chunk of users don't know what OS they're running.
Perhaps you should propose a feature detection for it. It could look like the media source API for resolutions and formats for web video - except the same for binary formats for software.
Websites presenting me with a big button to download a "reasonable" default and hiding everything else behind a small link that I have to go hunting gor is really annoying. There is nothing reasonable about it. Don't think that you know better what your users want then the users themselves.
Optimizing for the 99% use-case is fairly normal and reasonable.
iOS users will almost always install apps via the AppStore. Most Windows-users are probably not interested in a DMG. Are you really going to argue against that?
I agree that taking away options based on OS-detection is a seriously nasty UX anti-pattern though.
> iOS users will almost always install apps via the AppStore. Most Windows-users are probably not interested in a DMG. Are you really going to argue against that?
It may be true in the case of iOS and Android, because they are so locked down. However, on more powerful platforms that Windows, Mac and Linux are it isn't. I may want to run it in a VM, or not install it, but place it somewhere on a shared drive, or anything that a non-handicapped OS is capable of facilitating and many of these things will mean I will want a binary not meant to be run by my native OS. Sometimes it happens that one of my devices will break and so I want to use another one to download something that will help me fix the issue. But now I'm going to have to go full Sherlock Holmes on a website that thinks it knows better what I'm looking for.
In fairness, highlighting the right button for your OS and showing an 'other downloads' button is really a 'You' problem that probably only affects less than half of 1% of users. Almost all sites also show a 'other OS downloads' button. But this is all meaningless, as shown above, UA will be replaced by a client hint property.
- Server log analytics will not be able to provide OS/Browser stats.
- Default download pages (Detect OS automatically and redirect to the platform-specific download pages) would not work.
- "m." Sites: there are still some sites that sniff the UA string and redirect users to mobile site. It's looking for patterns like "/mobile/" in UA, and don't require a lot of updates as the post argues.
I think you'll still be able to tell what OS and browser it is, just not the specific major or minor version because they'll be 85.0.0 (if that's the version when this happens) forever. All of the examples you've put there will still be achievable with the out-of-date UA string.
> It provides the information in small increments, so servers are only exposed to the information they need and request, rather than being exposed to the full gamut of the UA string even if they are just trying to figure out one detail about the browser.
So, an extra RT if your app needs to sense which UA it’s talking to.
the proposed implementation leaves me questioning how useful the CH headers are for initial impressions of a web property if only the browser and version are sent by default, and more info is only provided on follow-up requests, assuming the user agrees to some permission popup they don't understand. it'll be the "this site uses cookies" nonsense / nuisance all over again. it'll finally push end users to install the native reddit app without those constant annoyances.
i have quite a few ux enhancements that need to know or guestimate the screen size, if the device is touch capable and OS (especially for form input styling). i dont see how it is possible without an additional delay, fetch request and some annoying permission popup to deliver a good experience in the brave new CH world.
a media query will not tell me what OS it is so i can infer the metrics of native fonts and form controls. i would have to use JS and create sentinel elements to make the measurements, which would still be a form of slower, shittier sniffing.
also, a media query or any other client side method prevents me from delivering exactly the final content that's necessary and not have to do additional reflow-inducing js to get the end result.
Requires some complicated conversations about how client hints should work in a privacy-respecting way, but this is still unquestionably the right move.
The arguments we're seeing are around what the replacement should be (or if there should be a replacement at all). But either of those scenarios are still better than keeping user agents as they are.
I have mixed feelings on this. On the other hand, I knew some people above me who are still stuck on useragent sniffing as the "way to go."
On the other hand, I did use it in combination with other techniques to provide some real information. At one point I had a hard-to-even-track kind of problem, with difficulty even looking for common trends. I created a help form for the user. Please fill out when this occurred, where you were (this mattered), what OS were you running, what browser, and so forth. What I found using the UA (and backed by other tricks) was that many users, young users who were enrolled in college, were not only unaware of their browser or operating system version but the OS itself.
Its utility began its decline when the browser makers began copying one another wholesale. It's kind of a shame. Capability-testing is better but for some troubleshooting applications, having something particular to pin your problems to is handy.
I know no one RTFAs (I usually don't even RTFA), but in this case, it's really egregious. The whole discussion is going back and forth about UA sniffing with no one mentioning that the damn proposal is entirely based on adding UA Client Hints as a replacement. It makes this whole multi-hundred comment discussion pretty much valueless. Please, other commenters: read this link before discussing whether you're caremad about "like Gecko" in the UA string!
This was the near top level comment I saw to mention it. I don’t think it’s crazy to collapse a huge thread where the first several levels of comments aren’t relevant.
This seems particularly unhelpful for bug reporting systems. Being able to see a specific bug is only happening on Chromium 78 for Linux can save a lot of time and frustration. In theory the user can fill in the details on a bug report themselves, but it's hard enough to persuade them to copy paste the crash message we generate for them at the moment.
There should be better ways to identify the browser than the user agent. The user agent is a legacy artifact, have you seen how many "compatibility" parts are in there? Like why does the chrome user agent say "Mozilla" in iy?
There's not really a good server side way to do it unfortunately. All the current alternatives to my knowledge require js.
It depends on your application but the only real alternative is to use a js analytics library to collect the statistics. The good news is that they're more accurate, can give you more information, and there's plenty of self hosted options like Matomo.
But it is yet another thing the client as to load, and adblockers will likely skew your metrics.
298 comments
[ 3.6 ms ] story [ 218 ms ] thread> The above abuse makes it desirable to freeze the UA string and replace it with a better mechanism.
UA sniffing should have died out a long time ago. It's frustrating that it's 2020 and I'm still having my browsing experience broken for no other reason than the site doesn't like my UA.
So come on, it’s 2020 and we still have to explain idioms to each other :)
[1] I believe FF includes in the download length the time a request spends sitting in the request queue, but I can't exactly remember.
Safari has SameSite bugs and it's infuriating that after years they don't even update the bug reports, let alone fix those.
https://bugs.webkit.org/show_bug.cgi?id=200345
What I think would be best is two things: A feature advertising string (preferably with some versioning in itself), and a new User Agent string that is just that: The user agent. As in, "This is Firefox 72.2.813". Maybe the OS in there too. I think a non-trivial part of the problem with the UA today is "Hey... I'm Firefox. Also, I'm Mozilla. Also, I'm edge 3, and Internet Explorer 6, and today I think I'm also Chrome. Also, I may be a monkey with a banana plugged in to an ethernet cable." Part of the problem is that people abuse the user agent, but part of the problem is that the user agent string is also a pack of outright, outrageous, self-contradictory lies. A truthful field may still have significant utility.
One of the things I've learned over the years in software engineering is that you should do you best to make sure that your system doesn't contain lies, or, failing that, treat them like any other issues that you should contain at the edges rather than letting it run riot through your system. For instance, if you decide to let customers have 10GB of bandwidth free before you start charging them, you should never accomplish that goal by tweaking the bandwidth counting system to report 7.5GB of usage as 0GB. That is a lie. Tell the truth about the usage and let the billing system apply the discount, which is itself a truth as well. You always pay for lies in the system in the long run. It is a very common pattern in coding as well where some function author figures out how to assemble the correct lie to some other function to make it do what the author things they want, but in the long run you're better off telling the truth and fixing the code to work with the truth. Otherwise, well, you really do end up with a tangled web of code. Which is... exactly what we've gotten with UA-detection-based code. We won't necessarily get the same mess if there's a new UA that isn't a lie.
I'm not claiming this will lead to utopia. Minority forks of popular browsers will fail to pick up the necessary ad-hoc bug fixes this way, for instance. My claim is more like in the long run, the best thing for everybody is just to have a user agent that tells the truth, even if on occasion in the short term you experience occasional problems.
(One way to at least partially achieve this is to standardize the new truthful agent string to something like (\w+) ((\d+.)+\d) (\w+), and specify it as "non-conforming true user agent strings MUST be entirely rejected and treated as being absent", so anyone who tries to be Firefox and Chrome and also a monkey ends up being nobody in particular.)
Also I think you're missing why the user agent lies now. It wasn't done on a lark. It's because it's been abused so much by web devs. Both by sloppy regular expressions and deliberately. For example, even big names like Microsoft and Google have at times used it to unnecessarily deliver poorer browsing experiences to people not using their browser.
Sure browsers could have stuck to their guns and been "honest" about their UA. But honesty is no comfort to users of the browser when they find websites break for no reason. The average user is more likely to blame the browser than the website.
The parent of my post is correct; in practice we're still going to need the occasional ability to shim in browser-specific fixes, because even if the browsers do their best, they're going to inadvertently lie in the future and claim to support WebVR1.0 in Firefox 92, but, whoops, actually it crashes the entire browser if you try to do anything serious in it. Or, whoops, Firefox 92 does do a pretty decent job of WebVR1.0 but I need some attribute they overlooked. Or any number of similar little fixups. We know from experience from the field in the real world that we're talking about crashing bugs here at times; this is real thing that has happened. Whatever proposal gets implemented should deal with this case too.
If we standardized on the format like I suggested at the end of my post, it would go a long ways towards preventing future browsers from mucking up the field. If you just get "$BROWSER $VERSION $OS" in a rigid specification, and if the major browsers are sure to conform to that, and the major frameworks enforce it, it'll be enough to prevent it from becoming a problem in the future. It won't stop Joe Bob's Bait Shack & Cloud Services from giving their client a custom browser and/or server that abuses it, but there's no stopping them from doing things like that no matter what you do, so shrug.
Culturally, you should prefer to use feature detection. Most developers would never need to use anything else. But when Amazon makes its new whizbang WebVR1.0 front-end in 2024, they may need the ability to blacklist a particular browser. Lacking that ability may actually prevent them from being able to ship, if shipping will result in some non-trivial fraction of the browsers claiming "web-vr/1.0.0" will in fact crash, and they have nothing they can do about it.
Besides... they will find a way to blacklist the browser. Honestly "prevent anyone from ever knowing what version of the browser is accessing your site" is not something you can accomplish. If you don't give them some type of user agent in the header, it doesn't mean the Amazon engineers are just going to throw their hands up and fail to ship. They will do something even more inadvisable than user agent sniffing, because you "cleverly" backed them into a corner. If necessary, they will examine the order of headers, details of the TLS negotiation, all sorts of things. See "server fingerprinting" in the security area. You can't really stop it. Might as well just give it to them as a header. But this time, a clean, specified, strict one based on decades of experience, instead of the bashed-together mess that is User-Agent.
Or, to put it really shortly, the fact that a bashed-together User-Agent header has been a disaster is not sufficient proof that the entire idea of sending a User Agent is fundamentally flawed. You can't separate from the current facts whether the problem is that User Agent sniffing is always 100% guarantee totally black&white no shades of grey mega-bad, or if it's the bashed-together nature of the field that is the problem.
https://www.chromium.org/updates/same-site/incompatible-clie...
Why not drop just that and leave nothing but the exact HTML and JavaScript engine names and versions like "Blink a.b, V8 x.y"?
And what about robots? Will the Googlebot give the UA up as well?
Using a completely new string format in the same field (or removing it entirely) breaks a lot of sites that'll never be fixed.
Freezing it prevents this. And if we're freezing and creating a new system then why not go for something queryable without all the baggage?
I expect no UA header to be even less usable now that sites are more paranoid and app-ified, so instead I use a random one. That still confuses some sites...
The only legitimate use case I can think of is exception tracking, it is valuable to know which browser caused the exception.
Beyond that, a website should never, ever rely on the UA for anything.
In my app Dictanote (https://dictanote.co) - which uses Chrome's speech-to-text API, I have no way to distinguish Brave/Vivaldi and user doesn't understand why its not working :/
To determine if a browser has the text to speech API just check if the webkitSpeechRecognition object exists (if that’s the one you’re using). It will exist in Chrome and will not exist in other browsers that lack the feature.
An incorrect assumption stated as fact with such great confidence. Good job.
Even on the face of it, any engineer would know this can't be true.
[1] https://developers.google.com/web/updates/2014/01/Web-apps-t...
This specific API only works in Google Chrome unfortunately. So we need to stop people from trying in another browser and getting frustrated as to why its not working.
To add to OP's point, this thing in Safari also comes to mind as an example of something that isn't easy to detect and address outside of UA detection: https://github.com/vitr/safari-cookie-in-iframe
Deprecating UA Strings and moving towards UA Client Hints seems like a move in the right direction though.
Assuming these iframes will never work means degrading the experience for all users, when only users on Safari are actually affected. Detecting the UA and branching based on that is a much more pragmatic solution.
- It's not likely to work anytime soon
- "The property doesn't exist or throws an exception or does something else that Chrom{e,ium} doesn't do" provides the necessary fingerprinting
- Adding the feature back later can just be chalked up to "growing pains", it doesn't really set especially unique precedent
The kind of testing you are complaining you can't do is exactly why user agent is broken, what if you test for Chrome and then Brave/Vivaldi start supporting the speech-to-text API and your broken website still says "sorry, you need Chrome" for no reason?
"Somethings wrong! You might be using Chrome and having connectivity issues, or you might be using Brave who have a dispute with Google about usage of their speech recognition API. In the former case, get better WiFi, and in the latter case there's nothing you can do, switch browsers."
Getting rid of user agent strings is great, as long as we get a better way to determine browser capabilities that doesn't require some kind of special feature checking library...
So, while UA hints could potentially supply more information than the current UA string - each item needs to be explicitly requested by the site meaning the browser can make a choice on what to return. This may depend on user's preferences, level of trust in a site, the amount of identifying information already provided to the site, etc.
You say this as though the ad industry cares.
> So, while UA hints could potentially supply more information than the current UA string - each item needs to be explicitly requested by the site meaning the browser can make a choice on what to return.
Let me introduce you to useragent switchers.
The replacement is strictly worse. Simply freezing the user agent solves things well.
They don't have a choice? The point about passive vs active is that it places control with the browser/user where they didn't have it before. You'll be able to respond to some hints and ignore others.
> Let me introduce you to useragent switchers.
And what's the adoption rate of those, I wonder... less than 1% of users? This client hints standard will make it a lot more reasonable for non-power users to control what information is being disclosed, should they wish.
So, you are saying that every time someone wants to test browser compatibility, the browser will prompt the user?
No, they're not doing that. Which means that the information is in the hands of anyone that cares. It just isn't in Apache server logs by default.
> And what's the adoption rate of those, I wonder... less than 1% of users?
About as high as the dynamic equivalent will be.
Which is why not replacing the useragent string is the only option that makes things better.
It's exactly the same as the User-Agent header we had, but worse.
UA was used for tracking? With this new standard, just ask the user agent to include all details in its Accept-CH header.
UA was used for feature detection? People will use this new standard to do feature detection.
And it's worse because there's legitimate uses of UA sniffing, and JS won't have access to it anymore - TFA wants to deprecate navigator.userAgent, so only the webserver would have access to user agent details? Why?
That becomes an explicit choice by the site to request more information, it's up to the client/browser how it responds to that. Fewer bits of information are exposed by default.
> JS won't have access to it anymore - TFA wants to deprecate navigator.userAgent, so only the webserver would have access to user agent details? Why?
I should have linked to the top-level repo with the explainer (https://github.com/WICG/ua-client-hints) as it's not immediately clear from the spec, but access to the hint values is provided via getUserAgent()
Why does edge say "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"? Because people kept abusing it for feature detection.
https://webaim.org/blog/user-agent-string-history/
As such, I think if detection comes in some other form, it might be harder to trick some sites into working properly.
The default level of information exposed drops to just the browser name and major version, which is only sent to sites on HTTPS and with JavaScript enabled.
Additional hints are only sent on subsequent requests by the browser if the site sends the matching header in its initial response and the browser chooses to send a value. The current set of proposed hints define the same amount of information exposed Chrome's User-Agent string.
I think the only winning move is not to play: Freeze the User-Agent and do not provide a replacement. Or at the very least, make the replacement based on actual feature bits, not version numbers and device models.
Just as with adblockers users will be nagged about "please turn that dial to 11". On average nothing will improve except for users who are able enough to get around those shenanigans even now.
Just as one example, active-permissions that can be revoked after being granted aren't perfect, but are a big step up over manifests, because they're more work to exploit and often allow users to retroactively change permissions after an app checks if they're allowed.
Not to pick on the Privacy Budget specifically, but I worry that proposals like this don't really get that larger principle yet -- that it's still something we haven't quite internalized in the privacy community. If a site exceeds the privacy budget, it shouldn't get told. It should just get misinformation.
It's like autoplay permissions. Autoplay permissions on web audio are awful, because you can just keep trying things until you get around the restriction. What would be better is to auto-mute the tab, because that would be completely invisible to code running on the page.
The only privacy-conscious way would be no feature-detection at all or a very coarse-grained approach like "I support HTML2021".
You can't lie that you do support something, but you can lie in the opposite direction. And for sites that legitimately need that feature to function, you don't get much benefit -- if a site genuinely needs Chrome's Native File access, saying that you don't have it just means the site won't work.
But there's a grey area past that, which is that sites that don't need a feature, but are just using it to fingerprint, can have that feature broken without repercussion. If a news site requests Native File Access, and I say "I don't support that", then whatever.
This puts engineers in an interesting position. You can't just break your site whenever the full range of features that every browser supports aren't available, because:
A) You want to support a wide range of browsers, and if your news site doesn't work with multiple browsers you're just losing potential market.
B) A fingerprinting metric that just rejects every browser that doesn't support everything is not an effective fingerprinter. At that point, we basically have the corse-grained approach you're talking about.
The problem with this approach is that when a site requests capabilities, you need some way to figure out whether or not they're actually required, and whether or not you can lie about them. Permission prompts are... well, there are probably UXs that work, but most of them are also probably too annoying to use. In practice, I suspect that manually tagging sites is not an insurmountable task -- adblockers already kind of do this today.
One thing to push for with Client Hints is that it really, really needs to be an API that's interceptable and controllable by web extensions.
The same thing is true of fonts today -- if you lie and say you already have a font that you don't, congratulations, your text rendering is broken. But you can still lie about not having fonts, and you can still standardize your installed fonts to a smaller subset to make your browser less unique.
I often hear proposals that the web should just be for static documents, and I'm fine with that, but very rarely are those proposals followed up with alternative ways for ordinary people to run untrusted code. The assumption seems to be that if the web didn't exist, users would instead be responsibly vetting every binary on their computer, rather than downloading them en-mass from dozens of sources. And just looking at the smartphone app market, I don't think that assumption is true.
Again, not to say that a better alternative platform couldn't exist, but who's working on it? The native desktop platforms I see almost all do a worse job than the web at protecting against fingerprinting. It's almost universally better for privacy to use Facebook in a browser instead of downloading their native phone app.
What we need are standardized protocols, strict customer protection laws and trustworthy software repositories so users can get software they can trust without having to vet it.
> Again, not to say that a better alternative platform couldn't exist, but who's working on it?
~Nobody is working on such a platform because the app web exists. Remove it and there is a lot more incentive to create a replacement. Regression to the mean alone practically guarantees that it will be superior.
I don't believe this is a practical philosophy given the way that ordinary people use both the web and apps today. If you can convince me that you have a plan to make everyone en-mass stop installing the Facebook App, I might be persuaded to change my mind. But I regard the advice, "don't run untrusted code" to be a bit like saying, "stick to abstinence to avoid pregnancy". The advice isn't technically wrong, we just have good evidence that it doesn't work for society in general.
I also think that "untrusted" is being used as a really broad catch-all here. Trust isn't binary. I trust a calculator app to give me the correct answer to a multiplication problem, I don't trust it to store my banking information.
What sandboxes do are they allow us to set up boundaries for apps that we partially trust. Of course, sandboxes don't remove the need for consumers to be taught not to blindly trust everything. But they're not designed to solve that problem, just to make it easier to manage. There is no way to remove the need to educate consumers; trust is too broad of a topic to divide every app into a single "trusted" or "untrusted" bucket. So what sandboxes do is embrace that grey area rather than ignore it.
In the physical world, if I'm securing a bank or an organization, there are going to be people I distrust so much that they can't even enter the premises. There are going to be people I trust enough to walk around while supervised. There are going to be people I trust enough to be unsupervised, but not enough to give them keys to my vault or server room. And finally there will be a minuscule number of people I trust to have full access to everything.
Imagine if instead, our policy was, "you shouldn't let untrustworthy people into your building in the first place, so security measures past that point are useless." Would we be able to build a database of 'trustworthy' people who could have access to the vaults of every bank they walked into?
When we get rid of sandboxes, we're still using a permissions system -- all we've done is made that permissions system less granular and harder to moderate, because we've removed our ability to say nuanced things like, "I trust Facebook to connect to the Internet, but not to read my contacts."
And how much of a hurdle is that, exactly?
That's an embedded chromium browser and according to [1] you should really be switching to oauth for login, not faking a UA string.
[1] https://security.googleblog.com/2019/04/better-protection-ag...
"However, one form of phishing, known as “man in the middle” (MITM), is hard to detect when an embedded browser framework (e.g., Chromium Embedded Framework - CEF) or another automation platform is being used for authentication. MITM intercepts the communications between a user and Google in real-time to gather the user’s credentials (including the second factor in some cases) and sign in."
In other words, the OAuth sign in is intended for applications which want to access the Google APIs in some way.
qutebrowser (as an application) doesn't need access to your Google Account in any way. It doesn't care about your Google authentication data, and it doesn't use Google APIs.
While browser and JavaScript engines are likely to continue being detectable due to behavior and performance for the foreseeable future, it's probably possible to make the OS, browser frontend and exact engine version undetectable.
UA hints solve nothing, even after reading the spec
The "main advantages" section for client hints all seem to apply there.
Edit: OK, this isn't progressive enhancement. But it's still a major problem you have to sniff the User-Agent for. I don't want to sniff UAs either but when Chrome is wont to change how fundamental parts of the web work like cookies it's sometimes necessary. I know UA hints will still allow you to do this, but requiring a second request is going to make things like redirect pages difficult to implement.
In the interest of fingerprinting a server would request this every time.
> It provides the information in small increments, so servers are only exposed to the information they need and request
Then the server would need and request the most comprehensive list possible in order to fingerprint someone with better granularity than a UA string could.
I'm not against this as I appreciate the value of this kind on information for the developers. It would be done one way or another. But why is this billed as mainly a privacy move? Nothing suggests it intrinsically offers better privacy when facing a website configured to fingerprint you. It actually looks like it gives even more granular info over which the user has less control than they used to with the UA.
* Exception Tracking - The User Agent string is usually attached to exceptions to aid in reproduction.
* Outdated Client Detection - Primarily in internal dashboards in BYOD environments, I've seen the server display an error when a known outdated/insecure browser connects.
There, I just told you to upgrade without sniffing UA.
It is ridiculous for every website to be babysitting their users software, when we already have operating systems that keep software up to date, package managers that keep software up to date, and software that keeps itself up to date.
If I need to make a really important website that can't lose users like that, I just need to make damn sure it works with any relevant browser without any UA sniffing hacks that will bite back down the line.
If I'm making some bleeding edge crap that really needs a bleeding edge browser, I'm sure I can find a place for that sentence where most users will find it, and if I lose a few users, it probably isn't that big of a deal. Chances are I'd lose more users anyway due to many not wanting or being able to update for any number of reasons.
That would avoid feature detection, but still create epochs to disallow insecure browsers
--
For exception tracking, the best I can suggest is getting screen size + other features; then storing them, ready to pass with the exception
I suggest UA string be a bitmask of features. Then feature detection should stop being broken
Extra bits could be used for js-on/js-off, and is-bot/is-human
--
Ah I see they're kind of doing the bitmask, but keeping a round-trip, and making things complicated (though I realize latest http standards can probably remove those round-trips in the average case)
I'd still suggest the bitmask for non-sensitive information, and have everything else simply js-tested as it currently is
Maybe is-user-blind might be a nice bit too, since canvas based websites could switch to the dom, or whatever
If we have those bits, then the user can make a set of choices once, for every site, and we get rid of cookie pop-ups
-- Websites could still ask if they want/need to do something that violates those choices
Like what?
> every website ask for permission in order to enable basic functionality
I don't believe that purely functional cookies require GDPR permission - that's covered by "provide services to the user". It's the ones which are functionality to third parties not the user which are the problem.
Ah, I didn't realize that. Well, that does sound much more reasonable.
> Necessary cookies
> Necessary cookies enable core functionality such as security, network management, and accessibility. You may disable these by changing your browser settings, but this may affect how the website functions.
> Analytics cookies [toggle On/Off]
> We'd like to set Google Analytics cookies to help us to improve our website by collecting and reporting information on how you use it. The cookies collect information in a way that does not directly identify anyone. For more information on how these cookies work, please see our 'Cookies page'.
The implication is that a consent dialog would not be required if they weren't using Google Analytics or any other third-party.
I typically read those warnings as reminders that I should open the site in a FF container.
That one was tried with the DNT bit - of course users ended up en masse setting it to "do not track" by default. Sites won't accept that.
This is exactly what you aren't supposed to do
Either send the wasm optimistically & fallback to js on error, or send it reactively with js+branch
A http header bitmask set by javascript:testWASM(), would be equivalent to what you suggest...
But avoid the js, and making a second trip to the server
https://wicg.github.io/ua-client-hints/
I was surprised that the post doesn't contain any example user agent strings for a sample of how they are expected to look starting from the different milestones.
Safari could say "Sorry - <div> tags aren't working in the current release", and all big websites would switch to other html tags in a matter of weeks.
Safari has been very wary of this break-neck speed of how Chrome imposes Google's view of standards onto the world. WebKit team has even walked away from proposals when concerns were not addressed (sadly, I can't find the link in GitHub issues right now. Edit: [1]).
And yes, lack of manpower is another pressing concern.
[1] https://github.com/WICG/construct-stylesheets/issues/45#issu...
"Now I consider this issue as an absolute show stopper. I don't think we want to ever implement this feature in WebKit unless this issue is resolved."
How do I know which (binary) download to offer my users?
Edit: How do I provide reasonable defaults when the user’s OS actually matters?
That’s a load of horseshit if I’ve ever heard it.
Regular users are not technical and if you ask them to make a choice based on technical matters (Windows, MAC, Linux, Apple AppStore, Android AppStore, Chrone WebStore, whatever), where you instead could have had -1 clear choice- already presented by default in a big nice button, you will lose conversions or increase tech-support costs.
The evidence for this is so overwhelming that knowingly trying to ignore that makes it look like you have an agenda or horse to grind.
In what other places of application development should we also stop providing reasonable defaults and force the user to choose instead?
- country? city?
- localization?
- number for local support hotline?
- email of logged in user?
- type of currency for payments?
No? Then why is having a reasonable default for downloads bad?
But anyways, the service should not make it difficult to download if it detects a wrong computer; it should allow it always, and should not be so difficult by trying to hide it.
Perhaps you should propose a feature detection for it. It could look like the media source API for resolutions and formats for web video - except the same for binary formats for software.
iOS users will almost always install apps via the AppStore. Most Windows-users are probably not interested in a DMG. Are you really going to argue against that?
I agree that taking away options based on OS-detection is a seriously nasty UX anti-pattern though.
It may be true in the case of iOS and Android, because they are so locked down. However, on more powerful platforms that Windows, Mac and Linux are it isn't. I may want to run it in a VM, or not install it, but place it somewhere on a shared drive, or anything that a non-handicapped OS is capable of facilitating and many of these things will mean I will want a binary not meant to be run by my native OS. Sometimes it happens that one of my devices will break and so I want to use another one to download something that will help me fix the issue. But now I'm going to have to go full Sherlock Holmes on a website that thinks it knows better what I'm looking for.
- Server log analytics will not be able to provide OS/Browser stats.
- Default download pages (Detect OS automatically and redirect to the platform-specific download pages) would not work.
- "m." Sites: there are still some sites that sniff the UA string and redirect users to mobile site. It's looking for patterns like "/mobile/" in UA, and don't require a lot of updates as the post argues.
So, an extra RT if your app needs to sense which UA it’s talking to.
the proposed implementation leaves me questioning how useful the CH headers are for initial impressions of a web property if only the browser and version are sent by default, and more info is only provided on follow-up requests, assuming the user agrees to some permission popup they don't understand. it'll be the "this site uses cookies" nonsense / nuisance all over again. it'll finally push end users to install the native reddit app without those constant annoyances.
i have quite a few ux enhancements that need to know or guestimate the screen size, if the device is touch capable and OS (especially for form input styling). i dont see how it is possible without an additional delay, fetch request and some annoying permission popup to deliver a good experience in the brave new CH world.
also, a media query or any other client side method prevents me from delivering exactly the final content that's necessary and not have to do additional reflow-inducing js to get the end result.
The arguments we're seeing are around what the replacement should be (or if there should be a replacement at all). But either of those scenarios are still better than keeping user agents as they are.
On the other hand, I did use it in combination with other techniques to provide some real information. At one point I had a hard-to-even-track kind of problem, with difficulty even looking for common trends. I created a help form for the user. Please fill out when this occurred, where you were (this mattered), what OS were you running, what browser, and so forth. What I found using the UA (and backed by other tricks) was that many users, young users who were enrolled in college, were not only unaware of their browser or operating system version but the OS itself.
Its utility began its decline when the browser makers began copying one another wholesale. It's kind of a shame. Capability-testing is better but for some troubleshooting applications, having something particular to pin your problems to is handy.
https://wicg.github.io/ua-client-hints/
You should still get this information, but more privacy oriented manner.
UA is the only thing I get in server logs to discover how much attention I should pay to support particular browsers.
https://wicg.github.io/ua-client-hints/
It depends on your application but the only real alternative is to use a js analytics library to collect the statistics. The good news is that they're more accurate, can give you more information, and there's plenty of self hosted options like Matomo.
But it is yet another thing the client as to load, and adblockers will likely skew your metrics.