Ask HN: How to deal with GDPR / cookie notices in the context of a crawler?
The latter ones present a issue to my crawler. I cannot access the content of the page without accepting those notices.
Things I'm considering to bypass those notices:
* US IP address (easy to implement, but some websites also display those notices to US IP's)
* Heuristics to detect those notices and accept them programatically (takes some time to implement - while a couple of vendors (i.e. OneTrust) offer off-the-shelf solutions which are easy to identify and automate, there are also many custom made solutions, so the system would need understand the concept of a consent form and how to bypass it - some forms only require the press of the right button, others involve checkboxes/radio buttons). To collect test data one solution might be to visit a set of websites once with an US IP, once with an EU IP and/or with different user agents (browser or googlebot).
Do you have any ideas how to approach this problem? Or are you even utilizing some techniques already and are willing to share them?
51 comments
[ 3.5 ms ] story [ 77.9 ms ] threadOverall I feel like the intent of these is correct, but the execution is terrible. I'd much rather have say a badge in the address bar of the browser (similar to the https badge) saying a site was gdpr compliant and used cookies then a popup everywhere.
This would also be much easier to implement. Just add a GDPR-Compliant header (or similar), and the browser already knows if there are cookies.
I think the actual issue is that browsers are open source/non-commercial things that are usually owned by companies in the US (a place where EU laws do not directly apply). They would have to ask nicely instead of making it a law.
Can anyone just write an RFC?
I'd expect the push to come from browser vendors. Either by blocking ads that aren't in those elements by default, or feeding them garbage data.
"user-visible text or image(s) meant to advertise a product or service, separate from the primary purpose or offer of the page in which it is embedded"
... or something. IANAL.
This won't really hit people doing self-serve or custom ads much, but it'll cover the vast majority of the web.
Unfortunately sites have decided not to comply with this portion.
In more philosophical terms, how am I supposed to feel about profit-optimizing websites that I don't want to pay for choosing to be shitty to me? Meh.
Anyway it's not so simple, as links are not what they used to be anymore either on many websites. All those shorteners and opaque tracking redirects... bleh.
There is https://www.i-dont-care-about-cookies.eu/ project. Still, it's the most annoying on phone, where I see now solutions.
Having said that, Prop 65 warnings usually sit quietly on some wall or paper somewhere, annoying only the business owners that have to print/place them, whereas the GDPR/cookie warnings thrust themselves in my face every damn time I visit a complying website in a new session. Very annoying in a way Prop 65 warnings are not.
The problem is websites have apparently decided it's safe not to comply with that portion, presumably knowing everyone would opt out.
So I don't know that they would lose any challenge on those grounds.
more like "Hey we have to tell you we're doing this evil thing, ok?"
Obviously the 'negative' response is to hit the back button on your browser.
I think there's a spectrum, but I think the vast majority of the cookie notices I've encountered have been implemented in a sneaky way that runs counter to the spirit of the law.
The spirit of the law is that sites should explain how the data is used in a way that the layperson can understand, and it should be clear to the layperson that (in most cases) the site is legally obligated to give you a way to say "no."
As it stands, most GDPR notices give you a choice between "OK" and "more info" (where the "no" option is hidden) or between "Yes" and a subtle X in the upper left (because upper right would be too obvious). And they don't tell you that by clicking "Yes", you are consenting to having your information brokered and sold to innumerable advertisers.
I think that's a dirty UX trick, and that for the purposes of getting consent the "no" button actually should be an obvious "no" button.
The reason the badge isn't possible is that GDPR did the right thing to enforce privacy by default, and all of the sites that want to monetize your data for advertising have to get your explicit consent. So you get all of these notices because they have to and want to ask you. Were it not for advertising, you GDPR would have been a pretty peaceful transition with a few exceptions like "oh yeah we keep crash reports, you're okay with that right?"
I mean, most of them already violate that clause where consent for advertising or tracking should be opt-in, not opt-out. X implying consent is just an extension of that.
The problem is that using cookies for advertising does require you to ask, so how do you conflate the two?
I've seen some sites phrase it like "we use cookies to personalize your experience". You can interpret that to mean session cookies, but you can also interpret that to mean marketing.
I hope EU regulators end up actually going after people for doing sneaky crap like this.
I have cookies disabled by default. Whenever a site does not work without cookies (news sites, travel sites, and blog sites especially), I open them in a guest mode. Still feel, its a bit tedious thing to do, but works for me.
Oh well, it's still better than before.
If they will track me, they’ll be breaking the law, and we both know it. And I can access the sites just fine.
If it’s different on your PC could be ‘coz I’m in Montenegro, it’s Europe but we are not yet in the EU.
To me the publishers are currently grasping at straws. Also, much like it is a complete failure of design to ask permissions for all of your stuff in a mobile app at the beginning, it is so with websites. A GDPR consent could as well be integrated into the signup process.
I doubt that EU will prosecute companies for this, but to me they are clearly acting in bad faith towards one piece of legislation that actually tries to somehow protect peoples privacy.
It is only useless for those with at least half a brain. For the rest, it does kinda work. Sure, the gains are tiny, but when you're desperate (which advertisers are, because otherwise they would've taken a more respectable career path building products people actually love instead of spamming the world), every little bit helps.
The real solution would be better user education to make them distrust advertising (because if a product is so great then why would someone pay money to put it in front of you?) and then this cancerous industry will die off by itself. There will be no more incentive for tracking because even the best tracking in the world won't help when your target market immediately sees you as a scam.
Personally I have build a very good product (got great reviews on a few respectable sites and above average conversion ration) but in the end I am quite desperate because it is actually very hard to get people to know about it without an advertising budget.
The inverse is true. Instead of a simple "opt in" / "opt out" dialogue, most sites are deliberately employing various UX dark patterns to coerce users into agreeing to tracking against their will. These range from the annoying: settings -> untick several checkboxes, to the plainly non-compliant: accept all tracking, or leave the site.
I hope that enforcement gets into gear soon, and takes a very dim view of these practises.
Of course, it failed because it was getting some sort of GDPR page at the podcast feed URL. I'm wondering if there was some way around this, because it's not like podcatchers can opt into something via an RSS feed...can they? I'm pretty sure I passed headers only accepting feed content-types, but even that wasn't enough.
Sure I can host elsewhere, but I just didn't care enough about the project to do that. But if there's a way around this, then I might pick it up again.
The RSS reader (TTRSS) gained a sorta-unofficial plugin that would fake the GDPR consent cookie.
I did contact support and they promised to fix this bug.
Though overall, for your podcast it's a bug too. You don't need GDPR consent to display a RSS feed document at all unless you sell IP addresses to third parties.