Ask HN: Has anyone leveraged GDPR to overturn automated bans?
There are many headlines of people getting their <FAANG/Big Company> accounts banned and losing access to a lot of important documents or services. Often, trying to talk to support on any of these companies is akin to talking to a wall.
However, GDPR has a clause stating that "The data subject shall have the right not to be subject to a decision based solely on automated processing". Which would mean that any EU/EEA citizen should have the right to have the decision reviewed by a human.
Has anyone successfully overturned a banned account using this method?
76 comments
[ 3.4 ms ] story [ 173 ms ] threadNow if you don’t believe them then you’d need to take them to court and show why you think that’s not the case.
Which I guess means my question is why don’t you believe them and how likely is it that they are lying when they claim thy appeals are reviewed by a human?
Here's the HN story: https://news.ycombinator.com/item?id=30060405
Screenshots of trying to "appeal" (Request a review) from when I recreated the issue show pretty clearly there is no human involved: https://imgur.com/a/5YHQtLi
This wasn't an account ban, so I don't know how well it fits the GDPR language. Though I'd be surprised if this was somehow the only "fully automated account action" FAANG type companies are doing.
This is like running a restaurant in the US, and not being able to discriminate by race. You're not required to run a restaurant, and certainly aren't required to run one that gives away free food, but if you are certain obligations come attached.
I'd also argue that your use of the phrase "fundamental human right" is misleading. Europe can and does require you do things for reasons other than respecting fundamental human rights. So does pretty much every other law making authority.
I'd also like to point out that these laws don't just come out of nowhere in a vacuum, to be interpreted without any further context. In EU we have recitals and guidelines to give context and support the interpretation of regulations.
If you're interested, do read Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679 (wp251rev.01).
https://ec.europa.eu/newsroom/article29/items/612053/en
Here's what it says about human intervention: "Any review must be carried out by someone who has the appropriate authority and capability to change the decision. The reviewer should undertake a thorough assessment of all the relevant data, including any additional information provided by the data subject."
However, I could make the case that losing an account which holds years of your private correspondence and is your point of contact for private exchange, services you rely on (including where bills, account recovery emails, policy changes, warnings & alerts, 2fa codes, and other very important messages are sent), potential employers or clients, and which doubles as a login for other services (see openid) and so on, can have a significant effect on your life and could potentially fall under "decisions that deny someone an employment opportunity or put them at a serious disadvantage" or (admittedly vague) "lead to the exclusion or discrimination of individuals."
Some of the other examples in the guidelines seem mild by comparison (e.g. getting a reduced limit on credit card).
My perspective is colored by both having lost access to an email account and also being denied a credit card application; the former was a much bigger problem.
1) Check whether the output of the machine learning model outputs Yes or No.
2) If yes, ban.
3) If no, no ban.
This is not a human review process. The review process is algorithmic, the human is only involved to relay the result.
That might be as simple as checking for the existence of legal documents claiming copyright infringement, or as reading a web page stating “we already removed X other copies of this file”.
Neither is a fail-safe way of doing such a review, but doing a thorough review might be expensive even for Google. Does anybody know how many such reviews they do each day?
It might also be a bug on their tooling to assist human reviewers.
Regardless -- and I know this is a "how the world should be, not how it is" type thing -- I really think the initial decision should not be allowed to be made by an algorithm. At the very most, an algorithm should be allowed to flag something for human review, but no action is taken until the human has a chance to review it and decide if the flag is warranted or not.
Also, just the absurdity that a human would review a file containing only "1" and decide the decision to flag it was correct.
https://twitter.com/googledrive/status/1486038872928792576
Problems shouldn't get fixed just because they got enough likes and reshares on Twitter.
Why would we believe them? It's Google's responsibility to prove their assertion, versus regulators taking them for their (not so good) word. The default should be the assumption that the corporation is being dishonest.
Can just mean some low-paid Amazon Mechanical Turk worker clicked on "Yes".
Lots of leeway for FAANG/BigCo management to wriggle out of that one. "Sure, Jones in Legal gets an email notification every time an account is banned and has the option to review it."
I can only imagine the lobbying and "negotiation" that takes place to have legislators water down the requirement for real human beings to review or respond to such bans.
> The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.
Emphasis mine. This would not include the vast majority of automated bans. It's more meant as a way to prevent e.g. automated police action via algorithmic selection.
While not tested by the courts, there is a plausible argument that "similarly significantly affects him or her" might apply to bans that impact your ability to earn a living. So streamers getting banned from YouTube, or AdWords bans for businesses where that's their main source of revenue. Bans that are lower-stakes than that get harder to justify under Article 22.
One example of a legal effect is cancellation of a contract. Examples of significant effect include automatic refusal of an online credit application, and e-recruiting practices without any human intervention.
Advertising is in scope too: "For example, someone known or likely to be in financial difficulties who is regularly targeted with high interest loans may sign up for these offers and potentially incur further debt."
Pricing is in scope too: "Automated decision-making that results in differential pricing based on personal data or personal characteristics could also have a significant effect if, for example, prohibitively high prices effectively bar someone from certain goods or services."
Finally, there's an example of profiling reducing a credit card limit. "This could mean that someone is deprived of opportunities based on the actions of others."
Anecdotally, getting kicked out of my email account has had far bigger effects on me than being rejected my credit card application.
https://ec.europa.eu/newsroom/article29/items/612053/en
> Paragraph 1 shall not apply if the decision...is necessary for...performance of, a contract between the data subject and a data controller
Which I can see applying as they probably have something in the ToS to enforce here.
It also allows automated decision making to comply with EU law. I don't know EU copyright law well enough, maybe Google has a responsibility to take down that data under copyright law and so this exception applies too.
The regulators are useless (especially the Irish one which seems happy to shield big tech scum from having to comply with the law) which confirms my own experience raising complaints with the ICO (the UK privacy regulator).
For example, the author you linked to is demanding a portable copy of all his personal data from all sources, which Facebook has no GDPR obligation to give him. He seems to have been misled by a form letter he found, which incorrectly conflates Article 15 data access (isn't required to be portable) and Article 20 data access (isn't required to include data that he didn't initially provide).
GDPR enforcement has been extremely lacking as demonstrated by the web being littered by non-compliant data processing consent forms. A compliant consent form should make the "decline" option as prominent as the "accept" one - the vast majority of services currently don't comply (including big names like Google or Facebook) and entire businesses such as TrustArc have been built on providing non-compliant consent forms as a service.
For GDPR enforcement to be considered serious, the fines amounts should be higher than the profits of companies built on abusing user data. If we look at https://www.enforcementtracker.com/?insights we can see that 1,6 billion euros has been handed out so far over a period of 4 years across the entire EU. How much does Google or Facebook profit in a year?
The entire experience of reporting violations is also a major problem and suggests the regulators (at least the UK one) aren't actually interested in enforcing the regulation. The process with the ICO requires that you first get in touch with the company and try to resolve your concern. This takes time & admin work on your behalf and a malicious actor can drag out the process for months. But let's assume that after you've done that and haven't gotten anywhere, escalating to the ICO merely results in them sending a letter. And when the company ignores that too, guess what happens? Another letter which they will promptly ignore too.
This sets the example that breaching the GDPR does pay, because not only reporting a violation requires so much commitment that the vast majority of people won't bother, but even once the violation is reported, the response from the ICO isn't actually an effective deterrent either.
Nothing in the text of the GDPR, nor of any regulatory guidance that I've seen, suggests that the "decline" option has any particular UI requirements beyond merely being present. Again, while I don't want to claim that this or any other regulatory process is perfect, I think the primary reason people in privacy circles find it so frustrating is that they keep trying to enforce things that aren't actually GDPR requirements.
https://ico.org.uk/for-organisations/guide-to-data-protectio... :
> Consent means offering individuals real choice and control. Genuine consent should put individuals in charge, build trust and engagement, and enhance your reputation.
> Consent requires a positive opt-in. Don’t use pre-ticked boxes or any other method of default consent.
> Be specific and ‘granular’ so that you get separate consent for separate things. Vague or blanket consent is not enough.
> Make it easy for people to withdraw consent and tell them how.
https://ico.org.uk/for-organisations/guide-to-data-protectio... :
> What is an unambiguous indication (by statement or clear affirmative action)?
> It must be obvious that the individual has consented, and what they have consented to. This requires more than just a confirmation that they have read terms and conditions – there must be a clear signal that they agree. If there is any room for doubt, it is not valid consent. [emphasis mine]
At this point you could already argue that unless the decline option is as prominent (if not more) than the accept option then the user didn't actually intend to consent and just couldn't figure out how to decline.
> Consent should be given by a clear affirmative act [...] Silence, pre-ticked boxes or inactivity should not therefore constitute consent.
> The key point is that all consent must be opt-in consent, ie a positive action or indication – there is no such thing as ‘opt-out consent’. Failure to opt out is not consent as it does not involve a clear affirmative act. You may not rely on silence, inactivity, default settings, pre-ticked boxes or your general terms and conditions, or seek to take advantage of inertia, inattention or default bias in any other way. All of these methods also involve ambiguity – and for consent to be valid it must be both unambiguous and affirmative. It must be clear that the individual deliberately and actively chose to consent. [emphasis mine]
Seems like that's cut and clear.
Don't get me wrong, I'd really like companies to design this way on principles of general user friendliness, but I don't see much evidence that anyone involved in creating the GDPR intended to require it.
Yeah, I definitely ignore that law, and I wish 100% of website owners did. It feels to me like 99% of them follow it.
Making your website worse is just a what certain analytics providers want you to do so you keep paying for their services.
https://github.blog/2020-12-17-no-cookie-for-you/
It's only worse for the user when the cookie notification is blocking the content, there is no "no, I don't agree" button or clicking it means clicking trough 100 extra toggles.
The DSA proposal also has language that appears to be intended to make such headers legally binding: "In order to avoid fatiguing recipients who refuse to consent, terminal equipment settings that signal an objection to processing of personal data should be respected."
One might be forgiven for assuming that the law was actually intending to accomplish the reverse of the stated goal. It gives site owners tons of explicit opt-ins that nobody can complain about, even though they were coerced.
DNT is utterly ignored to the point it’s officially deprecated in various browsers and the W3C working group for it disbanded:
https://en.wikipedia.org/wiki/Do_Not_Track
> It gives site owners tons of explicit opt-ins that nobody can complain about, even though they were coerced.
Coerced assent is expressly forbidden by GDPR, so we absolutely can and are complaining about this.
17 companies were fined for GDPR violations just this month. Last year, Amazon was fined €746,000,000, Google €150,000,000, Facebook €60,000,000.
https://www.enforcementtracker.com/
The 60M Facebook fine is a welcome development but my point still stands - how much did Facebook profit from breaching the regulation for the 4 years since it's been in effect? That fine should've had a few extra zeros at the end to actually serve its role, otherwise it's just a very small cost of doing business.
I could have and maybe should have just let it go, but it really got under my skin. I first tried out of band approaches to contacting somebody there. I didn't reach anybody, and you quickly realize how everybody else on the Internet just assumes you must either be lying or not telling the full story. Maybe it's just acceptable losses while doing business at scale.
So I finally just emailed them a polite GDPR request containing some spiel about Article 15(h), how I have the right to request my personal data, and also have the right to correct any inaccuracies in it, which must be the case since I committed no such fraudulent actions. I also requested a full list of all their data subprocessors, which I couldn't actually find listed anywhere on their site.
I'm not a lawyer, and I don't know if my request hit all the right notes or not. But literally one hour later, I got my account unlocked with a personal apology.
For what it's worth I also let them know that I'm not really looking to circumvent their systems, and I'm sure they have to deal with a lot of bad actors. But there really needs to be a better way to reach somebody to fix things when automated systems go wrong.
I also have the feeling that this approach would fall on deaf ears for big FAANGs, and there really needs to be some high profile ruling to put the fear in them.
I have observed the same. When I evaluate service providers, I'm curious to know how they handle dispute with customers.. it's quite depressing to see that on most online forums, it usually goes straight into victim blaming. You must have violated the TOS, you must be doing something sketchy, you're not telling the whole story, you're just holding a grudge so get over it, you're just entitled, etcetra. There's very little sympathy, and no giving benefit of the doubt.
> But literally one hour later, I got my account unlocked with a personal apology.
Congrats! This is a lovely anecdote, thank you so much for sharing.
And I'd note that I am very very certain that I've made mistakes that stupid over the years.
1) Hanlon's razor (do not attribute to malice that which can be explained by stupidity)
2) It was discovered (unfortunately missing citation atm :< ) that people who fall for spam scams that exploit gullibility do so "completely" - most see the scam for what it is, but the ones that do fall for it will kind of double down, defend their actions and see through their part of the "deal" because in their mind it Will Work. So it's almost like there's a super-thin line somewhere where everyone rubberbands to one or the other extreme ("are you serious, that's a scam" vs "are you serious, of course this is real") depending on whether That Last Single Piece Of Straw is on the haystack or not. (I just realize you could substitute "a scam" for "fake" and potentially explain a substantial percentage of conspiracy theorists... hmm)
Even hashes of your email address or payment data should be something you should be able to request they must delete.
If a person revokes consent for their personal data to be used, the data must be deleted if "there is no other legal ground for the processing". But if a data processor has an overriding "legitimate interest" in storing data about you, then they have legal grounds to do so without your consent. The details of this will vary depending on the situation (and the jurisdiction) but, for example, fraud prevention is explicitly called out as a legitimate interest.
https://law.stackexchange.com/questions/37882/google-adwords...
Mostly I’m scared of ‘multifactor’ where email access is considered a form of identity, but I’m not sure what else
I was trying to get my matchmaking data out of Activision Blizzard and they flat out refused, saying my data was their property
their exact response was:
> "the information requested are trade secret and/or intellectual property needed to preserve our game integrity"
I complained to the regulator, who agreed with my assessment, but to enforce it I'd have to go to court
seems the GDPR is basically useless
it's privacy theatre, nothing more
(thank god)
the data protection regulators have no ability to create rules... their job is (supposedly) to enforce it
1. Arguably your matchmaking data is someone else's as well. Meaning, they'd be potentially exposing other people's data to you.
2. Arguably you don't own the matchmaking data. You only own the initial request for matchmaking. The end result is actually a product of their proprietary algorithm. You didn't generate it.
Perhaps it might be a good idea getting in to touch with a privacy campaigner, or if the European equivalent of ACLU exists, and have them test this in court because it affects two different and important aspects.