> We tend to think of machines, in particular smart machines, as somehow cold, calculating and unbiased. We believe that self-driving cars will have no preference during life or death decisions between the driver and a random pedestrian. We trust that smart systems performing credit assessments will ignore everything except the genuinely impactful metrics, such as income and FICO scores. And we understand that learning systems will always converge on ground truth because unbiased algorithms drive them.
I absolutely hate this myth, and I suspect it's just an excuse used by technologists (consciously or not) to wash their hands clean of taking social responsibility.
There is no such thing as innocent technology, because it is made by and for humans, and with a purpose. Which means that before we even built anything we have already left the domain of "is" and entered "ought". Technology "encodes" values by being an answer to how we attempt to implement some intended purpose, and thus becomes entwined with the questions of responsibility that we all struggle with.
Here's a great anthology of writings on that topic: Inside the Politics of Technology[0], mostly written from a post-phenomenological[1] perspective.
Excerpt from the introduction:
> It was eventually proven that the pilot acted as capably as possible, so he was not to blame. But that only partly settled the question. After this first clarification, however important it was to the pilot, the accident could still be ascribed either to human error, e.g., false instructions from the control tower,
poor maintenance, or lack of knowledge about the migration cory patterns of birds, or, alternatively, to technical deficiencies, e.g., fuel problems or engine failure.
> In the end, a year after the crash, an official research report established the “real” cause of the crash: the snapping-off of a 10-cm metal pin that regulated the position of a fin in one of the 13 cogs in the rotor of the F-16. This set off a chain reaction demolishing the cogs one by one, ending up in a complete breakdown of the engine.
> But even then the problem of humanity versus technology was not solved. Who could be blamed for this technical defect: service engineers, the Air Force, or the manufacturer? The latter was ultimately left holding the bag. But still then, was it a production fault or a designer’s error? Again,different actors and different technicalities are involved.
> Apparently, a definitive dividing line between technical and human causes cannot be drawn. However technical the cause of the crash appeared to be, human beings always come along with the technicalities – and vice versa. Purely technical causes are just as illusory as purely human faults.
The idea of machines being unbiased and innocent is not just naive, but outright damaging.
[1] A friend of mine explained post-phenomenology as follows: "My understanding is that for Verbeek and others like him in general, postphenomenology continues the core traditions of phenomenology but with a deliberate emphasis on social science methods and empiricism over philosophical arguments - a positivist Heideggerian phenomenology"
Are there any other writings you could point me to? I've never heard of post-phenomenology, but I quite like the perspective. The concept of AI certainly isn't an unfamiliar one, so I imagine there'd be some interesting takes on that -- hopefully from a realistic, non-AGI perspective.
Hmm.. Well, if I'm completely honest I can't say I dove too deep into the material; this just happened to be part of the theory/philosophy that I had to study during my design bachelor/master that really resonated with me. But "What Things Do Philosophical Reflections on Technology, Agency, and Design" by Peter-Paul Verbeek is a touchpoint when it comes to post-phenomenology, AFAIK.
Here is a paper titled Automated Inference on Criminality using Face Images [0]. The system is trained on existing conviction records. Inheriting all the prejudices and biases of our current justice system.
Probably being a Uighur or Tibetan in their respective regions regions. Also faces might correspond to certain disadvantaged ethnicities or groups. Or they might even correspond to a genotype that is expressed as a phenotype, so a genetic basis. We have identified a certain gene that makes aggression much more likely. And apparently it's more prevalent in African Americans in the US than others. But even that has context. In certain countries beating someone up, i.e. physical assault, is glossed over. Crime is highly subjective in terms of what people are willing to put up with... anything more than thou shalt not kill is highly subjective, and even that's ignored by military all around the world all the time.
Good quote:
Indeed, that was an apt and true reply which was given to Alexander the Great by a pirate who had been seized. For when that king had asked the man what he meant by keeping hostile possession of the sea, he answered with bold pride, "What do you mean by seizing the whole earth; because I do it with a petty ship, I am called a robber, while you who does it with a great fleet are styled emperor".
There is a similar quote in one of Poul Anderson's Flandry novels.
"Crime is entirely a matter of degree. If you kill your neighbor and steal his property, you are a murderer and a thief. But if you gather together an army of lusty fellows in the name of honor and glory, kill a few million people, take their land, and hit up the survivors for taxes, then you are a conqueror and a hero, and your name goes into the history books."
It's scary that despite parts of their website appearing to be non-too-subtle satire, all the media evidence suggests they're dead serious about this...
The authors clearly need help marketing their research.
They should have marketed paper as "guiding face beautification for honest signal using discriminative(pun intended) classifier" [similar to this SIGGRAPH paper http://leyvand.com/beautification/]. That would have sold very well. People are uneasy with labeling faces as criminal but its fine to label unattractive/creepy <--> potentially criminal. They would even get a buzzfeed article or TedX Talk about guiding plastic surgery using AI.
Bias is in the initial conditions of the net. It's just not deliberately biased by the author. But it still has bias. "Closing your eyes" to the initial content/state of the net does not make the initial state go away.
I think we need to stop using the word "bias" and "prejudice". The term "bias" has a real, well-defined meaning in statistics, and both of these terms imply incorrect decisions are being systematically made.
Only one example in this article is actually about incorrect decisions. Most examples are about unbiased learning systems doing exactly what they are designed to do, but people other than the designers wishing they did different things.
For example, consider "conflicting goals bias". This isn't a bias - the system is maximizing clicks exactly as desired. It's just that some random third party wishes the system were actually trying to mitigate a nonexistent psychological problem (namely stereotype threat) instead [1].
What they call "similarity bias" is the same thing. The system is attempting to show people stories they like (and probably does a good job at it), but the author wishes instead that the selection of stories was closer to what a journalist might choose.
Another social "bias" - namely "redlining"/redundant encoding/etc is actually the elimination of statistical bias. A lot of input metrics - e.g. SAT score, FICO score, etc - are biased in favor of non-Asian minorities (and against Asians) [2] and machine learning algorithms designed to find hidden patterns discover this bias and fix it.
Conflating "someone is doing X but I wish they did Y" with bias is not useful. It's also not useful to conflate the elimination of socially desirable biases with introducing new biases.
Yeah, the word bias seems to be overloaded with many meanings. I actually wonder if there is any philosophical difference between the word "knowledge" and the word "bias".
But about machine learning... a system can learn from a set of outcomes which are the result of a "biased" system, that is to say tainted with an incorrect Bayesian prior which is not properly corrected, such as let's say courts in a racist society or whatever. It would learn the same bias. Because its goal is to maximize compatibility with the outcomes that the humans did. So it perpetuates those weights.
The problem is that we don't know whether the human decisions matched the reality. "Did the person commit the crime" for example. We might have to wait until more unbiased estimators for such activity come along, and throw away old historical data.
It's sort of like when Black-Scholes became a self-fulfilling prophecy for valuing derivatives, but only after it became widespread. The market started using Black-Scholes to value derivatives, so it became the best model to predict the value of derivatives. But until then, other models might have fit the historical data better.
Yeah, the word bias seems to be overloaded with many meanings. I actually wonder if there is any philosophical difference between the word "knowledge" and the word "bias".
If you interpret the word "bias" statistically - rather than the way innumerate reporters trying to sound profound use it - then it's pretty straightforward to do so.
"Knowledge" represents a fixed snapshot of your information about the world.
"Bias" represents a tendency for your knowledge to systematically differ from reality even as more data is gathered.
In programming terms, knowledge is the data in your DB at a fixe dtime. Bias is a tendency for my inserts to fail while yours succeed.
But about machine learning... a system can learn from a set of outcomes which are the result of a "biased" system, that is to say tainted with an incorrect Bayesian prior which is not properly corrected, such as let's say courts in a racist society or whatever. It would learn the same bias. Because its goal is to maximize compatibility with the outcomes that the humans did. So it perpetuates those weights.
If the goal of a system is to predict human behaviors, then it will in fact do that. That's not "bias", that's just building a system with the goal of matching human behavior and getting what you asked for.
However, if the inputs to your algorithm are biased predictors of the output your outputs can still be unbiased. The tendency of machine learning systems is to detect hidden patterns in the data; biased inputs are just another pattern.
I do think it's important for people to be more clear about the goals they have for these systems. Do you want them to be accurate regardless of anything else, or do you want them to accomplish particular social goals by deliberately bending the results? There's nothing necessarily wrong with the latter, though you will inevitably find once you're being clear about the latter that even people who are generally ideologically aligned with each other will discover they have different and mutually conflicting goals...
Anyhow, either way, until you have a clear specification about goals you can't determine whether you're accomplishing them. A vague goal of being "nondiscriminatory" doesn't cut it for computers... "nondiscriminatory" is a set of possibilities, not a unique specification. (And if you disagree about my claim it's a set, just try to sit down with one of the aforementioned ideologically-aligned people and try to hammer it out to a coding level of detail.)
I.e. every algorithm will be "discriminatory" unless you predictor is perfect (i.e. gets 100% of decisions right). It's just a question of which kind of discriminatory they are. The same is true of any human decision process.
Unfortunately this means that we will be subjected to a slew of innumerate reporters posting "XXX is discriminatory" articles, regardless of how decisions are made.
Given the findings on stereotype accuracy are some of the best-replicated in psychology, I've always found stereotype threat a humorous idea. It implies stereotypes are accurate because they trick us into perpetuating them. Can you think of a more absurd conspiracy theory? It's in keeping with the very popular notion that you can change reality with words, and thus is commonly said to be true in hopes of making it so.
> The term "bias" has a real, well-defined meaning in statistics, and both of these terms imply incorrect decisions are being systematically made.
Bias is not a word that was invented by statisticians[1], it's a word that statisticians adopted. It's just as legitimate to say that when a machine learning technique is trained with biased input that it produces biased output as it is to say that some particular machine learning method produces biased output from unbiased input. Either way, it's perfectly fine to say that the use of that technique results in bias even if that bias is in an aspect of the output that the algorithm wasn't selected to optimize.
It seems that what you're arguing is that the algorithm shouldn't be blamed in these cases, but in addition, you're declaring that there's no such thing as prejudice or bias, and implying that what the algorithms are doing is pulling some (uncomfortable for gatekeepers) platonic truth out of the ether (without any acknowledgement of the possibility of biased input.) Or more simply, you're using this article as an excuse to push a favorite "race realist" agenda which you use to justify a purist libertarianism (and which I agree it requires.)
The term "bias" and "prejudice" even outside statistics are defined by the dictionary as being about getting the wrong results: "preconceived opinion that is not based on reason or actual experience".
This agrees more or less with the statistical term, it's just less precise. That's also NOT what machine learning algorithms do.
If you want to declare an algorithm biased because it optimizes what it was designed to optimize rather than some random tangential goal, then the term has become meaningless. Similarly, my cell phone is broken because it can only make calls and is ineffective at pulling trains. A locomotive is also broken because it doesn't make tacos.
It seems that what you're arguing is that the algorithm shouldn't be blamed in these cases, but in addition, you're declaring that there's no such thing as prejudice or bias...what the algorithms are doing is pulling some (uncomfortable for gatekeepers) platonic truth out of the ether (without any acknowledgement of the possibility of biased input.)
I have no idea how you could possibly think I made this claim given that I explicitly listed two examples of algorithms which are biased. I then hinted at how algorithms can eliminate this bias. Consider rereading what I wrote.
Or more simply, you're using this article as an excuse to push a favorite "race realist" agenda which you use to justify a purist libertarianism (and which I agree it requires.)
I have no idea why you think "race realist" agendas require purist libertarianism, or vice versa. That's a random political tangent and totally unrelated to this conversation. I'm beginning to think you are seeking to derail the conversation rather than discussing statistics in good faith.
In the hopes that I've misunderstood you, I will nevertheless provide you with a good faith clarification. I made no normative claims regarding libertarianism or anything else. The only normative claim I've made is that we should not wrongfully apply an existing term (bias) to completely unrelated concepts (having undesired social outcomes).
> That's also NOT what machine learning algorithms do.
A machine learning algorithm run on non-representative input could be said to lack experience with the portion of input that isn't represented in the training set.
E.g. I think it's pretty obvious that machine learning algorithms exhibit far more vernacular-bias than purely logical reasoning techniques.
> I made this claim given that I explicitly listed two examples of algorithms which are biased.
Your parent is pointing out the very real possibility that technically-unbiased algorithms can produce vernacularly-biased outputs. I.e., that statistic's definition of "bias" does not sufficiently capture the notion of bias as it's used in the vernacular.
I don't think you've refuted that claim.
At the end of the day, your approach toward side-stepping the issue of vernacular bias is intellectually lazy. Instead of tackling the problem head on, you're hiding behind a mathematical object that happens to have the same name as the actual thing under discussion.
> I'm beginning to think you are seeking to derail the conversation rather than discussing statistics in good faith.
I think your characterization of bias in terms of statistic's technical definition is already skimming the edges of good faith. You know what the reporter means when they say "bias", and that's not the meaning that statisticians use in technical settings.
I know what the reporter means: "bias" refers to "outcomes the reporter dislikes and thinks he can generate clickbait with". I'm explicitly advocating against this practice and in favor of using clear and precise language instead.
At the end of the day, your approach toward side-stepping the issue of vernacular bias is intellectually lazy. Instead of tackling the problem head on, you're hiding behind a mathematical object that happens to have the same name as the actual thing under discussion.
I'm not side stepping the issue. I'm advocating in favor of describing the problem with clear language - once that's done we can begin useful discussion on how to address it.
In precise terms, what do you think the problem is?
> Your parent is pointing out the very real possibility that _technically-unbiased algorithms can produce vernacularly-biased outputs_. I.e., that statistic's definition of "bias" does not sufficiently capture the notion of bias as it's used in the vernacular.
> "bias" refers to "outcomes the reporter dislikes and thinks he can generate clickbait with"
That's quite a straw man for someone who claims others are trying to derail a conversation.
> I'm explicitly advocating against this practice and in favor of using clear and precise language instead.
The language you're suggesting masks the problem.
It's nice that statisticians came up with a mathematical object that happens to have a politically charged name, but the conflation you're making is not an argument, and doesn't address the underlying (inherently political) problem.
> In precise terms, what do you think the problem is?
The precise problem is that things like skin color etc. often have strong predictive power. We've recognized this unfortunate fact -- and its societal implications -- and decided to create laws and social norms that regulate (either restrict or sometimes even require) the use of these properties in certain high-impact decisions.
Unfortunately, algorithms sometimes use these features to make decisions in settings where a typical human wouldn't (or might even get into legal hot water if they did).
Notice that an algorithm can exhibit this behavior without being (statistically) biased.
Now, perhaps you think this problem isn't actually a problem. And you can certainly make the argument that any form of vernacular-discrimination should be OK as long as it doesn't exhibit statistical bias. But that's a separate conversation, and you can't defend that position by playing with definitions. And if you're not making that argument, then it should be clear why vernacular-bias differs from statistical-bias.
You and pessimizer have just demonstrated pretty clearly why Techcrunch's language choices are bad.
Pessimizer thinks the article is discussing inaccurate conclusions and that race realism is wrong ( https://news.ycombinator.com/item?id=13158797 ). You think race realism is accurate but we should ignore it. You both read the same article but drew totally different conclusions due to it's imprecise language.
Many examples in the article don't fit your description of the problem at all (e.g. difficulties in image processing, Tay the trolled chatbot, filter bubbles), so I don't think you are correctly describing what the article means by "bias" either. I can't figure it out either; the best I can come up with is "AI-related things that make the author have negative feelings".
I didn't argue in favor of or against your mysterious "vernacular-discrimination" - how could I when I don't even know what it is? The only thing that I argued is that the imprecise language used by Techcrunch is confusing people and that this is bad.
You seem to be missing a key point, which is that the definitions of racial bias are and have always been contentious and up for debate. The definitions have always been in flux and have evolved heavily over time. This is basically true for many politically contentious topics, as different parties repeatedly try to redefine and reframe terms to their advantage.
If you insist on precise language, then we basically can never have any conversation about race, racial bias, or any other topic where the terminology hasn't been completely nailed down.
The only thing I'm advocating for in this thread is clear thinking and clear communication - in particular I oppose overloading a perfectly well defined and clear statistical term.
Similarly, I advocate against new-age uses of the word "energy" - energy is a specific conserved quantity, not some mysterious unobservable thing that emanates from crystals.
What benefit do we gain when other people can't even understand the words we use?
> in particular I oppose overloading a perfectly well defined and clear statistical term.
The overloading happened the other way around...
> What benefit do we gain when other people can't even understand the words we use?
Similarly, what benefit do we gain when we use a term of art in settings where most of the audience doesn't have the technical background to be quite sure what they're even discussing?
Also, there is a notion that's distinct from statistical bias and that is highly political and controversial. Today we can say "bias" and most everyone will know we're talking about that topic. Whatever word or phrase we use in place of "bias" will be just as muddy and ill-defined. And probably some mathematician will come along and build some incomplete model and name if after that new word. And then we'll have this conversation again. The problem you're trying to avoid is unavoidable.
It's 100% clear from the article that 'bias' is in the political, or common sense of the word. It's an existing, established usage, and not one that the authors simply made up.
I fail to see why you insist that the authors use a word in its statistical sense when they clearly aren't making a statistical argument -- especially since the the statistical sense of the word bias is ultimately derived from the common usage, and not the other way around. The use of 'bias' in the sense of 'prejudice' in English dates from the 1570s, well before the development of statistics [1].
If anything, we should complain that the statisticians overloaded a perfectly good English word, instead of coming up with some Greek or Latin neologism for their own technical use.
> Similarly, I advocate against new-age uses of the word "energy" - energy is a specific conserved quantity,
Sounds like a stilted way to look at the world. If someone says 'I'm feeling really energetic today', would you object on the grounds that they haven't shown that their biochemical energy reserves are larger than normal, or that their body is burning calories at a faster rate than usual?
If you read your source, you'll realize the term "bias" referred to a slant in the floor of a ball game providing an advantage to one player. This slant would (I'm inferring) cause one player to have an advantage over another equally matched player depending on which side of the slant they are on.
This agrees perfectly with the statisticians definition, albeit applied to one particular example. Specifically, the slanted ball game field would provide a biased measurement (in the statistical sense) of player ability. Score 1 for the mathematicians.
I agree with you that this techcrunch reporter is using the term in some other mysterious "vernacular" way. It's this usage that I'm objecting to since it's misleading and confusing.
If someone says 'I'm feeling really energetic today', would you object on the grounds that they haven't shown that their biochemical energy reserves are larger than normal, or that their body is burning calories at a faster rate than usual?
If an article were discussing how coal is more "energetic" than nuclear energy, but was actually referring to energetic in the psychological sense (or maybe in the hippy energy crystal sense), I would in fact object. That would be similarly misleading.
> This agrees perfectly with the statisticians definition, albeit applied to one particular example. Specifically, the slanted ball game field would provide a biased measurement (in the statistical sense) of player ability. Score 1 for the mathematicians.
That's silly. It's not a competition between mathematicians and all other people for the 'correct', Platonic definition of a word -- a word is defined by how people use and interpret it. Logical, mathematical deduction from first principles is not the only way to look at the world.
Unless I'm mistaken, I think your objection can be summarized as 'we shouldn't talk about things that we don't have airtight definitions for (like racial bias or racial prejudice), because it might confuse people'.
I happen to strongly disagree, since grappling with fuzzy definitions is precisely how we get to having better definitions. Otherwise, we should give up on ever talking about anything except math, where we have clear axioms to work with.
Even in math, people were profitably doing calculus before we had clear definitions for infinitesimals, integrals, derivatives, or proofs that our procedures produced correct results. The logical machinery and deductive logic often comes after people spent time grappling with fuzzy, intuitive things with unclear definitions.
Unless I'm mistaken, I think your objection can be summarized as 'we shouldn't talk about things that we don't have airtight definitions for (like racial bias or racial prejudice), because it might confuse people'.
Specifically, the word "bias" makes virtually everyone think of one particular concept. I claim expanding the word "bias" to describe two separate concepts misleads people into thinking the second concept is the same as the first.
My sole objection is with confusing people by conflating multiple disparate concepts under one label, which is already a perfectly good label that clearly refers to something else (according to everything besides throwaway's mysterious and uncited "vernacular definition").
> You think race realism is accurate but we should ignore
No, I do not think race realism is accurate. I just think that race realism is not an adequate defense of racial discrimination, AND ALSO that race realism is inaccurate.
Pessimizer and I agree.
> Many examples in the article don't fit your description of the problem at all
You asked me what I think the problem is, not what I think the article's author thinks the problem is. The fact that I emphasize different things is not proof that I misunderstand the article's author.
> The only thing that I argued is that the imprecise language used by Techcrunch is confusing people and that this is bad.
Who, exactly, is supposed to be confused?
Subject matter experts should know the difference, and if they don't, that's really their own problem.
The public at large doesn't even know what statistical bias is, so why in the world would they be confused?
No matter how confusing the current state of affairs is, it would be far more confusing if suddenly everyone starting using bias to mean statistical bias. Mostly because 99.9% of the people using the term wouldn't have an adequate grasp of mathematics to even know to which object they are referring. So if your priority is clarity, then keeping around the messy moralized and politicized vernacular notion of bias is far preferable.
> I didn't argue in favor of or against your mysterious "vernacular-discrimination" - how could I when I don't even know what it is?
Not everything in the world can be mathematically formalized. Most things don't even have fixed meaning over time. If your criticism is genuinely about imprecision, you've got a lot of much lower hanging fruit than "discrimination" and "bias"...
The precise problem is that things like skin color etc. often have strong predictive power...Notice that an algorithm can exhibit this behavior without being (statistically) biased. - throwaway729, 2 hours ago https://news.ycombinator.com/item?id=13159539
The fact that I emphasize different things is not proof that I misunderstand the article's author.
Fine - what do you believe the author thinks "bias" means? I.e. what is the underlying concept behind all his examples?
Who, exactly, is supposed to be confused?
Pessimizer, for one, who thinks the problem is inaccurate predictions. 2 hours ago you disagreed with this, but 22 minutes ago you agreed. I'm getting kind of confused.
If your criticism is genuinely about imprecision, you've got a lot of much lower hanging fruit than "discrimination" and "bias"...
Sure, but this is my field. I give talks about misuse of p-values and the multiple comparison problem, even though there might be other issues worth worrying about. It's just what interests me. The fact that bigger problems exist doesn't mean we shouldn't fix this one.
I take race realism to be the sort of thing that posits an inherent relationship between skin color/other physical features, and other innate/intrinsic/genetic properties such as intelligence or criminality -- especially as a prior and when the trait is normative and negative. So "dark skinned people are less intelligent" is race realism, but "dark skinned people are treated poorly by the political establishment" is not race realism. (We're far afield, I don't mean for this thread to devolve into a definition or debate about race realism -- I'm just saying what I take race realism to mean.)
The important point isn't a precise characterization. Rather, the important point is this:
It's possible for race to be predictive without there being inherent differences between people of different physical attributes. Setup a system in which I take away all green people's life savings at the end of the day with probability .99 if they don't have a house and .5 if they do have a house. Both numbers are .2 for purple people. An unbiased lender can choose not to lend to green people without exhibiting statistical bias -- in fact, color has huge predictive power.
But that predictive power comes from the social system we setup specifically to screw green people, not from inherent differences that are causally related to color posited by race realism.
But this is an extremely obvious observation, so perhaps you meant something different by race realism? In any case, I don't think the disagreement you posited really exists.
The rest of the points you're making are covered elsewhere.
I deal with Real numbers every day, but I don't barge into discussions of epistemology and demand my axioms be used in discussions of a vaguely related but definitely different topic. And I'm careful in pop lectures to point out that Real numbers are just a mathematical object and no more more "real" than natural numbers.
If people in your statistics lectures don't know what bias means, then you should account for this in your teaching and lecturing. I know my statistics lecturer did.
When there exist two sets of meanings for a word -- one vernacular, the other technical -- we shouldn't try to insist that the technical meaning over-ride the vernacular meaning. That's almost always a losing battle.
Insisting that a system isn't biased in the vernacular sense due to a particular technical notion of bias completely misses the point. And we almost never have enough understanding about the world to make a definitive case that the system isn't biased in the vernacular sense.
My claim is, however, that when discussing statistics we should stick to the statistics terms. If we want to discuss a new concept then we should make up a new word for it.
To do otherwise risks having people get confused as to what underlying concepts are actually being discussed. For instance, go read the HN comments here: https://news.ycombinator.com/item?id=13004790 Why risk such confusion when we can just use two different words for two different concepts?
However, you are also wrong about the vernacular sense, at least if the dictionary is to be believed:
Bias: prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair.
Prejudice: preconceived opinion that is not based on reason or actual experience.
This are just informal versions of the statistical term.
To translate this to another topic, if we are discussing global warming I don't think we should use the term "warm" to reference "feelings in my heart". Why would we want to confuse whether we are discussing the temperature of the earth or my feelings?
If this were a conversation about a statistics paper or textbook, or a particular ML algorithm, you might have a point.
But we're not discussing statistics. We're discussing the social impact of ML algorithms.
> Why risk such confusion when we can just use two different words for two different concepts?
The problem with this is that we'll always have that confusion because most people aren't statistics experts.
So it's up to the domain experts to know when one meaning or the other is intended. That's true in every domain.
> However, you are also wrong about the vernacular sense, at least if the dictionary is to be believed
Dictionary definitions don't fully capture the vernacular meaning of words -- especially politically charged words such as "bias".
> This are just informal versions of the statistical term.
But when real people use the term bias in real conversations, they aren't typically approximating the statistical meaning. Rather, they are typically referring to some a priori ethical limitations on decision making.
For example, most people would say that taking skin color into account when approving a housing loan is a form of bias. But they don't mean that in any particularly technical way. There absolutely is moral and political content in the term "bias".
For example, when equal lending laws were passed, an unbiased algorithm definitely would've considered race when choosing the outcome of a loan application. African Americans were systematically discriminated against in employment decisions, and often the first to be let go when hard times hit. So a completely unbiased loan officer could come to the statistically valid conclusion that people of color were far more likely to default. None-the-less, a vase majority of people today would agree that the loan officer was exhibiting racial bias. And they're not coming to that conclusion due to innumeracy or some vacuous personal preference. They're perceiving a vast network of causal relationships between decisions that reinforce a systematic bias, even when many of those decisions aren't locally statistically biased
> I don't think we should use the term "warm" to reference "feelings in my heart".
And I agree, but I agree because non-experts know and understand the difference. And because the term "global warming" with the alternative meaning isn't already in wide-spread vernacular use.
> But when real people use the term bias in real conversations, they aren't typically approximating the statistical meaning. Rather, they are typically referring to some a priori ethical limitations on decision making.
What many common people mean by bias is approximately the statistical meaning. For example, if the algorithm accurately predicts the number of white defendants who fail to appear in court but overestimates the number of black defendants who fail to appear, people in common usage will call that bias. Which it is, in the statistical sense.
But if a higher percentage of black defendants actually fail to appear than white defendants, and the algorithm accurately predicts that outcome, that isn't bias in the statistical sense. And the people who try to conflate those things are doing this:
The second thing is not only not bias, it's not morally objectionable. Statistical bias is the thing which is morally objectionable. Calling something else "bias" is not just inaccurate, it leads directly to a false moral conclusion.
> What many common people mean by bias is approximately the statistical meaning.
And many people don't. The difference is that I'm not asserting I have the one true definition, but rather recognizing that "bias" is a term with inherent political and moral content. And that pretending that this isn't (or shouldn't be) the case just because statisticians use the term in a technical setting is extraordinarily intellectually lazy.
Most people are not statisticians. No serious mathematician has ever asserted that the axioms of Real numbers settle all epistemological debates. Like-wise, I see no reason why a statistical term of art should be used to clear up the various denotations and connotations surrounding "bias" and "discrimination".
> that isn't bias in the statistical sense.
That depends on what the goal of the algorithm is.
If the goal is to accurately predict whether people will appear, then it's working.
But if the goal is to accurately predict when people will choose not to show up, then there's certainly a chance that it's not working. For example, in the case where there's some significant correlation between race and access to transportation infrastructure. (Which isn't so much the case today, but certainly was 60 or so years ago.)
The distinction between choice and circumstance isn't immaterial -- the latter is not a flight risk even if they gum up the legal system via delays. So if the judge is interpreting "90% won't show" as "90% flight risk" in the above hypothetical world, then there's a huge problem. You could argue that this is the judge's problem, and I won't necessarily disagree. But UI bugs are still bugs, and this particular form of UI bug is pretty specific to ML systems.
> And the people who try to conflate those things are doing this
Or, perhaps they're far better at mathematical modeling of complex systems than you give them credit for; e.g., by seeing ways in which locally statistically unbiased decisions can magnify the impact of bias that exists in other parts of a system.
We will never have a complete mathematical model of the world and the history of human civilization, so I don't see any reason why we should insist that all bias needs to be what a statistician calls bias in his workplace.
> The second thing is not only not bias, it's not morally objectionable
This is a separate discussion I don't want to have in this thread, because we're already barely on-topic.
But suffice it to say that conflating technical and non-technical definitions is not a reasonable defense of this position.
> And many people don't. The difference is that I'm not asserting I have the one true definition, but rather recognizing that "bias" is a term with inherent political and moral content. And that pretending that this isn't (or shouldn't be) the case just because statisticians use the term in a technical setting is extraordinarily intellectually lazy.
Your assertion seems to be that not using the same word to describe separate and distinguishable things in a context where the distinction matters is intellectually lazy.
> If the goal is to accurately predict whether people will appear, then it's working.
> But if the goal is to accurately predict when people will choose not to show up, then there's certainly a chance that it's not working. For example, in the case where there's some significant correlation between race and access to transportation infrastructure. (Which isn't so much the case today, but certainly was 60 or so years ago.)
The goal is to predict whether people will appear. The bail-bondsman has to know whether to issue the bail-bond.
If your issue is transportation infrastructure then the only way to solve it is by fixing the transportation infrastructure, not by bankrupting the bail-bondsman.
> This is a separate discussion I don't want to have.
> But suffice it to say that conflating technical and non-technical definitions is not a reasonable defense of this position.
That's not the point -- the problem is that conflating technical and non-technical definitions is a way to avoid having that discussion.
Effectively everyone agrees that statistical bias is wrong. If you're on the wrong end of it you're being treated unfairly. If you can convince someone that something is this "bias" when everyone agrees bias is wrong then you've convinced them that the thing is wrong.
But if all you've done is broaden the scope of what bias means to encompass something that not everyone agrees is wrong then you've accomplished nothing of virtue. And more than likely misled some number of people who haven't managed to piece together that the "bias" they understand to be wrong and the "bias" you understand to be present are non-overlapping.
Effectively everyone agrees that statistical bias is wrong. If you're on the wrong end of it you're being treated unfairly.
That's not quite true. An unbiased credit scoring algorithm would penalize blacks, subtracting 50 pts or so from their FICO score.
Is it fair to reject a loan because of the ethnic background of the applicant? That seems unfair to me.
Then again forcing a lender to give out loans that he knows will default also seems unfair, to say nothing of the higher interest rates that Asians must pay in order to subsidize deadbeat blacks.
I don't quite know what "fair" means here. What I do know is that the term "bias" is used to mislead; to hide the fact that whatever choice we make, there is a cost to be paid.
> Is it fair to reject a loan because of the ethnic background of the applicant? That seems unfair to me.
I think you're drawing the wrong conclusion from this. Unless we think that black people are inherently less credit worthy (i.e. are genetically inferior), then what's really happening is that race is acting as a proxy for some other things that aren't being considered. Maybe living in a neighborhood with drugs and gang violence, or lack of social ties to more affluent people who could be leaned on in tough times, etc.
So we have a bias, which we know is wrong, and we have two ways to fix it. One is the cheater's way out, which is to just take race directly into account. The other is to find the set of actually-causative factors that correlate with race which if you took them into account would make it so that nothing more would be gained by also taking race into account, and use those instead.
And given the choice between those two ways of solving the wrongful statistical bias, the second one is clearly preferred. Predicting via causative factors is fairer than predicting via correlative factors -- it's more accurate. You don't introduce a new bias against the black small business owner who doesn't live in a bad neighborhood and does have financially-advantageous social ties.
This is true of all predictions based on correlations, but there is a resource trade-off here. It may be more economical to collect data on things that correlate than things that cause. And causation is more predictive than correlation, but correlation is more predictive than nothing.
We can't pretend there aren't political considerations here. We're going to spend more resources to use causative factors instead of using race, for entirely political reasons, when we might not for some other factor. But the unfairness isn't to not use race (assuming we do actually find and use the underlying factors instead), it's only that we have to use more than the typical resources in order to get a more accurate result without using race. Which is a necessary evil given the politics.
But just leaving the known bias without finding and using the underlying factors to mitigate it is clearly wrong.
I think you're drawing the wrong conclusion from this. Unless we think that black people are inherently less credit worthy (i.e. are genetically inferior), then what's really happening is that race is acting as a proxy for some other things that aren't being considered.
"Inherent" is always with respect to a particular data set - if there is no data in the data set that is more predictive than a given factor, you might as well treat that as inherent when doing the data analysis - it won't change your algorithm at all.
Maybe living in a neighborhood with drugs and gang violence, or lack of social ties to more affluent people who could be leaned on in tough times, etc.
Could be. However, due to housing separation, this will also be correlated with race. Regulators consider using factors like this to be "redlining" and strongly frown on it. (Historically this was not true; in fact regulators drew the original red line.)
The other is to find the set of actually-causative factors that correlate with race which if you took them into account would make it so that nothing more would be gained by also taking race into account, and use those instead.
That certainly might be possible. If you can solve this, and those actually-causative factors do not redundantly encode race (e.g. no redlining), you have built the next unicorn startup. Your customers will be every single bank in America.
> "Inherent" is always with respect to a particular data set
I meant it with respect to the hypothetical data set containing all possible data. Either there is a quite large disparity in the data which is actually caused by race and so can't be eliminated without considering it, or it's really mostly or entirely caused by some other factors that only correlate with race which we can hypothetically find and use instead.
> Could be. However, due to housing separation, this will also be correlated with race. Regulators consider using factors like this to be "redlining" and strongly frown on it. (Historically this was not true; in fact regulators drew the original red line.)
Traditional redlining was unconditionally denying loans for properties in black neighborhoods. Weighting neighborhoods based on their actual repayment history is something else entirely.
> That certainly might be possible. If you can solve this, and those actually-causative factors do not redundantly encode race (e.g. no redlining), you have built the next unicorn startup. Your customers will be every single bank in America.
Any factor that reduces racial bias is obviously going to correlate at least somewhat with race. It's mathematically necessary. But you can easily tell if a factor does more than redundantly encode race if considering race and that factor predicts more accurately than only considering race.
Determining what the factors actually are would require having the data, which I don't have.
> to say nothing of the higher interest rates that Asians must pay in order to subsidize deadbeat blacks
When I read this kind of thing in your comments it makes me wonder whether you aren't trying to smuggle in inflammatory phrases that you don't need, just to get away with it. Pejorative+race seems like such a troll trick to me (and there are others). If there are "deadbeats" being subsidized they would obviously not all be of one flavor.
I'm using "deadbeat" to refer to people who don't pay back their loans. That's both the literal dictionary use of the term and also a common term in the industry. Why do you consider it "inflammatory"?
Dang, I understand that you get a lot of flags whenever I point out unpleasant facts. I'm sorry if this bothers you. Should I just accept that HN is no longer a place where uncomfortable realities can be discussed, and move on?
If there are "deadbeats" being subsidized they would obviously not all be of one flavor.
They are disproportionately and predictably of one flavor. That's the entire reason this is a non-simple issue.
Based on eyeballing the graph, consider a group which is 50% black and 50% Asian. Every person has a 600 FICO score. Blacks with a 600 FICO score have about a 40% default rate while Asians have about a 20% default rate.
To lend $100 to each member of this group at a fixed interest rate (i.e. we ignore race) and break even we need to charge a 43% interest rate to the mixed group.
The average Asian will pay back $114.40, while the average black will pay back $85.8.
Defaulters come in both flavors - in a group of 100 blacks and 100 Asians, there are 20 Asian defaulters and 40 black defaulters. So the 80 Asians who are not defaulters (using a more awkward word since you object to "deadbeat") are paying higher interest rates to make up for predictable losses caused by the 40 black defaulters.
If we did not bar the use of racial information then we'd charge blacks a 66% interest rate and Asians a 25% interest rate. This would be statistically unbiased (i.e. it removes omitted variable bias https://en.wikipedia.org/wiki/Omitted-variable_bias ) and would not result in a predictable transfer of wealth from Asians to blacks. Individually this does seem unfair.
I am expressing no opinion on which policy is better. I'm pointing out that whatever we do will be in some sense unfair. The only position I'm pushing is numeracy and clear thinking that acknowledges the very real tradeoffs.
(To discuss examples in the article, one could repeat using similar calculations with ad delivery or recidivism. I just go with lending because it's mathematically equivalent and there's lots of published work on it that I can cite.)
Can I ask why you've opted to compare Asians and Blacks (thus maximizing the gap, without noting that the plurality population of White people fall inside that gap) and why you've opted to make your comparison at 600, a point that again works to increase the gap, while the paper itself pointedly focuses on the 620 prime-rate cutoff at which the "Black" default rate change sharply changes while the "Asian" default rate starts leveling off?
That's the fine art of eyeballing a graph; choose numbers that correspond to the ticks on the graph so you have an obvious place to put the ruler.
If you choose a different FICO cutoff or whites instead of Asians, the result remains directionally the same. You might get whites paying $110 vs blacks paying $90 (the gap narrows as FICO goes way up) but the same basic point arises.
Feel free to play with the numbers yourself if you think my arithmetic is wrong or nonrepresentative. If you find something I'd love to see your corrections.
A wise person once said this: "Which numbers do you disagree with? What do you think the more appropriate number would be? Change the numbers and rerun the script, so you can see if your argument even matters."
What happened to that guy? I miss him. He made HN a better place.
I didn't find black default rates at FICO=620, maybe you have sharper eyes than me. Feel free to change the numbers and redo the calculation. If you get something substantially different, lets discuss.
[edit, in response to your edit: Asians to whites is an equivalent comparison and the result is the same. Asian borrowers subsidize white deadbeats.]
He discovered that a lot of the time, the people whose calm rational-seeming posts he was occasionally sticking up for were really just taking pleasure in needling HN with edgy race and gender baiting. It was a disappointment.
The 620 default rate is on the very next page after the one you cited.
Chris, this is another thing you do. I tried to word my questions carefully, so as not to dismissively score points off you on this buried thread nobody is paying attention to you. It was you that personalized this, with your "where did that person go" rhetoric.
You asked a question --- orthogonal to my questions, which you really still haven't answered --- and then feigned offense when I answered it. If you didn't want to talk about "emotions", well, neither did I. Why'd you bring them up?
Here is another way to read the questions I asked: "perhaps the answers to these questions explain why a normal reader might snag on the way you worded your previous comment and have to ask themselves whether they were taking race-troll bait". Not: "Chris should make different points", but rather, "here are some tools Chris might think about using to avoid giving the impression that he's race-trolling."
So again, I'd ask:
* Why go through extra effort to find a data point less favorable to black people than the one the document already presents?
* If the point is that there are public policy debates to be had about fairness versus demographic blinding, why pick on black people in the first place? A comparison between Asians and whites makes the same point, without requiring you to ever write the words "deadbeat blacks".
Since you seem to be attempting to discuss things honestly, I'll give you my reply.
Answer 1: The document doesn't provide the number you claim it does, or if it does I still can't find it. Further, that number wouldn't substantively change my point and I'm pretty sure you know this.
Answer 2: Mainly because we all know a major chunk of the actual debate is about black people, and I don't see a reason to obscure what we are really discussing. But I freely acknowledge that you can get a similar gap between whites and Asians, particularly around FICO=500.
I'm very deliberately trying NOT to obscure the real issue, namely that you can't have both fairness and accuracy. You can't even have multiple kinds of fairness - pick one and you lose another. Unfortunately that's just how the math comes out.
Also, I do believe that in the contemporary US Asian people are directly discriminated against more than anyone else. They are also the highest performing on a variety of common metrics (academics, income, financial responsibility) and this is primarily due to difficult to change traits (some combination of culture and biology). Just mentioning it explicitly if you thought I was trying to obscure this point.
It's not my contention that the 600 number would substantively change your point. It's my contention that you went through extra effort to find a data point that put black people in a worse light. I was curious why you would have spent the effort to do that.
Meanwhile, you were ostensibly making a level-headed argument about how there may be tradeoffs between discrimination and fairness; that to entirely eliminate discrimination we mathematically might be required to force some (say) ethnicities subsidize others. You could have made that point in any number of ways without using the words "deadbeat blacks" --- as I pointed out, the same point was available to you in a comparison between whites and Asians.
As you can see from the thread, the objection wasn't to the underlying point you were trying to make, but to the way you chose to make it. Simply put: a reasonable reader could come to the conclusion that you were not only trying to make a point about a tension between non-discrimination and fairness/accuracy, but also trying to use that argument as a device to smuggle in an ethnic slur.
I don't think you're a racist, by the way. I think something else is going on with you. Maybe I'm wrong. But as you are to statistics, I am to the oeuvre of 'yummyfajitas comments on Hacker News: they are an area of my expertise.
My contention is that you are confusing yourself, and they don't really provide P(default|black, 620) and P(default|white,620) in the paper. The only number I could find is P(default|620)=82% which is not enough for my calculation.
I choose not to couch my facts in language meant to obscure the point, or avoid choosing factually correct examples that again hide the unpleasant bits of reality. We won't figure out the right way to deal with reality if we pretend the ugly bits aren't really there.
I don't think you're a racist, by the way. I think something else is going on with you.
It's very simple. On values I'm mainly an old school left wing individualist - I view ethnicity as holding no moral weight and groups as having no inherent rights that they don't inherit from their individual members.
Unfortunately, on racial issues, the white supremacists seem to be factually correct (mostly). The current left wing approach to this is to shame and ostracize anyone who dares to point out the emperor has no clothes. I originally interpreted your comments to be an attempt in that direction.
Meanwhile, UKIP/scary Hungarians/etc are pointing out the lies and people are following them.
I.e. I favor intellectualism and analytical philosophy over dogma and (secularized) religion.
Perhaps HN isn't the best place to persistently litigate the validity, merits and implications of The Bell Curve, which is the direction that many of your arguments (this thread among them) seem to take.
Of course, The Bell Curve, and the arguments contained within are a best assumption as to what "on racial issues, the white supremacists seem to be factually correct (mostly)" is referring to.
Also, you tend to fall back on claiming to be a harbinger of unadulterated knowledge ("I choose not to couch my facts in language meant to obscure the point") while ignoring the implicit (or explicit, https://news.ycombinator.com/item?id=13170988 ) invective contained in your language. I see no loss of descriptive accuracy w.r.t. the article between "deadbeat blacks" and "African-American defaulters", yet using the latter avoids the minefield of historical connotations that the former has.
You are either missing, or deliberately ignoring the part where the word choice for your arguments matter.
Surely you know that, in current discourse, individually the word "deadbeat" means something, and the word "blacks" means something, but the combined "deadbeat blacks" means something much greater than the sum of its parts, because of the societal context in which it is uttered.
To claim that you are just using that combination as just the exact sum of its parts is disingenuous, especially from someone who claims to favor analytical philosophy. Go read Paul Grice.
I realize that people often choose to read more into their words than should be read. I'm explicitly arguing against this. The "social context" means that there are ideas we can't discuss without being socially lumped in with deplorable people - for instance the fact that blacks don't pay back their loans as much as whites, that they are more likely to be deadbeats/defaulters/etc. (As tptacek noted, there is no similar stigma about observing that whites are deadbeats relative to Asians.)
This creates a significant level of friction on one side of a discussion, and makes it harder for us to come to truth.
This also means that the only people who do discuss these issues are the deplorable folks. I explicitly want to reclaim the unpleasant facts so that the deplorable folks don't have a monopoly on truth.
I want to say "we should import hundreds of thousands of Syrian refugees, a few hundred white girls will be raped as a result, and tens of thousands of lives will be saved. Good tradeoff." I want to say "black on white crime is the most common form of interracial violence, but we should still treat black people as equal citizens with civil rights. However our expectations of equal levels of police violence against blacks and whites might be unreasonable."
Lying/shaming/attacking the people speaking the truth isn't sustainable. When you lie to people and they notice the truth, they start to turn to turn to UKIP/Jobbik/other such folks. As a globalist and a universalist I don't want to live in that world.
I know you realize that every street corner preacher, PizzaGate activist, and Zero Hedge writer uses the same line about how they're being punished for speaking the truth. Why is your special pleading so much more interesting than theirs?
Getting to truth is important. However, the way you want to go about it makes it very expensive for your audience.
You want to say things that are true in a socially unacceptable way, and then leave it to us to do the work of questioning you extensively to figure out whether you are someone who is well versed in statistics and meant "deadbeat" "blacks", or if you are a Stormfront-style white supremacist and meant "deadbeat blacks".
This thread is an example of that. If you want to be an effective persuader, you have to talk at your audience's level. If you want a giant thread on HN where your audience walks away believing exactly what they believed before they encountered you, keep doing what you are doing.
> That's not the point -- the problem is that conflating technical and non-technical definitions is a way to avoid having that discussion.
On this we can 100% agree.
Choosing "bias" to mean "statistical bias", and then actively ignoring anything else that is currently called bias, is an extremely political choice.
Are you seriously asserting that naming clashes validate political opinions, or that some large group of people's political opinions would suddenly change if only they knew the statistical meaning of bias?
No one said we should ignore this other concept. All I said is that we should name it something else.
Are you seriously asserting that naming clashes validate political opinions, or that some large group of people's political opinions would suddenly change if only they knew the statistical meaning of bias?
Yes, I think this use of "bias" is meant to mislead. Specifically, I think the term "bias" is used to make people think machines are drawing incorrect conclusions (that's what "bias" means in the dictionary and in most common uses of the term).
There are two separate concepts:
1) An algorithm refuses to issue loans to blacks even though those blacks would pay the loans back.
2) An algorithm refuses to issue loans to blacks because they would not pay the loans back.
You want to use the term "bias" to describe both situations; I think this will mislead people. Specifically most people will think "bias" means we live in situation (1). I want to use "bias" to refer to (1) and to use a separate term to refer to (2).
Replace the word 'algorithm' with 'person' or 'institution'.
The politics around racial discrimination and bias means that in reality, few people do the work to tease apart (1) and (2). In particular, I think many people are extremely uncomfortable with the consequences that (2) being true entail.
Hence 'racial bias' usually has a simpler meaning: something is biased if the decision conditioned on race is not identical to the unconditional decision. That encompasses both (1) and (2).
You seem to ascribe some sort of malicious motive to the authors, when what you really have a problem with is how the country talks about race and racial bias in general.
The point is that in general, statistically unbiased algorithms will not lead to racially unbiased (in the strict conditional outcome = unconditional outcome sense) results. You might think this is completely obvious as a practitioner in the field, but it is far from obvious to the general public. Hence why it's fair to point that out to the public.
(edit: Either confusion isn't a real problem, or else the confusion is inevitable and some new term will become the politically charged one with lots of contested meanings. In either case, the correct solution from your perspective should be more statistics education for more people rather than trying to change terminology.)
> All I said is that we should name it something else
As I said elsewhere, some mathematician would come up with a concept, name it after that new thing, and then we're back here all over again because "bias" is inherently moral and political and has shades of meaning informed by moral and political opinion.
You won't have solved the precision problem, or even the overloaded terms problem.
> Specifically, I think the term "bias" is used to make people think machines are drawing incorrect conclusions.. I think this will mislead people
I agree "bias" is a politically powerful word.
I don't agree, however, that people fail to know and understand the difference between (1) and (2).
And I also think all people know that the term "bias" could mean both, and manage to communicate with one another using the full range of their vocabulary whenever it's unclear what's being discussed. See e.g. internet discussions around stop and frisk or WoD or racial profiling at airports.
People understand the difference between (1) and (2), even when bias is used to refer to both.
I might be wrong about that. A far more important prediction: I don't think there's anyone -- on either side of the debate -- whose position can be explained by their misunderstanding of the statistical meaning of "bias".
People understand the statistical meaning (or not), and call (2) bias regardless. Because they aren't referring to a mathematical object, they're referring to the effect something has within a larger social and political context. A context that cannot be perfectly modeled, and so is necessarily vague and difficult to pin down.
The fact that "bias" can have both meanings isn't a real problem. People are good communicators and, when they want, can easily distinguish between (1) and (2).
Saying that (2) is "not bias" is definitely a political statement when taking the common definition of bias. People are free to agree or disagree with that statement, but arguing via way of naming clash isn't a compelling argument (not saying you're making that argument, but GP is I think).
> Why do you want to conflate these two situations?
I'm perplexed by your examples (1) and (2) since the article doesn't actually discuss loaning to defaulting black people in any way shape or form. But what "separate term" would you consider to be more appropriate to describe ML systems' tendency to generate results unrepresentative of wider reality as a result of the characteristics of the subset it's trained on or inadequacies in its specification, which is what the article actually discusses?
> I'm perplexed by your examples (1) and (2) since the article doesn't actually discuss loaning to defaulting black people in any way shape or form.
It's an example. The article mentions calculating recidivism rates to determine parole, which is a similar problem if you prefer that one.
> But what "separate term" would you consider to be more appropriate to describe ML systems' tendency to generate results unrepresentative of wider reality as a result of the characteristics of the subset it's trained on or inadequacies in its specification, which is what the article actually discusses?
Most of the examples in the article have nothing to do with "bias" and are really the principal-agent problem. Misalignment of incentives.
Some website wants to maximize clicks rather than act in the user's interest, they create a machine to maximize clicks, the machine maximizes clicks. It's doing what they wanted it to do -- it's just not doing what you wanted it to do.
This category of problem has almost nothing to do with ML in particular. It's like identifying that charlatans often wear cotton and writing an article on the link between cotton and charlatans.
> This category of problem has almost nothing to do with ML in particular.
The article is identifying a particular misconception that people might have, which is because a computer did it, the result will be (racially) unbiased.
It might be obvious to you that's not the case, since you're familiar with the ML techniques in question, but still valuable to a lay audience to make that point crystal clear.
News algorithms tending to classifies stories as relevant reading based on relatively crude analysis of word frequencies and headline clickthroughs (as opposed to, say, whether the story added additional information or an alternative perspective, both of which would typically be negatively weighted by such an algorithm) is much more an ML problem than a principal-agent problem, unless one believes that "maximising clicks" is literally all anyone applying this technique aims to do, and that's the only reason why ML systems tend to optimise for simple goals like click-maximization rather than complex ones like breadth and salience of relevant information
There are obviously still unsolved problems in ML. But Facebook has billions of dollars. It isn't that they couldn't solve the problem if they wanted to. Yet they haven't.
I'm perplexed by your examples (1) and (2) since the article doesn't actually discuss loaning to defaulting black people in any way shape or form.
Sorry, s/loan/allow out on bail/ or s/loan/show advertisement/ to go with examples in the article. Mathematically it's the same problem. I just default to thinking about loans since I've worked on loans and since most of the academic work on the problem is about lending.
But what "separate term" would you consider to be more appropriate to describe ML systems' tendency to generate results unrepresentative of wider reality...
Thanks - you've perfectly illustrated my point as to why the term "bias" is misleading. The term "bias" leads you to believe the algorithms are generating results unrepresentative of wider reality, which simply isn't the case.
> The term "bias" leads you to believe the algorithms are generating results unrepresentative of wider reality, which simply isn't the case.
You can make a strong argument that, for example, the FB news feed algorithm leads to results that are unrepresentative of reality, because its input data (user clicks and likes) do not sufficiently represent reality accurately.
Or from the article: "Nikon’s confusion about Asian faces and HP’s skin tone issues in their face recognition software both seem to be the product of learning from skewed example sets." Nowhere here is the algorithm being blamed, but we have an example of results that at least some people find objectionable due to patterns in the input data.
Just to take a particular example, the algorithm's output has marked someone as blinking when they aren't, they're just Asian. That's a particular example of results being unrepresentative of reality.
As much as you want to talk about the algorithms in isolation, using ML also includes figuring out what input data to use and also how to properly interpret the output.
> The term "bias" leads you to believe the algorithms are generating results unrepresentative of wider reality, which simply isn't the case.
But unlike the loan example they are; that's precisely my point. In the loan example the algorithms (or actuaries' calculations) are often entirely accurate in identifying that for whatever reason $ISBLACK is associated with statistically significant increased likelihood of default after controlling for other variables, which may result in an unbiased estimation model assessing a particular person falls into an unacceptable default risk category (which FWIW isn't the same thing as accurately predicting a particular person won't pay, which is a more dangerous conflation of concepts...)
That doesn't apply to the examples discussed in the article - where Asians aren't blinking (or statistically more likely to be blinking), people trying to game a Twitter bot's corpus of conversations are not representative of human interaction patterns, nobody seriously believes social media bubbles represent reality and even the very important recidivism risk model is 50% more likely to be a false positive if a suspected repeat offender is black, and less likely to be a false negative if the individual categorised as low risk is white.
> even the very important recidivism risk model is 50% more likely to be a false positive if a suspected repeat offender is black, and less likely to be a false negative if the individual categorised as low risk is white.
That's actually the expected consequence of having different recidivism rates.
If some of the predictions of who will recidivate are wrong at random and a higher percentage of black people were predicted to recidivate then a higher percentage of black people will have been incorrectly predicted to recidivate.
If some of the predictions of who won't recidivate are wrong at random and a higher percentage of white people were predicted not to recidivate then a higher percentage of white people will have been incorrectly predicted not to recidivate.
In other words, X * error rate is more than Y * error rate because X is more than Y.
Say I have two categories, programming articles and non- programming articles, and some other data about each article. And I want to predict whether the article will be interesting or not. And I want to be fair to interesting non-programming articles by having the same proportion of false negatives to correct positives in the non-programming subset of articles as in the programming subset of articles.
Is there a technical term for that in statistics?
It's like trying to get a representative sample, but only representative in one specific way (topic), and deliberately non representative in another (interestingness)
I think this could get at one of the things people mean, and
it might be interesting to see how this trades off against overall accuracy or representativeness in other categories.
With risk of spoiling the Haiku, could someone explain this in more detail?
What is Sussman suggesting that Minsky do instead: wire the neural net in a fixed way instead of randomly? What does that have to do with the difference between viewing the room with your eyes open vs. closed?
The koan is commonly accompanied by the following:
"What I actually said was, 'If you wire it randomly, it will still have preconceptions of how to play. But you just won't know what those preconceptions are.'" -- Marvin Minsky
The funny thing about bias is that it was invented, technically, as a way of... improving the accuracy of data in audio recordings. By adding something (current or frequency) to the signal you could increase the quality of the output.
Without chasing the metaphor too much, it's also interesting to considerer that, maybe, adding something, yes a bias, to the data would also increase their quality.
Why did authors (e.g. Azimov) came up with fundamental robotic laws?
102 comments
[ 4.4 ms ] story [ 162 ms ] threadI absolutely hate this myth, and I suspect it's just an excuse used by technologists (consciously or not) to wash their hands clean of taking social responsibility.
There is no such thing as innocent technology, because it is made by and for humans, and with a purpose. Which means that before we even built anything we have already left the domain of "is" and entered "ought". Technology "encodes" values by being an answer to how we attempt to implement some intended purpose, and thus becomes entwined with the questions of responsibility that we all struggle with.
Here's a great anthology of writings on that topic: Inside the Politics of Technology[0], mostly written from a post-phenomenological[1] perspective.
Excerpt from the introduction:
> It was eventually proven that the pilot acted as capably as possible, so he was not to blame. But that only partly settled the question. After this first clarification, however important it was to the pilot, the accident could still be ascribed either to human error, e.g., false instructions from the control tower, poor maintenance, or lack of knowledge about the migration cory patterns of birds, or, alternatively, to technical deficiencies, e.g., fuel problems or engine failure.
> In the end, a year after the crash, an official research report established the “real” cause of the crash: the snapping-off of a 10-cm metal pin that regulated the position of a fin in one of the 13 cogs in the rotor of the F-16. This set off a chain reaction demolishing the cogs one by one, ending up in a complete breakdown of the engine.
> But even then the problem of humanity versus technology was not solved. Who could be blamed for this technical defect: service engineers, the Air Force, or the manufacturer? The latter was ultimately left holding the bag. But still then, was it a production fault or a designer’s error? Again,different actors and different technicalities are involved.
> Apparently, a definitive dividing line between technical and human causes cannot be drawn. However technical the cause of the crash appeared to be, human beings always come along with the technicalities – and vice versa. Purely technical causes are just as illusory as purely human faults.
The idea of machines being unbiased and innocent is not just naive, but outright damaging.
[0] Free PDF: https://oapen.org/download?type=document&docid=340207
[1] A friend of mine explained post-phenomenology as follows: "My understanding is that for Verbeek and others like him in general, postphenomenology continues the core traditions of phenomenology but with a deliberate emphasis on social science methods and empiricism over philosophical arguments - a positivist Heideggerian phenomenology"
http://www.psupress.org/books/titles/0-271-02539-5.html
[0] https://arxiv.org/abs/1611.04135
It would be great to hear about this; most of us are not from China and are unfamiliar with such things.
Good quote:
Indeed, that was an apt and true reply which was given to Alexander the Great by a pirate who had been seized. For when that king had asked the man what he meant by keeping hostile possession of the sea, he answered with bold pride, "What do you mean by seizing the whole earth; because I do it with a petty ship, I am called a robber, while you who does it with a great fleet are styled emperor".
I wonder how well they did it though, it seems to may that the non-criminal faces in Fig 10, have a happier expression.
The page describing (defending?) the "theory behind our technology" is amazing. In brief,
1. Personality is somewhat heritable.
2. Face shape is also determined by genes.
3. ???
4. Profit.
There is even an explicit positive call-out to phrenology. The mind boggles.....
They should have marketed paper as "guiding face beautification for honest signal using discriminative(pun intended) classifier" [similar to this SIGGRAPH paper http://leyvand.com/beautification/]. That would have sold very well. People are uneasy with labeling faces as criminal but its fine to label unattractive/creepy <--> potentially criminal. They would even get a buzzfeed article or TedX Talk about guiding plastic surgery using AI.
Also they must include this image for laughs
https://img.buzzfeed.com/buzzfeed-static/static/2014-11/1/13...
http://www.catb.org/jargon/html/koans.html#id3141241
closing my eyes makes my eyes insensitive to the state of the room, it does not make the room empty;
randomizing my network makes it insensitive to the biases of the AI, it does not mean there are no biases.
I'm finding it hard to figure this one out...
Is it possible to not be affected by so many nested layers of bias?
Only one example in this article is actually about incorrect decisions. Most examples are about unbiased learning systems doing exactly what they are designed to do, but people other than the designers wishing they did different things.
For example, consider "conflicting goals bias". This isn't a bias - the system is maximizing clicks exactly as desired. It's just that some random third party wishes the system were actually trying to mitigate a nonexistent psychological problem (namely stereotype threat) instead [1].
What they call "similarity bias" is the same thing. The system is attempting to show people stories they like (and probably does a good job at it), but the author wishes instead that the selection of stories was closer to what a journalist might choose.
Another social "bias" - namely "redlining"/redundant encoding/etc is actually the elimination of statistical bias. A lot of input metrics - e.g. SAT score, FICO score, etc - are biased in favor of non-Asian minorities (and against Asians) [2] and machine learning algorithms designed to find hidden patterns discover this bias and fix it.
Conflating "someone is doing X but I wish they did Y" with bias is not useful. It's also not useful to conflate the elimination of socially desirable biases with introducing new biases.
[1] Stereotype threat has repeatedly failed to replicate. Funnel plots suggest it only existed to begin with due to publication bias. http://www.sciencedirect.com.sci-hub.cc/science/article/pii/... https://dl.dropboxusercontent.com/u/85192141/2013-ganley.pdf https://en.wikipedia.org/wiki/Stereotype_threat#Failures_to_... https://replicationindex.wordpress.com/tag/stereotype-threat...
[2] See for example Figure 7 here: https://drive.google.com/file/d/0B-wQVEjH9yuhanpyQjUwQS1JOTQ...
But about machine learning... a system can learn from a set of outcomes which are the result of a "biased" system, that is to say tainted with an incorrect Bayesian prior which is not properly corrected, such as let's say courts in a racist society or whatever. It would learn the same bias. Because its goal is to maximize compatibility with the outcomes that the humans did. So it perpetuates those weights.
The problem is that we don't know whether the human decisions matched the reality. "Did the person commit the crime" for example. We might have to wait until more unbiased estimators for such activity come along, and throw away old historical data.
It's sort of like when Black-Scholes became a self-fulfilling prophecy for valuing derivatives, but only after it became widespread. The market started using Black-Scholes to value derivatives, so it became the best model to predict the value of derivatives. But until then, other models might have fit the historical data better.
If you interpret the word "bias" statistically - rather than the way innumerate reporters trying to sound profound use it - then it's pretty straightforward to do so.
"Knowledge" represents a fixed snapshot of your information about the world.
"Bias" represents a tendency for your knowledge to systematically differ from reality even as more data is gathered.
In programming terms, knowledge is the data in your DB at a fixe dtime. Bias is a tendency for my inserts to fail while yours succeed.
But about machine learning... a system can learn from a set of outcomes which are the result of a "biased" system, that is to say tainted with an incorrect Bayesian prior which is not properly corrected, such as let's say courts in a racist society or whatever. It would learn the same bias. Because its goal is to maximize compatibility with the outcomes that the humans did. So it perpetuates those weights.
If the goal of a system is to predict human behaviors, then it will in fact do that. That's not "bias", that's just building a system with the goal of matching human behavior and getting what you asked for.
However, if the inputs to your algorithm are biased predictors of the output your outputs can still be unbiased. The tendency of machine learning systems is to detect hidden patterns in the data; biased inputs are just another pattern.
I wrote a blog post earlier this year explaining this in detail: https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
I do think it's important for people to be more clear about the goals they have for these systems. Do you want them to be accurate regardless of anything else, or do you want them to accomplish particular social goals by deliberately bending the results? There's nothing necessarily wrong with the latter, though you will inevitably find once you're being clear about the latter that even people who are generally ideologically aligned with each other will discover they have different and mutually conflicting goals...
Anyhow, either way, until you have a clear specification about goals you can't determine whether you're accomplishing them. A vague goal of being "nondiscriminatory" doesn't cut it for computers... "nondiscriminatory" is a set of possibilities, not a unique specification. (And if you disagree about my claim it's a set, just try to sit down with one of the aforementioned ideologically-aligned people and try to hammer it out to a coding level of detail.)
https://arxiv.org/pdf/1609.05807v1.pdf
I.e. every algorithm will be "discriminatory" unless you predictor is perfect (i.e. gets 100% of decisions right). It's just a question of which kind of discriminatory they are. The same is true of any human decision process.
Unfortunately this means that we will be subjected to a slew of innumerate reporters posting "XXX is discriminatory" articles, regardless of how decisions are made.
Bias is not a word that was invented by statisticians[1], it's a word that statisticians adopted. It's just as legitimate to say that when a machine learning technique is trained with biased input that it produces biased output as it is to say that some particular machine learning method produces biased output from unbiased input. Either way, it's perfectly fine to say that the use of that technique results in bias even if that bias is in an aspect of the output that the algorithm wasn't selected to optimize.
It seems that what you're arguing is that the algorithm shouldn't be blamed in these cases, but in addition, you're declaring that there's no such thing as prejudice or bias, and implying that what the algorithms are doing is pulling some (uncomfortable for gatekeepers) platonic truth out of the ether (without any acknowledgement of the possibility of biased input.) Or more simply, you're using this article as an excuse to push a favorite "race realist" agenda which you use to justify a purist libertarianism (and which I agree it requires.)
[1] http://etymonline.com/index.php?term=bias
This agrees more or less with the statistical term, it's just less precise. That's also NOT what machine learning algorithms do.
If you want to declare an algorithm biased because it optimizes what it was designed to optimize rather than some random tangential goal, then the term has become meaningless. Similarly, my cell phone is broken because it can only make calls and is ineffective at pulling trains. A locomotive is also broken because it doesn't make tacos.
It seems that what you're arguing is that the algorithm shouldn't be blamed in these cases, but in addition, you're declaring that there's no such thing as prejudice or bias...what the algorithms are doing is pulling some (uncomfortable for gatekeepers) platonic truth out of the ether (without any acknowledgement of the possibility of biased input.)
I have no idea how you could possibly think I made this claim given that I explicitly listed two examples of algorithms which are biased. I then hinted at how algorithms can eliminate this bias. Consider rereading what I wrote.
If you want more details on how algorithms correct biased inputs (it's not based on "ether") read the blog post I linked to downthread: https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
Or more simply, you're using this article as an excuse to push a favorite "race realist" agenda which you use to justify a purist libertarianism (and which I agree it requires.)
I have no idea why you think "race realist" agendas require purist libertarianism, or vice versa. That's a random political tangent and totally unrelated to this conversation. I'm beginning to think you are seeking to derail the conversation rather than discussing statistics in good faith.
In the hopes that I've misunderstood you, I will nevertheless provide you with a good faith clarification. I made no normative claims regarding libertarianism or anything else. The only normative claim I've made is that we should not wrongfully apply an existing term (bias) to completely unrelated concepts (having undesired social outcomes).
A machine learning algorithm run on non-representative input could be said to lack experience with the portion of input that isn't represented in the training set.
E.g. I think it's pretty obvious that machine learning algorithms exhibit far more vernacular-bias than purely logical reasoning techniques.
> I made this claim given that I explicitly listed two examples of algorithms which are biased.
Your parent is pointing out the very real possibility that technically-unbiased algorithms can produce vernacularly-biased outputs. I.e., that statistic's definition of "bias" does not sufficiently capture the notion of bias as it's used in the vernacular.
I don't think you've refuted that claim.
At the end of the day, your approach toward side-stepping the issue of vernacular bias is intellectually lazy. Instead of tackling the problem head on, you're hiding behind a mathematical object that happens to have the same name as the actual thing under discussion.
> I'm beginning to think you are seeking to derail the conversation rather than discussing statistics in good faith.
I think your characterization of bias in terms of statistic's technical definition is already skimming the edges of good faith. You know what the reporter means when they say "bias", and that's not the meaning that statisticians use in technical settings.
At the end of the day, your approach toward side-stepping the issue of vernacular bias is intellectually lazy. Instead of tackling the problem head on, you're hiding behind a mathematical object that happens to have the same name as the actual thing under discussion.
I'm not side stepping the issue. I'm advocating in favor of describing the problem with clear language - once that's done we can begin useful discussion on how to address it.
In precise terms, what do you think the problem is?
> Your parent is pointing out the very real possibility that _technically-unbiased algorithms can produce vernacularly-biased outputs_. I.e., that statistic's definition of "bias" does not sufficiently capture the notion of bias as it's used in the vernacular.
That's quite a straw man for someone who claims others are trying to derail a conversation.
> I'm explicitly advocating against this practice and in favor of using clear and precise language instead.
The language you're suggesting masks the problem.
It's nice that statisticians came up with a mathematical object that happens to have a politically charged name, but the conflation you're making is not an argument, and doesn't address the underlying (inherently political) problem.
> In precise terms, what do you think the problem is?
The precise problem is that things like skin color etc. often have strong predictive power. We've recognized this unfortunate fact -- and its societal implications -- and decided to create laws and social norms that regulate (either restrict or sometimes even require) the use of these properties in certain high-impact decisions.
Unfortunately, algorithms sometimes use these features to make decisions in settings where a typical human wouldn't (or might even get into legal hot water if they did).
Notice that an algorithm can exhibit this behavior without being (statistically) biased.
Now, perhaps you think this problem isn't actually a problem. And you can certainly make the argument that any form of vernacular-discrimination should be OK as long as it doesn't exhibit statistical bias. But that's a separate conversation, and you can't defend that position by playing with definitions. And if you're not making that argument, then it should be clear why vernacular-bias differs from statistical-bias.
Pessimizer thinks the article is discussing inaccurate conclusions and that race realism is wrong ( https://news.ycombinator.com/item?id=13158797 ). You think race realism is accurate but we should ignore it. You both read the same article but drew totally different conclusions due to it's imprecise language.
Many examples in the article don't fit your description of the problem at all (e.g. difficulties in image processing, Tay the trolled chatbot, filter bubbles), so I don't think you are correctly describing what the article means by "bias" either. I can't figure it out either; the best I can come up with is "AI-related things that make the author have negative feelings".
I didn't argue in favor of or against your mysterious "vernacular-discrimination" - how could I when I don't even know what it is? The only thing that I argued is that the imprecise language used by Techcrunch is confusing people and that this is bad.
If you insist on precise language, then we basically can never have any conversation about race, racial bias, or any other topic where the terminology hasn't been completely nailed down.
Similarly, I advocate against new-age uses of the word "energy" - energy is a specific conserved quantity, not some mysterious unobservable thing that emanates from crystals.
What benefit do we gain when other people can't even understand the words we use?
It's almost as if you want to use a Motte-and-Bailey argument to confuse people ( http://slatestarcodex.com/2014/11/03/all-in-all-another-bric... ). Which is of course something that people do on this very topic: https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
The overloading happened the other way around...
> What benefit do we gain when other people can't even understand the words we use?
Similarly, what benefit do we gain when we use a term of art in settings where most of the audience doesn't have the technical background to be quite sure what they're even discussing?
Also, there is a notion that's distinct from statistical bias and that is highly political and controversial. Today we can say "bias" and most everyone will know we're talking about that topic. Whatever word or phrase we use in place of "bias" will be just as muddy and ill-defined. And probably some mathematician will come along and build some incomplete model and name if after that new word. And then we'll have this conversation again. The problem you're trying to avoid is unavoidable.
I fail to see why you insist that the authors use a word in its statistical sense when they clearly aren't making a statistical argument -- especially since the the statistical sense of the word bias is ultimately derived from the common usage, and not the other way around. The use of 'bias' in the sense of 'prejudice' in English dates from the 1570s, well before the development of statistics [1].
If anything, we should complain that the statisticians overloaded a perfectly good English word, instead of coming up with some Greek or Latin neologism for their own technical use.
[1] http://www.etymonline.com/index.php?term=bias
> Similarly, I advocate against new-age uses of the word "energy" - energy is a specific conserved quantity,
Sounds like a stilted way to look at the world. If someone says 'I'm feeling really energetic today', would you object on the grounds that they haven't shown that their biochemical energy reserves are larger than normal, or that their body is burning calories at a faster rate than usual?
This agrees perfectly with the statisticians definition, albeit applied to one particular example. Specifically, the slanted ball game field would provide a biased measurement (in the statistical sense) of player ability. Score 1 for the mathematicians.
I agree with you that this techcrunch reporter is using the term in some other mysterious "vernacular" way. It's this usage that I'm objecting to since it's misleading and confusing.
If someone says 'I'm feeling really energetic today', would you object on the grounds that they haven't shown that their biochemical energy reserves are larger than normal, or that their body is burning calories at a faster rate than usual?
If an article were discussing how coal is more "energetic" than nuclear energy, but was actually referring to energetic in the psychological sense (or maybe in the hippy energy crystal sense), I would in fact object. That would be similarly misleading.
That's silly. It's not a competition between mathematicians and all other people for the 'correct', Platonic definition of a word -- a word is defined by how people use and interpret it. Logical, mathematical deduction from first principles is not the only way to look at the world.
Unless I'm mistaken, I think your objection can be summarized as 'we shouldn't talk about things that we don't have airtight definitions for (like racial bias or racial prejudice), because it might confuse people'.
I happen to strongly disagree, since grappling with fuzzy definitions is precisely how we get to having better definitions. Otherwise, we should give up on ever talking about anything except math, where we have clear axioms to work with.
Even in math, people were profitably doing calculus before we had clear definitions for infinitesimals, integrals, derivatives, or proofs that our procedures produced correct results. The logical machinery and deductive logic often comes after people spent time grappling with fuzzy, intuitive things with unclear definitions.
No. I give a concrete example of what I want in this comment: https://news.ycombinator.com/item?id=13161470
Specifically, the word "bias" makes virtually everyone think of one particular concept. I claim expanding the word "bias" to describe two separate concepts misleads people into thinking the second concept is the same as the first.
My sole objection is with confusing people by conflating multiple disparate concepts under one label, which is already a perfectly good label that clearly refers to something else (according to everything besides throwaway's mysterious and uncited "vernacular definition").
No, I do not think race realism is accurate. I just think that race realism is not an adequate defense of racial discrimination, AND ALSO that race realism is inaccurate.
Pessimizer and I agree.
> Many examples in the article don't fit your description of the problem at all
You asked me what I think the problem is, not what I think the article's author thinks the problem is. The fact that I emphasize different things is not proof that I misunderstand the article's author.
> The only thing that I argued is that the imprecise language used by Techcrunch is confusing people and that this is bad.
Who, exactly, is supposed to be confused?
Subject matter experts should know the difference, and if they don't, that's really their own problem.
The public at large doesn't even know what statistical bias is, so why in the world would they be confused?
No matter how confusing the current state of affairs is, it would be far more confusing if suddenly everyone starting using bias to mean statistical bias. Mostly because 99.9% of the people using the term wouldn't have an adequate grasp of mathematics to even know to which object they are referring. So if your priority is clarity, then keeping around the messy moralized and politicized vernacular notion of bias is far preferable.
> I didn't argue in favor of or against your mysterious "vernacular-discrimination" - how could I when I don't even know what it is?
Not everything in the world can be mathematically formalized. Most things don't even have fixed meaning over time. If your criticism is genuinely about imprecision, you've got a lot of much lower hanging fruit than "discrimination" and "bias"...
No, I do not think race realism is accurate. - throwaway729, 22 minutes ago https://news.ycombinator.com/item?id=13160709
Whoops!
The fact that I emphasize different things is not proof that I misunderstand the article's author.
Fine - what do you believe the author thinks "bias" means? I.e. what is the underlying concept behind all his examples?
Who, exactly, is supposed to be confused?
Pessimizer, for one, who thinks the problem is inaccurate predictions. 2 hours ago you disagreed with this, but 22 minutes ago you agreed. I'm getting kind of confused.
If your criticism is genuinely about imprecision, you've got a lot of much lower hanging fruit than "discrimination" and "bias"...
Sure, but this is my field. I give talks about misuse of p-values and the multiple comparison problem, even though there might be other issues worth worrying about. It's just what interests me. The fact that bigger problems exist doesn't mean we shouldn't fix this one.
I take race realism to be the sort of thing that posits an inherent relationship between skin color/other physical features, and other innate/intrinsic/genetic properties such as intelligence or criminality -- especially as a prior and when the trait is normative and negative. So "dark skinned people are less intelligent" is race realism, but "dark skinned people are treated poorly by the political establishment" is not race realism. (We're far afield, I don't mean for this thread to devolve into a definition or debate about race realism -- I'm just saying what I take race realism to mean.)
The important point isn't a precise characterization. Rather, the important point is this:
It's possible for race to be predictive without there being inherent differences between people of different physical attributes. Setup a system in which I take away all green people's life savings at the end of the day with probability .99 if they don't have a house and .5 if they do have a house. Both numbers are .2 for purple people. An unbiased lender can choose not to lend to green people without exhibiting statistical bias -- in fact, color has huge predictive power.
But that predictive power comes from the social system we setup specifically to screw green people, not from inherent differences that are causally related to color posited by race realism.
But this is an extremely obvious observation, so perhaps you meant something different by race realism? In any case, I don't think the disagreement you posited really exists.
The rest of the points you're making are covered elsewhere.
I deal with Real numbers every day, but I don't barge into discussions of epistemology and demand my axioms be used in discussions of a vaguely related but definitely different topic. And I'm careful in pop lectures to point out that Real numbers are just a mathematical object and no more more "real" than natural numbers.
If people in your statistics lectures don't know what bias means, then you should account for this in your teaching and lecturing. I know my statistics lecturer did.
Insisting that a system isn't biased in the vernacular sense due to a particular technical notion of bias completely misses the point. And we almost never have enough understanding about the world to make a definitive case that the system isn't biased in the vernacular sense.
To do otherwise risks having people get confused as to what underlying concepts are actually being discussed. For instance, go read the HN comments here: https://news.ycombinator.com/item?id=13004790 Why risk such confusion when we can just use two different words for two different concepts?
However, you are also wrong about the vernacular sense, at least if the dictionary is to be believed:
Bias: prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair.
Prejudice: preconceived opinion that is not based on reason or actual experience.
This are just informal versions of the statistical term.
To translate this to another topic, if we are discussing global warming I don't think we should use the term "warm" to reference "feelings in my heart". Why would we want to confuse whether we are discussing the temperature of the earth or my feelings?
But we're not discussing statistics. We're discussing the social impact of ML algorithms.
> Why risk such confusion when we can just use two different words for two different concepts?
The problem with this is that we'll always have that confusion because most people aren't statistics experts.
So it's up to the domain experts to know when one meaning or the other is intended. That's true in every domain.
> However, you are also wrong about the vernacular sense, at least if the dictionary is to be believed
Dictionary definitions don't fully capture the vernacular meaning of words -- especially politically charged words such as "bias".
> This are just informal versions of the statistical term.
But when real people use the term bias in real conversations, they aren't typically approximating the statistical meaning. Rather, they are typically referring to some a priori ethical limitations on decision making.
For example, most people would say that taking skin color into account when approving a housing loan is a form of bias. But they don't mean that in any particularly technical way. There absolutely is moral and political content in the term "bias".
For example, when equal lending laws were passed, an unbiased algorithm definitely would've considered race when choosing the outcome of a loan application. African Americans were systematically discriminated against in employment decisions, and often the first to be let go when hard times hit. So a completely unbiased loan officer could come to the statistically valid conclusion that people of color were far more likely to default. None-the-less, a vase majority of people today would agree that the loan officer was exhibiting racial bias. And they're not coming to that conclusion due to innumeracy or some vacuous personal preference. They're perceiving a vast network of causal relationships between decisions that reinforce a systematic bias, even when many of those decisions aren't locally statistically biased
> I don't think we should use the term "warm" to reference "feelings in my heart".
And I agree, but I agree because non-experts know and understand the difference. And because the term "global warming" with the alternative meaning isn't already in wide-spread vernacular use.
What many common people mean by bias is approximately the statistical meaning. For example, if the algorithm accurately predicts the number of white defendants who fail to appear in court but overestimates the number of black defendants who fail to appear, people in common usage will call that bias. Which it is, in the statistical sense.
But if a higher percentage of black defendants actually fail to appear than white defendants, and the algorithm accurately predicts that outcome, that isn't bias in the statistical sense. And the people who try to conflate those things are doing this:
http://slatestarcodex.com/2014/11/03/all-in-all-another-bric...
The second thing is not only not bias, it's not morally objectionable. Statistical bias is the thing which is morally objectionable. Calling something else "bias" is not just inaccurate, it leads directly to a false moral conclusion.
And many people don't. The difference is that I'm not asserting I have the one true definition, but rather recognizing that "bias" is a term with inherent political and moral content. And that pretending that this isn't (or shouldn't be) the case just because statisticians use the term in a technical setting is extraordinarily intellectually lazy.
Most people are not statisticians. No serious mathematician has ever asserted that the axioms of Real numbers settle all epistemological debates. Like-wise, I see no reason why a statistical term of art should be used to clear up the various denotations and connotations surrounding "bias" and "discrimination".
> that isn't bias in the statistical sense.
That depends on what the goal of the algorithm is.
If the goal is to accurately predict whether people will appear, then it's working.
But if the goal is to accurately predict when people will choose not to show up, then there's certainly a chance that it's not working. For example, in the case where there's some significant correlation between race and access to transportation infrastructure. (Which isn't so much the case today, but certainly was 60 or so years ago.)
The distinction between choice and circumstance isn't immaterial -- the latter is not a flight risk even if they gum up the legal system via delays. So if the judge is interpreting "90% won't show" as "90% flight risk" in the above hypothetical world, then there's a huge problem. You could argue that this is the judge's problem, and I won't necessarily disagree. But UI bugs are still bugs, and this particular form of UI bug is pretty specific to ML systems.
> And the people who try to conflate those things are doing this
Or, perhaps they're far better at mathematical modeling of complex systems than you give them credit for; e.g., by seeing ways in which locally statistically unbiased decisions can magnify the impact of bias that exists in other parts of a system.
We will never have a complete mathematical model of the world and the history of human civilization, so I don't see any reason why we should insist that all bias needs to be what a statistician calls bias in his workplace.
> The second thing is not only not bias, it's not morally objectionable
This is a separate discussion I don't want to have in this thread, because we're already barely on-topic.
But suffice it to say that conflating technical and non-technical definitions is not a reasonable defense of this position.
Your assertion seems to be that not using the same word to describe separate and distinguishable things in a context where the distinction matters is intellectually lazy.
> If the goal is to accurately predict whether people will appear, then it's working.
> But if the goal is to accurately predict when people will choose not to show up, then there's certainly a chance that it's not working. For example, in the case where there's some significant correlation between race and access to transportation infrastructure. (Which isn't so much the case today, but certainly was 60 or so years ago.)
The goal is to predict whether people will appear. The bail-bondsman has to know whether to issue the bail-bond.
If your issue is transportation infrastructure then the only way to solve it is by fixing the transportation infrastructure, not by bankrupting the bail-bondsman.
> This is a separate discussion I don't want to have.
> But suffice it to say that conflating technical and non-technical definitions is not a reasonable defense of this position.
That's not the point -- the problem is that conflating technical and non-technical definitions is a way to avoid having that discussion.
Effectively everyone agrees that statistical bias is wrong. If you're on the wrong end of it you're being treated unfairly. If you can convince someone that something is this "bias" when everyone agrees bias is wrong then you've convinced them that the thing is wrong.
But if all you've done is broaden the scope of what bias means to encompass something that not everyone agrees is wrong then you've accomplished nothing of virtue. And more than likely misled some number of people who haven't managed to piece together that the "bias" they understand to be wrong and the "bias" you understand to be present are non-overlapping.
That's not quite true. An unbiased credit scoring algorithm would penalize blacks, subtracting 50 pts or so from their FICO score.
Is it fair to reject a loan because of the ethnic background of the applicant? That seems unfair to me.
Then again forcing a lender to give out loans that he knows will default also seems unfair, to say nothing of the higher interest rates that Asians must pay in order to subsidize deadbeat blacks.
I don't quite know what "fair" means here. What I do know is that the term "bias" is used to mislead; to hide the fact that whatever choice we make, there is a cost to be paid.
I think you're drawing the wrong conclusion from this. Unless we think that black people are inherently less credit worthy (i.e. are genetically inferior), then what's really happening is that race is acting as a proxy for some other things that aren't being considered. Maybe living in a neighborhood with drugs and gang violence, or lack of social ties to more affluent people who could be leaned on in tough times, etc.
So we have a bias, which we know is wrong, and we have two ways to fix it. One is the cheater's way out, which is to just take race directly into account. The other is to find the set of actually-causative factors that correlate with race which if you took them into account would make it so that nothing more would be gained by also taking race into account, and use those instead.
And given the choice between those two ways of solving the wrongful statistical bias, the second one is clearly preferred. Predicting via causative factors is fairer than predicting via correlative factors -- it's more accurate. You don't introduce a new bias against the black small business owner who doesn't live in a bad neighborhood and does have financially-advantageous social ties.
This is true of all predictions based on correlations, but there is a resource trade-off here. It may be more economical to collect data on things that correlate than things that cause. And causation is more predictive than correlation, but correlation is more predictive than nothing.
We can't pretend there aren't political considerations here. We're going to spend more resources to use causative factors instead of using race, for entirely political reasons, when we might not for some other factor. But the unfairness isn't to not use race (assuming we do actually find and use the underlying factors instead), it's only that we have to use more than the typical resources in order to get a more accurate result without using race. Which is a necessary evil given the politics.
But just leaving the known bias without finding and using the underlying factors to mitigate it is clearly wrong.
"Inherent" is always with respect to a particular data set - if there is no data in the data set that is more predictive than a given factor, you might as well treat that as inherent when doing the data analysis - it won't change your algorithm at all.
Maybe living in a neighborhood with drugs and gang violence, or lack of social ties to more affluent people who could be leaned on in tough times, etc.
Could be. However, due to housing separation, this will also be correlated with race. Regulators consider using factors like this to be "redlining" and strongly frown on it. (Historically this was not true; in fact regulators drew the original red line.)
The other is to find the set of actually-causative factors that correlate with race which if you took them into account would make it so that nothing more would be gained by also taking race into account, and use those instead.
That certainly might be possible. If you can solve this, and those actually-causative factors do not redundantly encode race (e.g. no redlining), you have built the next unicorn startup. Your customers will be every single bank in America.
I meant it with respect to the hypothetical data set containing all possible data. Either there is a quite large disparity in the data which is actually caused by race and so can't be eliminated without considering it, or it's really mostly or entirely caused by some other factors that only correlate with race which we can hypothetically find and use instead.
> Could be. However, due to housing separation, this will also be correlated with race. Regulators consider using factors like this to be "redlining" and strongly frown on it. (Historically this was not true; in fact regulators drew the original red line.)
Traditional redlining was unconditionally denying loans for properties in black neighborhoods. Weighting neighborhoods based on their actual repayment history is something else entirely.
> That certainly might be possible. If you can solve this, and those actually-causative factors do not redundantly encode race (e.g. no redlining), you have built the next unicorn startup. Your customers will be every single bank in America.
Any factor that reduces racial bias is obviously going to correlate at least somewhat with race. It's mathematically necessary. But you can easily tell if a factor does more than redundantly encode race if considering race and that factor predicts more accurately than only considering race.
Determining what the factors actually are would require having the data, which I don't have.
When I read this kind of thing in your comments it makes me wonder whether you aren't trying to smuggle in inflammatory phrases that you don't need, just to get away with it. Pejorative+race seems like such a troll trick to me (and there are others). If there are "deadbeats" being subsidized they would obviously not all be of one flavor.
Dang, I understand that you get a lot of flags whenever I point out unpleasant facts. I'm sorry if this bothers you. Should I just accept that HN is no longer a place where uncomfortable realities can be discussed, and move on?
If there are "deadbeats" being subsidized they would obviously not all be of one flavor.
They are disproportionately and predictably of one flavor. That's the entire reason this is a non-simple issue.
Let me take numbers from 7 of this paper: https://drive.google.com/file/d/0B-wQVEjH9yuhanpyQjUwQS1JOTQ...
Based on eyeballing the graph, consider a group which is 50% black and 50% Asian. Every person has a 600 FICO score. Blacks with a 600 FICO score have about a 40% default rate while Asians have about a 20% default rate.
To lend $100 to each member of this group at a fixed interest rate (i.e. we ignore race) and break even we need to charge a 43% interest rate to the mixed group.
The average Asian will pay back $114.40, while the average black will pay back $85.8.
Defaulters come in both flavors - in a group of 100 blacks and 100 Asians, there are 20 Asian defaulters and 40 black defaulters. So the 80 Asians who are not defaulters (using a more awkward word since you object to "deadbeat") are paying higher interest rates to make up for predictable losses caused by the 40 black defaulters.
If we did not bar the use of racial information then we'd charge blacks a 66% interest rate and Asians a 25% interest rate. This would be statistically unbiased (i.e. it removes omitted variable bias https://en.wikipedia.org/wiki/Omitted-variable_bias ) and would not result in a predictable transfer of wealth from Asians to blacks. Individually this does seem unfair.
I am expressing no opinion on which policy is better. I'm pointing out that whatever we do will be in some sense unfair. The only position I'm pushing is numeracy and clear thinking that acknowledges the very real tradeoffs.
(To discuss examples in the article, one could repeat using similar calculations with ad delivery or recidivism. I just go with lending because it's mathematically equivalent and there's lots of published work on it that I can cite.)
If you choose a different FICO cutoff or whites instead of Asians, the result remains directionally the same. You might get whites paying $110 vs blacks paying $90 (the gap narrows as FICO goes way up) but the same basic point arises.
Feel free to play with the numbers yourself if you think my arithmetic is wrong or nonrepresentative. If you find something I'd love to see your corrections.
If the directional comparison was all you cared about, why not compare Asians to Whites?
https://news.ycombinator.com/item?id=6726852
What happened to that guy? I miss him. He made HN a better place.
I didn't find black default rates at FICO=620, maybe you have sharper eyes than me. Feel free to change the numbers and redo the calculation. If you get something substantially different, lets discuss.
[edit, in response to your edit: Asians to whites is an equivalent comparison and the result is the same. Asian borrowers subsidize white deadbeats.]
The 620 default rate is on the very next page after the one you cited.
If you are simply expressing your distaste for me, your emotions are noted.
You asked a question --- orthogonal to my questions, which you really still haven't answered --- and then feigned offense when I answered it. If you didn't want to talk about "emotions", well, neither did I. Why'd you bring them up?
Here is another way to read the questions I asked: "perhaps the answers to these questions explain why a normal reader might snag on the way you worded your previous comment and have to ask themselves whether they were taking race-troll bait". Not: "Chris should make different points", but rather, "here are some tools Chris might think about using to avoid giving the impression that he's race-trolling."
So again, I'd ask:
* Why go through extra effort to find a data point less favorable to black people than the one the document already presents?
* If the point is that there are public policy debates to be had about fairness versus demographic blinding, why pick on black people in the first place? A comparison between Asians and whites makes the same point, without requiring you to ever write the words "deadbeat blacks".
Answer 1: The document doesn't provide the number you claim it does, or if it does I still can't find it. Further, that number wouldn't substantively change my point and I'm pretty sure you know this.
Answer 2: Mainly because we all know a major chunk of the actual debate is about black people, and I don't see a reason to obscure what we are really discussing. But I freely acknowledge that you can get a similar gap between whites and Asians, particularly around FICO=500.
I'm very deliberately trying NOT to obscure the real issue, namely that you can't have both fairness and accuracy. You can't even have multiple kinds of fairness - pick one and you lose another. Unfortunately that's just how the math comes out.
Also, I do believe that in the contemporary US Asian people are directly discriminated against more than anyone else. They are also the highest performing on a variety of common metrics (academics, income, financial responsibility) and this is primarily due to difficult to change traits (some combination of culture and biology). Just mentioning it explicitly if you thought I was trying to obscure this point.
Meanwhile, you were ostensibly making a level-headed argument about how there may be tradeoffs between discrimination and fairness; that to entirely eliminate discrimination we mathematically might be required to force some (say) ethnicities subsidize others. You could have made that point in any number of ways without using the words "deadbeat blacks" --- as I pointed out, the same point was available to you in a comparison between whites and Asians.
As you can see from the thread, the objection wasn't to the underlying point you were trying to make, but to the way you chose to make it. Simply put: a reasonable reader could come to the conclusion that you were not only trying to make a point about a tension between non-discrimination and fairness/accuracy, but also trying to use that argument as a device to smuggle in an ethnic slur.
I don't think you're a racist, by the way. I think something else is going on with you. Maybe I'm wrong. But as you are to statistics, I am to the oeuvre of 'yummyfajitas comments on Hacker News: they are an area of my expertise.
I choose not to couch my facts in language meant to obscure the point, or avoid choosing factually correct examples that again hide the unpleasant bits of reality. We won't figure out the right way to deal with reality if we pretend the ugly bits aren't really there.
I don't think you're a racist, by the way. I think something else is going on with you.
It's very simple. On values I'm mainly an old school left wing individualist - I view ethnicity as holding no moral weight and groups as having no inherent rights that they don't inherit from their individual members.
Unfortunately, on racial issues, the white supremacists seem to be factually correct (mostly). The current left wing approach to this is to shame and ostracize anyone who dares to point out the emperor has no clothes. I originally interpreted your comments to be an attempt in that direction.
Meanwhile, UKIP/scary Hungarians/etc are pointing out the lies and people are following them.
I.e. I favor intellectualism and analytical philosophy over dogma and (secularized) religion.
I hope the old tptacek who discussed ideas comes back some day. I miss him.
Of course, The Bell Curve, and the arguments contained within are a best assumption as to what "on racial issues, the white supremacists seem to be factually correct (mostly)" is referring to.
Also, you tend to fall back on claiming to be a harbinger of unadulterated knowledge ("I choose not to couch my facts in language meant to obscure the point") while ignoring the implicit (or explicit, https://news.ycombinator.com/item?id=13170988 ) invective contained in your language. I see no loss of descriptive accuracy w.r.t. the article between "deadbeat blacks" and "African-American defaulters", yet using the latter avoids the minefield of historical connotations that the former has.
Surely you know that, in current discourse, individually the word "deadbeat" means something, and the word "blacks" means something, but the combined "deadbeat blacks" means something much greater than the sum of its parts, because of the societal context in which it is uttered.
To claim that you are just using that combination as just the exact sum of its parts is disingenuous, especially from someone who claims to favor analytical philosophy. Go read Paul Grice.
This creates a significant level of friction on one side of a discussion, and makes it harder for us to come to truth.
This also means that the only people who do discuss these issues are the deplorable folks. I explicitly want to reclaim the unpleasant facts so that the deplorable folks don't have a monopoly on truth.
I want to say "we should import hundreds of thousands of Syrian refugees, a few hundred white girls will be raped as a result, and tens of thousands of lives will be saved. Good tradeoff." I want to say "black on white crime is the most common form of interracial violence, but we should still treat black people as equal citizens with civil rights. However our expectations of equal levels of police violence against blacks and whites might be unreasonable."
Lying/shaming/attacking the people speaking the truth isn't sustainable. When you lie to people and they notice the truth, they start to turn to turn to UKIP/Jobbik/other such folks. As a globalist and a universalist I don't want to live in that world.
Getting to truth is important. However, the way you want to go about it makes it very expensive for your audience.
You want to say things that are true in a socially unacceptable way, and then leave it to us to do the work of questioning you extensively to figure out whether you are someone who is well versed in statistics and meant "deadbeat" "blacks", or if you are a Stormfront-style white supremacist and meant "deadbeat blacks".
This thread is an example of that. If you want to be an effective persuader, you have to talk at your audience's level. If you want a giant thread on HN where your audience walks away believing exactly what they believed before they encountered you, keep doing what you are doing.
On this we can 100% agree.
Choosing "bias" to mean "statistical bias", and then actively ignoring anything else that is currently called bias, is an extremely political choice.
Are you seriously asserting that naming clashes validate political opinions, or that some large group of people's political opinions would suddenly change if only they knew the statistical meaning of bias?
Are you seriously asserting that naming clashes validate political opinions, or that some large group of people's political opinions would suddenly change if only they knew the statistical meaning of bias?
Yes, I think this use of "bias" is meant to mislead. Specifically, I think the term "bias" is used to make people think machines are drawing incorrect conclusions (that's what "bias" means in the dictionary and in most common uses of the term).
There are two separate concepts:
1) An algorithm refuses to issue loans to blacks even though those blacks would pay the loans back.
2) An algorithm refuses to issue loans to blacks because they would not pay the loans back.
You want to use the term "bias" to describe both situations; I think this will mislead people. Specifically most people will think "bias" means we live in situation (1). I want to use "bias" to refer to (1) and to use a separate term to refer to (2).
Why do you want to conflate these two situations?
The politics around racial discrimination and bias means that in reality, few people do the work to tease apart (1) and (2). In particular, I think many people are extremely uncomfortable with the consequences that (2) being true entail.
Hence 'racial bias' usually has a simpler meaning: something is biased if the decision conditioned on race is not identical to the unconditional decision. That encompasses both (1) and (2).
You seem to ascribe some sort of malicious motive to the authors, when what you really have a problem with is how the country talks about race and racial bias in general.
The point is that in general, statistically unbiased algorithms will not lead to racially unbiased (in the strict conditional outcome = unconditional outcome sense) results. You might think this is completely obvious as a practitioner in the field, but it is far from obvious to the general public. Hence why it's fair to point that out to the public.
> All I said is that we should name it something else
As I said elsewhere, some mathematician would come up with a concept, name it after that new thing, and then we're back here all over again because "bias" is inherently moral and political and has shades of meaning informed by moral and political opinion.
You won't have solved the precision problem, or even the overloaded terms problem.
> Specifically, I think the term "bias" is used to make people think machines are drawing incorrect conclusions.. I think this will mislead people
I agree "bias" is a politically powerful word.
I don't agree, however, that people fail to know and understand the difference between (1) and (2).
And I also think all people know that the term "bias" could mean both, and manage to communicate with one another using the full range of their vocabulary whenever it's unclear what's being discussed. See e.g. internet discussions around stop and frisk or WoD or racial profiling at airports.
People understand the difference between (1) and (2), even when bias is used to refer to both.
I might be wrong about that. A far more important prediction: I don't think there's anyone -- on either side of the debate -- whose position can be explained by their misunderstanding of the statistical meaning of "bias".
People understand the statistical meaning (or not), and call (2) bias regardless. Because they aren't referring to a mathematical object, they're referring to the effect something has within a larger social and political context. A context that cannot be perfectly modeled, and so is necessarily vague and difficult to pin down.
The fact that "bias" can have both meanings isn't a real problem. People are good communicators and, when they want, can easily distinguish between (1) and (2).
Saying that (2) is "not bias" is definitely a political statement when taking the common definition of bias. People are free to agree or disagree with that statement, but arguing via way of naming clash isn't a compelling argument (not saying you're making that argument, but GP is I think).
> Why do you want to conflate these two situations?
I don't think this is a real problem.
It's an example. The article mentions calculating recidivism rates to determine parole, which is a similar problem if you prefer that one.
> But what "separate term" would you consider to be more appropriate to describe ML systems' tendency to generate results unrepresentative of wider reality as a result of the characteristics of the subset it's trained on or inadequacies in its specification, which is what the article actually discusses?
Most of the examples in the article have nothing to do with "bias" and are really the principal-agent problem. Misalignment of incentives.
Some website wants to maximize clicks rather than act in the user's interest, they create a machine to maximize clicks, the machine maximizes clicks. It's doing what they wanted it to do -- it's just not doing what you wanted it to do.
This category of problem has almost nothing to do with ML in particular. It's like identifying that charlatans often wear cotton and writing an article on the link between cotton and charlatans.
The article is identifying a particular misconception that people might have, which is because a computer did it, the result will be (racially) unbiased.
It might be obvious to you that's not the case, since you're familiar with the ML techniques in question, but still valuable to a lay audience to make that point crystal clear.
Sorry, s/loan/allow out on bail/ or s/loan/show advertisement/ to go with examples in the article. Mathematically it's the same problem. I just default to thinking about loans since I've worked on loans and since most of the academic work on the problem is about lending.
But what "separate term" would you consider to be more appropriate to describe ML systems' tendency to generate results unrepresentative of wider reality...
Thanks - you've perfectly illustrated my point as to why the term "bias" is misleading. The term "bias" leads you to believe the algorithms are generating results unrepresentative of wider reality, which simply isn't the case.
You can make a strong argument that, for example, the FB news feed algorithm leads to results that are unrepresentative of reality, because its input data (user clicks and likes) do not sufficiently represent reality accurately.
Or from the article: "Nikon’s confusion about Asian faces and HP’s skin tone issues in their face recognition software both seem to be the product of learning from skewed example sets." Nowhere here is the algorithm being blamed, but we have an example of results that at least some people find objectionable due to patterns in the input data.
Just to take a particular example, the algorithm's output has marked someone as blinking when they aren't, they're just Asian. That's a particular example of results being unrepresentative of reality.
As much as you want to talk about the algorithms in isolation, using ML also includes figuring out what input data to use and also how to properly interpret the output.
But unlike the loan example they are; that's precisely my point. In the loan example the algorithms (or actuaries' calculations) are often entirely accurate in identifying that for whatever reason $ISBLACK is associated with statistically significant increased likelihood of default after controlling for other variables, which may result in an unbiased estimation model assessing a particular person falls into an unacceptable default risk category (which FWIW isn't the same thing as accurately predicting a particular person won't pay, which is a more dangerous conflation of concepts...)
That doesn't apply to the examples discussed in the article - where Asians aren't blinking (or statistically more likely to be blinking), people trying to game a Twitter bot's corpus of conversations are not representative of human interaction patterns, nobody seriously believes social media bubbles represent reality and even the very important recidivism risk model is 50% more likely to be a false positive if a suspected repeat offender is black, and less likely to be a false negative if the individual categorised as low risk is white.
That's actually the expected consequence of having different recidivism rates.
If some of the predictions of who will recidivate are wrong at random and a higher percentage of black people were predicted to recidivate then a higher percentage of black people will have been incorrectly predicted to recidivate.
If some of the predictions of who won't recidivate are wrong at random and a higher percentage of white people were predicted not to recidivate then a higher percentage of white people will have been incorrectly predicted not to recidivate.
In other words, X * error rate is more than Y * error rate because X is more than Y.
Is there a technical term for that in statistics?
It's like trying to get a representative sample, but only representative in one specific way (topic), and deliberately non representative in another (interestingness)
I think this could get at one of the things people mean, and it might be interesting to see how this trades off against overall accuracy or representativeness in other categories.
"What are you doing?", asked Minsky.
"I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied.
"Why is the net wired randomly?", asked Minsky.
"I do not want it to have any preconceptions of how to play", Sussman said.
Minsky then shut his eyes.
"Why do you close your eyes?" Sussman asked his teacher.
"So that the room will be empty."
At that moment, Sussman was enlightened.
What is Sussman suggesting that Minsky do instead: wire the neural net in a fixed way instead of randomly? What does that have to do with the difference between viewing the room with your eyes open vs. closed?
"What I actually said was, 'If you wire it randomly, it will still have preconceptions of how to play. But you just won't know what those preconceptions are.'" -- Marvin Minsky
To criticize them by highlighting their failings in the stories they wrote, in order to show us friendly AI is not an easy problem?