"she shows how the algorithms [...] are based on exactly the sort of shallow and volatile type of data sets that informed those faulty mortgage models in the run up to 2008"
so the fault is of the technology, right?
Well there is a paradox here, if the models were faulty, then why was every bank scrambling to offload the mortgages to someone else? If the models said the mortgages were sound, why didn't the banks want to hold onto them as a cash cow? Could it be the data spoke but no-one wanted to hear what it said?
The banks were holding higher tranches of the securitised mortgage pools. While those pools had traditional levels of defaults, the tranches had their expected value. As the defaults accelerated beyond historic levels, they plummeted in value. In both cases the same model said the instrument had/didn't have value. As these instruments are fair value accounted the fact that they were still bringing in cash didn't help any.
However, as the mortgage market started to go south, the default models underestimated the level of defaults, particularly in the equity release market. This underestimation was endemic in the industry at the time, from the banks to the regulators and ratings agencies.
There is, of course, a very strong argument to say that the ignorance was willful, everyone was on quite a nicer earner and reacted late to the evidence as it started appearing.
So there wasn't really a paradox. The banks were holding tranches while the expected defaults were within historic levels. Once it became clear that that wasn't a good predictor anymore, they tried to limit the effect but it turned out that so many participants were in the same boat that there weren't enough buyers.
I have to say I'm not entirely sure what this has to do with ad analytics but I'm not writing a book on the subject so maybe that's not too surprising.
Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness. Here's the last paragraph of the article:
"""
O’Neil also proposes updating existing civic rights oriented legislation to make it clear that it encompasses computerized algorithms. One thing that her book has already made quite clear – far from being coolly scientific, Big Data comes with all the biases of its creators. It’s time to stop pretending that people wielding the most numbers necessarily have the right answers.
Conversely, what if the numbers are unbias and we are in fact the bias ones. What if, for example, Irish Americans are actually more prone to purchase Guinness beer because we feel that it has a slightly higher alcohol content then all those other beers?
Tongue and cheek response, but "wielding numbers"? Politicians don't misuse statistics to get what they want, people misuse statistics to get what we want. We shouldn't ban statistics and we shouldn't ignore the fact that both "sides" are equally susceptible.
> Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness.
Uhm, racial segregation is still a thing. Even if the signal of race is gone the geography of where a person lives could be signal enough to discriminate.
Disallowing the use of race as a signal in machine learning
That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race" directly encoded as a feature, but that is insufficient.
See slide 22 from [1], further in depth discussion discussion from [2] or just this feature (which many systems could automatically discover):
Feature6578 = Loc=EastOakland && Income<10k
I don't know what the solution to this is, but it's a pretty hard problem, and even the best intention are insufficient.
Delip proposes in [1] to introduce a "fairness constraint" where the probability of a favourable outcome in the majority class is the same as in a protected class. This sounds sensible, but make optimising systems much harder.
The problem is deciding which features are reasonable to include. Yes, location and income can probably predict race to a certain precision, and if you're building a race-detector-by-proxy, you'd want to include those. But what if you're evaluating eligibility for a mortgage? Certainly location and income are critical. A black person in East Oakland with an income below $10k is going to struggle with a mortgage, but so is a white person with those characteristics (and a black person in Connecticut making $250k is going to do fine and so is a white person). But if more black people than white legitimately are in a situation where they might struggle to service a mortgage, more black people than white would get flagged as such by an objective algorithm. Does that make the algorithm racist? The slide seems to imply so (although without speaker-notes it's difficult to ascertain).
The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?
There isn't a simple answer here - like in most non-trivial data science/ML questions.
The only think I think worth pointing out is that in the mortgage example the current location of a person isn't as major an influence as income. The fairness test here seems somewhat reasonable: a person on $10K/y who lives in Oakland should probably have the same outcome as someone on $10K/y from some very white area - all other things being equal.
The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?
Yes, I think so? Or at least it can be audited (in theory at least - although see sentencing guideline systems for the problems with this).
Race is often very predictive. Blacks don't perform worse simply because they have income < $10k. Blacks perform worse on many measures even if you hold all that stuff equal. Here are a few of the many studies reproducing this:
If you were right, then the machine learning algorithm would quickly determine that race doesn't matter. Redundant encoding, direct encoding, etc would be irrelevant - the algorithm would ignore it as noise.
The problem is that the algorithm is supposed to uncover hidden patterns that predict loan default. But then there is one specific hidden pattern that is socially and legally taboo to detect, yet also highly predictive.
(Incidentally, if you can solve this problem in lending, it's easily a unicorn startup.)
> Disallowing the use of race as a signal in machine learning / ad targeting could help intercultural awareness
Many a time, algorithms infer race from secondary or tertiary variables. This problem is compounded by two more factors - (1) Unable to explain WHY an algorithm has reached a decision aka interpretability and (2) Focus of algorithm building to maximize accuracy regardless of other valid costs. There is some pretty interesting work going on this area.
The irony is that Times is partnered with advertisers torun these exact algorithms on the readers of this very article. In fact, they have anti-adblocking scripts to prevent circumvention, which unfortunately breaks reader mode.
A wide array of concerns, but I stopped reading when the article explains that algorithms are going to keep people from being aware that poor people exist. If the only awareness someone has of poverty is from the Internet ads or any other content, then they really don't know anything about it at all. This sounds more like the people that protested computers in the 60's because they are used for war.
> add more police to his neighborhood because of higher violent crime rates will necessarily be more likely to be targeted for any petty violation
He'll also be less likely to be the victim of violent crime in an area with a proven record for violent crime. For the price of avoiding 'petty' crimes, it seems like a good trade-off.
That's a solid theoretical argument. However, it is contradicted by the empirical evidence.
What usually happens is that the police issue fines so frequently that they become considered an occupying force by the populace. In turn, the populace withholds information from the police, and both violent crimes and petty violations rise.
Police statistics make quantitative conclusions difficult, since they are so often fudged for political purposes. They rise when the police need more funding, and fall when they need political capital. Murder rates are generally considered the only reasonably accurate figures.
On the ground, however, the reality is very clear.
Here's an example, and the study cited here has been replicated in many American cities.[0]
For the petty fines and eroded trust, this is from an advisory letter sent from the Justice Deparment to the judiciary of all fifty states:
“Individuals may confront escalating debt; face repeated, unnecessary incarceration for nonpayment despite posing no danger to the community; lose their jobs; and become trapped in cycles of poverty that can be nearly impossible to escape,” Gupta and Foster wrote. “Furthermore, in addition to being unlawful, to the extent that these practices are geared not toward addressing public safety, but rather toward raising revenue, they can cast doubt on the impartiality of the tribunal and erode trust between local governments and their constituents.”
In a world where we have police, we want to deploy it against crime. If the police isn't fit for purpose, that's a different kettle of fish.
(Also, the next bit of the quote, "Yet neighborhoods more likely to commit white collar crime aren’t targeted in this way", suggests a very loose grasp of how policing works. No, they're not, because patrolling the street in front of an office building isn't the least bit effective against white collar crime.)
Because it's not really detectable by patrolling without in-depth investigation. A patrol cop staring at e.g. a stockbroker doing insider trading would not deter it because that wouldn't be visible even as he would be doing the trades. A patrol cop sitting in all Madoff's meetings and sales presentations wouldn't prevent the Madoff scam from succeeding. There's nothing to suggest even getting a warrant without analysis of the actual trade data/accounting information/etc, which cannot really be done by most nonspecialized police officers.
I would suggest, however, that this is not an argument for less police but instead an argument for fewer laws that are only infrequently and inconsistently enforced. It is these laws that give police the opportunity to "issue fines so frequently" and to give affected populations (rightfully or not) the impression that they are singled out. (Laws against loitering or the consumption of alcohol in public come into mind, to a lesser degree probably also the criminalization of marijuana consumption.)
Right... but those laws only ever get revoked when they hit the rich with them. And for every one you revoke there's ten more wealthy/connected people with sob stories that could have been prevented if X were illegal who are contacting their elected representative.
I agree with you on principal but IMO society on Mars will be having this debate (I hope I'm wrong).
There is a lot of interesting research on the societal impact and the (potential) problems caused by the use of large-scale data analysis and artificial intelligence, e.g. by Kate Crawford at MSFT Research.
Last year I gave a talk on this at 32C3 where I tried to elucidate some of the problems that can pop up when using data analysis to automate things that were previously done by humans:
Personally I see a lot of potential in data analysis & artificial intelligence, but like all technologies they pose significant risks as well. What we really need therefore (IMHO) is to teach ethics to people that work with data, and establish organizations and methods that ensure data analysis isn't used to do harm (deliberately or not).
> What we really need therefore (IMHO) is to teach ethics to people
The issue with almost all organized teaching (as opposed to when you don't even know you are doing it, e.g. to your children, every day) is that it only targets conscious thinking. Do you already see the problem with teaching "ethics" (and empathy)? It's the wrong brain area. People will ace the tests but how they actually act won't change. Another big reason for why teaching ethics (the way it is usually done) is that much of behavior is environment-driven. So you can teach someone all you want, if you then place them in a cut-throat competitive "result-driven" environment you can see empathy and ethics quietly sneak out the back door.
Furthermore, ethics is way too cultural and subjective to be taught. Even if we could all agree on macro-rules, eg. hurting peoples is bad whatever hurt they infliged, the more specifics (and micro) we become, the more divergences will arise. And as everythng, our perception of what is "ethics" will evolve.
Being deliberately obtuse goes a long way to delay a logical ending. Flashback to 2004 and the same type of debate/affirmations were posited to Dr Mandelbrot's apocalyptic warnings[0][1] of the imminent subprime crisis.
Instead of learning lessons, the players continue to 'print money out of thin air'[2] and deny any fault for the last collapse. The rationale seems to be, 2007 was an extreme culmination of events that wouldn't happen again in millions of years. Problem is, in a universe of infinite variables, an infinite number of combinations exist and could manifest anytime a given set of vulnerabilities and unanticipated outlier events present themselves concurrently.
People learning or not learning ethics doesn't really have much to do with it. Read up on the Fraud Triangle. Unethical job tasks follow a similar pattern. Perceived need ("I might lose my job if I refuse") and rationalizations ("the results of this flawed algorithm surely won't get taken seriously") get you two-thirds of the way there. The third leg, opportunity, is inherent in this sort of job. So "ethics" are really beside the point when the workers confront the reality of their situation.
> What we really need therefore (IMHO) is to teach ethics to people that work with data
Studies seem to indicate that studying and teaching ethics does not make you more inclined towards ethical behavior.[1] On top of that, from personal experience in business ethics courses, "ethical" has an extremely suspicious equivalence with "the thing which limits the company's exposure to liability."
The ethics being discussed here, though, are, well... not obvious. Political, really. Are you going to be able to get serious agreement about what the correct decisions are in cases like these? By actual argument, and not by shouting down one's opponents?
I found this article problematic. First the assertion that "big data comes with all the biases of its creators". Isn't this exactly what big data (which I believe is a term used here to reference some sort of predictive model or machine learning) is supposed to avoid - ie. Making decisions based on evidence and not bias?
Maybe it's the cynic in me but someone who worked in a hedge fund, started yet another advertising tech company and now releases a book about the evils of "big data" abusing the poor... guess I'm biased.
Humans still make decisions about what data to include, how to model it, how to interpret the results and what to do about it.
I firmly believe in the power of data to help people make better decisions, but decision making is always a human process, even if much of the work is delegated to a machine.
Big data somehow being unbiased and impartial is a myth. The algorithms are still a product of humans. One notable example is a Google algorithm for labeling objects in photos that tagged black people as gorillas [1]. Now I don't have particular insight into the development process behind this algorithm, but this result may have occurred because the training data used did not include enough photos of black people.
Another example (from an old version of Picasa -- I'm not trying to pick on Google but I have firsthand experience of this) is when a friend of mine had photos of two different Asian people in his collection, but Picasa always recognized person B as being person A, presumably because it couldn't distinguish features unique to the faces of Asian people.
What evidence do you have that one specific classification error resulting in a tweetstorm was the result of bias at all? I've been mislabeled as a tank in at least one image processing demo (admittedly, this was well before the era of deep learning). Image classification errors happen.
Also, it's a bit strange that you suggest ML algos are racist against Asians. You do know that Asians are wildly overrepresented among people building such algos, right?
Consider the possibility that some classification problems are actually more difficult than others. As one example, darker objects are harder to distinguish than bright ones and there are very clear mathematical reasons for that. That's true even in fields where questions of "racism" don't make sense, e.g. in MRI (magnetic resonance imaging is non-optical, but contrast still matters): https://www.chrisstucchio.com/pubs/wf_segment.pdf
I did not make a blanket statement that all ML algos are racist against Asians. My point was to convey that if the training data do not accurately represent the population, the algorithm will carry the biases of that inaccurate representation: the algorithm is only as good as the data it is trained upon (for supervised learning).
As a sibling comment stated, humans are deeply involved in all aspects of designing and interpreting an algorithm's results. To the extent that we are flawed, so will our algorithms be.
Another example I heard recently from a conference was that of analyzing blighted homes in areas. The underlying data was generated by human inspectors who turned out to have given passes to violations in more affluent areas and fines in less affluent areas. The fines compounded the problem of blight as the residents could not afford to pay them and ended up causing a feedback loop of blight increasing in neighborhoods identified as blighted.
I am frankly confused as to what is so controversial about the statement that algorithms created by humans are not free from human biases. I apologize if I have only added confusion rather than a productive analysis.
I did not make a blanket statement that all ML algos are racist against Asians. My point was to convey that if the training data do not accurately represent the population, the algorithm will carry the biases of that inaccurate representation: the algorithm is only as good as the data it is trained upon (for supervised learning).
But that's not true. Inadequate training data will cause variance, not bias. This variance can have any direction.
In a linear prediction scenario (i.e. any go/no go decision process), which exactly what Cathy O'Neil is criticizing, the effect could just as easily be "more loans for black people" as "less loans for black people".
In a multidimensional predictor, it does directly decrease accuracy but that's all. "Black guy -> gorilla" is just as likely as "yummyfajitas -> tank".
Also, realistically, if we want to think about your black guy -> gorilla example, most likely if insufficient training data was the problem, then it was the number of gorillas that was too few. Google has vastly more black humans in their training set than gorillas - a quick google search suggests there are only 100,000 gorillas in the world, most of whom are probably unphotographed.
As a sibling comment stated, humans are deeply involved in all aspects of designing and interpreting an algorithm's results. To the extent that we are flawed, so will our algorithms be.
Sure, algorithms are flawed. But this does not imply that bias must be present proportionally to ours. Insofar as we do good statistics, human bias is mitigated and can even go away. Science works.
As a classical example, consider Morton's analysis of human skulls. Morton was a super racist guy trying to prove how white people are superior. But he did an unbiased analysis and accurately measured skull volume - filling a skull with buckshot is a pretty unbiased process.
I am frankly confused as to what is so controversial about the statement that algorithms created by humans are not free from human biases.
What's controversial is that this is the idea of "original sin" but applied to science. And most of the proposed mechanisms are simply mathematically wrong. Also many of the examples cited by proponents of this original sin concept are also wrong.
Thanks for the interesting article. Although, from my reading, it seems to support the importance of both data handling and statistics in avoiding bias. That is, a bias in the dataset (or which subset is analyzed) would subsequently bias the statistical results (Box 1 and 2).
Regarding big data and ML, I guess the question is then what kind of biases exist in data collection, something that seems dependent on the details and history each particular dataset.
If we train a mortgage engine on Redlining [0] data, it will then red line. Algorithms are not somehow "fair" because there isn't directly a human in the path. They are as "human as we are" but only proxy and at much lower fidelity.
Not to Godwin the thread ... but big data has been used for immoral purposes since well, big data [1].
The even larger danger, is that people who don't know better, will simply "train a model" and having no overt ill-will, will still create a biased algorithm.
This is simply not true. If we train a mortgage engine on redlining data, it will only reproduce the red line if it's predictive. I.e., the algorithm will only reproduce the red line if the red line is statistically valid.
The fact is that machine learning takes biased inputs and produces unbiased outputs, and this is a pretty normal occurrence. If I tell you that my algorithm to target advertisements treated `displayedInterest x isMobile` with 15% more weight than `displayedInterest x isDesktop` because it corrected for bias induced by latency on mobile connections, you'd think nothing of it.
> This is simply not true. If we train a mortgage engine on redlining data, it will only reproduce the red line if it's predictive. I.e., the algorithm will only reproduce the red line if the red line is statistically valid.
If you train the algorithm to make the same decisions humans did (which is reasonably common when the humans are considered to be effective experts), it will make the same decisions humans did.
If you're comparing outcomes then you'll do better, but even then bias can affect outcomes - if members of group A was sold higher-interest mortgages than equivalent members of group B, then your dataset will show that group B has higher default rates and your algorithm will learn this, even though the difference was only there due to human bias.
But you don't train the algorithm to make the same decision humans did. You train it to maximize your objective function.
I algorithmically trade the stock market. I don't train my model by asking whether it trades the way I would. The whole point is for it to do a better job than me! I train the model to maximize my returns [1] in backtests and simulations - if it trades differently than I do, so much the better!
[1] More precisely, volatility adjusted returns, and I also penalize model complexity to avoid overfitting.
> But you don't train the algorithm to make the same decision humans did. You train it to maximize your objective function.
In the idea case you would, sure. In practice mortgages take 25 years to deliver outcome data and the point isn't necessarily to get better outcomes, the mortgage provider would be content to get exactly the same outcomes (or even slightly worse outcomes) if it let them replace a large number of human employees with an automated system.
In most fields of human endeavour you can't backtest, because you don't have access to the outcomes of the decisions you didn't make. And the data often aren't as clear-cut or objective as market prices. The stock market is great but it's atypical of modern "big data" in many ways.
In most fields of human endeavor you do have an objective function better than "do what humans do".
Mortgages are very specifically an area where you do. First of all, there is historical data.
Second of all, you can backtest well before 25 years. A couple of weeks ago I wrote a blog post explaining specifically how to make measurements in the presence of delayed reactions - I'm discussing a situation involving sensor networks, literally the same mathematics would work for mortgage default or refinance: https://www.chrisstucchio.com/blog/2016/delayed_reactions.ht...
Third, mortgage lenders can often backtest alternate decisions because there are pretty straightforward relationships between decisions. Some of them are even mechanistic, e.g. refinance_risk(interest_rate) and default_prob(interest_rate) are monotonically increasing.
You seem to think we are living in the exact specific dark age necessary to make your morality play poignant and relevant. That's about as silly as Star Trek landing on all sorts of alien planets, each one designed to highlight one specific social issue from USA 1966.
There's no historical data on the default rates of people who didn't get mortgages, because they didn't get mortgages.
I'm willing to believe that theoretical solutions exist. I know from direct personal experience in the big data/lending industry that they are not always applied. If you are claiming that real-world lenders never train their models on human decisions then you are simply wrong.
Nice zinger - I hope you weren't saving it for too long. But I'm not going to get into witticisms or personal arguments.
Sure, but there is plenty of historical data on people who marginally got mortgages. If those people are solidly profitable, as opposed to marginally profitable, you try a marginally more expansive lending criteria.
I.e., if I know f(0) = 10, f(1) = 9, f(2) = 8, but I don't have data on f(3), it's worth running a bit of an experiment to see if f(3) is actually 7.
(I happen to know from experience I can't talk about that this analysis is regularly done, albeit keeping the Lucas Critique in mind.)
I don't claim that real world use of ML is perfect. I claim that "bias" (in the sense of making wrong decisions due to race) is a statistical problem and the solution is simply better algorithms rather than Cathy O'Neil's statistical nihilism.
You're right within the boundaries of what you're considering, meaning testing the profitability of marginal borrowers, etc. But you're missing the big picture question about the use of ML in making political decisions (and yes, the question of who gets mortgages is very much a political one).
For example, in the earliest days of the creation of the US national mortgage market, conforming to FHA standards meant following more or less explicit racial guidelines (not to mention car-promoting platting and urban planning guidelines). That amounted to a political decision to reward people who invested (and/or resided) in white suburbia with greater liquidity and hence price appreciation.
Focusing on the marginal profitability of individual loans ignore the bigger issue, which is that choosing the rules often encodes a much larger set of political preferences or goals. Then, the critique of using opaque algos is, for me, less that they hide the use of some verboten input variable like race or sex, but that by focusing on the algos we ignore the bigger question.
(it's a subclass of what I call the scalar fallacy, where any real world outcome being reduced to a scalar always collapses a multidimensional space into a single dimension, and that projection contains a huge set of silent preferences)
I'm aware that political processes surrounding loans were racially biased. That's not evidence that a profit maximizing algorithm will be. That's really just a claim that "this was once politicized, therefore it must remain political forever".
The whole benefit of using algorithms is that you make your goals explicit and non-opaque; you just look at the objective function. Opacity of the models isn't really a big deal - machine learning is fundamentally the mathematics of studying opaque models. It's the objective function that should be transparent.
If the objective function says "minimize defaults", that's your explicit goal. If the objective function says "maximize # of blacks getting loans", that's what you'll do. But it will do one or the other, unless you explicitly set your objective function to a x "minimize defaults" + b x "maximize blacks".
But unfortunately, an optimization system won't give you what you want unless you explicitly put what you want in the objective function, something our modern activists (probably including Cathy O'Neil) refuse to do.
You're right within the boundaries of what you're considering, meaning testing the profitability of marginal borrowers, etc. But you're missing the big picture question about the use of ML in making political decisions (and yes, the question of who gets mortgages is very much a political one).
For example, in the earliest days of the creation of the US national mortgage market, conforming to FHA standards meant following more or less explicit racial guidelines (not to mention car-promoting platting and urban planning guidelines). That amounted to a political decision to reward people who invested (and/or resided) in white suburbia with greater liquidity and hence price appreciation.
Focusing on the marginal profitability of individual loans ignore the bigger issue, which is that choosing the rules often encodes a much larger set of political preferences or goals. Then, the critique of using opaque algos is, for me, less that they hide the use of some verboten input variable like race or sex, but that by focusing on the algos we ignore the bigger question.
(it's a subclass of what I call the scalar fallacy, where any real world outcome being reduced to a scalar always collapses a multidimensional space into a single dimension, and that projection contains a huge set of silent preferences)
> The fact is that machine learning takes biased inputs and produces unbiased outputs
That is not necessarily true, because it assumes that the biased inputs have no influence on the outputs.
Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often.
Cause and effect are very hard for an algorithm to distinguish when it's the clustered input variables that are linked together for some opaque reason. The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other.
Sure, more training data could help the algorithm distinguish between the two, but there may not be enough training data that shows the inverse -- low-interest loans given to minorities -- if the historical data is biased enough.
Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often.
But if the relationship is actually `defaultProb = a x interest_rate + b`, then the algorithm will reflect that. I discuss this case explicitly in the blog post I linked to in the section "What if black people don't perform as well?", and explicitly give an example of a linear model NOT doing what you claim it will do.
The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other.
No, it wouldn't because you'd also have cases of white people with high interest who defaulted. The model using race as a predictive variable would fail to fit those data points. I explicitly discuss this case in my blog post.
You are correct that in some cases you don't have enough training data. But in these cases, there is no particular reason for bias to have a particular sign. Again, as I discuss in the blog post I linked, why would Captain Kirk be biased against "black on left guy" vs "black on right guy"?
Insufficient training data causes variance, not bias, and could just as easily result in bias in favor of blacks. (In fact, I link to a number of examples where it does.)
> Insufficient training data causes variance, not bias
Key point.
If one were to compare the two high-interest groups, a small sample for one group would be reflected in a large confidence interval around its mean, a greater overlap in the CIs of both group means, and less confidence that a real difference exists.
red-line - "The term refers to the presumed practice of mortgage lenders of drawing red lines around portions of a map to indicate areas or neighborhoods in which they do not want to make loans. Redlining on a racial basis has been held by the courts to be an illegal practice."
What's interesting is if at some point in future regulators extend "red line" to mean variable highly correlated to a protected class. Because if your using a variable and not checking how it correlates to a protected class and making financial based decisions on it. Couldn't you in affect transform the decision into a graph "women" on on side "men" on another(or what ever your favorite protected class is).
I appreciate this analysis, but it's important to note that not everyone wants unbiased model output. Historically, redlining was intentional, and it took laws to stop it. A major risk of automating mortgage engines is that redlining could easily be reinstated (in the form of model parameters) in a way that is both hard to detect and easily deniable.
1) I hate black people more than I like money, but blacks do pay back their loans, so I'll redline even if it costs me money.
2) I like money and am neutral towards blacks, but blacks are deadbeats so redlining gets me more money.
(There are also 2 other cases. "I love/am neutral towards blacks and they pay back their loans", which results in no disparate impact. Also "I hate blacks and they don't pay back their loans" which is equivalent to (2).)
Statistical algorithms solve (1), yet somehow disparate impact isn't fixed. (2) is unspeakable to the modern left - only racists could say that - yet statistics continually reproduces conclusion (2). So obviously the problem must be with the algorithms.
There is also "I am neutral to blacks and view them as, ceteris paribus, equally likely to pay back loans, but white people on average are not neutral toward them, and particularly are not neutral to living in close proximity to them, and the decrease in property values when black people are allowed in otherwise white neighborhoods makes both their default and the default of white people in the neighborhood more likely, and makes deficiency more likely in the event of default of either."
(Statistical models that properly account track all-factors likelihood of default on a particular loan do not solve this.)
True, a model which predicts reality will in fact predict reality.
I've never seen any evidence that your hypothesis is true, or even a model that would account for it, but I'm hardly an expert in the field. Assuming this is more than speculation/theoretical nitpicking, do you have citations on the effects you describe?
If your theory is more than just idle speculation, this is a great way for non-racists to engage in house price arbitrage insofar as they wish to consume housing. Do you know of any neighborhoods in NYC where I can exploit this arbitrage?
> I've never seen any evidence that your hypothesis is true
If the key element of it wasn't true at one point, blockbusting would never have been a viable strategy. If you want to present arguments that race relations have changed in enough since then that whites no longer prefer living without blacks in the immediate vincinity so that such neighborhoods command a price premium and diversity causes price declines, I'm prepared to examine your evidence.
> If your theory is more than just idle speculation, this is a great way for non-racists to engage in house price arbitrage insofar as they wish to consume housing.
If you mean "white non-racists can get cheaper housing by voluntarily choosing to live where black people live", that's been a well-known way for non-racist whites to optimize home affordability since at least when my parents were young adults in the 1960s.
Of course, the downside in jurisdictions where there is a significant constituency of racist whites (which is, of course, any place where this strategy is particularly effective) is that non-racist whites then also get to experience the downside of the way racist whites distribute public services among different neighborhoods, which depending on personal utility functions may more than offset the value of the cost savings on the real estate itself. But, still, the strategy isn't new.
Blockbusting does not have to be caused by white people being non-neutral towards blacks. It can also be caused by white people, perhaps accurately, believing that the neighborhood will change in other undesirable ways (e.g. higher crime).
Similarly, while I'm well aware of (and have practiced in the past) living in black neighborhoods to save money, it was not obviously a racial-arbitrage strategy. The neighborhoods I lived in were undesirable and cheap for a variety of reasons (reduced amenities, higher crime), and it's far from clear how much of a racial delta there was. I don't have any reason to believe that a low amenity, high crime, entirely white neighborhood would have had a different price.
I always viewed it as a crime arbitrage strategy; being 6'6", somewhat muscular and wearing shoes with holes in them, I don't view myself as much of a crime target.
So yes, I'd like to see evidence of either a) whites being non-neutral towards blacks in the statistical algorithms era (e.g. 1995+) and b) a statistical default or prepayment model that incorporates network effects of this sort (or anything mathematically similar).
And incidentally, I'm asking about the networking model because it sounds super interesting. Insofar as network models are reliable enough to use in lending, I'd love to know about it. This would directly affect some of my work if I could port such models into the tech space.
You like money and are neutral toward blacks but there exists a reverse Keynesian beauty contest where you accurately predict that others will redline and hence drive down liquidity and prices, thus making lending less attractive. Hence you are "blamelessly" (but predictably) racist.
You like money and are neutral toward blacks but there exists a reverse Keynesian beauty contest where you accurately predict that others will redline and hence drive down liquidity and prices, thus making lending less attractive. Hence you are "blamelessly" (but predictably) racist.
The article emphasizes the fact that data driven decisions amplify not only efficiencies of the modern society, but also its inefficiencies. Which is probably true for any technology.
On a related note, a Princeton professor and some colleagues just published a blog post and a research paper documenting that "language contains human biases, and so will machines trained on language corpora:" https://news.ycombinator.com/item?id=12356111
> [algorithms] decide who gets access to credit and who pays higher insurance premiums, as well as who will receive online advertising for luxury handbags versus who’ll be targeted by predatory ads for for-profit universities.
As opposed to what? Subjective decisions made by humans based on biases and personal preferences with no semblance of reason or fairness?
> for example, who lives in an area targeted by crime fighting algorithms that add more police to his neighborhood because of higher violent crime rates will necessarily be more likely to be targeted for any petty violation, which adds to a digital profile that could subsequently limit his credit, his job prospects, and so on.
So having more cops in a high-crime neighborhood is somehow a bad thing, because the presence of more police actually confirms the fact that there is more of a need of police in this neighborhood?
> this technology was actually siloing people into online gated communities where they no longer had to even acknowledge the existence of the poor,
So targeted advertising to the needs of people is bad because... I can't even make sense of this one.
> the presence of more police actually confirms the fact that there is more of a need of police in this neighborhood?
If we were to enforce all laws we would need massively more police in every neighborhood. When police are sent to flood a poor neighborhood they end up ticketing poor residents for a bunch of "violations" that they would also find if they went to the rich neighborhoods. But if they ever actually did issue all those ticky-tack tickets in the rich neighborhoods residents would be up in arms contacting their city council members and it would end quickly. In the poor neighborhoods nobody ever hears about it and the residents that can least afford it take the financial hits.
And how would you prove that hypothesis? With some data and an algorithm. The problem you speak of can be fixed by improving the model, not abandoning it.
> they end up ticketing poor residents for a bunch of
> "violations" that they would also find if they went
> to the rich neighborhoods.
I'm not sure what data you have to support this. But check out violent crime rates. Larceny-thefts. Burglary/Property theft. Motor Vehicle Theft. Aggrevated assult. Robbery (mugging, stickups. etc).
These very popular crimes aren't so popular in nicer neighborhoods. Rich neighborhoods have their own problems, but let's not appeal to the spirit of lawlessness (cops are bad, rules are bad, unfairly arrested, etc.)
> So having more cops in a high-crime neighborhood is somehow a bad thing, because the presence of more police actually confirms the fact that there is more of a need of police in this neighborhood?
Did you understand what you quoted? The author is saying that citations for petty offences (which practically everyone commits with some non-zero frequency, intentionally or not) have a higher chance of affecting future prospects of people in communities where there are more police. the Author isn't saying police are bad in general, rather that the negative effects of police presence are more significant in communities with higher police presence.
> So targeted advertising to the needs of people is bad because... I can't even make sense of this one.
The message isn't that hard to get, come on. The author is saying (assuming you think like them) that ignorance of suffering is a bad thing. You can say something like "why should I care about people I don't know", and I would understand and even agree to a large part with that statement, but this isn't James Joyce here, you can make sense of it.
> the Author isn't saying police are bad in general, rather that the negative effects of police presence are more significant in communities with higher police presence.
Yes, but are there any positive effects of police presence? Does this outweight the negative effects?
> The author is saying (assuming you think like them) that ignorance of suffering is a bad thing
So seeing poorly targeted ads tells you that people are suffering? It is the role of corporations to inform you that there are people who have different needs than you?
> Yes, but are there any positive effects of police presence? Does this outweight the negative effects?
Certainly I would agree that police have positive effects as well. Your second question really is the crux of the issue, and I can't answer that.
> So seeing poorly targeted ads tells you that people are suffering? It is the role of corporations to inform you that there are people who have different needs than you?
Suffering was perhaps poor word choice. Obviously it's not the responsibility of some faceless corporation that I be informed about what's going on in the world, but it would be nice if they didn't cut down incidental information the masses might receive so it fits in their little corner of the world.
> petty offences (which practically everyone commits with
> some non-zero frequency,
This isn't Josef K in Kafka's Der Process. Most people don't have any trouble avoiding arrestable behavior.
Lawlessness is not a virtue.
>ignorance of suffering
Ads for bad for-pay universitites are good to remind us of... suffering? And sorry, but even "awareness of suffering" isn't virtuous either.
> Most people don't have any trouble avoiding arrestable behavior.
I think we've pretty well established the past couple of years that there is no lower bound on what's considered "arrestable behavior" when it comes to minorities in the US.
> If there's corruption, that's still a crime. Why not figure out how to highlight it and stop it?
That sounds nice, but in practice, white collar crime, often committed by white people, has much lower incarceration rates than simple drug crimes. Even in Seattle, where its now legal to posses pot but not legal to smoke it publicly (yet this is routinely ignored), we found that black people were being incarcerated at a much higher rate than white people for public smoking of pot. An investigation found that there was one freaking cop who went around arresting tons of black people for this 'crime'.
>> When you find bugs in your code do you throw up your hands and proclaim that it could be anywhere so it's too hard to find and fix?
My rule of thumb for reasoning about things other than code: your code is like your code, and nothing else is quite like your code.
Ergo, don't expect that you can fix everything that's wrong with the Western civilisation just 'cause you can debug something you hacked together in a week or so.
You jumped immediately to an assumption of higher crime in the neighborhood, while the article pointed out the existence of white-collar crime in other neighborhoods.
An interesting similar practical application was differential sentencing for drug dealing outside versus drug dealing inside. White and black Americans use illegal drugs at essentially the same rate, with white Americans pulling ahead in several categories historically. Suburban Americans more often deal drugs inside houses, while urban Americans are more likely to deal drugs outside than suburban Americans. Due in part to redlining practices, black people are more likely to live in urban areas and white people in suburban areas. So just send a bunch of cops to urban neighborhoods and you'll catch plenty of drug dealers and give them higher sentences, while letting all the white drug dealers go and giving them lower sentences if their behavior is egregious enough that you've got to arrest them.
This example doesn't even get into differential treatment for white-collar crime (scamming your investors) vs stealing meat from a grocery store. All those articles we saw on HN about scamming your employees and investors? MotionLoft, 1for.one? Those folks are "just doing business" and most commenters seemed to implicitly condone the scamming behavior -- sure it's unfortunate that it happens but you know business -- while no one here would condone stealing meat from a grocery store, and the meat-lifter would be more likely to get caught since there are cops at the doors of those grocery stores. Do we need cops in the startup scene if scams are happening, though?
The bad thing is not targeting "needs," as above, it's targeting perceived needs or exploiting differences in a way that contributes to injustice. I can see how you might not care about targeted advertising that allows you to ignore the lives of the poor -- that's fairly normal and in the US we really rely on it for political stability -- but ads that only show high-cost high-interest low-placement educational options for poor people push out ads from your local community college, targeted ads contribute to the political polarization we're seeing, targeted ads show higher-paying jobs to men than women regardless of qualifications. There is information asymmetry in a lot of these situations and in several ways: if the poor person trying to look up education doesn't already know about the low-cost legit CC and the information is buried after a page of for-profit ads, they may not understand all their options, and just as bad, we who earn a bit more may not know that's happening, because when we look up education to see if our teen could take multivariable calc nearby, we will get the CC right at the top of the page and the for-profit schools, not targeting us, won't show up! And then we can grumble with a clean conscience that poor people are so dumb they can't even look at the first search result...
Your answer for addressing indoor drug dealing is to not arrest outdoor drug dealing when you see it?
Allow stealing meat because somebody else got away with scamming?
We disallow theft because theft-free is a better way to live. We have decided that scamming your investors is also punishable as fraud.
Don't throw up your hands as if these are impossible issues. Live life and make your decisions to build your community into the places where you want to live.
> So having more cops in a high-crime neighborhood is somehow a bad thing, because the presence of more police actually confirms the fact that there is more of a need of police in this neighborhood?
That's the essence of the Security vs Liberty argument. Yes, on one hand it is a good thing to have more security in an area which needs it; but will you accept say, being filmed 24/7 for more security, with the tradeoff that this same data can be analysed by third parties for any reason whatsoever (ala Minority Report's shopping mall scene).
It is not an easy problem and we must decide for ourselves which tradeoffs we will accept, because it is seldom free technology with no strings attached.
Well this is specifically for high-crime neighborhoods. I would accept less privacy and more policing in high-crime neighborhoods because that reduces the incentives of crime as the perpetrator would be more likely to be caught. When you're talking about marginal benefit (crime rate from 2/100,000 -> 1/100,000), I would be less willing to give up liberties. There is some tipping point, and we can argue where that is, but I still believe in the value of old-fashioned policing to get us to an acceptable risk
I agree with you that for most of these points, the algorithm is unlikely to be less rational or more biased than a human (and the crime-fighting algorithm example doesn't seem to make a good point against the algorithm at all).
However, for something like a credit approval algorithm, there is a valid issue of centralisation - if there are problems with the algorithm, a large number of people will be affected negatively by the same algorithm. While humans are biased and imperfect, you can find some who are less biased, or call their manager and see if they can justify themselves, etc. I can imagine situations where it's hard to escape or find recourse when you're faced with an algorithm that's put you in an unfavourable situation.
I'm hoping that, given enough data to train on and enough validation to make sure they are working optimally, this will not really be a big issue, but I can understand if some people are concerned about that aspect.
I've always taken it for granted that discrimination and inequality would be the natural result of any optimization/ML algorithm, especially when maximizing for profit or any other utilitarian goal. I wonder why people would be surprised by this logical outcome given that it has been long known that statistical discrimination is profitable, and companies rarely have incentives to improve equality and diversity.
I'm surprised the term "disparate impact" hasn't come up. I'm too well versed in anti-discrimination law, but it seems like this is the term for exactly what they're trying to avoid.
I find it insulting when statements start with "Mathematician says", "Experts say". Whoever does this thinks little of his readers. They count on that prefix to make their story more convincing. Sadly, it probably works.
Jerr Thorpe actually has a tool called floodwatch[1] that keeps an eye on your targeted ads and shares at the meta level to try and hold advertisers to the rules. Sadly for it to work you have to browse the internet with no ad-blockers.
Big data has the potential to create either unprecedented equality or inequality. Transparency is a key word in this post (I often Ctrl+F to see how many matches for "transparen" I see in this type of posts).
When it comes to software, one policy to make things more transparent is open-sourcing. The question is opening what and to what degree. I feel strongly that organizations today could and should be much more open and are instead absurdly opaque.
It's our responsibility as a society to talk about transparency as key for trust and equality.
Transparency of pricing models does not immediately result in better outcomes for consumers. In some cases, it may actually result in worse outcomes.
Consider when there are few companies within an industry, like rare metal mining. If they chose to use the same, open-source pricing model, and any changes to that pricing model would be reflected by all companies, the optimal pricing model would be to price at the monopoly level -- to raise the price to where it would be if there were no competition and a single firm in the marketplace.
In the case of lending, however, different firms have different levels of risk acceptance. Many firms (big banks) avoid high-risk borrowers. A few firms (pay-day lenders) are willing to lend to those borrowers, but only at a high interest rate. Somewhere in the middle is a collection of scrupulous lending opportunities.
Any regulation that would restrict the acceptable interest rates to a narrow band would reduce not just the (unscrupulous) loan sharks but also the banks lending to high-risk customers.
This would necessarily result in loss of borrowing ability to those labeled "high-risk".
Is this a desirable outcome? Maybe, maybe not. But it should certainly be discussed.
It's true openness isn't free of side effects. I think a mining company that has exclusive possession over a public resource should be rewarded with the value creation (mining) and not by possession (natural resources). A government should designate the natural resources to public welfare and is ultimately achieved by openness. A resource greedy mining company would rather dispense with transparency in this case.
Regarding the borrower problem. This is a common fear of revealing one weaknesses by openness. I believe society will invent new alternatives to support the new needs. Again, it seems openness is positive overall, despite new challenges it presents.
An interesting point touched on in the article is the extent to which feedback from a predictive algorithm contributes to its later predictions being biased. If police are targeting a specific neighborhood because it is flagged as being a high crime area, then there is likely to be a higher arrest rate for innocent people too, i.e. a higher false positive rate. Unless there is a good way of differentiating false and true positives (which seems unlikely in a justice system heavily focused on plea bargains), then a cycle of over-policing may ensue. A good predictive algorithm would need to account for this type of temporal autocorrelation.
> If police are targeting a specific neighborhood because it is flagged as being a high crime area, then there is likely to be a higher arrest rate for innocent people
Police officers could be employed to periodically monitor a sample of the AI outputs and rate them. Humans can still be in the loop, just not at the lowest level of the loop. I'd have much more confidence in an AI that passes the tests and is independently monitored by humans, than in individuals taking the same decisions. An AI would be much more balanced, informed by more examples and tested to check its biases. Humans can always invoke discretion to justify their choices.
I'd expect diffusion of responsibility or just plain laziness to happen with a human in the loop.
(Side note: This thread is talking about 'broken window' policy.)
Could that be a personnel training problem? The data in such a case would be saying there is a high number of crime victims, even if that number is correlated with the number of people committing crimes. And unless there's a corpse, the reports generally come from victims, not surveillance of criminals committing acts.
Officers making themselves visible on the street and checking to make sure people are safe and finding out from them where dangers might come from, is different than directly treating everyone on the street like they're potential criminals.
Just like programs only know to do what they're coded to do, algorithms only learn what's already in their dataset. It's up to data scientists and analysts to scrub their dataset of potentially discriminatory data. If you're contractually obligated to do otherwise, you should either find another job or do the right thing anyway. If you choose the latter and someone takes action against you, then you should take it to court.
I was in a situation like this in my own work and I regret not standing up for the right thing now...
110 comments
[ 5.8 ms ] story [ 172 ms ] threadThe banks were holding higher tranches of the securitised mortgage pools. While those pools had traditional levels of defaults, the tranches had their expected value. As the defaults accelerated beyond historic levels, they plummeted in value. In both cases the same model said the instrument had/didn't have value. As these instruments are fair value accounted the fact that they were still bringing in cash didn't help any.
However, as the mortgage market started to go south, the default models underestimated the level of defaults, particularly in the equity release market. This underestimation was endemic in the industry at the time, from the banks to the regulators and ratings agencies.
There is, of course, a very strong argument to say that the ignorance was willful, everyone was on quite a nicer earner and reacted late to the evidence as it started appearing.
So there wasn't really a paradox. The banks were holding tranches while the expected defaults were within historic levels. Once it became clear that that wasn't a good predictor anymore, they tried to limit the effect but it turned out that so many participants were in the same boat that there weren't enough buyers.
I have to say I'm not entirely sure what this has to do with ad analytics but I'm not writing a book on the subject so maybe that's not too surprising.
"""
O’Neil also proposes updating existing civic rights oriented legislation to make it clear that it encompasses computerized algorithms. One thing that her book has already made quite clear – far from being coolly scientific, Big Data comes with all the biases of its creators. It’s time to stop pretending that people wielding the most numbers necessarily have the right answers.
Tongue and cheek response, but "wielding numbers"? Politicians don't misuse statistics to get what they want, people misuse statistics to get what we want. We shouldn't ban statistics and we shouldn't ignore the fact that both "sides" are equally susceptible.
Uhm, racial segregation is still a thing. Even if the signal of race is gone the geography of where a person lives could be signal enough to discriminate.
That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race" directly encoded as a feature, but that is insufficient.
See slide 22 from [1], further in depth discussion discussion from [2] or just this feature (which many systems could automatically discover):
I don't know what the solution to this is, but it's a pretty hard problem, and even the best intention are insufficient.Delip proposes in [1] to introduce a "fairness constraint" where the probability of a favourable outcome in the majority class is the same as in a protected class. This sounds sensible, but make optimising systems much harder.
[1] http://deliprao.com/archives/129
[2] https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?
The only think I think worth pointing out is that in the mortgage example the current location of a person isn't as major an influence as income. The fairness test here seems somewhat reasonable: a person on $10K/y who lives in Oakland should probably have the same outcome as someone on $10K/y from some very white area - all other things being equal.
The flip-side is that under a human-biased system, the first white person might get the mortgage based on being white, and the second black person might be denied. That would be reversed under an algorithmic system only considering legitimate factors, and surely that's progress?
Yes, I think so? Or at least it can be audited (in theory at least - although see sentencing guideline systems for the problems with this).
Race is often very predictive. Blacks don't perform worse simply because they have income < $10k. Blacks perform worse on many measures even if you hold all that stuff equal. Here are a few of the many studies reproducing this:
http://ftp.iza.org/dp8733.pdf http://www.mindingthecampus.org/2010/09/the_underperformance... https://randomcriticalanalysis.wordpress.com/2015/05/16/on-c... https://randomcriticalanalysis.wordpress.com/2015/11/22/on-t...
If you were right, then the machine learning algorithm would quickly determine that race doesn't matter. Redundant encoding, direct encoding, etc would be irrelevant - the algorithm would ignore it as noise.
The problem is that the algorithm is supposed to uncover hidden patterns that predict loan default. But then there is one specific hidden pattern that is socially and legally taboo to detect, yet also highly predictive.
(Incidentally, if you can solve this problem in lending, it's easily a unicorn startup.)
Many a time, algorithms infer race from secondary or tertiary variables. This problem is compounded by two more factors - (1) Unable to explain WHY an algorithm has reached a decision aka interpretability and (2) Focus of algorithm building to maximize accuracy regardless of other valid costs. There is some pretty interesting work going on this area.
http://fairness.haverford.edu/
The only possible conclusion is that we should ban Time magazine. Or some such nonsense.
He'll also be less likely to be the victim of violent crime in an area with a proven record for violent crime. For the price of avoiding 'petty' crimes, it seems like a good trade-off.
What usually happens is that the police issue fines so frequently that they become considered an occupying force by the populace. In turn, the populace withholds information from the police, and both violent crimes and petty violations rise.
Police statistics make quantitative conclusions difficult, since they are so often fudged for political purposes. They rise when the police need more funding, and fall when they need political capital. Murder rates are generally considered the only reasonably accurate figures.
On the ground, however, the reality is very clear.
For the petty fines and eroded trust, this is from an advisory letter sent from the Justice Deparment to the judiciary of all fifty states:
“Individuals may confront escalating debt; face repeated, unnecessary incarceration for nonpayment despite posing no danger to the community; lose their jobs; and become trapped in cycles of poverty that can be nearly impossible to escape,” Gupta and Foster wrote. “Furthermore, in addition to being unlawful, to the extent that these practices are geared not toward addressing public safety, but rather toward raising revenue, they can cast doubt on the impartiality of the tribunal and erode trust between local governments and their constituents.”
http://mobile.nytimes.com/2012/06/29/nyregion/new-york-polic...
This seems to be what he is referring to.
(Also, the next bit of the quote, "Yet neighborhoods more likely to commit white collar crime aren’t targeted in this way", suggests a very loose grasp of how policing works. No, they're not, because patrolling the street in front of an office building isn't the least bit effective against white collar crime.)
Do you have any sources for this assertion? Have there been any studies?
Because nobody will make a study to verify that the sky is blue if there isn't at a minimum some dissenting opinion.
How much white-collar crime occurs on the street? Even violent crime off the street won't be helped much by street patrols.
I agree with you on principal but IMO society on Mars will be having this debate (I hope I'm wrong).
Last year I gave a talk on this at 32C3 where I tried to elucidate some of the problems that can pop up when using data analysis to automate things that were previously done by humans:
https://www.youtube.com/watch?v=iRY9IceaVig
Personally I see a lot of potential in data analysis & artificial intelligence, but like all technologies they pose significant risks as well. What we really need therefore (IMHO) is to teach ethics to people that work with data, and establish organizations and methods that ensure data analysis isn't used to do harm (deliberately or not).
Instead of learning lessons, the players continue to 'print money out of thin air'[2] and deny any fault for the last collapse. The rationale seems to be, 2007 was an extreme culmination of events that wouldn't happen again in millions of years. Problem is, in a universe of infinite variables, an infinite number of combinations exist and could manifest anytime a given set of vulnerabilities and unanticipated outlier events present themselves concurrently.
[0] The (Mis)Behavior Of Markets 2004
http://www.goodreads.com/book/show/665134.The_Mis_Behavior_o...
[1] 2006 reminder
http://www.ft.com/cms/s/2/5372968a-ba82-11da-980d-0000779e23...
[2]The Quants
Great Monday Morning QB analysis on what went wrong & why it will be repeated(hint:greed +ability +hubris)
http://www.goodreads.com/book/show/7495395-the-quants
edit: fixed links
Studies seem to indicate that studying and teaching ethics does not make you more inclined towards ethical behavior.[1] On top of that, from personal experience in business ethics courses, "ethical" has an extremely suspicious equivalence with "the thing which limits the company's exposure to liability."
[1] http://www.faculty.ucr.edu/~eschwitz/SchwitzPapers/BehEth-14...
Maybe it's the cynic in me but someone who worked in a hedge fund, started yet another advertising tech company and now releases a book about the evils of "big data" abusing the poor... guess I'm biased.
I firmly believe in the power of data to help people make better decisions, but decision making is always a human process, even if much of the work is delegated to a machine.
Another example (from an old version of Picasa -- I'm not trying to pick on Google but I have firsthand experience of this) is when a friend of mine had photos of two different Asian people in his collection, but Picasa always recognized person B as being person A, presumably because it couldn't distinguish features unique to the faces of Asian people.
[1] http://www.cnet.com/news/google-apologizes-for-algorithm-mis...
Also, it's a bit strange that you suggest ML algos are racist against Asians. You do know that Asians are wildly overrepresented among people building such algos, right?
Consider the possibility that some classification problems are actually more difficult than others. As one example, darker objects are harder to distinguish than bright ones and there are very clear mathematical reasons for that. That's true even in fields where questions of "racism" don't make sense, e.g. in MRI (magnetic resonance imaging is non-optical, but contrast still matters): https://www.chrisstucchio.com/pubs/wf_segment.pdf
As a sibling comment stated, humans are deeply involved in all aspects of designing and interpreting an algorithm's results. To the extent that we are flawed, so will our algorithms be.
Another example I heard recently from a conference was that of analyzing blighted homes in areas. The underlying data was generated by human inspectors who turned out to have given passes to violations in more affluent areas and fines in less affluent areas. The fines compounded the problem of blight as the residents could not afford to pay them and ended up causing a feedback loop of blight increasing in neighborhoods identified as blighted.
I am frankly confused as to what is so controversial about the statement that algorithms created by humans are not free from human biases. I apologize if I have only added confusion rather than a productive analysis.
But that's not true. Inadequate training data will cause variance, not bias. This variance can have any direction.
In a linear prediction scenario (i.e. any go/no go decision process), which exactly what Cathy O'Neil is criticizing, the effect could just as easily be "more loans for black people" as "less loans for black people".
In a multidimensional predictor, it does directly decrease accuracy but that's all. "Black guy -> gorilla" is just as likely as "yummyfajitas -> tank".
Also, realistically, if we want to think about your black guy -> gorilla example, most likely if insufficient training data was the problem, then it was the number of gorillas that was too few. Google has vastly more black humans in their training set than gorillas - a quick google search suggests there are only 100,000 gorillas in the world, most of whom are probably unphotographed.
As a sibling comment stated, humans are deeply involved in all aspects of designing and interpreting an algorithm's results. To the extent that we are flawed, so will our algorithms be.
Sure, algorithms are flawed. But this does not imply that bias must be present proportionally to ours. Insofar as we do good statistics, human bias is mitigated and can even go away. Science works.
As a classical example, consider Morton's analysis of human skulls. Morton was a super racist guy trying to prove how white people are superior. But he did an unbiased analysis and accurately measured skull volume - filling a skull with buckshot is a pretty unbiased process.
http://journals.plos.org/plosbiology/article?id=10.1371/jour...
I am frankly confused as to what is so controversial about the statement that algorithms created by humans are not free from human biases.
What's controversial is that this is the idea of "original sin" but applied to science. And most of the proposed mechanisms are simply mathematically wrong. Also many of the examples cited by proponents of this original sin concept are also wrong.
Regarding big data and ML, I guess the question is then what kind of biases exist in data collection, something that seems dependent on the details and history each particular dataset.
http://www.theverge.com/2016/3/24/11297050/tay-microsoft-cha...
Not to Godwin the thread ... but big data has been used for immoral purposes since well, big data [1].
The even larger danger, is that people who don't know better, will simply "train a model" and having no overt ill-will, will still create a biased algorithm.
[0] http://www.encyclopedia.chicagohistory.org/pages/1050.html
[1] https://en.wikipedia.org/wiki/IBM_and_the_Holocaust
I discuss this in detail here, with numerical examples: https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
The fact is that machine learning takes biased inputs and produces unbiased outputs, and this is a pretty normal occurrence. If I tell you that my algorithm to target advertisements treated `displayedInterest x isMobile` with 15% more weight than `displayedInterest x isDesktop` because it corrected for bias induced by latency on mobile connections, you'd think nothing of it.
If you train the algorithm to make the same decisions humans did (which is reasonably common when the humans are considered to be effective experts), it will make the same decisions humans did.
If you're comparing outcomes then you'll do better, but even then bias can affect outcomes - if members of group A was sold higher-interest mortgages than equivalent members of group B, then your dataset will show that group B has higher default rates and your algorithm will learn this, even though the difference was only there due to human bias.
I algorithmically trade the stock market. I don't train my model by asking whether it trades the way I would. The whole point is for it to do a better job than me! I train the model to maximize my returns [1] in backtests and simulations - if it trades differently than I do, so much the better!
[1] More precisely, volatility adjusted returns, and I also penalize model complexity to avoid overfitting.
In the idea case you would, sure. In practice mortgages take 25 years to deliver outcome data and the point isn't necessarily to get better outcomes, the mortgage provider would be content to get exactly the same outcomes (or even slightly worse outcomes) if it let them replace a large number of human employees with an automated system.
In most fields of human endeavour you can't backtest, because you don't have access to the outcomes of the decisions you didn't make. And the data often aren't as clear-cut or objective as market prices. The stock market is great but it's atypical of modern "big data" in many ways.
Mortgages are very specifically an area where you do. First of all, there is historical data.
Second of all, you can backtest well before 25 years. A couple of weeks ago I wrote a blog post explaining specifically how to make measurements in the presence of delayed reactions - I'm discussing a situation involving sensor networks, literally the same mathematics would work for mortgage default or refinance: https://www.chrisstucchio.com/blog/2016/delayed_reactions.ht...
Third, mortgage lenders can often backtest alternate decisions because there are pretty straightforward relationships between decisions. Some of them are even mechanistic, e.g. refinance_risk(interest_rate) and default_prob(interest_rate) are monotonically increasing.
You seem to think we are living in the exact specific dark age necessary to make your morality play poignant and relevant. That's about as silly as Star Trek landing on all sorts of alien planets, each one designed to highlight one specific social issue from USA 1966.
I'm willing to believe that theoretical solutions exist. I know from direct personal experience in the big data/lending industry that they are not always applied. If you are claiming that real-world lenders never train their models on human decisions then you are simply wrong.
Nice zinger - I hope you weren't saving it for too long. But I'm not going to get into witticisms or personal arguments.
I.e., if I know f(0) = 10, f(1) = 9, f(2) = 8, but I don't have data on f(3), it's worth running a bit of an experiment to see if f(3) is actually 7.
(I happen to know from experience I can't talk about that this analysis is regularly done, albeit keeping the Lucas Critique in mind.)
I don't claim that real world use of ML is perfect. I claim that "bias" (in the sense of making wrong decisions due to race) is a statistical problem and the solution is simply better algorithms rather than Cathy O'Neil's statistical nihilism.
For example, in the earliest days of the creation of the US national mortgage market, conforming to FHA standards meant following more or less explicit racial guidelines (not to mention car-promoting platting and urban planning guidelines). That amounted to a political decision to reward people who invested (and/or resided) in white suburbia with greater liquidity and hence price appreciation.
Focusing on the marginal profitability of individual loans ignore the bigger issue, which is that choosing the rules often encodes a much larger set of political preferences or goals. Then, the critique of using opaque algos is, for me, less that they hide the use of some verboten input variable like race or sex, but that by focusing on the algos we ignore the bigger question.
(it's a subclass of what I call the scalar fallacy, where any real world outcome being reduced to a scalar always collapses a multidimensional space into a single dimension, and that projection contains a huge set of silent preferences)
The whole benefit of using algorithms is that you make your goals explicit and non-opaque; you just look at the objective function. Opacity of the models isn't really a big deal - machine learning is fundamentally the mathematics of studying opaque models. It's the objective function that should be transparent.
If the objective function says "minimize defaults", that's your explicit goal. If the objective function says "maximize # of blacks getting loans", that's what you'll do. But it will do one or the other, unless you explicitly set your objective function to a x "minimize defaults" + b x "maximize blacks".
The tricky bit is that by doing this, we are explicitly and verifiably lowering the bar. Just witness how much flak people get for discussing this: https://techcrunch.com/2015/11/03/twitter-engineering-manage...
But unfortunately, an optimization system won't give you what you want unless you explicitly put what you want in the objective function, something our modern activists (probably including Cathy O'Neil) refuse to do.
For example, in the earliest days of the creation of the US national mortgage market, conforming to FHA standards meant following more or less explicit racial guidelines (not to mention car-promoting platting and urban planning guidelines). That amounted to a political decision to reward people who invested (and/or resided) in white suburbia with greater liquidity and hence price appreciation.
Focusing on the marginal profitability of individual loans ignore the bigger issue, which is that choosing the rules often encodes a much larger set of political preferences or goals. Then, the critique of using opaque algos is, for me, less that they hide the use of some verboten input variable like race or sex, but that by focusing on the algos we ignore the bigger question.
(it's a subclass of what I call the scalar fallacy, where any real world outcome being reduced to a scalar always collapses a multidimensional space into a single dimension, and that projection contains a huge set of silent preferences)
That is not necessarily true, because it assumes that the biased inputs have no influence on the outputs.
Suppose the training data is a bank that has historically given higher-interest loans to minorities. Those loans will default more often, and so the algorithm could "correctly" conclude that minorities default more often.
Cause and effect are very hard for an algorithm to distinguish when it's the clustered input variables that are linked together for some opaque reason. The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other.
Sure, more training data could help the algorithm distinguish between the two, but there may not be enough training data that shows the inverse -- low-interest loans given to minorities -- if the historical data is biased enough.
But if the relationship is actually `defaultProb = a x interest_rate + b`, then the algorithm will reflect that. I discuss this case explicitly in the blog post I linked to in the section "What if black people don't perform as well?", and explicitly give an example of a linear model NOT doing what you claim it will do.
The fact is, both statements in this example would be true: "higher interest is correlated with more defaults" and "minority borrowers are correlated with defaults." The algorithm doesn't know one from the other.
No, it wouldn't because you'd also have cases of white people with high interest who defaulted. The model using race as a predictive variable would fail to fit those data points. I explicitly discuss this case in my blog post.
You are correct that in some cases you don't have enough training data. But in these cases, there is no particular reason for bias to have a particular sign. Again, as I discuss in the blog post I linked, why would Captain Kirk be biased against "black on left guy" vs "black on right guy"?
Insufficient training data causes variance, not bias, and could just as easily result in bias in favor of blacks. (In fact, I link to a number of examples where it does.)
Key point.
If one were to compare the two high-interest groups, a small sample for one group would be reflected in a large confidence interval around its mean, a greater overlap in the CIs of both group means, and less confidence that a real difference exists.
What's interesting is if at some point in future regulators extend "red line" to mean variable highly correlated to a protected class. Because if your using a variable and not checking how it correlates to a protected class and making financial based decisions on it. Couldn't you in affect transform the decision into a graph "women" on on side "men" on another(or what ever your favorite protected class is).
1) I hate black people more than I like money, but blacks do pay back their loans, so I'll redline even if it costs me money.
2) I like money and am neutral towards blacks, but blacks are deadbeats so redlining gets me more money.
(There are also 2 other cases. "I love/am neutral towards blacks and they pay back their loans", which results in no disparate impact. Also "I hate blacks and they don't pay back their loans" which is equivalent to (2).)
Statistical algorithms solve (1), yet somehow disparate impact isn't fixed. (2) is unspeakable to the modern left - only racists could say that - yet statistics continually reproduces conclusion (2). So obviously the problem must be with the algorithms.
(Statistical models that properly account track all-factors likelihood of default on a particular loan do not solve this.)
I've never seen any evidence that your hypothesis is true, or even a model that would account for it, but I'm hardly an expert in the field. Assuming this is more than speculation/theoretical nitpicking, do you have citations on the effects you describe?
If your theory is more than just idle speculation, this is a great way for non-racists to engage in house price arbitrage insofar as they wish to consume housing. Do you know of any neighborhoods in NYC where I can exploit this arbitrage?
If the key element of it wasn't true at one point, blockbusting would never have been a viable strategy. If you want to present arguments that race relations have changed in enough since then that whites no longer prefer living without blacks in the immediate vincinity so that such neighborhoods command a price premium and diversity causes price declines, I'm prepared to examine your evidence.
> If your theory is more than just idle speculation, this is a great way for non-racists to engage in house price arbitrage insofar as they wish to consume housing.
If you mean "white non-racists can get cheaper housing by voluntarily choosing to live where black people live", that's been a well-known way for non-racist whites to optimize home affordability since at least when my parents were young adults in the 1960s.
Of course, the downside in jurisdictions where there is a significant constituency of racist whites (which is, of course, any place where this strategy is particularly effective) is that non-racist whites then also get to experience the downside of the way racist whites distribute public services among different neighborhoods, which depending on personal utility functions may more than offset the value of the cost savings on the real estate itself. But, still, the strategy isn't new.
Similarly, while I'm well aware of (and have practiced in the past) living in black neighborhoods to save money, it was not obviously a racial-arbitrage strategy. The neighborhoods I lived in were undesirable and cheap for a variety of reasons (reduced amenities, higher crime), and it's far from clear how much of a racial delta there was. I don't have any reason to believe that a low amenity, high crime, entirely white neighborhood would have had a different price.
I always viewed it as a crime arbitrage strategy; being 6'6", somewhat muscular and wearing shoes with holes in them, I don't view myself as much of a crime target.
So yes, I'd like to see evidence of either a) whites being non-neutral towards blacks in the statistical algorithms era (e.g. 1995+) and b) a statistical default or prepayment model that incorporates network effects of this sort (or anything mathematically similar).
And incidentally, I'm asking about the networking model because it sounds super interesting. Insofar as network models are reliable enough to use in lending, I'd love to know about it. This would directly affect some of my work if I could port such models into the tech space.
You like money and are neutral toward blacks but there exists a reverse Keynesian beauty contest where you accurately predict that others will redline and hence drive down liquidity and prices, thus making lending less attractive. Hence you are "blamelessly" (but predictably) racist.
You like money and are neutral toward blacks but there exists a reverse Keynesian beauty contest where you accurately predict that others will redline and hence drive down liquidity and prices, thus making lending less attractive. Hence you are "blamelessly" (but predictably) racist.
As opposed to what? Subjective decisions made by humans based on biases and personal preferences with no semblance of reason or fairness?
> for example, who lives in an area targeted by crime fighting algorithms that add more police to his neighborhood because of higher violent crime rates will necessarily be more likely to be targeted for any petty violation, which adds to a digital profile that could subsequently limit his credit, his job prospects, and so on.
So having more cops in a high-crime neighborhood is somehow a bad thing, because the presence of more police actually confirms the fact that there is more of a need of police in this neighborhood?
> this technology was actually siloing people into online gated communities where they no longer had to even acknowledge the existence of the poor,
So targeted advertising to the needs of people is bad because... I can't even make sense of this one.
If we were to enforce all laws we would need massively more police in every neighborhood. When police are sent to flood a poor neighborhood they end up ticketing poor residents for a bunch of "violations" that they would also find if they went to the rich neighborhoods. But if they ever actually did issue all those ticky-tack tickets in the rich neighborhoods residents would be up in arms contacting their city council members and it would end quickly. In the poor neighborhoods nobody ever hears about it and the residents that can least afford it take the financial hits.
I'm not sure what data you have to support this. But check out violent crime rates. Larceny-thefts. Burglary/Property theft. Motor Vehicle Theft. Aggrevated assult. Robbery (mugging, stickups. etc).
These very popular crimes aren't so popular in nicer neighborhoods. Rich neighborhoods have their own problems, but let's not appeal to the spirit of lawlessness (cops are bad, rules are bad, unfairly arrested, etc.)
It makes sense for police to be where serious crime occurs.
Did you understand what you quoted? The author is saying that citations for petty offences (which practically everyone commits with some non-zero frequency, intentionally or not) have a higher chance of affecting future prospects of people in communities where there are more police. the Author isn't saying police are bad in general, rather that the negative effects of police presence are more significant in communities with higher police presence.
> So targeted advertising to the needs of people is bad because... I can't even make sense of this one.
The message isn't that hard to get, come on. The author is saying (assuming you think like them) that ignorance of suffering is a bad thing. You can say something like "why should I care about people I don't know", and I would understand and even agree to a large part with that statement, but this isn't James Joyce here, you can make sense of it.
Yes, but are there any positive effects of police presence? Does this outweight the negative effects?
> The author is saying (assuming you think like them) that ignorance of suffering is a bad thing
So seeing poorly targeted ads tells you that people are suffering? It is the role of corporations to inform you that there are people who have different needs than you?
Certainly I would agree that police have positive effects as well. Your second question really is the crux of the issue, and I can't answer that.
> So seeing poorly targeted ads tells you that people are suffering? It is the role of corporations to inform you that there are people who have different needs than you?
Suffering was perhaps poor word choice. Obviously it's not the responsibility of some faceless corporation that I be informed about what's going on in the world, but it would be nice if they didn't cut down incidental information the masses might receive so it fits in their little corner of the world.
This isn't Josef K in Kafka's Der Process. Most people don't have any trouble avoiding arrestable behavior.
Lawlessness is not a virtue.
>ignorance of suffering Ads for bad for-pay universitites are good to remind us of... suffering? And sorry, but even "awareness of suffering" isn't virtuous either.
It's what you DO about what you know.
I think we've pretty well established the past couple of years that there is no lower bound on what's considered "arrestable behavior" when it comes to minorities in the US.
Really? I wouldn't throw in the towel.
If there's corruption, that's still a crime. Why not figure out how to highlight it and stop it?
When you find bugs in your code do you throw up your hands and proclaim that it could be anywhere so it's too hard to find and fix?
Do you not fix the easy-to-see bugs because that's somehow unfair? Do you not fix the harder-to-find bugs becuase your biased?
You make the software operate according to your vision and purpose. You and go out of your way to ensure order and correctness.
No need to shrug off harder-to-solve social issues as impossible.
That sounds nice, but in practice, white collar crime, often committed by white people, has much lower incarceration rates than simple drug crimes. Even in Seattle, where its now legal to posses pot but not legal to smoke it publicly (yet this is routinely ignored), we found that black people were being incarcerated at a much higher rate than white people for public smoking of pot. An investigation found that there was one freaking cop who went around arresting tons of black people for this 'crime'.
This is the world we live in.
Edited to fix a typo
If you are having problems with drugs contact me and I'll give you details about a clinic in Seattle.
My rule of thumb for reasoning about things other than code: your code is like your code, and nothing else is quite like your code.
Ergo, don't expect that you can fix everything that's wrong with the Western civilisation just 'cause you can debug something you hacked together in a week or so.
For example, "breathing while black."
An interesting similar practical application was differential sentencing for drug dealing outside versus drug dealing inside. White and black Americans use illegal drugs at essentially the same rate, with white Americans pulling ahead in several categories historically. Suburban Americans more often deal drugs inside houses, while urban Americans are more likely to deal drugs outside than suburban Americans. Due in part to redlining practices, black people are more likely to live in urban areas and white people in suburban areas. So just send a bunch of cops to urban neighborhoods and you'll catch plenty of drug dealers and give them higher sentences, while letting all the white drug dealers go and giving them lower sentences if their behavior is egregious enough that you've got to arrest them.
This example doesn't even get into differential treatment for white-collar crime (scamming your investors) vs stealing meat from a grocery store. All those articles we saw on HN about scamming your employees and investors? MotionLoft, 1for.one? Those folks are "just doing business" and most commenters seemed to implicitly condone the scamming behavior -- sure it's unfortunate that it happens but you know business -- while no one here would condone stealing meat from a grocery store, and the meat-lifter would be more likely to get caught since there are cops at the doors of those grocery stores. Do we need cops in the startup scene if scams are happening, though?
The bad thing is not targeting "needs," as above, it's targeting perceived needs or exploiting differences in a way that contributes to injustice. I can see how you might not care about targeted advertising that allows you to ignore the lives of the poor -- that's fairly normal and in the US we really rely on it for political stability -- but ads that only show high-cost high-interest low-placement educational options for poor people push out ads from your local community college, targeted ads contribute to the political polarization we're seeing, targeted ads show higher-paying jobs to men than women regardless of qualifications. There is information asymmetry in a lot of these situations and in several ways: if the poor person trying to look up education doesn't already know about the low-cost legit CC and the information is buried after a page of for-profit ads, they may not understand all their options, and just as bad, we who earn a bit more may not know that's happening, because when we look up education to see if our teen could take multivariable calc nearby, we will get the CC right at the top of the page and the for-profit schools, not targeting us, won't show up! And then we can grumble with a clean conscience that poor people are so dumb they can't even look at the first search result...
Your answer for addressing indoor drug dealing is to not arrest outdoor drug dealing when you see it?
Allow stealing meat because somebody else got away with scamming?
We disallow theft because theft-free is a better way to live. We have decided that scamming your investors is also punishable as fraud.
Don't throw up your hands as if these are impossible issues. Live life and make your decisions to build your community into the places where you want to live.
That's the essence of the Security vs Liberty argument. Yes, on one hand it is a good thing to have more security in an area which needs it; but will you accept say, being filmed 24/7 for more security, with the tradeoff that this same data can be analysed by third parties for any reason whatsoever (ala Minority Report's shopping mall scene).
It is not an easy problem and we must decide for ourselves which tradeoffs we will accept, because it is seldom free technology with no strings attached.
However, for something like a credit approval algorithm, there is a valid issue of centralisation - if there are problems with the algorithm, a large number of people will be affected negatively by the same algorithm. While humans are biased and imperfect, you can find some who are less biased, or call their manager and see if they can justify themselves, etc. I can imagine situations where it's hard to escape or find recourse when you're faced with an algorithm that's put you in an unfavourable situation.
I'm hoping that, given enough data to train on and enough validation to make sure they are working optimally, this will not really be a big issue, but I can understand if some people are concerned about that aspect.
https://en.wikipedia.org/wiki/Disparate_impact
[1] https://floodwatch.o-c-r.org/
When it comes to software, one policy to make things more transparent is open-sourcing. The question is opening what and to what degree. I feel strongly that organizations today could and should be much more open and are instead absurdly opaque.
It's our responsibility as a society to talk about transparency as key for trust and equality.
Consider when there are few companies within an industry, like rare metal mining. If they chose to use the same, open-source pricing model, and any changes to that pricing model would be reflected by all companies, the optimal pricing model would be to price at the monopoly level -- to raise the price to where it would be if there were no competition and a single firm in the marketplace.
In the case of lending, however, different firms have different levels of risk acceptance. Many firms (big banks) avoid high-risk borrowers. A few firms (pay-day lenders) are willing to lend to those borrowers, but only at a high interest rate. Somewhere in the middle is a collection of scrupulous lending opportunities.
Any regulation that would restrict the acceptable interest rates to a narrow band would reduce not just the (unscrupulous) loan sharks but also the banks lending to high-risk customers.
This would necessarily result in loss of borrowing ability to those labeled "high-risk".
Is this a desirable outcome? Maybe, maybe not. But it should certainly be discussed.
Regarding the borrower problem. This is a common fear of revealing one weaknesses by openness. I believe society will invent new alternatives to support the new needs. Again, it seems openness is positive overall, despite new challenges it presents.
Police officers could be employed to periodically monitor a sample of the AI outputs and rate them. Humans can still be in the loop, just not at the lowest level of the loop. I'd have much more confidence in an AI that passes the tests and is independently monitored by humans, than in individuals taking the same decisions. An AI would be much more balanced, informed by more examples and tested to check its biases. Humans can always invoke discretion to justify their choices.
[1]: https://jeremykun.com/2015/07/13/what-does-it-mean-for-an-al...
Officers making themselves visible on the street and checking to make sure people are safe and finding out from them where dangers might come from, is different than directly treating everyone on the street like they're potential criminals.
They were found: http://www.military.com/daily-news/2015/04/03/army-seeks-to-...
I was in a situation like this in my own work and I regret not standing up for the right thing now...