From the article: "... the researchers are concerned that AI could end up perpetuating racial biases. They are especially concerned that they could not figure out precisely how the machine-learning models were able to identify race, even from heavily corrupted and low-resolution images."
What is the closest thing to a "stack trace" to debug and understand AI neural network output? I'm new to this and curious of where to start looking to further understand how one would reverse engineer the logic or reasoning behind AI output.
With very deep models based on Convolution (CNNs, the article is talking about AI on a computer vision problem and CNNs are the standard for these problems) you loose almost all explainability. There might be ways to partially mitigate that like adding additional "heads" to the model to output some debugging-like data, or run the image in patches and see where the model is more confident about a particular metric of interest. Both these options are not optimal as you still don't see the inner workings of the model in an interpretable way.
Transformers models [1], on the other hand, are the de-facto standard for Natural Language Processing/Understanding problems and recent studies [2] show that they can be applied to computer vision tasks. By construction, transformers let you access the so called "attention heads", which carry a good deal of information about what the model believes is important about the input. So that might be a promising route for some explainability in computer vision tasks, but we are quite a bit far away from fully understanding what is going on inside a deep learning model.
Masking out parts of the image and monitoring how each part changes the prediction. This lets you porduce and "importance map" about which parts of the image contribute more to the prediction.
The Artificial Neural Network is essentially just a huge set of formulas involving millions and sometimes even billions of values. Debugging in conventional software engineering sense of having a breakpoint and looking at values does not work.
ShAP is primarily an instance based explainer (one image = one explanation) but if you run it over multiple instances it is possible to gather global model insights on the data. The internals of the model are still quite unexplainable compared to decision trees or anything a human can code (horrible code aside).
While not directly related to explainability their work is on providing bounds on how much can corruption of the input still provide valid output. Very interesting and practically relevant as well.
Finally there is also common sense. If race was a large factor in prediction then the model will implicitly learn to predict races. I am not in medicine and do not know how much it is but if it is then the only way to not learn race prediction is to make race not correlated to the targets.
The quite interesting technical and analytical problem of Deep Learner I suppose, aside.
Well, there are medical differences between races.
The problem is, unlike some of those between the sexes, they are largely based on social history, not biology.
Additionally, these are statistical values that relate to groups of individuals.
The question is whether or not automated racial classification improves the treatment of individuals.
Another aspect is, a medical diagnosis does not consist of an X-ray alone, but of a conglomeration of assured methodologies and medical experience of the physician.
Which raises the question of whether or not, through classification by race, biases of the practitioners are reinforced or not.
> Well, there are medical differences between races. The problem is, unlike some of those between the sexes, they are largely based on social history, not biology.
And once again, this sort of pseudoscience masquerading as science emerges, simply because it fits the American cultural meme that race should not have any inherent impact on outcomes. Hell, the article in the post contradicts that point too
Race, as it is right now, is a surrogate variable for a number of genetic factors. It's certainly less precise than some sort of genetic marker analysis, but it's low cost and notable improvement in performance means that it still has a distinct role to play, especially since we're nowhere near post-scarcity.
The argument that race is a social construct with no objective source is also not a good counter to this. While true, it also discounts the strong relation between declared race and genetic history. Increasing levels of mixing may reduce this correlation though, requiring a revisit in the future.
> Even when the resolution of the scan was reduced to 4 x 4 pixels, the predictions were still better than random chance – and by the time resolution was increased to 160 x 160 pixels, accuracy was over 95 percent.
> but that it appears likely that it will be remarkably difficult to debias these systems
They are not using the term bias correctly. If the model learns these different modes, as opposed to a mean between the modes, it means the model has low bias.
13 comments
[ 3.7 ms ] story [ 37.4 ms ] threadWhat is the closest thing to a "stack trace" to debug and understand AI neural network output? I'm new to this and curious of where to start looking to further understand how one would reverse engineer the logic or reasoning behind AI output.
Transformers models [1], on the other hand, are the de-facto standard for Natural Language Processing/Understanding problems and recent studies [2] show that they can be applied to computer vision tasks. By construction, transformers let you access the so called "attention heads", which carry a good deal of information about what the model believes is important about the input. So that might be a promising route for some explainability in computer vision tasks, but we are quite a bit far away from fully understanding what is going on inside a deep learning model.
[1] https://arxiv.org/abs/1706.03762 [2] https://arxiv.org/abs/2010.11929
http://cnnlocalization.csail.mit.edu/
Can this help?
The current state of the art for analysis is ShAP: - https://github.com/slundberg/shap
ShAP is primarily an instance based explainer (one image = one explanation) but if you run it over multiple instances it is possible to gather global model insights on the data. The internals of the model are still quite unexplainable compared to decision trees or anything a human can code (horrible code aside).
There is a group at ETH doing work on adversarial attacks: - https://www.sri.inf.ethz.ch/publications/
While not directly related to explainability their work is on providing bounds on how much can corruption of the input still provide valid output. Very interesting and practically relevant as well.
Finally there is also common sense. If race was a large factor in prediction then the model will implicitly learn to predict races. I am not in medicine and do not know how much it is but if it is then the only way to not learn race prediction is to make race not correlated to the targets.
Well, there are medical differences between races.
The problem is, unlike some of those between the sexes, they are largely based on social history, not biology.
Additionally, these are statistical values that relate to groups of individuals.
The question is whether or not automated racial classification improves the treatment of individuals.
Another aspect is, a medical diagnosis does not consist of an X-ray alone, but of a conglomeration of assured methodologies and medical experience of the physician.
Which raises the question of whether or not, through classification by race, biases of the practitioners are reinforced or not.
https://magazine.ucsf.edu/whats-wrong-and-right-race-medicin...
And once again, this sort of pseudoscience masquerading as science emerges, simply because it fits the American cultural meme that race should not have any inherent impact on outcomes. Hell, the article in the post contradicts that point too
Race, as it is right now, is a surrogate variable for a number of genetic factors. It's certainly less precise than some sort of genetic marker analysis, but it's low cost and notable improvement in performance means that it still has a distinct role to play, especially since we're nowhere near post-scarcity.
The argument that race is a social construct with no objective source is also not a good counter to this. While true, it also discounts the strong relation between declared race and genetic history. Increasing levels of mixing may reduce this correlation though, requiring a revisit in the future.
These populations do not correspond to that what is understood under the term 'race' which is much older and uses phenotypic phenomena.
These were largely used to identify exploitable human resources.
Funnily enough, ancient societies were not so picky about them and thus probably did not need a classifier to determine them.
They are not using the term bias correctly. If the model learns these different modes, as opposed to a mean between the modes, it means the model has low bias.