This is bewildering and concerning. I absolutely applaud the authors for releasing their code and asking the public to have a look, as well as raising the alarm. Given the degree of due diligence they've performed, it seems too much to hope that there's just some really nightmarish data leakage occurring; but even if that were the case, the difficulty of identifying its origin or even determining the true characteristics of what the models are doing is extremely problematic regardless of whether this particular instance of bias can be explained.
> Racial identity is a social, legal, and political construct that consists of our own perceptions of our race, and how other people see us.
This seems a pretty US-centric conception. I for one wouldn't know how to assign a "racial identity" to myself, and most of my acquitances would surely say the same thing. I doubt that this study holds much meaning outside the USA.
Now if it was about genetic ancestry it would make some sense (and not really be surprising). But if it is about "self-reported identity" I guess it's only finding the correlation of that with actual genetically transmitted features.
> This seems a pretty US-centric conception. I for one wouldn't know how to assign a "racial identity" to myself, and most of my acquitances would surely say the same thing. I doubt that this study holds much meaning outside the USA.
I’m outside of the US (Australian born, resident in Europe for many years) and I wouldn’t find this difficult. If asked, I would assign myself a racial identity of white/European/caucasian. If someone else was asked what race I was, they’d probably say about the same.
I don’t think that my race is an important part of my identity, mind you, but it is ‘real’ in that it impacts how I am treated by others (that is, it has social consequences).
White Europeans are not Caucasian, because they are not from the Caucasus. There are actual Caucasians so stop abusing that term which was incorrectly applied to Europeans and others based on some outdated pseudoscience. And even if you do use it in the absurd old "Caucasoid" sense, it includes many people like Middle Easterners, Indians etc.
I can think of some reasons why this might be a problem, but the article doesn't seem to make a strong case? Without extrapolating a bit with my imagination, the only conclusion I can draw is "medical ai considers race in it's diagnosis, and it is neither a good nor a bad thing". They seem to leave it as a given that it's bad, but tbh as a layman I could imagine arguments both ways[1]. An article from a professional like this one is where I would expect to find them, along with a summary that effectively "picks a side", with a justification for picking that side.
1: Eg: using race as a clue that some reading might indicate a disease common in that race could be good, on the other side there might be some very expensive scan that could throw off the diagnosis because the training data is skewed towards old white men by the sheer price of doing the scan.
I'm sympathetic to the questions being asked, but I think the author is too quick to assume the worst here.
The key assumption is shown in the diagram that "shows a hypothetical pneumonia detection model that, during optimization, has learned to recognize racial identity." -- the author asserts that this is obviously incorrect and dangerous.
Look at the HPF 50 results, and the fact that the location being imaged is nearly irrelevant -- it's clear that image shape features are irrelevant to the results. It seems like the signal itself is picking up (is modified by) racial indicators. If that's true, it only makes sense that being good at detecting pneumonia is going to require correcting for that signal. That makes (this use of) AI anti-racist in a very important way.
Disclaimer: Worth noting that this paper is apparently pre-print [1] and was submitted to the Lancet, who are not particularly reliable in recent times (flaws hydroxychloroquine study [2], stamped out early investigation in COVID lab-leak possibility [3]).
Regarding the blog and paper:
> AI can trivially learn to identify the self-reported racial identity of patients to an absurdly high degree of accuracy
If it is 'trivial' for a model to learn race, then what they are detecting is likely pretty obvious.
In their paper they use 'private and public datasets' [4]. There is probably exactly where the problem occurs. People of the similar race tend to live in groups together, if you collect data in some area, there's a good chance you source data from the same ethnic pool. What it appears they are detecting is the difference between different machines and practices. To test this, they should collect their own medical data on the same machine. They also need to figure out how to remove the bias of the medical staff themselves.
Another assumption is that just because the medical doctors cannot see differences in the anatomy of races, it doesn't mean there isn't one. One of the commentators on this blog indicated that dark skin may affect the absorption of x-rays, for example.
> AI does learn to do this when trained for clinical tasks
Just because AI models can detect race from medical data, doesn't mean they will. They may in the early stages learn such simplistic relationships, but as accuracy increases, it will be forced to look for more complex classifiers/indicators.
Finally, this blog really skirts around the important thing - how does this affect patient care? Is it beneficial to actually have race-based medical plans? We know that some medicines affect different races - for example in pain and pain management [5].
I think the point to be made here isn't 'racism' but the possibility of bias that people training models should be aware of. Some of this bias will have a positive outcome, some will have a negative outcome.
Nothing in this paper seems surprising. Of course self-reported race/ethnicity corresponds very well to ancestry, and ancestry is related to different traits, and these traits are detected in medical images. The social justice concerns brought up do not really seem relevant, the system is operating as expected.
I sometimes think 'race' is nothing more than a fantastical vanity construct. Really its tribalism. Furthermore, I find it hard as well to comprehend how it holds weight in the medical industry. Race is not real science. Race is entertainment science. AI is mostly entertainment science.
Genetic ancestry is not just a fantastical vanity construct. Genetic ancestry from different populations is real, and while it doesn't correspond exactly to our social "race" categories, it often does. We need to be careful about how we define these ancestral groupings and limit potential negative social impacts from acknowledging their differences, but we can't deny their existence.
10 comments
[ 99.7 ms ] story [ 1298 ms ] threadThis seems a pretty US-centric conception. I for one wouldn't know how to assign a "racial identity" to myself, and most of my acquitances would surely say the same thing. I doubt that this study holds much meaning outside the USA.
Now if it was about genetic ancestry it would make some sense (and not really be surprising). But if it is about "self-reported identity" I guess it's only finding the correlation of that with actual genetically transmitted features.
I’m outside of the US (Australian born, resident in Europe for many years) and I wouldn’t find this difficult. If asked, I would assign myself a racial identity of white/European/caucasian. If someone else was asked what race I was, they’d probably say about the same.
I don’t think that my race is an important part of my identity, mind you, but it is ‘real’ in that it impacts how I am treated by others (that is, it has social consequences).
1: Eg: using race as a clue that some reading might indicate a disease common in that race could be good, on the other side there might be some very expensive scan that could throw off the diagnosis because the training data is skewed towards old white men by the sheer price of doing the scan.
The key assumption is shown in the diagram that "shows a hypothetical pneumonia detection model that, during optimization, has learned to recognize racial identity." -- the author asserts that this is obviously incorrect and dangerous.
Look at the HPF 50 results, and the fact that the location being imaged is nearly irrelevant -- it's clear that image shape features are irrelevant to the results. It seems like the signal itself is picking up (is modified by) racial indicators. If that's true, it only makes sense that being good at detecting pneumonia is going to require correcting for that signal. That makes (this use of) AI anti-racist in a very important way.
This is nonetheless a fascinating result!
Regarding the blog and paper:
> AI can trivially learn to identify the self-reported racial identity of patients to an absurdly high degree of accuracy
If it is 'trivial' for a model to learn race, then what they are detecting is likely pretty obvious.
In their paper they use 'private and public datasets' [4]. There is probably exactly where the problem occurs. People of the similar race tend to live in groups together, if you collect data in some area, there's a good chance you source data from the same ethnic pool. What it appears they are detecting is the difference between different machines and practices. To test this, they should collect their own medical data on the same machine. They also need to figure out how to remove the bias of the medical staff themselves.
Another assumption is that just because the medical doctors cannot see differences in the anatomy of races, it doesn't mean there isn't one. One of the commentators on this blog indicated that dark skin may affect the absorption of x-rays, for example.
> AI does learn to do this when trained for clinical tasks
Just because AI models can detect race from medical data, doesn't mean they will. They may in the early stages learn such simplistic relationships, but as accuracy increases, it will be forced to look for more complex classifiers/indicators.
Finally, this blog really skirts around the important thing - how does this affect patient care? Is it beneficial to actually have race-based medical plans? We know that some medicines affect different races - for example in pain and pain management [5].
I think the point to be made here isn't 'racism' but the possibility of bias that people training models should be aware of. Some of this bias will have a positive outcome, some will have a negative outcome.
[1] https://www.bibsonomy.org/bibtex/c3e2ede2f835d35c08c54a67095...
[2] https://www.nbcnews.com/health/health-news/lancet-retracts-l...
[3] https://www.washingtonexaminer.com/news/wuhan-lab-collaborat...
[4] https://arxiv.org/pdf/2107.10356.pdf
[5] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3654683/