I think the data the paper presents doesn't warrant the conclusion.
On the contrary, being "not biased" is predefined here as it would require you to use the same amount of active/passive voice and terms for violence without regard to what is actually happening.
That can be dangerously wrong. It could also mean that Palestinians commit more violence for example. Doesn't mean the NYT is biased. You would need to quantify reality first to be able to compare it to language used.
"disproportionate use of passive voice..." seems to be a healthy definition of bias to me. I did not have the time to properly analyse all 33000 articles myself, so I'm not in a position to validate the data.
Seems quite natural to me. When dealing with sensitive issues, you may want to use less assertive instruments of speech, like "20 people were killed on the Israeli side" rather than "Palestinians killed 20 Israelis." For proper research you'd need to compare at least against other pieces of military conflicts reporting.
Putting numbers on the suffering is important work, but I don't like the conclusion that are drawn from it. They are mostly invalid.
If Israel attacks Palestine and kills a citizen, Palestine retaliates and kills 5 in return. Should they have stopped after killing just one person? It that a non-biased approach? What is the ethical approach? Kill just one? Kill nobody? Kill as much as you can?
Doesn't allow for conclusions about bias either, especially not in an armed conflict.
I don’t think your linked ground truth is useful if the author wanted to try to establish reality or a “non-biased” baseline.
I don’t see any attempt in the paper to try to do this or to use your source, or others.
The number of fatalities would be hard, I think, to try to show what the violent wording would be used since it quantifies deaths (eg, 200 deaths in a single incident would be reported different than 200 events with one death each).
But perhaps trying to determine a correlation of reporting to deaths, or reporting to events. Or any attempt.
The biggest problem I have with the paper is that there’s no null hypothesis to disprove, so it “proves bias” without establishing or even estimating what a non-biased corpus looks like.
This seems like an interesting claim. One way to validate it might be to run this analysis on the same topic on different news sources and see if the relative biases of the sources matches our intuitions.
For example, I would expect to find that Fox News uses more violent descriptions of BLM protestors, and that the Washington Post uses fewer; I would expect the converse when describing the Jan 6 US Capitol riots.
This kind of comparison doesn't say whether the BLM protests are more or less violent than the Capitol riots, but it would say whether such an analysis hews to our intuition about relative bias between news sources.
This sounds pretty biased itself, first of all: we would need to compare the reported news to "neutral" facts, if the Hamas actually did initiated let's say 20% more violence (illustrative figure) than Israel then it would make sense that roughly 20% reports were made to report such events. ML in this case is just automated bias, you can't look at real world in absolute, a better thing (maybe) would be to compare the NYT's reports to other new outlets.
Obviously this depends a lot on how you define the metric.
It seems to me (superficially) that "like-for-like" metrics such as "use of passive voice" should be fairly independent of reality; for a similar example, see the American tendency to refer to "officer-involved shootings" (but, in a clear case of bias, to never say "dog-involved bitings").
On the other hand, as you note, "use of violent terms" could be a reflection of real-world differences in violent actions, so it might be a more fraught metric.
I'm really intrigued by the idea that we can bring this kind of quantitative analysis to bear to reveal hidden bias, though; it sounds like it can be a powerful approach. But you're certainly right about potential pitfalls.
> I prove a history of disproportionate bias against Palestinians in the New York Times.
This is a really bold statement for any analysis and seems odd to come put of academic work that’s supervised.
This preprint is submitted to the Journal of Palestinian Studies so it will be interesting to see how their peer review checks this out.
I don’t quite understand this conclusion based on the paper as it seems like the author looked for existence of some words and for use of active vs passive voice.
I don’t understand how active vs passive voice is a sign of bias unless you control for many other factors not mentioned. Are specific authors more likely, is time period affected, are other topics similar, etc etc. I don’t get the logic that this shows a bias in general and don’t see anything presented about why these 33k articles are different.
Similarly, I’m not sure how tracking the mention of Palestinian or Israeli people just by word count means anything. Wouldn’t this be impacted by events, this isn’t addressed in the paper. Are there more deaths or attacks in different groups. This seems important to address, but I couldn’t find anything. I’m also not sure how the 33k articles were selected so whether articles were included or excluded would impact these results.
Similarly not sure if the sentiment analysis of the NLP kits she used is similar or dissimilar from NYT in general.
And the determination of violent language seems like a manual exercise with no validation and would like to know more about the methods used to create the set of violent words. I don’t see the actual training_data.csv that the author created and would be interested to see what was actually made since that file seems to be the output of a manual process to sample 500 articles and pick words.
These are good points. I think the claims are a bit bold.
I do think the "active vs passive voice" is an interesting metric, since it seems intuitively a bit harder to know why that would differ for the two groups.
As I noted below, in the US it's common to note that the phrase "officer-involved shooting" (i.e., passive voice) is often used to describe cases where police officers shoot someone, but "dog-involved biting" is rarely used to describe the case where a dog bites someone. This strikes me as a convincing (though perhaps not ironclad) argument for how such metrics—which are seemingly independent of the rate of mentions (which itself could be reflecting world events)—can shed light on authorial bias.
I think you could construct similar "event-independent" metrics; for example, are victims of violence more or less likely to be named when they are of one group or another? Are direct quotes from spokespeople more likely to be used from one group or another? Etc.
I generally argue against trying to make all journalism just "impartial he-said, she-said", so I'm a little uncomfortable with this exercise, but, again, the "officer-involved shooting" case is one where very clear language bias seems to affect the discourse.
I think the active/passive thing is worth expiration. But to 1) conclude that passive=bad seems to need evidence for why that is and 2) is there a significant difference for the Israeli/Palestine article set.
Both of those seem just stated without evidence so it seems odd that the MIT supervisors would have a student write this us.
It's obvious. Any entity or agency that sides with the Zionist state, which was formed on stolen and occupied Palestinian lands, is biased. The tides finally seem to be changing, and truth is starting to dawn on people.
Love that students are thinking about how they can apply what they learn to real problems.
However, choosing such a charged topic needs to be managed by an advisor - @h_jackson_ is about to get a lot more than she probably bargained for in terms of people arguing good faith, twisting her results to their narrative, and several arm chair scientists that are going to tear her down despite probably never actually reading what she's worked on.
Is her analysis perfect, no. Is it likely flawed, yea probably. This is where a professor can try to tone down the flames and push toward what might be next in the science of getting a right answer.
i.e. when you go in with this belief to quote her "The results of this study are now as relevant as ever. Today, the New York Times continues its legacy of Palestinian erasure."
how can one expect the results to be anything but something that reflects that. either the author would continue to tweak and tweak and tweak till they get the results desired or simply not publish something that contradicts their worldview.
While I'm a big fan of this NLP analysis, it's disingenuous to state ML can "prove" something. Journalistic bias is fundamentally a subjective judgement; 90% accurate ML models don't make the claim true.
24 comments
[ 0.35 ms ] story [ 73.0 ms ] threadI think the data the paper presents doesn't warrant the conclusion.
On the contrary, being "not biased" is predefined here as it would require you to use the same amount of active/passive voice and terms for violence without regard to what is actually happening.
That can be dangerously wrong. It could also mean that Palestinians commit more violence for example. Doesn't mean the NYT is biased. You would need to quantify reality first to be able to compare it to language used.
After trying to look it up I found this article (2003):
http://itre.cis.upenn.edu/~myl/languagelog/archives/000236.h...
I thought that funny.
But I think it remains true, that you cannot infer bias by only observing language.
Also oddly, the paper didn’t include a statistical analysis of whether these differences are even significant.
Yes, we need to have a ground truth with regard to violence committed. But that data is already published elsewhere
https://www.un.org/unispal/document/auto-insert-208380/
If Israel attacks Palestine and kills a citizen, Palestine retaliates and kills 5 in return. Should they have stopped after killing just one person? It that a non-biased approach? What is the ethical approach? Kill just one? Kill nobody? Kill as much as you can?
Doesn't allow for conclusions about bias either, especially not in an armed conflict.
I don’t see any attempt in the paper to try to do this or to use your source, or others.
The number of fatalities would be hard, I think, to try to show what the violent wording would be used since it quantifies deaths (eg, 200 deaths in a single incident would be reported different than 200 events with one death each).
But perhaps trying to determine a correlation of reporting to deaths, or reporting to events. Or any attempt.
The biggest problem I have with the paper is that there’s no null hypothesis to disprove, so it “proves bias” without establishing or even estimating what a non-biased corpus looks like.
For example, I would expect to find that Fox News uses more violent descriptions of BLM protestors, and that the Washington Post uses fewer; I would expect the converse when describing the Jan 6 US Capitol riots.
This kind of comparison doesn't say whether the BLM protests are more or less violent than the Capitol riots, but it would say whether such an analysis hews to our intuition about relative bias between news sources.
It seems to me (superficially) that "like-for-like" metrics such as "use of passive voice" should be fairly independent of reality; for a similar example, see the American tendency to refer to "officer-involved shootings" (but, in a clear case of bias, to never say "dog-involved bitings").
On the other hand, as you note, "use of violent terms" could be a reflection of real-world differences in violent actions, so it might be a more fraught metric.
I'm really intrigued by the idea that we can bring this kind of quantitative analysis to bear to reveal hidden bias, though; it sounds like it can be a powerful approach. But you're certainly right about potential pitfalls.
This is a really bold statement for any analysis and seems odd to come put of academic work that’s supervised.
This preprint is submitted to the Journal of Palestinian Studies so it will be interesting to see how their peer review checks this out.
I don’t quite understand this conclusion based on the paper as it seems like the author looked for existence of some words and for use of active vs passive voice.
I don’t understand how active vs passive voice is a sign of bias unless you control for many other factors not mentioned. Are specific authors more likely, is time period affected, are other topics similar, etc etc. I don’t get the logic that this shows a bias in general and don’t see anything presented about why these 33k articles are different.
Similarly, I’m not sure how tracking the mention of Palestinian or Israeli people just by word count means anything. Wouldn’t this be impacted by events, this isn’t addressed in the paper. Are there more deaths or attacks in different groups. This seems important to address, but I couldn’t find anything. I’m also not sure how the 33k articles were selected so whether articles were included or excluded would impact these results.
Similarly not sure if the sentiment analysis of the NLP kits she used is similar or dissimilar from NYT in general.
And the determination of violent language seems like a manual exercise with no validation and would like to know more about the methods used to create the set of violent words. I don’t see the actual training_data.csv that the author created and would be interested to see what was actually made since that file seems to be the output of a manual process to sample 500 articles and pick words.
I do think the "active vs passive voice" is an interesting metric, since it seems intuitively a bit harder to know why that would differ for the two groups.
As I noted below, in the US it's common to note that the phrase "officer-involved shooting" (i.e., passive voice) is often used to describe cases where police officers shoot someone, but "dog-involved biting" is rarely used to describe the case where a dog bites someone. This strikes me as a convincing (though perhaps not ironclad) argument for how such metrics—which are seemingly independent of the rate of mentions (which itself could be reflecting world events)—can shed light on authorial bias.
I think you could construct similar "event-independent" metrics; for example, are victims of violence more or less likely to be named when they are of one group or another? Are direct quotes from spokespeople more likely to be used from one group or another? Etc.
I generally argue against trying to make all journalism just "impartial he-said, she-said", so I'm a little uncomfortable with this exercise, but, again, the "officer-involved shooting" case is one where very clear language bias seems to affect the discourse.
Both of those seem just stated without evidence so it seems odd that the MIT supervisors would have a student write this us.
It's obvious. Any entity or agency that sides with the Zionist state, which was formed on stolen and occupied Palestinian lands, is biased. The tides finally seem to be changing, and truth is starting to dawn on people.
However, choosing such a charged topic needs to be managed by an advisor - @h_jackson_ is about to get a lot more than she probably bargained for in terms of people arguing good faith, twisting her results to their narrative, and several arm chair scientists that are going to tear her down despite probably never actually reading what she's worked on.
Is her analysis perfect, no. Is it likely flawed, yea probably. This is where a professor can try to tone down the flames and push toward what might be next in the science of getting a right answer.
i.e. when you go in with this belief to quote her "The results of this study are now as relevant as ever. Today, the New York Times continues its legacy of Palestinian erasure."
how can one expect the results to be anything but something that reflects that. either the author would continue to tweak and tweak and tweak till they get the results desired or simply not publish something that contradicts their worldview.
That said, the paper itself is a good read.
http://web.mit.edu/hjackson/www/The_NYT_Distorts_the_Palesti...