contains text and metadata scraped from 244 websites
tagged as "bullshit" here by the BS Detector Chrome Extension
by Daniel Sieradski. Warning: I did not modify the list
of news sources from the BS Detector so as not to introduce
my (useless) layer of bias
Just a list of domains compiled by a guy. If you want to know more about the level of thinking behind the list of domains, this[1] page linked from the BS detector readme gives the general idea.
I would say it's the product of a very earnest idealist who thinks he has solved the problem because he hasn't spent enough time thinking about the problem to understand the ways in which it is non-trivial.
It includes Breitbart and Drudge Report as fake. Those sites may have a bias you disagree with but no more so than, say, The Daily Kos or Mother Jones.
Can anyone identify any left-of-centre site urls in the list
Completely unsourced material is not news, regardless of if we 'agree' with it. If completely unsourced material is labeled 'news' then this material could reasonably be called 'fake news'.
Breitbart's ex-editor-at-large quit because they "shaped the company into Trump’s personal Pravda, to the extent that he abandoned and undercut his own reporter, Breitbart News’ Michelle Fields, in order to protect Trump’s bully campaign manager, Corey Lewandowski, who allegedly assaulted Michelle."
Did anything like that happen at the sites you name?
At the NYT: "By and large, talented reporters scrambled to match stories with what internally was often called “the narrative.” We were occasionally asked to map a narrative for our various beats a year in advance, square the plan with editors, then generate stories that fit the pre-designated line."
The entire Michelle Fields thing was coordinated attack bullshit that sites like Vox and Mother Jones swallowed uncritically, one might even call it this "fake news" everybody is talking about.
This data was collected using a recently developed tool called the BS Detector [0] which aims to automatically classify fake news. As the labels have not been validated by humans, I question how much utility this will have for developing additional fake news detection systems. Any model trained on this data is really just learning the parameters of the BS Detector, not ground truth fake news.
Classify? I'd expect classification to be able to detect whether something is fake or not regardless of the source. Meanwhile this tool is nothing more than just a simple lookup of sites that the author deemed to be biased or fake.
None of these are backed up with any proof regarding the claim that these sites provide "fake news" - most of these doesn't even have a reason on why these sites are there. If the author cannot provide a reason for these sites to be there then for what reason should I trust the author?
To re-iterate what ManlyBread wrote above, the BS Detector tool uses a fixed list [1] of domains that are deemed to convey 'fake news'; there is no neural network / algorithm / heuristics being used in the tool. Just a blacklist with several categories, curated by humans.
Yeah, it'll definitely need human filtering/tagging prior to training any serious model, but it's neat to have a corpus of stories to start to work from.
I think the biggest issue in "fake news" detection is going to be the model-builders leaking as little of their biases into the model as possible. Objective truth is a tricky thing, and any model which proclaims to be able to represent The Truth should be viewed with extreme suspicion.
I'm livid about the abundant existence and spread of 'fake news' and the inability people have to fact-check, but, I also want to play devil's advocate here.
1) This data set mostly relies on a data-set called the "BS Detector" which describes itself as "[a] hastily assembled...proof of concept...[that] searches all links on a given webpage for references to unreliable sources, checking against a manually compiled list of domains...[to] address the proliferation of fake news". While I agree with their choices in sites, the concept of manual curation deciding the legitimacy of news is horrifying and something that we hoped the internet would move us away from.
2) We run into this problem where we want places like Facebook to censor 'fake news'. But, I imagine most of us can agree that as autonomous moral agents, we'd never want out access to information censored. Rather we'd want the tools to make an informed decision ourselves. In accepting that, you'd have to extend that same right of autonomy to other individuals no matter how 'wrong' you think they may be.
A) We can't manually curate legitimacy list. As that leads to more abuse, and let's face it people who read fake news wouldn't trust your list anyway.
B) We can't let tech companies filter out websites based on their interpretation that leads to more bias especially politically.
C) We can't let crowd-sourced intelligence do it without being locked into the tyranny of the majority, which as we've seen, spreads fake news.
In a time when traditional news agencies are hurting for views, lost in the digital noise and often slower to respond to breaking news than niche outlets, we can't simply give them the benefit of the doubt either. (That would also enshrine them as some sort of news aristocracy which emerging outlets and voices couldn't compete with).
It seems rather than try and interpret the news as fake based on the piece/site itself, we need to take a step back and look and how on the consumption platform (browser, news channel, social media) you can display relevant facts to people.
[Unsubstantiated claim warning] I imagine most credible journalists have at one time another cited an irrefutable fact. A raw figure from a report, a percent by which a poll was won, or the weight of the Eiffel Tower. It seems that with data sets like WikiData and Firebase (old dump, not the new mobile platform) we would be able to locate claims of fact within the writings of journalists and outlets and compare those against the irrefutable fact.
If a journalist or outlet has a number of facts that are incorrectly stated well outside a margin of error, a little pop-up displays those to the user letting them decide how to interpret that.
I know we want to label things as 'Fake' or 'Manipulative' media, and I do think it should be a criminal offense to knowingly publish false information outside the protections of entertainment/satire/art etc. But, no matter how we spin it, categorizing media is more destructive than it is helpful.
We need to find ways to put the educated decision back in the hands of moral agents who are struggling to find facts in an overwhelming sea of noise.
I can see elements of danger if improperly used, but, from my point of view it is no more dangerous than libel laws or any other law which requires explicitly "knowing" what one is doing.
A law like that is dangerous only if the burden of proof is on the journalist. Which in our modern society should never be the case. I admit, sadly there seems to many cases now where innocent until proven guilty isn't as strongly held.
It is tremendously hard to prove that someone 'knew' the information they were publishing was false (outside of an explicit confession or conversation) and it absolutely should be hard. But, such a law may make people who are in it for the money at least give pause.
I can't think of a case where if it isn't fiction, satire, humor, commentary etc etc, and you know it to be false information that you choose to spread as fact, that you aren't doing so with malicious intent to harm others or to personally benefit from the exploitation of others? The social contract of society exists primarily for protection from such actions, and its tool of protection is the law.
If there are cases you can think of where that wouldn't be malicious or exploitative I'm really interested. While I still hold true on the idea that the most important thing with fake news is to empower people to make their own autonomous and informed decisions, I'd quite like to be wrong about criminalizing the spread of false information - as I am a fan of as few laws as possible to get the job done (as long as the job is done well.)
I think before seriously proposing a new law, you should think deeply and long about, at least:
- how exactly the law will be phrased
- how exactly the law will be enforced
- what the penalties shall be
- all the ways you can reasonably anticipate that the law may be misused, whether obvious or subtle
After all that thinking is finished, do a review of the history of laws that are similar to the one that you have proposed and see where and when such laws have existed and how they have worked out in practice.
If, after doing all of that, you still support your original proposal, I will be very much taken aback.
You put forth that a statement was a dangerous idea. When asked why you can't simply state that you believe things should be thought about "deeply and long". That is not a necessary and sufficient condition for things being a "rather dangerous idea."
Let's look at your logic for a second:
1) From the get go you're off with a form of an appeal to a stone (argumentum ad lapidem.) You've suggest that a conclusion is absurd (or in this case dangerous) without any statements as to why or how.
2) You've got an irrelevant conclusion (Ignoratio elenchi) where in you've assumed prima facie that any proposal of a law that does not meet your list of criteria must ultimately be false regardless of its content. (Don't judge a book by its cover?)
3) When asked for why you've shifted the burden of proof (onus probandi) by instead of trying to state why you believe it is false, telling me that if I go research I will most likely change my position.
4) Finally you make an appeal to emotion (in the form of ab absurdo) in suggesting that your conclusion is so simply logical that you would be taken aback if after following your steps I don't believe the same as you. Thus belittling the original argument.
None of this takes into account the fact that it is clear I was not proposing a full new law, nor that your steps would prevent there from being any origin point or idea from which a new law can be born.
I readily welcome feedback on my thoughts, I think the ability to create a dialogue around ideas is crucial. We only grow and craft better ideas when we make mistakes and people point them out. My original statement on the law was brash and fully born from emotion, and in its simple one-sentence state it absolutely is filled with problematic generalizations. I welcome feedback on that, and am happy to dialogue about it. But, if you believe something to be wrong, or off, or dangerous then engage in an open dialogue about it.
When society shuns away from discussions and usurps their idea as 'better' or 'right' with no substantiation because the other person is 'dumb' or 'wrong' or 'evil' or 'uneducated' that is when we reach an impasse where no one wants to listen to people who don't share their ideals. We polarize one another and that leads to animosity and distrust that drives a wedge further between us.
You are right that a one-sentence law about publishing false information would be highly dangerous, but, if you're going to call people out about it online, at least state why you believe that and have an open discussion about it. We'll all be better off for it.
You've made a category error. My indirect suggestion that one should think more deeply before proposing a law was not intended as an argument. It was intended to prompt you to think more deeply.
Do not feel I am attacking you. I simply believe that you have experienced the "there oughtta be a law" emotional reflex to an unpleasant situation and that on further reflection you will realize that that actually won't work well.
Tensions rise with Russia over Ukraine/Syria/Baltics/Eastern Europe/etc. The Pentagon draws up contingency plans for responding to an armed conflict in any one of those areas.
When questioned by a reporter, the White House responds "We have no plans to go to war with Russia."
Should the White House Press Secretary go to jail?
I sympathize with the information-libertarian position that holds that all information should be free, truthful, & accessible to everyone. But there's a really big pragmatic problem in that when individual soundbites are pulled out and shared, they lose context. People don't understand the circumstances behind the facts, and they bring all of their own prejudices and emotions along. Oftentimes, reporting the truth brings about a worse outcome (in some cases, a disastrous outcome) than lying to the public.
It's not just the government either - corporations, institutions, and individuals lie all the time too, and often for good reasons. I was a Google employee when the Nexus line was under development, and I knew that the official press statement of "We are not and have no plans to develop a phone" was false. The reason for it was that if news of the Nexus line leaked, it would've jeopardized supplier relationships such that there wouldn't be a phone. In some cases you can actually get into logical paradoxes where a statement is true only if everyone believes that it is false and vice versa.
I can entirely concede the case for secrecy in these situations.
Obviously, my statement is not a full proposal for a specific law, but rather a starting point. There is certainly a level of nuance (as there is with all laws), but, I think it comes down to the difference between publishing content you know to be false for matters of personal gain, manipulation, malice, etc.
To be a practical law it would need some sort of clause akin to separate the 'actus rea' from the 'mens rea' as well as the impact.
Consider cases in libel. The 'actus rea' (the actual act of committing libel) and saying something defamatory about someone that you believe to be true, is different than the libel which has the 'mens rea' (forethought of the action) in which you made claims you knew to be false but did so for a negative impact against someone else.
The 'actus rea' alone is often swept under the rug with a slap on the wrist, unless the plaintiff proves substantial damages from the act itself. That qualifier can make it a criminal act even without the 'mens rea'.
I agree, there is a lot of nuance to tease out in such a legal concept. My retort to the other commentator was more that if you are going to say something is a bad idea, you have to put forth why and not just put forth rhetoric!
But, once again, I wholeheartedly agree with your examples. I couldn't call for a law that compels the truth. Just something that prevents false publication with the intent to deceive people that it is real for the purposes of malice or ill-gotten self-gain.
The dataset doesn't contain the article URLs which makes further analysis harder. With the URLs you could look at who is linking to the article or website and use that as an indication of authority.
32 comments
[ 2.7 ms ] story [ 83.1 ms ] threadAs per the author of the dataset on kaggle:
I would say it's the product of a very earnest idealist who thinks he has solved the problem because he hasn't spent enough time thinking about the problem to understand the ways in which it is non-trivial.
https://www.inverse.com/article/23781-bs-detector-facebook-f...
Can anyone identify any left-of-centre site urls in the list
Completely unsourced material is not news, regardless of if we 'agree' with it. If completely unsourced material is labeled 'news' then this material could reasonably be called 'fake news'.
Neither Breitbart or Drudge Report are marked as fake in the dataset.
Did anything like that happen at the sites you name?
At the NYT: "By and large, talented reporters scrambled to match stories with what internally was often called “the narrative.” We were occasionally asked to map a narrative for our various beats a year in advance, square the plan with editors, then generate stories that fit the pre-designated line."
http://deadline.com/2016/11/shocked-by-trump-new-york-times-...
It looks like they intentionally included right and left-biased partisan news sites as well as right and left-biased hoax sites
I think this is important because any model you train should be able to distinguish between overt bias and actual fakery
[0] https://github.com/selfagency/bs-detector
None of these are backed up with any proof regarding the claim that these sites provide "fake news" - most of these doesn't even have a reason on why these sites are there. If the author cannot provide a reason for these sites to be there then for what reason should I trust the author?
For example:
[1] https://github.com/selfagency/bs-detector/blob/master/chrome...I think the biggest issue in "fake news" detection is going to be the model-builders leaking as little of their biases into the model as possible. Objective truth is a tricky thing, and any model which proclaims to be able to represent The Truth should be viewed with extreme suspicion.
1) This data set mostly relies on a data-set called the "BS Detector" which describes itself as "[a] hastily assembled...proof of concept...[that] searches all links on a given webpage for references to unreliable sources, checking against a manually compiled list of domains...[to] address the proliferation of fake news". While I agree with their choices in sites, the concept of manual curation deciding the legitimacy of news is horrifying and something that we hoped the internet would move us away from.
2) We run into this problem where we want places like Facebook to censor 'fake news'. But, I imagine most of us can agree that as autonomous moral agents, we'd never want out access to information censored. Rather we'd want the tools to make an informed decision ourselves. In accepting that, you'd have to extend that same right of autonomy to other individuals no matter how 'wrong' you think they may be.
A) We can't manually curate legitimacy list. As that leads to more abuse, and let's face it people who read fake news wouldn't trust your list anyway.
B) We can't let tech companies filter out websites based on their interpretation that leads to more bias especially politically.
C) We can't let crowd-sourced intelligence do it without being locked into the tyranny of the majority, which as we've seen, spreads fake news.
In a time when traditional news agencies are hurting for views, lost in the digital noise and often slower to respond to breaking news than niche outlets, we can't simply give them the benefit of the doubt either. (That would also enshrine them as some sort of news aristocracy which emerging outlets and voices couldn't compete with).
It seems rather than try and interpret the news as fake based on the piece/site itself, we need to take a step back and look and how on the consumption platform (browser, news channel, social media) you can display relevant facts to people.
[Unsubstantiated claim warning] I imagine most credible journalists have at one time another cited an irrefutable fact. A raw figure from a report, a percent by which a poll was won, or the weight of the Eiffel Tower. It seems that with data sets like WikiData and Firebase (old dump, not the new mobile platform) we would be able to locate claims of fact within the writings of journalists and outlets and compare those against the irrefutable fact.
If a journalist or outlet has a number of facts that are incorrectly stated well outside a margin of error, a little pop-up displays those to the user letting them decide how to interpret that.
I know we want to label things as 'Fake' or 'Manipulative' media, and I do think it should be a criminal offense to knowingly publish false information outside the protections of entertainment/satire/art etc. But, no matter how we spin it, categorizing media is more destructive than it is helpful.
We need to find ways to put the educated decision back in the hands of moral agents who are struggling to find facts in an overwhelming sea of noise.
That is a rather dangerous idea.
A law like that is dangerous only if the burden of proof is on the journalist. Which in our modern society should never be the case. I admit, sadly there seems to many cases now where innocent until proven guilty isn't as strongly held.
It is tremendously hard to prove that someone 'knew' the information they were publishing was false (outside of an explicit confession or conversation) and it absolutely should be hard. But, such a law may make people who are in it for the money at least give pause.
I can't think of a case where if it isn't fiction, satire, humor, commentary etc etc, and you know it to be false information that you choose to spread as fact, that you aren't doing so with malicious intent to harm others or to personally benefit from the exploitation of others? The social contract of society exists primarily for protection from such actions, and its tool of protection is the law.
If there are cases you can think of where that wouldn't be malicious or exploitative I'm really interested. While I still hold true on the idea that the most important thing with fake news is to empower people to make their own autonomous and informed decisions, I'd quite like to be wrong about criminalizing the spread of false information - as I am a fan of as few laws as possible to get the job done (as long as the job is done well.)
- how exactly the law will be phrased
- how exactly the law will be enforced
- what the penalties shall be
- all the ways you can reasonably anticipate that the law may be misused, whether obvious or subtle
After all that thinking is finished, do a review of the history of laws that are similar to the one that you have proposed and see where and when such laws have existed and how they have worked out in practice.
If, after doing all of that, you still support your original proposal, I will be very much taken aback.
Let's look at your logic for a second:
1) From the get go you're off with a form of an appeal to a stone (argumentum ad lapidem.) You've suggest that a conclusion is absurd (or in this case dangerous) without any statements as to why or how.
2) You've got an irrelevant conclusion (Ignoratio elenchi) where in you've assumed prima facie that any proposal of a law that does not meet your list of criteria must ultimately be false regardless of its content. (Don't judge a book by its cover?)
3) When asked for why you've shifted the burden of proof (onus probandi) by instead of trying to state why you believe it is false, telling me that if I go research I will most likely change my position.
4) Finally you make an appeal to emotion (in the form of ab absurdo) in suggesting that your conclusion is so simply logical that you would be taken aback if after following your steps I don't believe the same as you. Thus belittling the original argument.
None of this takes into account the fact that it is clear I was not proposing a full new law, nor that your steps would prevent there from being any origin point or idea from which a new law can be born.
I readily welcome feedback on my thoughts, I think the ability to create a dialogue around ideas is crucial. We only grow and craft better ideas when we make mistakes and people point them out. My original statement on the law was brash and fully born from emotion, and in its simple one-sentence state it absolutely is filled with problematic generalizations. I welcome feedback on that, and am happy to dialogue about it. But, if you believe something to be wrong, or off, or dangerous then engage in an open dialogue about it.
When society shuns away from discussions and usurps their idea as 'better' or 'right' with no substantiation because the other person is 'dumb' or 'wrong' or 'evil' or 'uneducated' that is when we reach an impasse where no one wants to listen to people who don't share their ideals. We polarize one another and that leads to animosity and distrust that drives a wedge further between us.
You are right that a one-sentence law about publishing false information would be highly dangerous, but, if you're going to call people out about it online, at least state why you believe that and have an open discussion about it. We'll all be better off for it.
Do not feel I am attacking you. I simply believe that you have experienced the "there oughtta be a law" emotional reflex to an unpleasant situation and that on further reflection you will realize that that actually won't work well.
Tensions rise with Russia over Ukraine/Syria/Baltics/Eastern Europe/etc. The Pentagon draws up contingency plans for responding to an armed conflict in any one of those areas.
When questioned by a reporter, the White House responds "We have no plans to go to war with Russia."
Should the White House Press Secretary go to jail?
I sympathize with the information-libertarian position that holds that all information should be free, truthful, & accessible to everyone. But there's a really big pragmatic problem in that when individual soundbites are pulled out and shared, they lose context. People don't understand the circumstances behind the facts, and they bring all of their own prejudices and emotions along. Oftentimes, reporting the truth brings about a worse outcome (in some cases, a disastrous outcome) than lying to the public.
It's not just the government either - corporations, institutions, and individuals lie all the time too, and often for good reasons. I was a Google employee when the Nexus line was under development, and I knew that the official press statement of "We are not and have no plans to develop a phone" was false. The reason for it was that if news of the Nexus line leaked, it would've jeopardized supplier relationships such that there wouldn't be a phone. In some cases you can actually get into logical paradoxes where a statement is true only if everyone believes that it is false and vice versa.
Obviously, my statement is not a full proposal for a specific law, but rather a starting point. There is certainly a level of nuance (as there is with all laws), but, I think it comes down to the difference between publishing content you know to be false for matters of personal gain, manipulation, malice, etc.
To be a practical law it would need some sort of clause akin to separate the 'actus rea' from the 'mens rea' as well as the impact.
Consider cases in libel. The 'actus rea' (the actual act of committing libel) and saying something defamatory about someone that you believe to be true, is different than the libel which has the 'mens rea' (forethought of the action) in which you made claims you knew to be false but did so for a negative impact against someone else.
The 'actus rea' alone is often swept under the rug with a slap on the wrist, unless the plaintiff proves substantial damages from the act itself. That qualifier can make it a criminal act even without the 'mens rea'.
I agree, there is a lot of nuance to tease out in such a legal concept. My retort to the other commentator was more that if you are going to say something is a bad idea, you have to put forth why and not just put forth rhetoric!
But, once again, I wholeheartedly agree with your examples. I couldn't call for a law that compels the truth. Just something that prevents false publication with the intent to deceive people that it is real for the purposes of malice or ill-gotten self-gain.