The author describes a "fake news detector AI", that is actually a "typically legitimate source of news" data model, combined with a fake news domain blacklist. It doesn't detect fake news. It detects whether a story possibly came from a source you find to typically be legitimate.
Yeah; I don't doubt that there's a lot of value in the collection of data they built, but it's hard to judge the value of "95%" accuracy without comparison to a baseline like `is_fake = !whitelist.includes(article_domain)`. That whitelist is basically how our brains work currently, and my own (biased, perhaps) accuracy rate is something close to 100%.
A "true fake news detector" would use only the text of the article -- without the URL. And so I agree that this article is kind of like fake news of the "misleading" variety. :)
OK. And? Is the point to be perfect, or is the point to be useful?
False fakes--and a wisely designed system would probably say "probably fake" in the first place, not outright "fake"--are less damaging than false truths overall.
>False fakes--and a wisely designed system would probably say "probably fake" in the first place, not outright "fake"--are less damaging than false truths overall.
That depends on what your goal is: to discover groundbreaking truths (which are almost always considered taboo/false at first) in order to improve society, or merely attempt to cover up apparent falsehoods as damage control.
In the latter case, such a system would be highly prone to enforcing the prexisting biases and prejudices of its developers onto the users, making it much more difficult for them to discover truths that the developers are ignorant of.
From the article, it does seem that the training and test sets were both generating from a dubious data collection process: scraping URLs they knew were fake/satire/real and then applying a label based upon the domain.
But, they had more features than just "source" in their model. so while it's possible their data collection method means that their model is way over-fit and basically just tests of proxies of "source", it's not prime facie obvious that this is the case...? Or am I missing something?
If the test data was collected in the same way that the training data was collected, this was a rather ridiculously round-about way of writing what should've been a 5 line perl script, and I'm really curious where the 5% error could've possibly even come from :-\",
Personally speaking: family, friends, and coworkers have long equated the "Fake News" label with the pap put out by mainstream media monopolies, and, the agendas pushed by trendy social networking corporations. Many of the people included this cohort are working class democrats and self avowed "liberals".
I find it odd that some think the problem with "fakeness" in news reporting is being caused by anything but monopolies over media outlets; curating sources and blacklisting hardly seems a solution. Try antitrust.
Nevertheless, this is of value since it distinguishes between articles with a lot of opinion and articles with little opinion. Even reputable newspapers like the NYT regularly publish news with a lot of opinion.
A more accurate way of detecting "fake news" would be interesting, but I fail to see how such a thing could be designed, past simple detection of wishy-washy and avoidant word patterns.
The OP does not say the label distribution of the training data; it's entirely likely that the split is not balanced 50/50, which would make "95% accuracy" as an indicator of quality misleading.
This is one of the reasons why I recommend that Medium thought pieces disclose their data and code instead of just saying "I did AI magic!" to sell a product (and they do charge for their product on their website).
The fakebox doesn’t detect fake news, it detects articles which are factual/real and everything else is labeled as “fake”.
Where’s the dataset? How did you verify the ground truth? Where are the annotation/labeling guidelines?
What’s the definition of factual/real articles? The dataset appears to be created by the author - which isn’t necessarily wrong but to paraphrase Karl Popper (in the context of human knowledge and scientific endeavors):
There are no ‘pure’ facts available; all observations are functions of subjective factors such as interests, expectations, wishes etc.
I'm with @wadkar on this. I think the Fake News Challenge Stage 1 (FNC-1) was a good step towards this effort. They acknowledge (almost) all of these concerns and start with Stance Detection as their first stage. In this problem, pairs of article headlines and body text were classified into {Agrees, Disagrees, Discusses, Unrelated}.
Constructively criticism to the OP: I'd suggest they read the nuance and discussions on the Fake News Challenge [0] and then look into their datasets + evaluation code [1] instead of hand-coding their own "biases" into a {"Fake news","Not-Fake-News"} binary classifier. Feel free to replace "Fake News Challenge" with any other similar effort so that OP isn't tasking themselves with the massive task of "Solving Fake News" all alone.
> I found myself drifting in my own interpretation of fake news, getting angry as I came across articles that I didn’t agree with, fighting hard against the urge to only pick ones I thought were right. What was right or wrong anyway?
A good question and I'm not surprised he went a bit crazy.
> The problem of truth is in a way easy to state: what truths are, and what (if anything) makes them true. But this simple statement masks a great deal of controversy. Whether there is a metaphysical problem of truth at all, and if there is, what kind of theory might address it, are all standing issues in the theory of truth. We will see a number of distinct ways of answering these questions.
What is "right" in news has become "a compelling narrative that appears plausible to the reader."
What is "wrong" in news is "an implausible narrative relative to the reader."
Plato deals with truth which is something entirely different, not in the same realm of the current state of news.
To be more specific, news can be one of four things to a reader: plausible and true, plausible and untrue, implausible and untrue, and implausible and true.
Most readers seem to more concerned with a stories' plausibility and not it's trueness.
Most importantly though is readers no longer value the "truth" component of news. They value whether or not the narrative aligns with their own view of the world.
Because what truly happened doesn't matter to most people, only that they have a way to make sense of it themselves. Even if it's a partially or completely false narrative.
The model is not based on domains. Just the text of the article. Can confirm there was an even number of real and notreal news examples. Data set was eventually broken into two categories; written with bias, or without bias. For example, a NYT Opinion piece was considered notreal news.
26 comments
[ 3.3 ms ] story [ 60.8 ms ] threadThe author describes a "fake news detector AI", that is actually a "typically legitimate source of news" data model, combined with a fake news domain blacklist. It doesn't detect fake news. It detects whether a story possibly came from a source you find to typically be legitimate.
This article is fake news.
A "true fake news detector" would use only the text of the article -- without the URL. And so I agree that this article is kind of like fake news of the "misleading" variety. :)
False fakes--and a wisely designed system would probably say "probably fake" in the first place, not outright "fake"--are less damaging than false truths overall.
That depends on what your goal is: to discover groundbreaking truths (which are almost always considered taboo/false at first) in order to improve society, or merely attempt to cover up apparent falsehoods as damage control.
In the latter case, such a system would be highly prone to enforcing the prexisting biases and prejudices of its developers onto the users, making it much more difficult for them to discover truths that the developers are ignorant of.
From the article, it does seem that the training and test sets were both generating from a dubious data collection process: scraping URLs they knew were fake/satire/real and then applying a label based upon the domain.
But, they had more features than just "source" in their model. so while it's possible their data collection method means that their model is way over-fit and basically just tests of proxies of "source", it's not prime facie obvious that this is the case...? Or am I missing something?
> Domain name - Some domains are known for hosting certain types of content, Fakebox knows about the most popular sites
Fake news sites typically register bespoke domains, though.
If the test data was collected in the same way that the training data was collected, this was a rather ridiculously round-about way of writing what should've been a 5 line perl script, and I'm really curious where the 5% error could've possibly even come from :-\",
I find it odd that some think the problem with "fakeness" in news reporting is being caused by anything but monopolies over media outlets; curating sources and blacklisting hardly seems a solution. Try antitrust.
https://en.wikipedia.org/wiki/Concentration_of_media_ownersh...
Then, in your account page, you scroll down and there are instructions for installing/running fakebox (and others).
The "demo" is actually a docker app that spins up a web application of some sort. I'm firing it up as I type this, and I'll checking it out soon-ish.
A more accurate way of detecting "fake news" would be interesting, but I fail to see how such a thing could be designed, past simple detection of wishy-washy and avoidant word patterns.
https://en.m.wikipedia.org/wiki/Evaluation_of_binary_classif...
This is one of the reasons why I recommend that Medium thought pieces disclose their data and code instead of just saying "I did AI magic!" to sell a product (and they do charge for their product on their website).
Where’s the dataset? How did you verify the ground truth? Where are the annotation/labeling guidelines?
What’s the definition of factual/real articles? The dataset appears to be created by the author - which isn’t necessarily wrong but to paraphrase Karl Popper (in the context of human knowledge and scientific endeavors):
There are no ‘pure’ facts available; all observations are functions of subjective factors such as interests, expectations, wishes etc.
http://plato.stanford.edu/entries/popper/#GrowHumaKnow
Constructively criticism to the OP: I'd suggest they read the nuance and discussions on the Fake News Challenge [0] and then look into their datasets + evaluation code [1] instead of hand-coding their own "biases" into a {"Fake news","Not-Fake-News"} binary classifier. Feel free to replace "Fake News Challenge" with any other similar effort so that OP isn't tasking themselves with the massive task of "Solving Fake News" all alone.
Disclaimer: I don't have any stake in FNC-1
References:
[0] http://www.fakenewschallenge.org/
[1] https://github.com/FakeNewsChallenge/fnc-1
A good question and I'm not surprised he went a bit crazy.
https://plato.stanford.edu/entries/truth/
> The problem of truth is in a way easy to state: what truths are, and what (if anything) makes them true. But this simple statement masks a great deal of controversy. Whether there is a metaphysical problem of truth at all, and if there is, what kind of theory might address it, are all standing issues in the theory of truth. We will see a number of distinct ways of answering these questions.
What is "wrong" in news is "an implausible narrative relative to the reader."
Plato deals with truth which is something entirely different, not in the same realm of the current state of news.
To be more specific, news can be one of four things to a reader: plausible and true, plausible and untrue, implausible and untrue, and implausible and true.
Most readers seem to more concerned with a stories' plausibility and not it's trueness.
Most importantly though is readers no longer value the "truth" component of news. They value whether or not the narrative aligns with their own view of the world.
Because what truly happened doesn't matter to most people, only that they have a way to make sense of it themselves. Even if it's a partially or completely false narrative.