Been experimenting with building tools to make fact checking more transparent and reusable for a few years now, this is the latest iteration of that project. Feedback appreciated!
Perhaps I am just being nitpicky, and I have no comments on the content or intent of the site (not my industry / area) but grey/blue text on the night sky blue background with white for emphasis really threw me whilst trying to read the text on the site. It may also have to do with Linux's crappy font rendering but I found it unreasonably hard to look at.
In my experience, there are few good colours that remain readable on text, and I personally find serif fonts visually displeasing in most cases, although there are some exceptions (https://www.nytimes.com/) and even then, the light grey on white background is pretty displeasing.
The goal is to enable journalists/bloggers to annotate their content with their fact checking. Allowing their readers to audit to and to contribute to the fact checking for themselves.
In other words, enabling journalists to "show their work".
Thanks for the feedback about the colours, I'll play around with it!
Is the primary idea for journalists to share the checks they did, or to have contributors help them check whether the things they mention are factual? The former is what I'd imagine when I hear "open source fact checking", the latter would be called "crowd-sourced fact checking", I think.
Just some feedback that might or might not be useful for your messaging, depending on your goals :)
For this to be useful at scale, you will need to not just design this for good-faith users, but also against bad-faith actors. If this catches on, it will become a target of coordinated disinformation campaigns. Have you been thinking about what to do against that?
Indeed, currently all "verified" reviewers are manually vetted.
Long term the goal is to have an open reputation score for every fact checker. I'm currently experimenting with different implementations of this, and seeing good promise.
How would the open reputation score mitigate bad-faith actors trying to exploit the system?
For example, I believe HN has a vote-ring detector, but I think there's also a decent amount of manual intervention from the mods. Amazon's reputation system is under constant attack through a variety of means (paying teams of people for fake reviews, going to the length of having real transactions on Amazon between fake-reviewer and the seller buying the review).
Neat tool and well done, but why would a journalist use this? In the day and age of 'fake news', it's pretty clear that journalists don't highly value checking facts.
While I quite like the concept, I think there's something of a catch 22 here.
'Fact-based' generally implies 'lesser impact', it's way easier to catch eyeballs with fakes and click-bait than with better stories, and part of the fake news issues is that professional success in journalism is dependent on impact, not on accuracy (a similar thing goes on in academia).
You'll have to figure out a way to increase the impact of writers with more accuracy (less negative fact checks maybe?) by promoting them in some shape or form. Have you considered top writers scoreboards or something similar?
IMHO reputation is important to build into the base level for a platform like this. Allowing anon registrations and commenting will eventually lead to a lot of noise and abuse of the platform for misinformation. I think some sort of verification of a person's qualifications on particular subject areas and then allowing them to only "fact check" on those qualified areas is required here to have an effective platform that will scale well.
Currently (for the prototype), you can start contributing, but your reviews only become public once I've vetted you and given you a fact checking test.
Long term the goal is to have an open reputation score for every fact checker. I'm currently experimenting with different implementations of this, and seeing good promise.
I would respectfully disagree. Introduce enough noise into the system and the cost of verification becomes so high that doing so is unrealistic for many or most people.
Also, on first glance the app looks great, well-polished, and the UX of reading an article and looking at sources for claim is nice :). Thanks for working on and sharing this!
Yes, but that's not the point. That merely shows that we still need to trust someone -- you.
Specifically, let's say you support belief A, and I want to add a whole bunch of facts that support belief B. You're still the gatekeeper to decide whether I get to be a contributor. There's no way for someone to know that you're going to approve equally-qualified contributors on different sides of an issue, so they're always going to wonder whether there are voices or facts being omitted.
(But if that's just the plan for beta, that's fine.)
Indeed, manual vetting is only for alpha. I should reiterate - the point of this type of open fact checking is you can always verify everything for yourself, so you don't need to trust my credentials.
Long term every review, even those that are flagged, will be accessible. Albeit, those that are flagged will be in a separate section, but the point is they will always be accessible, so that you never have to trust me or the community, you can always review for yourself.
> I should reiterate - the point of this type of open fact checking is you can always verify everything for yourself, so you don't need to trust my credentials.
Again, not for the omission of facts, or the omission of counter-evidence.
E.g, suppose you only let in contributors who agree with you. An article is posts with "Fact A," which you happen to agree with. A bunch of your hand-picked contributors put up evidence to support "Fact A." No one ever posts the evidence debunking "Fact A."
So there's no way to "verify" anything, because we can't see what's not there. Most readers will be satisfied that "Fact A" is well-sourced.
So it's not enough to say that the reader can verify everything and not need to trust you. I mean, they could do their own research to Google the evidence debunking Fact A, but they could do that anyway and the platform doesn't support them.
I'm not 100% sure that lack of specific subject matter expertise should prevent people from commenting. In many cases experts are the best people to go to, but in other cases they can represent entrenched ideas that are precisely the ones that need rethinking.
For an example, I have read writings of many experts in mental disorders that completely ignore changes in the DSM V. I do not know if the experts are not aware of the changes or don't agree with them, but the way mental disorders are classified underwent some significant changes and I can quote direct lines from the DSM V that strongly contradict the claims of experts.
> For an example, I have read writings of many experts in mental disorders that completely ignore changes in the DSM V.
Nitpick, but it's DSM 5; roman numerals were dropped after DSM IV.
> I do not know if the experts are not aware of the changes or don't agree with them
Almost certainly the latter. Lots of DSM V changes were, and are, controversial (even among people who acknowledge the existence of the problems the changes were intended to fix, the aisle solutions chosen are sometimes controversial.)
Equally important : Don't display reputation to users other than themselves. Humans will attributes higher trust level to individuals with higher reputation, despite them occasionally dropping false claims.
I think so. You could probably pick something better.
Its more of a Pavlovian reinforcement than reputation. You see what got you lots of points (what people liked) and you're more apt to do that in the future.
By keeping everyone else blind to someone's scores, you don't have as much popularity bias creeping in.
I think it is better to record the facts about people's opinions, than to count reputation points on identities.
For example, as ternary predicates:
1: water is wet
2: identity_1 corroborates 1
3: 2 by_way_of blind_faith
4: identity_2 denies 1
5: 4 by_way_of blind_faith
etc.
You can go as far with this as you want, and the consumer gets to decide what establishes or hurts credibility, and which statements of fact corroborated by what methods and by whom will either count toward or against the credibility of a specific form of statement on a specific topic by a specific set of identities.
For example, you could have predicates describing conflicts of interest, and identify the relevant conflicts of interest by querying for interests in the predicate in question to find what people you trust will say about the interest of individuals and organizations in corroborating or denying a particular claim.
You could also have predicates which show ways in which a statement is controversial in its general form, but is uncontroversial when refined. For example: liquid water can not be wetted with liquid water.
Disagree. The point is to link assertions to publicly available primary sources like court documents or suchlike and articulate the linkage clearly.It's well designed for that.
Because anon users won't link to "primary sources", which are really nothing of the sort? Because "primary sources" are always correct and trustworthy and never disagree?
1) Now that video deepfake is a thing, I currently only allow videos as primary sources in rare circumstances (such as the video being published by a verified account of the subject).
2) Only primary sources are accepted. Currently anyone can start contributing, but every claim is still reviewed by a team of verified fact checkers. I hope to scale this long term with a fact checking reputation score.
Haha indeed, but you can't just submit any document and have it considered a primary source on the site. For example, if you're making an assertion about a piece of legislation, the primary source is that legislation hosted on that government's website. If your assertion is about a regulatory action, the primary source is that action hosted on that regulatory agency's website, etc.
I'm happy to provide any journalist/blogger who wants to include this style of annotation on their site a js plug-in to do just that.
The whole fact checking process actually happens within the system, the facts you see on there were collaboratively fact checked by a team of 7 to 11 vetted fact checkers. So that may also provide an incentive for journalists.
Not only do you have to account for what those sources say, but also, a reporter reporting what an anonymous source said can be 100% factual (Yes, the anonymous, valid source was indeed reported accurately), but maybe the anonymous source is incorrect. The reporting is accurate. The source is not.
Basically, a lot of grey area. And with that comes the issue of anything not "fact checked" by a service like this could be dismissed. What harm does that do?
Not a dismissal of a service like this. Rather, just open questions.
Only public primary sources are allowed as evidence on SourcedFact.
Anonymous sources certainly have value within journalism, but the goal of SourcedFact is to make facts that can be proven beyond a reasonable doubt with public primary sources stand out from everything else.
However as others have said a reputation system for participants is absolutely key. Any platform like this has to assume it is going to be flooded by bots trying to influence opinion on behalf of corporations, foreign governments and activists.
A reputation where users gain reputation slowly (through work, by verifying sources, etc..) become more able to influence opinion, but the reputation drops quickly if they are found to be misleading people. How this actually works is harder than it seems if you want a truly robust system.
When I think of platforms like this I think of Galileo[0]. He was perceived as a troll or heretic in his time, but of course he was right. How do you create a platform that allows correct ideas to flourish even when they go against conventional wisdom?
I 100% agree that the reputation system will be they key to scaling this. I'm currently seeing promise with my experiments with reputation scoring.
As for "How do you create a platform that allows correct ideas to flourish even when they go against conventional wisdom?" Great question, however my goal here is a bit more modest, it's to create a platform that allows facts which are provable by public primary sources to stand out from everything else. I think a platform that focuses on simply that can add a lot of value to political discourse.
The bots would be a serious issue if this ever became big enough, since they can add to each other's reputation. Reddit probably has thousands of code-hours devoted to this, and bots are still a problem.
There is also the problem that humans will probably tend to upvote "facts" that they agree with, and visa-versa. I'm not sure that reputation is the solution.
And yet "expert" verification introduces trust. I'm not sure what the solution is.
Yes it's a super tough problem, not to be underestimated by anybody working on such a platform.
If there is any easily automatable way gaining reputation, then the bots will use that, and if you can easily pass reputation too, then the rep system won't hold.
> I'm not sure what the solution is.
Ditto, and I've ben pondering this on and off for years :/
So, your examples are all "reporting about court cases" and you use court documents as your citations for facts.
Court cases are as good as we have for determining facts, I suppose. (Though the error rate in court cases is quite high!) But 99.9999% of reporting isn't about court cases. Most reporting isn't about issues that will ever be litigated, and even those things that will be litigated, haven't been at the time of the reporting.
"Mitch McConnell had oatmeal for breakfast this morning, according to sources."
There, that's a fact. Now, tell me which court case I can refer to in order to verify that fact?
Fact-checking is an impossible endeavor. The essence of journalism is reporting things no one knows. That's what news is! This is completely at odds with "fact-checking". A good news story CANNOT be fact-checked by the public because if the public knew it, it wouldn't be news. There is an extremely limited genre of "news by trawling public but little-known documents", such as reporting on old court cases. Almost zero news fits into that category.
Call me a pedant, but I'd say OP is enabling "consensus checking"
I find the present day's fact checking wave is a misnomer. In a world where something is false, but sourced, most of what we call fact-checking would only reinforce mistruths.
e.g.
The ratio of circumference to diameter is exactly three to one [1][2][3]
[1]: the Bible.
[2]: pious scholars.
[3]: low-tech experiments.
Would be cool to see someone think carefully about what "real" "fact checking" would be. First-hand sources? Links to instructions for replicating findings?
Consensus checking is of course also an interesting problem, albeit a pretty different one.
I'm surprised that FAGAM haven't got into the crowd-sourced fact checking space. Seems like a great way to build a knowledge engine / AI, by getting humans to "connect the facts" while motivating (herding) them with the idea they are doing it to "preserve truth in an era of fake narratives", when actually they are just helping build an AI.
This is pretty awesome. Any time I open any article, the first thing I do is try to follow the trail of links/sources back to the primary sources and read that instead. If I can't find one, I don't trust the article.
"Open source" is absolutely a verb. "I open sourced my latest work project this month".
It's also a term that seems to be used similarly to the post title in contexts outside of software. For example, I subscribe to a podcast that covers arms control, where they frequently use the term "open source" to describe analysis derived publicly available sources and methods.
What problem does this solve that existing reputation systems don't? When I read a New Yorker article, for example, part of what my subscription is paying for is considerable skilled labor by a group of in-house fact checkers. How does moving this work to an outside free platform improve the situation?
Moreover, how do you replicate work like calling sources to verify statements before publication?
I wrote a short piece for Wired once and had to go back and forth with their fact checker on the most picayune details. In an article about balancing online work and travel I had mentioned in passing searching under a hotel bed for an outlet, and she wanted to know what hotel it was, what country it was in, when I stayed there, etc. The process was exhaustive and not something I would go through with a bunch of Internet randos. It was also done before publication and involved a fairly high level of trust.
The goal is to capture the fact checking process and enable any reader to review it for themselves. Certainly some publications currently have an excellent fact checking process, but very few make their fact checking open and transparent.
As for "how do you replicate work like calling sources to verify statements before publication?" Good question. Currently this is only designed to open up the fact checking which involves public primary sources. I think that a platform that focuses on doing that subset of fact checking really really well can add a lot of value to the public discourse.
Currently this is only designed to open up the fact checking which involves public primary sources.
Limiting the scope to this is wise. It's a tractable problem and it's better journalism. The types of journalists who write "this unnamed source implied that this other unnamed source had implied to another unnamed source..." articles won't like this sort of tool, but those journalists should be encouraged to change careers.
First sentence: "Some of Robert S. Mueller III’s investigators have told associates that Attorney General William P. Barr failed to adequately portray the findings of their inquiry and that they were more troubling for President Trump than Mr. Barr indicated, according to government officials and others familiar with their simmering frustrations."
This sort of crap gets swept under the rug within a couple of weeks, but we're certainly paying attention to it now.
What's the problem with that sentence? People speak to the press off the record all the time; it's part of the process. Unless the NYT doesn't actually have these sources, there's nothing wrong with it. Of course, all we have for proof is their reputation as a newspaper for integrity, and the word of any reporters involved, but reporters have often gone to jail rather than reveal anonymous sources.
1) I've never heard of Nicholas Fandos, Michael S. Schmidt, or Mark Mazzetti. I don't suspect them of anything horrible, but neither do they have "reputations" on which I can rely. Taibbi in his recent essay observed that reporters without established reputations get lower-quality anonymous tips (i.e., "lies") from insiders than better-known reporters get. In this case multiple no-name authors are actually less reputable than one no-name author would be, because we don't know which of the authors received the anonymous tip at the heart of this article.
2) The basic proposition addressed by that sentence is that "investigators are uncomfortable with Barr's summary", but the implication is that there is a problem with that summary. The rest of the article addresses the potential meaning of this "big, if true" implication but gets no closer to the question of whether that implication is true. The alleged feelings of randoms are not interesting to the average reader. If investigators or anyone else has specific knowledge of wrongdoing that should be made known to the public, they should contact reporters directly.
3) Even if the basic proposition were firmer, we're approaching it through too many layers of indirection. The reporter didn't talk to the investigators. He didn't talk to associates of the investigators. He talked rather to "government officials and others familiar", and not about what those officials and others had heard from the investigators but rather about what associates of the investigators had heard from the investigators. How the government officials and others heard about it is not specified. We're at least four layers deep here, and nothing believable is four layers deep. These reporters are under no threat of "going to jail", because even if the basic question were important no judge could decide that the reporters are close enough to the truth to justify contempt.
4) The basic question is not important. Eventually politicians of both parties will read the entire un-redacted report. The public will not, because grand jury testimony will be redacted to protect the reputations of Republican operatives whose transgressions were not judged to merit indictments while "sources and methods" will be redacted to protect Democratic FBI (and possibly NSA?) agents whose transgressions were under color of law. It's not surprising that Barr's report had a political slant. Everything in DC has a political slant. Maybe if they found the Lindbergh baby, this tottering tower of misdirection could be justified. Nobody here found the Lindbergh baby.
5) That this happens "all the time" is actually a serious problem. Every day, well-placed people insert self-interested opinions into the public discourse not as self-interested opinions but rather as journalistic fact. This has terrible effects on our society and on our military victims around the world. Soi-disant journalists should stop aiding this terrible process, and this tool could help them do that.
I haven't heard of the journalists who wrote the article. I have, however, heard of the New York Times. I trust the NYT to vet its reporters carefully, and to review relevant facts. If someone wants to go off on "liberal media fake news", well, fuck them. They're not interested in facts anyway, and there's no point in debating such idiots.
As for "specific knowledge of wrongdoing"... well. What's "wrongdoing" here? Barr presented a summary of the Mueller Report to the public. Apparently, people in the investigation itself are unhappy with that summary and consider it deceptive (yeah, no shit, saying the Trump administration misleading people is like saying water is wet).
Yes, this is multi-layered, anonymous leakage, BUT. BUT. Is it untrue? Are investigators actually fine with the Barr summary, and the sources are lying to the journalists? Probably not. So despite the abstraction layers, the gist of the story is most likely true. And the NYT certainly doesn't have to run a story that says "Mueller is totally fine with the Barr summary" next week, so they're not going to print it unless they believe it true. So smell tests are fine here.
And the basic question is important. It's not whether Barr lied, but rather whether he intentionally misled - or more to the point, if Mueller's team believes he misled. That is relevant to the public, and to me.
>Apparently, people in the investigation itself are unhappy with that summary
But the article doesn't even say that they heard the investigators say that. At best it's hearsay and gossip of n-th degree, at worst it's outright misrepresentation of the facts.
Look at the weasel words: "according to government officials and others familiar with their simmering frustrations."
Who are government officials in this usage? And what qualifies someone as "familiar" with the matter? How many people work as a government official in some capacity and could claim some vague familiarity with the Mueller report? Does a park ranger in Wyoming who follows political news qualify? It wouldn't be technically lying to cite them as a "government official familiar with the matter", right?
Stories are leaked through cutouts like this for good reason, politically. What I'm saying is that the layers don't make it untrue, and decent journalism will catch most misrepresentation.
You say "weasel words", I say "We think you should take this with a grain of salt". They're being explicitly clear that their sources are secondhand. What's your answer? That they shouldn't run the story at all?
Well, I think as you get more and more degrees removed from
a primary source, the closer the article gets to being gossip and/or somebody's opinion and not factual news. Which is fine, there's a market for stuff like that. But I don't think that's very sustainable if you want to maintain a reputation as a purveyor of fact-based reporting.
There's a reason hearsay isn't generally admissable in a court of law.
> There's a reason hearsay isn't generally admissable in a court of law.
The main reasons are that courts have compulsory process to attain testimony of primary witnesses which makes it generally unnecessary, and that courts are by design intended to work via adversarial process including cross examination, and a indirect witness can't be cross examined.
Needless to say, there are lots of exceptions to hearsay exclusions, too.
It's not hard to see parallels between the two situations. Both the court and the average consumer of news are trying to figure out what's true and what isn't. A real human being who is quoted in the paper will certainly be "cross-examined" when subsequent events show him to have been completely mistaken. DC-insider ventriloquists with no names will not be cross-examined so.
Absolutely, a story based entirely on odious unattributable anonymous innuendo like that we see here should not be published. If the "government officials and others" want this important information to get out, they can still talk to Rep. Schiff. He will repeat anything that hypes up RussiaRussiaRussia, and he's on TV all the time. If, in fact, he isn't the "government official" to which this piece refers in the first place.
I don't say this because I support Trump or voted for him (never did, never would). I say it because this whole multiyear journalistic dumpster fire has guaranteed his reelection. TFA is all about putting out the flames, which I do support.
I've never heard of Nicholas Fandos, Michael S. Schmidt, or Mark Mazzetti. I don't suspect them of anything horrible, but neither do they have "reputations" on which I can rely.
Two of these people have multiple Pulitzers between them. Schmidt is on tv practically daily. These are not low quality tips to no-name reporters - that 'fact' you made up to fit your preferred narrative. Or at least, did not bother to fact-check!
Thanks for the update; it's really hard to figure out who has won a Pulitzer! Schmidt, in particular, seems to show up in the pictures without his name appearing at the top of the prize announcement. I would have recognized the name "Ronan Farrow". [0] Is it meaningful that the intern is the first author? Might he have written the first line in the story, which as we've established is the only claim of any substance, such as it is?
>Unless the NYT doesn't actually have these sources, there's nothing wrong with it
It doesn't even have to come down to bad intentions or journalists intentionally lying about their anonymous sources, it's the fact that I have no way of reasonably knowing whether the NYT got tricked or lied to. Maybe their so-called "anonymous source" is just taking credulous journalists for a ride. I'm expected to just put blind faith in their opaque verification process and accept that anonymous sources are equally as valid as someone going on record?
Interestingly enough, the credibility (or lack thereof) of anonymous sources is an easily tractable problem that could be easily solvable with ring-signature cryptography [0] if there was the will to do so. And the trustless nature of that solution does not require journalists to be deemed the anointed arbiters of The Truth(tm). Alas, the social and cultural factors are just not there such that every government official, business leader, etc has a public crypto key.
Crypto aside (I think it's useless nonsense in this case, but that's me), my gripe with journalism isn't the facts... it's the stenography. Some serial liar (coughcoughTrumpcough) says the sky is falling because the Sun rises in the West and the idea that it rises in the East is liberal fake news, and the media dutifully reports the quote and doesn't call them on their shit. And then the debate over whether the Sun rises in the east or the west is treated as he-said-she-said opinion by the media.
That is a real problem. Not anonymous sources or fact-checking.
> What problem does this solve that existing reputation systems don't?
To turn its slogan into an analogy: the New Yorker is high quality closed source software. If you disagree with the New Yorker, you can't file a bug report.
You might say, "I wouldn't want to," either because of something on your end, or something on the New Yorker's end. Continuing the analogy, there are closed-source models, open-source-but-closed-development models, and open-source-community-driven models.
I'm typing this because the problem this solves for me is I often disagree with the news, and I would like to fact check them. Perhaps not--as you mention--by calling sources, but by citing other evidence.
I truly believe that post-publication fact checking is a losing battle. If you take a sampling of redaction tweets by media outlets and compare their retweet counts to the retweet counts of the original posting you'll see that that is very much the case.
Wikipedia talk pages are a perfect example of facts ignored and removed from articles to push an agenda. You can see how a narrative can be pushed with half-truths and selective editing. Same thing is happening in News, its agenda driven, not truth driven. Entertainment opinion pieces that pretends to be news. People are just bending the facts to support their personal politics.
If Wikipedia can't be factually accurate due to editors personal politics, how do you keep those personal politics out of a fact checker?
You're 100% correct in that one can cite completely valid facts and still push an agenda by omitting other facts. I'm actually working on a "fact comment section" feature where any reader can pin facts to an article (facts that still go through the same rigorous and open fact checking process as everything else). Facts which readers deem are important extra context.
However, I'd argue that even without such a feature, having an open standard for facts openly verified from primary sources will add a lot of value to current public discourse.
Wikipedia has flaws,too. 1. Your account need to have good history or social repo for your edits to be seriously considered by Wikipedia admins. 2. Admin can have their own set of bias and can decide to ignore edits.
Remember Wikipedia is a centralized and under control of single entity, fact that they rely on donation doesn't change anything.
There wasn't anything said about Wikipedia being a bad idea, but rather that it shouldn't be your sole source for fact-checking. I've learned that people only trust Wikipedia to the point at which it confirms their biases.
Part of the problem is that Wikipedia is generally pretty darn accurate. Everybody always talks about funny vandalism stories, but generally it's pretty good. The issue of course is that there's parts of it that definitely aren't correct.
This makes it easy to say "Well if it's accurate 99% of the time then it must confirm what I thought!".
Wikipedia, for a few high-profile current events each week, serves a function similar to the one proposed for sourcedfact; and it regularly results in deeper + more contextualized coverage than even the best outlets. However, WP's commitment to being a tertiary source means that _some_ secondary source has to properly interpret the primary sources, before that interpretation can be included.
This is more a communication standard than a complicated tech solution. A communication standard for fact checks from public primary sources.
Coding this wasn't actually that complicated, figuring out a design that enables readers to review the fact checking as easily as possible and enables fact checkers to collaborate easily took a lot of experimentation though. Although even that is still very much a work in progress.
Exactly. I would just appreciate footnotes in the articles I read that gave a detailed explanation of how much fact checking was done.
That way if a story needs to be expanded upon at a later date, the next journalist can dig deeper easily picking up the trail where the previous journalist left off.
I think there is a technical solution to this problem though. A structured graph of supporting facts (including an analog of identity for the sources and corroborators, pointing to the edges) is an excellent datastructure for maintaining a record of reasoning and evidence for a statement of fact.
There is, of course, the social issue of convincing people that this is important, but the underlying issue is in part a technical one. I don't think it's fair to disqualify a solution based on the fact that involves technical work.
I'm reminded of the oft-quoted phrase "you cannot reason someone out of a position they did not reason themselves into."
I am not trying to disqualify the solution, more like observing that the existence of provable facts has not yet been any sort of antidote for 'fake news' (whether that refers to the real deal, or the position many people take that any news they don't like is fake news).
I don't think this sort of effort has to be ironclad to be meaningful. The margins could be a big deal, here.
Some sort of attestation toolchain might enable people who care about evidence to produce more strongly-attested cases, and to get a better handle on what is and isn't sourced from others. It seems likely that using good attestation toolchains could create feedback loops for researchers, journalists, archivists, authors, film-makers, educators, students, and other people involved in knowledge production.
These feedback loops might change how they research, where they start, how often they have to re-invent wheels, and raise standards by enabling these people to compete on metrics that might drive more social value than clicks, engagement, shares, or ad revenue.
These effects could be important, even if you never budge the kind of person inclined to consume the information equivalent of junk-food.
Agreed, it is also a question of goals. Journalists can bend "facts" to support the argument they want. Title are 99% click-bait anymore. Sure there are some people who want to present fair and balanced view points but that takes a ton of work and time. Most "journalism" these days is get likes and move on.
Right. Many prominent newsrooms seem to regard fact checking as an obstacle to avoid.
Still, it seems valuable to make it easy for readers to view the supporting or contradictory evidence. That might’ve stymied many notable media hit-pieces. Of course, many newsrooms wouldn’t want this (at least I can’t imagine The Guardian and it’s ilk would want to draw attention to their own impoverished fact:BS ratios), so it would have to be a browser plugin or similar.
People post contradictory evidence to journalism they don't like all the time. Their "evidence" is usually conspiracy theory crap that can't pass even logic checking, much less fact checking.
Sure, but it's not easy to find all of the debate about a given assertion in a particular article, much less contribute to that debate (given that it's probably distributed across a bunch of social media venues).
One can easily imagine hovering over a given assertion (e.g., "Damore posted an anti-diversity screed") and see the fact-checks, perhaps ranked by fact-checker's credibility in order to suppress conspiracy theory crap as well as contribute to the debate.
Anyway, that there is lots of noise among the criticism doesn't mean that there isn't a lot of valid criticism. It's only been a few months since nearly the whole American newsmedia willfully mislead the nation, sending an angry mob after innocent kids, after all.
People (the audience, that is) have historically refused to pay a significant share of the cost for news, what has changed is that advertisers have stopped paying as much for news (because the share of people's attention debited to news, and therefore which news outlets have to sell access to, has dropped.)
Paid circulation used to often be important not so much because of direct revenue but because advertisers looked to it as an important measure of reach.
That's true as well. The effect is the same either way, though -- news organizations aren't exactly swimming in money these days. They have trouble just paying journalists, much less support staff for those journalists like fact-checkers.
Maybe that's part of it, but there are plenty of examples of the media deliberately misleading their audience. "Can't afford fact checkers" doesn't really explain that.
"These days" suggests that this is something new. Tabloids and magazines have been doing bunk journalism for decades. Now they are motivated by getting clicks to inflate ad costs instead of getting sales to inflate ad costs. The only thing that has changed is the delivery method. The reliable sources of journalism are unchanged as well. There are plenty of new online tabloids and "e-zines" but journalism still comes from the same places.
Fact checking was very effective for publications like Readers' Digest that routinely invited guest authors. The publication wanted a variety of authors to contribute, and they also had a reputation to defend.
If you've ever been interviewed by a modern journo, you know you just get the same question reworded a dozen different ways. The reason is that they decide what story they want to write up front, and then just go shopping for facts to support it. That story is usually dictated by market pressures, which presently reward sensationalism to feed the addiction of a dwindling number of listeners in an echo chambers.
And current fact checkers are typically checking material put out by others, which is completely broken. They have all the same incentives as anyone writing a hit / puff piece, except they put a phony "truth number" at the top and other sites will tally those up as though it means something.
Why does it seem like that to you? Do you assume that the project should solve the problem entirely in order for it to be of any help or have any value?
I'm a bit jealous because I had nearly the same idea back when I worked in public radio, but didn't have the time to implement it. Basically it was about having facts and quotes exist as embedded units within articles, all with their own individual records of verification and relationships to all the articles in which they were used. I'm really glad someone is taking a shot at this, even though I have my doubts as to whether journalists will take it seriously.
I think to solve the problem, you need to do it in steps, kind of like how chained logic works. You start with something simple: "Can we verify what someone says with public record?" Yes, here's how we do it.
As one problem is solved, it starts hinting at solutions to subsequent problems. This is not a problem with a silver bullet. It's going to take a fundamental shift in how we understand and verify truth.
Technical solutions solve social problems all the time. A lock is a technical solution to the social problem of theft, and much superior to a social solution.
While I like how the software ties together the sourcing, I feel like the following is lacking (and I realize this is an alpha, but its things to consider)
1. Anyone who answers a question regarding whether the claim is "fully" proven by a source or whether the source is valid for the claim must or should (debatable) provide proof of their own interpretation by citing either specific instances in the source in question supporting it. For example, in the Net Neutrality article, Yatz states this: The adopted principles in this statement are at the top of page 3 in a response to does the source fully prove the claim. It would be nice if that was linked to the actual location in the document or identified through highlighting or some other mechanism to get full context for the response. There also should be uses for outside sources to be attached when evaluating these questions that are suppose to validate a source. This is because someone could submit a response to these source validity questions and cite an external source or internal quote of the source in question for their reasoning that doesn't support their answer.
2. Having some sort of expertise verification is extremely important and should be weighted either separately as in Experts: YesExperts: No category and should be more important. It distinguishes this from random anon answers that have no training in how the source should be interpreted.
3. Having anonymous people do this seems rather dubious and the current platform seems like it would make it prone to Youtube comment syndrome. Maybe have some reputation system to gate keep like StackOverflow?
4. Sources usually require appropriate interpretation in order to be taken in the right context and be considered correct. Sometimes there's no single source or any source, philosophically, that fully proves a claim. I'm guessing that's what supporting documents are for?
5. Why is the original articles claim allowed to be edited?
302 comments
[ 3.0 ms ] story [ 216 ms ] threadIn my experience, there are few good colours that remain readable on text, and I personally find serif fonts visually displeasing in most cases, although there are some exceptions (https://www.nytimes.com/) and even then, the light grey on white background is pretty displeasing.
I wish you all the best in your endeavour!
In other words, enabling journalists to "show their work".
Thanks for the feedback about the colours, I'll play around with it!
Fact checking need a court like system, with judges. Contact me, if you interested. I have lot of real world experience with that.
1) only primary sources are accepted 2) the claim and primary source must be verified by a community of fact checkers
Definitely happy to chat though, will send you an email!
Just some feedback that might or might not be useful for your messaging, depending on your goals :)
It's clean and easy to use and I don't necessarily see a huge problem with "noise" diluting the "facts".
I certainly appreciate both the goal and work you've done to accomplish it.
Long term the goal is to have an open reputation score for every fact checker. I'm currently experimenting with different implementations of this, and seeing good promise.
For example, I believe HN has a vote-ring detector, but I think there's also a decent amount of manual intervention from the mods. Amazon's reputation system is under constant attack through a variety of means (paying teams of people for fake reviews, going to the length of having real transactions on Amazon between fake-reviewer and the seller buying the review).
'Fact-based' generally implies 'lesser impact', it's way easier to catch eyeballs with fakes and click-bait than with better stories, and part of the fake news issues is that professional success in journalism is dependent on impact, not on accuracy (a similar thing goes on in academia).
You'll have to figure out a way to increase the impact of writers with more accuracy (less negative fact checks maybe?) by promoting them in some shape or form. Have you considered top writers scoreboards or something similar?
Long term the goal is to have an open reputation score for every fact checker. I'm currently experimenting with different implementations of this, and seeing good promise.
Working on this over the last few years though, I've personally fact checked over 500 assertions.
Also, on first glance the app looks great, well-polished, and the UX of reading an article and looking at sources for claim is nice :). Thanks for working on and sharing this!
Specifically, let's say you support belief A, and I want to add a whole bunch of facts that support belief B. You're still the gatekeeper to decide whether I get to be a contributor. There's no way for someone to know that you're going to approve equally-qualified contributors on different sides of an issue, so they're always going to wonder whether there are voices or facts being omitted.
(But if that's just the plan for beta, that's fine.)
Long term every review, even those that are flagged, will be accessible. Albeit, those that are flagged will be in a separate section, but the point is they will always be accessible, so that you never have to trust me or the community, you can always review for yourself.
Again, not for the omission of facts, or the omission of counter-evidence.
E.g, suppose you only let in contributors who agree with you. An article is posts with "Fact A," which you happen to agree with. A bunch of your hand-picked contributors put up evidence to support "Fact A." No one ever posts the evidence debunking "Fact A."
So there's no way to "verify" anything, because we can't see what's not there. Most readers will be satisfied that "Fact A" is well-sourced.
So it's not enough to say that the reader can verify everything and not need to trust you. I mean, they could do their own research to Google the evidence debunking Fact A, but they could do that anyway and the platform doesn't support them.
Nitpick, but it's DSM 5; roman numerals were dropped after DSM IV.
> I do not know if the experts are not aware of the changes or don't agree with them
Almost certainly the latter. Lots of DSM V changes were, and are, controversial (even among people who acknowledge the existence of the problems the changes were intended to fix, the aisle solutions chosen are sometimes controversial.)
Its more of a Pavlovian reinforcement than reputation. You see what got you lots of points (what people liked) and you're more apt to do that in the future.
By keeping everyone else blind to someone's scores, you don't have as much popularity bias creeping in.
For example, as ternary predicates:
1: water is wet
2: identity_1 corroborates 1
3: 2 by_way_of blind_faith
4: identity_2 denies 1
5: 4 by_way_of blind_faith
etc.
You can go as far with this as you want, and the consumer gets to decide what establishes or hurts credibility, and which statements of fact corroborated by what methods and by whom will either count toward or against the credibility of a specific form of statement on a specific topic by a specific set of identities.
For example, you could have predicates describing conflicts of interest, and identify the relevant conflicts of interest by querying for interests in the predicate in question to find what people you trust will say about the interest of individuals and organizations in corroborating or denying a particular claim.
You could also have predicates which show ways in which a statement is controversial in its general form, but is uncontroversial when refined. For example: liquid water can not be wetted with liquid water.
Sure, if those with higher reputation have their claims promoted above others with poorer reputations.
1) Some primary sources are dubious, or manipulated. See the multiple versions of Acosta getting kicked out of a White House press briefing.
2) What happens when you have an article that repeatedly cites dubious secondary opinion pieces, but the reader accepts these pieces as fact?
It seems like its less for fact checking, which would happen prior to publication, than for fact proof?
The whole fact checking process actually happens within the system, the facts you see on there were collaboratively fact checked by a team of 7 to 11 vetted fact checkers. So that may also provide an incentive for journalists.
Not only do you have to account for what those sources say, but also, a reporter reporting what an anonymous source said can be 100% factual (Yes, the anonymous, valid source was indeed reported accurately), but maybe the anonymous source is incorrect. The reporting is accurate. The source is not.
Basically, a lot of grey area. And with that comes the issue of anything not "fact checked" by a service like this could be dismissed. What harm does that do?
Not a dismissal of a service like this. Rather, just open questions.
Anonymous sources certainly have value within journalism, but the goal of SourcedFact is to make facts that can be proven beyond a reasonable doubt with public primary sources stand out from everything else.
However as others have said a reputation system for participants is absolutely key. Any platform like this has to assume it is going to be flooded by bots trying to influence opinion on behalf of corporations, foreign governments and activists.
A reputation where users gain reputation slowly (through work, by verifying sources, etc..) become more able to influence opinion, but the reputation drops quickly if they are found to be misleading people. How this actually works is harder than it seems if you want a truly robust system.
When I think of platforms like this I think of Galileo[0]. He was perceived as a troll or heretic in his time, but of course he was right. How do you create a platform that allows correct ideas to flourish even when they go against conventional wisdom?
[0] https://en.wikipedia.org/wiki/Galileo_Galilei
As for "How do you create a platform that allows correct ideas to flourish even when they go against conventional wisdom?" Great question, however my goal here is a bit more modest, it's to create a platform that allows facts which are provable by public primary sources to stand out from everything else. I think a platform that focuses on simply that can add a lot of value to political discourse.
There is also the problem that humans will probably tend to upvote "facts" that they agree with, and visa-versa. I'm not sure that reputation is the solution.
And yet "expert" verification introduces trust. I'm not sure what the solution is.
If there is any easily automatable way gaining reputation, then the bots will use that, and if you can easily pass reputation too, then the rep system won't hold.
> I'm not sure what the solution is.
Ditto, and I've ben pondering this on and off for years :/
Court cases are as good as we have for determining facts, I suppose. (Though the error rate in court cases is quite high!) But 99.9999% of reporting isn't about court cases. Most reporting isn't about issues that will ever be litigated, and even those things that will be litigated, haven't been at the time of the reporting.
"Mitch McConnell had oatmeal for breakfast this morning, according to sources."
There, that's a fact. Now, tell me which court case I can refer to in order to verify that fact?
Fact-checking is an impossible endeavor. The essence of journalism is reporting things no one knows. That's what news is! This is completely at odds with "fact-checking". A good news story CANNOT be fact-checked by the public because if the public knew it, it wouldn't be news. There is an extremely limited genre of "news by trawling public but little-known documents", such as reporting on old court cases. Almost zero news fits into that category.
I find the present day's fact checking wave is a misnomer. In a world where something is false, but sourced, most of what we call fact-checking would only reinforce mistruths.
e.g.
Would be cool to see someone think carefully about what "real" "fact checking" would be. First-hand sources? Links to instructions for replicating findings?Consensus checking is of course also an interesting problem, albeit a pretty different one.
https://www.purplemath.com/modules/bibleval.htm
"transparency" is the word you are looking for. It's more general and predates "open source" by far.
It's also a term that seems to be used similarly to the post title in contexts outside of software. For example, I subscribe to a podcast that covers arms control, where they frequently use the term "open source" to describe analysis derived publicly available sources and methods.
Moreover, how do you replicate work like calling sources to verify statements before publication?
I wrote a short piece for Wired once and had to go back and forth with their fact checker on the most picayune details. In an article about balancing online work and travel I had mentioned in passing searching under a hotel bed for an outlet, and she wanted to know what hotel it was, what country it was in, when I stayed there, etc. The process was exhaustive and not something I would go through with a bunch of Internet randos. It was also done before publication and involved a fairly high level of trust.
As for "how do you replicate work like calling sources to verify statements before publication?" Good question. Currently this is only designed to open up the fact checking which involves public primary sources. I think that a platform that focuses on doing that subset of fact checking really really well can add a lot of value to the public discourse.
Currently this is only designed to open up the fact checking which involves public primary sources.
Limiting the scope to this is wise. It's a tractable problem and it's better journalism. The types of journalists who write "this unnamed source implied that this other unnamed source had implied to another unnamed source..." articles won't like this sort of tool, but those journalists should be encouraged to change careers.
Some might not realize that articles like that are fairly common. For an example, I went to NYT homepage and this was the first article: https://www.nytimes.com/2019/04/03/us/politics/william-barr-...
First sentence: "Some of Robert S. Mueller III’s investigators have told associates that Attorney General William P. Barr failed to adequately portray the findings of their inquiry and that they were more troubling for President Trump than Mr. Barr indicated, according to government officials and others familiar with their simmering frustrations."
This sort of crap gets swept under the rug within a couple of weeks, but we're certainly paying attention to it now.
2) The basic proposition addressed by that sentence is that "investigators are uncomfortable with Barr's summary", but the implication is that there is a problem with that summary. The rest of the article addresses the potential meaning of this "big, if true" implication but gets no closer to the question of whether that implication is true. The alleged feelings of randoms are not interesting to the average reader. If investigators or anyone else has specific knowledge of wrongdoing that should be made known to the public, they should contact reporters directly.
3) Even if the basic proposition were firmer, we're approaching it through too many layers of indirection. The reporter didn't talk to the investigators. He didn't talk to associates of the investigators. He talked rather to "government officials and others familiar", and not about what those officials and others had heard from the investigators but rather about what associates of the investigators had heard from the investigators. How the government officials and others heard about it is not specified. We're at least four layers deep here, and nothing believable is four layers deep. These reporters are under no threat of "going to jail", because even if the basic question were important no judge could decide that the reporters are close enough to the truth to justify contempt.
4) The basic question is not important. Eventually politicians of both parties will read the entire un-redacted report. The public will not, because grand jury testimony will be redacted to protect the reputations of Republican operatives whose transgressions were not judged to merit indictments while "sources and methods" will be redacted to protect Democratic FBI (and possibly NSA?) agents whose transgressions were under color of law. It's not surprising that Barr's report had a political slant. Everything in DC has a political slant. Maybe if they found the Lindbergh baby, this tottering tower of misdirection could be justified. Nobody here found the Lindbergh baby.
5) That this happens "all the time" is actually a serious problem. Every day, well-placed people insert self-interested opinions into the public discourse not as self-interested opinions but rather as journalistic fact. This has terrible effects on our society and on our military victims around the world. Soi-disant journalists should stop aiding this terrible process, and this tool could help them do that.
As for "specific knowledge of wrongdoing"... well. What's "wrongdoing" here? Barr presented a summary of the Mueller Report to the public. Apparently, people in the investigation itself are unhappy with that summary and consider it deceptive (yeah, no shit, saying the Trump administration misleading people is like saying water is wet).
Yes, this is multi-layered, anonymous leakage, BUT. BUT. Is it untrue? Are investigators actually fine with the Barr summary, and the sources are lying to the journalists? Probably not. So despite the abstraction layers, the gist of the story is most likely true. And the NYT certainly doesn't have to run a story that says "Mueller is totally fine with the Barr summary" next week, so they're not going to print it unless they believe it true. So smell tests are fine here.
And the basic question is important. It's not whether Barr lied, but rather whether he intentionally misled - or more to the point, if Mueller's team believes he misled. That is relevant to the public, and to me.
But the article doesn't even say that they heard the investigators say that. At best it's hearsay and gossip of n-th degree, at worst it's outright misrepresentation of the facts.
Look at the weasel words: "according to government officials and others familiar with their simmering frustrations."
Who are government officials in this usage? And what qualifies someone as "familiar" with the matter? How many people work as a government official in some capacity and could claim some vague familiarity with the Mueller report? Does a park ranger in Wyoming who follows political news qualify? It wouldn't be technically lying to cite them as a "government official familiar with the matter", right?
You say "weasel words", I say "We think you should take this with a grain of salt". They're being explicitly clear that their sources are secondhand. What's your answer? That they shouldn't run the story at all?
Well, I think as you get more and more degrees removed from a primary source, the closer the article gets to being gossip and/or somebody's opinion and not factual news. Which is fine, there's a market for stuff like that. But I don't think that's very sustainable if you want to maintain a reputation as a purveyor of fact-based reporting.
There's a reason hearsay isn't generally admissable in a court of law.
Got it.
The main reasons are that courts have compulsory process to attain testimony of primary witnesses which makes it generally unnecessary, and that courts are by design intended to work via adversarial process including cross examination, and a indirect witness can't be cross examined.
Needless to say, there are lots of exceptions to hearsay exclusions, too.
Absolutely, a story based entirely on odious unattributable anonymous innuendo like that we see here should not be published. If the "government officials and others" want this important information to get out, they can still talk to Rep. Schiff. He will repeat anything that hypes up RussiaRussiaRussia, and he's on TV all the time. If, in fact, he isn't the "government official" to which this piece refers in the first place.
I don't say this because I support Trump or voted for him (never did, never would). I say it because this whole multiyear journalistic dumpster fire has guaranteed his reelection. TFA is all about putting out the flames, which I do support.
Two of these people have multiple Pulitzers between them. Schmidt is on tv practically daily. These are not low quality tips to no-name reporters - that 'fact' you made up to fit your preferred narrative. Or at least, did not bother to fact-check!
[0] https://www.pulitzer.org/winners/new-york-times-reporting-le...
It doesn't even have to come down to bad intentions or journalists intentionally lying about their anonymous sources, it's the fact that I have no way of reasonably knowing whether the NYT got tricked or lied to. Maybe their so-called "anonymous source" is just taking credulous journalists for a ride. I'm expected to just put blind faith in their opaque verification process and accept that anonymous sources are equally as valid as someone going on record?
Interestingly enough, the credibility (or lack thereof) of anonymous sources is an easily tractable problem that could be easily solvable with ring-signature cryptography [0] if there was the will to do so. And the trustless nature of that solution does not require journalists to be deemed the anointed arbiters of The Truth(tm). Alas, the social and cultural factors are just not there such that every government official, business leader, etc has a public crypto key.
[0]: https://people.csail.mit.edu/rivest/pubs/RST01.pdf
That is a real problem. Not anonymous sources or fact-checking.
To turn its slogan into an analogy: the New Yorker is high quality closed source software. If you disagree with the New Yorker, you can't file a bug report.
You might say, "I wouldn't want to," either because of something on your end, or something on the New Yorker's end. Continuing the analogy, there are closed-source models, open-source-but-closed-development models, and open-source-community-driven models.
I'm typing this because the problem this solves for me is I often disagree with the news, and I would like to fact check them. Perhaps not--as you mention--by calling sources, but by citing other evidence.
Every news source has bug reports too. They're called "letters to the editor".
If Wikipedia can't be factually accurate due to editors personal politics, how do you keep those personal politics out of a fact checker?
However, I'd argue that even without such a feature, having an open standard for facts openly verified from primary sources will add a lot of value to current public discourse.
Remember Wikipedia is a centralized and under control of single entity, fact that they rely on donation doesn't change anything.
This makes it easy to say "Well if it's accurate 99% of the time then it must confirm what I thought!".
- OP made an open platform
- Someone said it could be a good idea because Wikipedia
- Someone else said that Wikipedia has flaws
Hence my comment.
Coding this wasn't actually that complicated, figuring out a design that enables readers to review the fact checking as easily as possible and enables fact checkers to collaborate easily took a lot of experimentation though. Although even that is still very much a work in progress.
That way if a story needs to be expanded upon at a later date, the next journalist can dig deeper easily picking up the trail where the previous journalist left off.
There is, of course, the social issue of convincing people that this is important, but the underlying issue is in part a technical one. I don't think it's fair to disqualify a solution based on the fact that involves technical work.
I am not trying to disqualify the solution, more like observing that the existence of provable facts has not yet been any sort of antidote for 'fake news' (whether that refers to the real deal, or the position many people take that any news they don't like is fake news).
Some sort of attestation toolchain might enable people who care about evidence to produce more strongly-attested cases, and to get a better handle on what is and isn't sourced from others. It seems likely that using good attestation toolchains could create feedback loops for researchers, journalists, archivists, authors, film-makers, educators, students, and other people involved in knowledge production.
These feedback loops might change how they research, where they start, how often they have to re-invent wheels, and raise standards by enabling these people to compete on metrics that might drive more social value than clicks, engagement, shares, or ad revenue.
These effects could be important, even if you never budge the kind of person inclined to consume the information equivalent of junk-food.
Still, it seems valuable to make it easy for readers to view the supporting or contradictory evidence. That might’ve stymied many notable media hit-pieces. Of course, many newsrooms wouldn’t want this (at least I can’t imagine The Guardian and it’s ilk would want to draw attention to their own impoverished fact:BS ratios), so it would have to be a browser plugin or similar.
One can easily imagine hovering over a given assertion (e.g., "Damore posted an anti-diversity screed") and see the fact-checks, perhaps ranked by fact-checker's credibility in order to suppress conspiracy theory crap as well as contribute to the debate.
Anyway, that there is lots of noise among the criticism doesn't mean that there isn't a lot of valid criticism. It's only been a few months since nearly the whole American newsmedia willfully mislead the nation, sending an angry mob after innocent kids, after all.
Or they simply can't afford to employ fact checkers anymore, now that people refuse to pay for news.
People (the audience, that is) have historically refused to pay a significant share of the cost for news, what has changed is that advertisers have stopped paying as much for news (because the share of people's attention debited to news, and therefore which news outlets have to sell access to, has dropped.)
Paid circulation used to often be important not so much because of direct revenue but because advertisers looked to it as an important measure of reach.
If you've ever been interviewed by a modern journo, you know you just get the same question reworded a dozen different ways. The reason is that they decide what story they want to write up front, and then just go shopping for facts to support it. That story is usually dictated by market pressures, which presently reward sensationalism to feed the addiction of a dwindling number of listeners in an echo chambers.
And current fact checkers are typically checking material put out by others, which is completely broken. They have all the same incentives as anyone writing a hit / puff piece, except they put a phony "truth number" at the top and other sites will tally those up as though it means something.
I'm a bit jealous because I had nearly the same idea back when I worked in public radio, but didn't have the time to implement it. Basically it was about having facts and quotes exist as embedded units within articles, all with their own individual records of verification and relationships to all the articles in which they were used. I'm really glad someone is taking a shot at this, even though I have my doubts as to whether journalists will take it seriously.
YazIAm: If you want any further inspiration, here's my old github issue where I wrote about this concept: https://github.com/SCPR/SCPRv4/issues/718
I don't know what all the criteria are, but you should consider applying for a Knight Foundation grant: https://knightfoundation.org/apply/
As one problem is solved, it starts hinting at solutions to subsequent problems. This is not a problem with a silver bullet. It's going to take a fundamental shift in how we understand and verify truth.
1. Anyone who answers a question regarding whether the claim is "fully" proven by a source or whether the source is valid for the claim must or should (debatable) provide proof of their own interpretation by citing either specific instances in the source in question supporting it. For example, in the Net Neutrality article, Yatz states this: The adopted principles in this statement are at the top of page 3 in a response to does the source fully prove the claim. It would be nice if that was linked to the actual location in the document or identified through highlighting or some other mechanism to get full context for the response. There also should be uses for outside sources to be attached when evaluating these questions that are suppose to validate a source. This is because someone could submit a response to these source validity questions and cite an external source or internal quote of the source in question for their reasoning that doesn't support their answer.
2. Having some sort of expertise verification is extremely important and should be weighted either separately as in Experts: Yes Experts: No category and should be more important. It distinguishes this from random anon answers that have no training in how the source should be interpreted.
3. Having anonymous people do this seems rather dubious and the current platform seems like it would make it prone to Youtube comment syndrome. Maybe have some reputation system to gate keep like StackOverflow?
4. Sources usually require appropriate interpretation in order to be taken in the right context and be considered correct. Sometimes there's no single source or any source, philosophically, that fully proves a claim. I'm guessing that's what supporting documents are for?
5. Why is the original articles claim allowed to be edited?