I guess the only way to combat this in fields where you aren't knowledgeable is to read many sources and synthesize your own conclusions? Alternately, you could accept that there are good and bad journalists, and trust some of them?
I get your point, but only ever trusting primary sources would require you to be an expert in everything. At some point you need to trust that someone else synthesized the data correctly. Or else you simply have no opinions on topics you don't know from first principles.
I have tried to treat information I gather from news sources as preliminary. When I relay information I've learned from these sources, I try to say "I read in the new york times" instead of repeating the information as fact. There's a lot of information that I know I'll never get around to researching thoroughly, but it still is good to be able to use as a working base of knowledge until what seems to be better info can replace it.
Critical thinking (and reading and listening) is a state of mind. Its not so much disbelieving everything you hear, so much as looking at what is communicated as a set of data points and one of perhaps many conclusions.
I don't take credit for this insight, I believe I first heard it from Don Knuth when we were discussing a research paper. The process is to try to ascertain what is the data in the communication, which of the pieces of information were observed or came from unbiased sources. Next is to think about what data might be missing, so if you're reading a paper that says "all of the people in our study could explain complexity theory" what other things about those people might have been considered and weren't? Their course of study? Their interests? How they were chosen?
Once you have an idea of what data is present, and perhaps a guess at what isn't, look at the conclusion or assertion in the paper. One flag to look for is if some missing data that should have been available would materially change the conclusion. An example of this was watching a news story of a single mother of two who had been unemployed for over a year. The story concluded that unemployment benefits needed to be extended because people "like this" were out of work. And yet they never mentioned her job although they did mention where she lived. So a bit of web searching we can turn up other bits of information about this person. She was unemployed, had filed and lost several workmens compensation claims against her former employer, had no additional training or schooling past high school, and was terminated from her previous job under a cloud of poor accounting records keeping. Of course adding in all of the information to the news story would not have made her a sympathetic figure so it wasn't included. I don't doubt the news producer, if pressed, would simply call her "an example" of the problem rather than a specific problem.
So critical reading and listening begins with "What is this person trying to communicate?" and the second is "How hard or honest are they being in making their argument?"
Fair warning here, if you are in a conversation with someone and start probing around the data question they will get mad at you. (I won't, I'll just jump into my reasoning, but I have found that isn't the common response)
The risk of being a critical thinker/reader/listener is that you risk rampant cynicism. This is especially true around sources which have a long history of shading the data to make their point. If you are not careful you throw out good data. I find it is useful and fun to deconstruct a news story with my wife. As she and I think differently about things we always find questions the other person missed (not that we always agree that they are the right questions, but they do keep the discussion interesting)
Every time I've had traditional media coverage of a project I've worked on, I've made this resolution. I'm not at all ungrateful for the coverage - each time, it made a difference, and it's lovely to be noticed - but even Wired, which should be pretty good at this, got major details wrong. (To their credit, TechCrunch, RWW et al have always reported accurately.)
Journalism is really important, but it's rare to see it done well. I really like long-form journalism outlets like the New Yorker for this reason.
I have something related:
- I read an article on hacker news
- I realize that the article is useless and/or completely undocumented
- I decide not to go to hacker news
- I read another article on hacker news
What I've found interesting is reading these tomes by people that pile up stuff from all sorts of various fields, e.g. Jared Diamond (Guns, Germs, Steel).
I'll read along happily, completely convinced, being amazed and thrilled at the depth and scope of his knowledge ... and then I'll happen across some little tidbit that's in a field I know about—and he screws it up, saying something silly/wrong in an authoritative tone.
It suddenly makes the entire thing come crashing down. I suddenly feel like I can't really trust anything he's said, because his obvious skill at sounding authoritative and confident, and the wide scope of his pronouncements, make it pretty obvious that there are probably other silly/wrong things in the book that I didn't get.
So I don't read Jared Diamond books anymore, entertaining may they be. Worse than that, I really can't read any of these sorts of books anymore. I can suspend my disbelief for fiction, but that doesn't work so well for non-fiction...
10 comments
[ 3.2 ms ] story [ 34.9 ms ] threadI don't take credit for this insight, I believe I first heard it from Don Knuth when we were discussing a research paper. The process is to try to ascertain what is the data in the communication, which of the pieces of information were observed or came from unbiased sources. Next is to think about what data might be missing, so if you're reading a paper that says "all of the people in our study could explain complexity theory" what other things about those people might have been considered and weren't? Their course of study? Their interests? How they were chosen?
Once you have an idea of what data is present, and perhaps a guess at what isn't, look at the conclusion or assertion in the paper. One flag to look for is if some missing data that should have been available would materially change the conclusion. An example of this was watching a news story of a single mother of two who had been unemployed for over a year. The story concluded that unemployment benefits needed to be extended because people "like this" were out of work. And yet they never mentioned her job although they did mention where she lived. So a bit of web searching we can turn up other bits of information about this person. She was unemployed, had filed and lost several workmens compensation claims against her former employer, had no additional training or schooling past high school, and was terminated from her previous job under a cloud of poor accounting records keeping. Of course adding in all of the information to the news story would not have made her a sympathetic figure so it wasn't included. I don't doubt the news producer, if pressed, would simply call her "an example" of the problem rather than a specific problem.
So critical reading and listening begins with "What is this person trying to communicate?" and the second is "How hard or honest are they being in making their argument?"
Fair warning here, if you are in a conversation with someone and start probing around the data question they will get mad at you. (I won't, I'll just jump into my reasoning, but I have found that isn't the common response)
The risk of being a critical thinker/reader/listener is that you risk rampant cynicism. This is especially true around sources which have a long history of shading the data to make their point. If you are not careful you throw out good data. I find it is useful and fun to deconstruct a news story with my wife. As she and I think differently about things we always find questions the other person missed (not that we always agree that they are the right questions, but they do keep the discussion interesting)
Journalism is really important, but it's rare to see it done well. I really like long-form journalism outlets like the New Yorker for this reason.
I'll read along happily, completely convinced, being amazed and thrilled at the depth and scope of his knowledge ... and then I'll happen across some little tidbit that's in a field I know about—and he screws it up, saying something silly/wrong in an authoritative tone.
It suddenly makes the entire thing come crashing down. I suddenly feel like I can't really trust anything he's said, because his obvious skill at sounding authoritative and confident, and the wide scope of his pronouncements, make it pretty obvious that there are probably other silly/wrong things in the book that I didn't get.
So I don't read Jared Diamond books anymore, entertaining may they be. Worse than that, I really can't read any of these sorts of books anymore. I can suspend my disbelief for fiction, but that doesn't work so well for non-fiction...
What do the experts on insulin can say about his books?