The examples show Fox News-levels of graph manipulation, i.e. maximum manipulation.
This is clearly done in an attempt to communicate a particular message. If you read the questions as "What message was the author trying to communicate?", which is reasonable, then the study could show that you were willing to repeat back the author's intended message.
In this sense, you weren't fooled by the graph - you understood exactly the message it was intended to convey.
A truncated graph is, in my opinion, completely appropriate for data where the relative differences or trends in data points is more relevant to the message than the absolute values - a temperature chart, say (where the absolute values are pretty much meaningless).
If you took data of a clearly relative type rather than a clearly absolute type, would you be able to say that viewers were misled in the exact opposite direction? That would be interesting.
Regardless of the type of graph, it's important to understand whether a difference of 5 units between one measure and the next is meaningful.
For example if you are measuring the length of optical fibre and the difference between the lengths of fibres is 5cm over 100km, is that important in the context? You'll already lose several metres of length in various loops and coils that are important for reliable function of that 100km length of fibre.
On the other hand when cutting frames for doors, the difference of a couple of millimetres over the two metre length of the side of the frame can make the difference between passing or failing the building envelope pressurisation test.
On the gripping hand when I'm shooting an arrow 50m to a target, what's of interest to me is not the distance from me to the arrow but the tiny variances in distance from the arrow to the bullseye. So measuring the vector from me to the arrow is pointless and the centimetre or two difference in length of those vectors is meaningless.
It's similar to the lack of context. For example budget costs without context so people freak out on total dollar spent instead of how it fits in the context of historical spend. I would argue that time series plots should be given for most large number items so people have contextual history instead of going straight to outrage.
Not that there is an incentive for most media organizations to present information in a reasonable way, would lower traffic and engagement.
This is also similar to providing massive numbers without any context. For example $x produces 2.3M tonnes more $waste than $y every year!!! Of course $y produces 8.2T tonnes a year so it isn't /that much/ of a delta.
Also vaguely related to my pet peeve of using large denominators to make numbers large and confusing. This factory produces 63M items a year! How about 2 per second, I can visualize 2.
> Also vaguely related to my pet peeve of using large denominators to make numbers large and confusing. This factory produces 63M items a year! How about 2 per second, I can visualize 2.
smaller denominators can also be misleading though. maybe this is just me, but I usually read the denominator as implying a time period over which the figure is stable. for example, before the pandemic, I drove about 40 miles/day. I really did drive 20 miles each way to work pretty much every day. another way of saying this is that I drove 1.6 miles/hour every work day. true on average but kinda misleading when I spent 23 hours out of the day not driving at all.
it might not matter much to your example, but the factory probably didn't produce two items every second. there were probably shift changes, or maybe it stopped production for a few hours every night. depending on market conditions, it might have produced more on some days than others, and so forth.
That is true. There are cases where the timeframe is important, but often it isn't. I definitely should have put "on average" in there. You can also be more clear and say "Produces 5 per second when running at full capacity, averaging 2 per second throughout the year" if that is appropriate for the info you are sharing.
There's a vague prescription in Thinking Fast and Slow that an objective person attempting transparency should present both versions of the numbers.
Example:
.15% or half a million Americans died. What policy changes if any should we enact to try to compromise between lives lost and GDP, as lower GDP compromises (for instance) our influence in the international community.
Versus:
Side 1: it was less than one in 800 people. This is hysteria.
Side 2: What do you mean hysteria, we lost HALF A MILLION PEOPLE and it could have been 3 million if we had done it your way.
Ed Tufte pointed out that Japan’s newspapers were far ahead of American media in terms of presenting a large number of meaningful visualizations. He also pointed out that the Communist Party of Japan was the leader amping Japanese media on presenting charts. Needless to say there are important cultural factors at play here.
your solution of time series for context seems like a form of appeal to authority. because it was x last year, x+5% can't be bad! (appealing to the authority of the budget maker in this case)
appropriate context would be a breakdown of what x consists of, not just what it was last time period, and how that changed over time. then viewers can decide for themselves what's reasonable as a whole budget based on its parts. appropriate visualizations can provide breakdowns like this without much added complication.
but as you noted, letting readers think for themselves is not in the interests of the media, whose power lies in the control they can exert on what we think.
The number of pixels representing quantities in a graph should be proportional to each other as what they represent. This also addresses perspective bias in slanted 3d pi charts
Curious what biases one might also find based on color intensity of bars & varying widths
Not sure how to balance out this idea with log scales, which can be useful, but require a bit more care in visually digesting
This is neither new nor surprising. I recommend the little book "How to lie with statistics" by Darrell Huff, first published 1954. Chapter 5, called "The Gee-Whiz Graph" dives into the matter of truncated bar graphs (among others).
This could be considered a subset of the more general problem of trying to figure out when two points on a graph are meaningfully different, as opposed to just visually different.
In my early physics training the importance of putting error bars on data points was stressed, allowing for a visual comparison of the difference between two points in standard deviations.
We have to use the technology: animate bars on the X/Y scale on the graph, speed of the animation should be tied to the current max value represented on the graph. This way, humans will naturally recognize the value the current bars represent.
Wonderful. We now have research to prove what has been well-known and obvious for many decades.
After all this exact form of misleading graph was one of the topics in the famous book How to Lie With Statistics written in 1954. And I don't think that the observation was new then.
Funny, I just added y-min: 0 to a lot of my grafana graphs today for this exact reason. It's really good to give yourself sufficient context when creating dashboards or you miss the bigger, more important picture.
20 comments
[ 3.0 ms ] story [ 42.1 ms ] threadhttps://psyarxiv.com/7aq4h/
The examples show Fox News-levels of graph manipulation, i.e. maximum manipulation.
This is clearly done in an attempt to communicate a particular message. If you read the questions as "What message was the author trying to communicate?", which is reasonable, then the study could show that you were willing to repeat back the author's intended message.
In this sense, you weren't fooled by the graph - you understood exactly the message it was intended to convey.
A truncated graph is, in my opinion, completely appropriate for data where the relative differences or trends in data points is more relevant to the message than the absolute values - a temperature chart, say (where the absolute values are pretty much meaningless).
If you took data of a clearly relative type rather than a clearly absolute type, would you be able to say that viewers were misled in the exact opposite direction? That would be interesting.
The problem are truncated bar graph. Using a truncated bar graph is never a good idea.
For example if you are measuring the length of optical fibre and the difference between the lengths of fibres is 5cm over 100km, is that important in the context? You'll already lose several metres of length in various loops and coils that are important for reliable function of that 100km length of fibre.
On the other hand when cutting frames for doors, the difference of a couple of millimetres over the two metre length of the side of the frame can make the difference between passing or failing the building envelope pressurisation test.
On the gripping hand when I'm shooting an arrow 50m to a target, what's of interest to me is not the distance from me to the arrow but the tiny variances in distance from the arrow to the bullseye. So measuring the vector from me to the arrow is pointless and the centimetre or two difference in length of those vectors is meaningless.
Not that there is an incentive for most media organizations to present information in a reasonable way, would lower traffic and engagement.
Also vaguely related to my pet peeve of using large denominators to make numbers large and confusing. This factory produces 63M items a year! How about 2 per second, I can visualize 2.
smaller denominators can also be misleading though. maybe this is just me, but I usually read the denominator as implying a time period over which the figure is stable. for example, before the pandemic, I drove about 40 miles/day. I really did drive 20 miles each way to work pretty much every day. another way of saying this is that I drove 1.6 miles/hour every work day. true on average but kinda misleading when I spent 23 hours out of the day not driving at all.
it might not matter much to your example, but the factory probably didn't produce two items every second. there were probably shift changes, or maybe it stopped production for a few hours every night. depending on market conditions, it might have produced more on some days than others, and so forth.
Example:
.15% or half a million Americans died. What policy changes if any should we enact to try to compromise between lives lost and GDP, as lower GDP compromises (for instance) our influence in the international community.
Versus:
Side 1: it was less than one in 800 people. This is hysteria.
Side 2: What do you mean hysteria, we lost HALF A MILLION PEOPLE and it could have been 3 million if we had done it your way.
Side 1: 1 in 800 is not that many.
Side 2: 1 in 800 is a lot, actually.
appropriate context would be a breakdown of what x consists of, not just what it was last time period, and how that changed over time. then viewers can decide for themselves what's reasonable as a whole budget based on its parts. appropriate visualizations can provide breakdowns like this without much added complication.
but as you noted, letting readers think for themselves is not in the interests of the media, whose power lies in the control they can exert on what we think.
Curious what biases one might also find based on color intensity of bars & varying widths
Not sure how to balance out this idea with log scales, which can be useful, but require a bit more care in visually digesting
> While discussions of misleading graphs are not new (e.g., Huff, 1954), empirical research on their assumed consequences is scattered across fields.
So on the surface, at least, this looks like just another paper belabouring the obvious.
His books are really good sources for learning to do this data-presentation well.
In my early physics training the importance of putting error bars on data points was stressed, allowing for a visual comparison of the difference between two points in standard deviations.
After all this exact form of misleading graph was one of the topics in the famous book How to Lie With Statistics written in 1954. And I don't think that the observation was new then.