When I started getting into reading pop psychology in my mid-20s, I was very lucky to have a dad who reminded me to be skeptical of this stuff, despite it all sounding very confident. Doubly lucky to have a brother-in-law who eagerly criticized Gladwell’s “Outliers.”
I was very appreciative of the Scott Adams podcast[1] when he pointed out that any story or study that is too "on the nose", that is, hits our confirmation biases too perfectly, needs to be approached with extra caution.
I can't recall enough of a quote to find it right now, but someone said something like "any story that's good enough to be repeated should be regarded with skepticism'.
It's the sort of thing that might be attributed to Mark Twain or Robert Heinlein, whether or not they said it.
That's a great quote. I tried and failed to find the source, but if it comes to mind some time between now and before you've forgotten you ever mentioned it, please consider adding it to the thread.
Similarly fun to have to spend days at work doing "Myers Briggs", "Clifton Strengths" or variations thereof. Then have it woven into unrelated conversations for months until the novelty wears off.
Biggest issue with Meyers Briggs is that it gives people the impression that types are discrete (i.e., traits have a bimodal distribution) when in fact the traits are normally distributed. And of course it's just out of date since MB was developed decades ago and is entrenched in the business world.
>> Fwiw, the Big 5 (OCEAN) personality inventory is one of the more robust findings in psychology (much more so than all the examples from the blog post)
That depends on who you ask:
Many investigations into the structure of individual differences theorize in terms of latent
variables, but rely on Principal Components Analyses (PCA) when it comes to the analysis of
empirical data. However, the extraction of a principal components structure, by itself, will not
ordinarily shed much light on the correspondence with a putative latent variable structure. The
reason is that PCA is not a latent variablemodel but a data reduction technique (e.g., Bartholomew,
2004). This is no problem as long as one does not go beyond the obvious interpretation of a principal
component, which is that it is a conveniently weighted sumscore. Unfortunately, however,
this is not the preferred interpretation among the enthusiastic users of principal components
analysis.
Consider, for instance, the personality literature,where people have discovered that executing
a PCA of large numbers of personality subtest scores, and selecting components by the usual
selection criteria, often returns five principal components. What is the interpretation of these
components? They are “biologically based psychological tendencies,” and as such are endowed
with causal forces (McCrae et al., 2000, p. 173). This interpretation cannot be justified solely on
the basis of a PCA, if only because PCA is a formative model and not a reflective one (Bollen
& Lennox, 1991; Borsboom, Mellenbergh, & Van Heerden, 2003). As such, it conceptualizes
constructs as causally determined by the observations, rather than the other way around (Edwards
& Bagozzi, 2000). In the case of PCA, the causal relation is moreover rather uninteresting;
principal component scores are “caused” by their indicators inmuch the same way that sumscores
are “caused” by item scores. Clearly, there is no conceivable way in which the Big Five could
cause subtest scores on personality tests (or anything else, for that matter), unless they were in
fact not principal components, but belonged to a more interesting species of theoretical entities;
for instance, latent variables. Testing the hypothesis that the personality traits in question are
causal determinants of personality test scores thus, at a minimum, requires the specification
of a reflective latent variable model (Edwards & Bagozzi, 2000). A good example would be a
Confirmatory Factor Analysis (CFA) model.
Now it turns out that, with respect to the Big Five, CFA gives Big Problems. For instance,
McCrae, Zonderman, Costa, Bond, & Paunonen (1996) found that a five factor model is not
supported by the data, even though the tests involved in the analysis were specifically designed
on the basis of the PCA solution. What does one conclude from this? Well, obviously, because
the Big Five exist, but CFA cannot find them, CFA is wrong. “In actual analyses of personality
data [. . .] structures that are known to be reliable [from principal components analyses] showed
poor fits when evaluated by CFA techniques. We believe this points to serious problems with
CFA itself when used to examine personality structure” (McCrae et al., 1996, p. 563).
Denny Borsboom, The attack of the psychometricians, Psychometrika. 2006 Sep;71(3):425-440
You describe a reasonable minority critique of the use of Big 5 for latent variable theorizing. That some "enthusistic users" of Big 5 mis-apply it or claim unjustified confidence in downstream conclusions does not at conflict with my comment that you are responding to. Indeed, a similar critique can be leveled even more strongly at the other purported psychological findings mentioned in the blog post.
The critique is that the existence of the Big 5 itself is the result of misapplication of PCA for a task where a latent variable model would be required, the identification of personality traits.
To clarify, personality traits are "latent" in the sense that they are not directly observable in personality test scores, so any analysis of test scores that simply aggregates test scores is not capable of detecting personality traits. PCA simply aggregates data to reduce the dimensionality of the data but it can't detect latent variables.
Using PCA to detect personality traits is like using a thermometer to measure atmospheric pressure. What's worse, psychometrists seem to have taken the reading on the thermometer and proclaimed "Atmostperhic pressure is 25 [degrees Celsius]!".
> The critique is that the existence of the Big 5 itself is the result of misapplication of PCA for a task where a latent variable model would be required, the identification of personality traits.
As stated, that critique is wrong in this sense: for a given population of, say, new employees at a business, PCA is sufficient to establish correlations between things that are easy to measure (Big 5 assessment test scores) and things that are much more expensive to measure but which the business cares about (how the employee will perform if assigned to a certain team). In this way, mere correlational data, without the construction of a latent variable model, is operationally useful. Objections that the tests don't really "detect personality" "directly" are semantical in this context.
Yes, if the business starts using (implicitly or explicitly) a causal model of latent personality variables to make claims or predictions that take them outside the population distribution for which the correlations have been measured, then they could be making a mistake. But this does not invalidate the use of Big 5 assessments in general.
> Using PCA to detect personality traits is like using a thermometer to measure atmospheric pressure. What's worse, psychometrists seem to have taken the reading on the thermometer and proclaimed "Atmostperhic pressure is 25 [degrees Celsius]!".
Given a fixed population (in this case, a population of weather events) where temperature and pressure are well correlated, it is completely reasonable to use a thermometer to estimate pressure, and to take concrete action based on this. By all means, criticize people for being insufficiently precise (e.g. for not stating what their error bars or what the relevant population is), but don't pretend that this completely invalidates everything and Big 5 should be ignored.
On typing: "other, intangible process factors, e.g., the preference of certain personality types for functional, static and strongly typed languages" stands out.
As he says, you may be able to show a slight benefit, but I would bet that if you controlled for that "personality type", the strong and replicable result would be "people who program in a language they like do a better job than those forced to use one they don't like."
FWIW, I don't think the timescales are correct with the longitudinal twin-exercise studies. I don't think anyone would predict that exercise today would have anything (or rather, very little) causal to do with mood 2 years from now. Sustained exercise over the course of a couple of weeks or more might for that period or for maybe a month or two afterward, but I doubt anything more than that.
I also am not entirely sure that twin studies are the best way to get at causality in terms of exercise and mood, because the exercise is nonrandom. I think someone might object in the sense that that's the point, but my perspective is that it's still not the same as randomly assigning exercise. There's a lot buried in their acknowledgment that a third factor could still account for the results, as that means a lot given the design.
FWIW, there's also evidence in the opposite direction using similar but different logic without twins, but using molecular genetic risk scores:
"The risk of depression was 22% higher among those at high genetic risk compared with those at low genetic risk (HR = 1.22, 95% CI: 1.14–1.30). Participants with high genetic risk and unfavorable lifestyle had a more than two-fold risk of incident depression compared with low genetic risk and favorable lifestyle (HR = 2.18, 95% CI: 1.84–2.58). There was no significant interaction between genetic risk and lifestyle factors (P for interaction = 0.69). Among participants at high genetic risk, a favorable lifestyle was associated with nearly 50% lower relative risk of depression than an unfavorable lifestyle (HR = 0.51, 95% CI: 0.43–0.60). We concluded that genetic and lifestyle factors were independently associated with risk of incident depression. Adherence to healthy lifestyles may lower the risk of depression regardless of genetic risk."
I guess the broader point is to be careful about making claims of BS (or anything else) based on a single study.
Radio stations near me seem to always be playing John Tesh “intelligence for your life” I’ve often wondered a) how many people take that advice and apply it and b) if you tried to apply all the advice what would become of you. I’d wager that you would have to start getting ready for bed immediately upon waking to just handle all the things that are recommended to do “before bed”
A very recent example: a made-up story about a hospital being overrun with patients who had overdosed on Ivermectin, such that they were unable to treat gunshot victims. This ran in Rolling Stone, the Guardian and other pubs. Your usual run of smart, skeptical urbane types accepted it uncritically.
Be careful when something you would like to be true comes along.
It’s also worth examining what happens after something like this is shown to be untrue. Some will own up to their mistake, most will not. But will continue banging on about “misinformation” and so forth.
The happiness/income debunking here is flawed. The use of logarithm for the graph axis to give the perspective of a linear relationship is flawed - it's also responding to a mis-summary of the research.
Also note that the graph axis for income tops out at around.. about that 75k/year figure. Whoops.
It should also be noted that 75k figure is a median, it changes based on cost of living.
It is not flawed. The subject is under study, continually. The debunking comes from https://www.pnas.org/content/118/4/e2016976118 where yes, many measures of happiness seem to increase linearly as log(income) increases. That means diminishing marginal utility of money, but it does not mean there's a cap - the relationship seems to continue at all income levels.
The main reason to believe the famous previous research was in good faith and could still miss this result is because the previous research used a measure of happiness which itself capped out, and so _could not detect_ changes in happiness after a certain level. If your instruments can't register any changes above some amount, it is no wonder your results level off and stop at that amount.
This new research could be wrong, of course. But there's no mis-summary here.
That's super interesting! I can see their study included asking to what extent the participants are experiencing certain negative/positive feelings (which may give some more context), but isn't their core measure of happiness also capped? They use a continuous response scale, but in terms of being capped, it shouldn't matter if it's measured on a 0-100 or 1-5 scale, no? The participants can't be happier than "extremely happy" in their study design as far as I can tell.
I’m sort of skeptical about these “turns out” stories. Generally when you follow the trail of turns out the original statement is kind of right but for the wrong reasons. It’s sort of like high school physics, kind of wrong, but right enough to be useful.
Take dunning-kruger for example. Yes, it was not shown that people of low ability rank themselves higher than those of high ability. But it was shown that people of low ability rank themselves higher than they should, and that makes dunning-kruger’s popsci version right enough in many circumstances where it is used to make a point.
Or take the findings on happiness. Yes, money can make you happier, contrary to prevailing opinion. But other factors affect happiness as well if not more, so if you have to choose between money and other things, often times the other things will make you happier. Again, the popsci version of “money doesn’t make you happy” is right enough to be useful.
Same thing with climate change skepticism. When you first start digging into the better skeptic arguments they seem to be onto something, but as you dig down into the details and peel back the layers of the onion, the popsci consensus view of climate change turns out to be right enough to be useful.
I’ve come to the conclusion that these false trueisms hang around because they are “good enough”. It turns out the turns outs are much less insightful than they seem.
Like every other popular thought/meme/idea though, it can be equally true that they stick around, not because they are "good enough" but rather because they support the opinion the person (or people) already held. "money doesn't make you happy" could just as easily be commonly believed because it helps people stay happy when they don't have much money, and to explain away not achieving more because "those rich folk aren't REALLY happy, but I am". Same with the statically typing -- we all know how opinionated people are about code, this just helps in the echo chambers too.
But also, just because something is "true enough" isn't the issue -- the issue is that these are touted as ways SCIENCE proved someone right, when in fact they are absolutely not "facts" like many believe, and so that can erode belief in science.
26 comments
[ 0.23 ms ] story [ 84.1 ms ] thread[1] https://www.scottadamssays.com/
I can't recall enough of a quote to find it right now, but someone said something like "any story that's good enough to be repeated should be regarded with skepticism'.
It's the sort of thing that might be attributed to Mark Twain or Robert Heinlein, whether or not they said it.
https://en.wikipedia.org/wiki/Big_Five_personality_traits
and the four Meyers Briggs "types" are quite correlated with four of the Big 5.
https://en.wikipedia.org/wiki/Myers%E2%80%93Briggs_Type_Indi...
Biggest issue with Meyers Briggs is that it gives people the impression that types are discrete (i.e., traits have a bimodal distribution) when in fact the traits are normally distributed. And of course it's just out of date since MB was developed decades ago and is entrenched in the business world.
That depends on who you ask:
Many investigations into the structure of individual differences theorize in terms of latent variables, but rely on Principal Components Analyses (PCA) when it comes to the analysis of empirical data. However, the extraction of a principal components structure, by itself, will not ordinarily shed much light on the correspondence with a putative latent variable structure. The reason is that PCA is not a latent variablemodel but a data reduction technique (e.g., Bartholomew, 2004). This is no problem as long as one does not go beyond the obvious interpretation of a principal component, which is that it is a conveniently weighted sumscore. Unfortunately, however, this is not the preferred interpretation among the enthusiastic users of principal components analysis.
Consider, for instance, the personality literature,where people have discovered that executing a PCA of large numbers of personality subtest scores, and selecting components by the usual selection criteria, often returns five principal components. What is the interpretation of these components? They are “biologically based psychological tendencies,” and as such are endowed with causal forces (McCrae et al., 2000, p. 173). This interpretation cannot be justified solely on the basis of a PCA, if only because PCA is a formative model and not a reflective one (Bollen & Lennox, 1991; Borsboom, Mellenbergh, & Van Heerden, 2003). As such, it conceptualizes constructs as causally determined by the observations, rather than the other way around (Edwards & Bagozzi, 2000). In the case of PCA, the causal relation is moreover rather uninteresting; principal component scores are “caused” by their indicators inmuch the same way that sumscores are “caused” by item scores. Clearly, there is no conceivable way in which the Big Five could cause subtest scores on personality tests (or anything else, for that matter), unless they were in fact not principal components, but belonged to a more interesting species of theoretical entities; for instance, latent variables. Testing the hypothesis that the personality traits in question are causal determinants of personality test scores thus, at a minimum, requires the specification of a reflective latent variable model (Edwards & Bagozzi, 2000). A good example would be a Confirmatory Factor Analysis (CFA) model.
Now it turns out that, with respect to the Big Five, CFA gives Big Problems. For instance, McCrae, Zonderman, Costa, Bond, & Paunonen (1996) found that a five factor model is not supported by the data, even though the tests involved in the analysis were specifically designed on the basis of the PCA solution. What does one conclude from this? Well, obviously, because the Big Five exist, but CFA cannot find them, CFA is wrong. “In actual analyses of personality data [. . .] structures that are known to be reliable [from principal components analyses] showed poor fits when evaluated by CFA techniques. We believe this points to serious problems with CFA itself when used to examine personality structure” (McCrae et al., 1996, p. 563).
Denny Borsboom, The attack of the psychometricians, Psychometrika. 2006 Sep;71(3):425-440
https://pubmed.ncbi.nlm.nih.gov/19946599/
In other words: bad statistics, badly applied.
To clarify, personality traits are "latent" in the sense that they are not directly observable in personality test scores, so any analysis of test scores that simply aggregates test scores is not capable of detecting personality traits. PCA simply aggregates data to reduce the dimensionality of the data but it can't detect latent variables.
Using PCA to detect personality traits is like using a thermometer to measure atmospheric pressure. What's worse, psychometrists seem to have taken the reading on the thermometer and proclaimed "Atmostperhic pressure is 25 [degrees Celsius]!".
As stated, that critique is wrong in this sense: for a given population of, say, new employees at a business, PCA is sufficient to establish correlations between things that are easy to measure (Big 5 assessment test scores) and things that are much more expensive to measure but which the business cares about (how the employee will perform if assigned to a certain team). In this way, mere correlational data, without the construction of a latent variable model, is operationally useful. Objections that the tests don't really "detect personality" "directly" are semantical in this context.
Yes, if the business starts using (implicitly or explicitly) a causal model of latent personality variables to make claims or predictions that take them outside the population distribution for which the correlations have been measured, then they could be making a mistake. But this does not invalidate the use of Big 5 assessments in general.
> Using PCA to detect personality traits is like using a thermometer to measure atmospheric pressure. What's worse, psychometrists seem to have taken the reading on the thermometer and proclaimed "Atmostperhic pressure is 25 [degrees Celsius]!".
Given a fixed population (in this case, a population of weather events) where temperature and pressure are well correlated, it is completely reasonable to use a thermometer to estimate pressure, and to take concrete action based on this. By all means, criticize people for being insufficiently precise (e.g. for not stating what their error bars or what the relevant population is), but don't pretend that this completely invalidates everything and Big 5 should be ignored.
As he says, you may be able to show a slight benefit, but I would bet that if you controlled for that "personality type", the strong and replicable result would be "people who program in a language they like do a better job than those forced to use one they don't like."
I also am not entirely sure that twin studies are the best way to get at causality in terms of exercise and mood, because the exercise is nonrandom. I think someone might object in the sense that that's the point, but my perspective is that it's still not the same as randomly assigning exercise. There's a lot buried in their acknowledgment that a third factor could still account for the results, as that means a lot given the design.
FWIW, there's also evidence in the opposite direction using similar but different logic without twins, but using molecular genetic risk scores:
https://www.nature.com/articles/s41398-021-01306-w
"The risk of depression was 22% higher among those at high genetic risk compared with those at low genetic risk (HR = 1.22, 95% CI: 1.14–1.30). Participants with high genetic risk and unfavorable lifestyle had a more than two-fold risk of incident depression compared with low genetic risk and favorable lifestyle (HR = 2.18, 95% CI: 1.84–2.58). There was no significant interaction between genetic risk and lifestyle factors (P for interaction = 0.69). Among participants at high genetic risk, a favorable lifestyle was associated with nearly 50% lower relative risk of depression than an unfavorable lifestyle (HR = 0.51, 95% CI: 0.43–0.60). We concluded that genetic and lifestyle factors were independently associated with risk of incident depression. Adherence to healthy lifestyles may lower the risk of depression regardless of genetic risk."
I guess the broader point is to be careful about making claims of BS (or anything else) based on a single study.
[0] https://www.littlebrown.com/titles/rutger-bregman/humankind/... [1] https://cjlm.ca/notes/humankind/
Be careful when something you would like to be true comes along.
It’s also worth examining what happens after something like this is shown to be untrue. Some will own up to their mistake, most will not. But will continue banging on about “misinformation” and so forth.
Also note that the graph axis for income tops out at around.. about that 75k/year figure. Whoops.
It should also be noted that 75k figure is a median, it changes based on cost of living.
The main reason to believe the famous previous research was in good faith and could still miss this result is because the previous research used a measure of happiness which itself capped out, and so _could not detect_ changes in happiness after a certain level. If your instruments can't register any changes above some amount, it is no wonder your results level off and stop at that amount.
This new research could be wrong, of course. But there's no mis-summary here.
Take dunning-kruger for example. Yes, it was not shown that people of low ability rank themselves higher than those of high ability. But it was shown that people of low ability rank themselves higher than they should, and that makes dunning-kruger’s popsci version right enough in many circumstances where it is used to make a point.
Or take the findings on happiness. Yes, money can make you happier, contrary to prevailing opinion. But other factors affect happiness as well if not more, so if you have to choose between money and other things, often times the other things will make you happier. Again, the popsci version of “money doesn’t make you happy” is right enough to be useful.
Same thing with climate change skepticism. When you first start digging into the better skeptic arguments they seem to be onto something, but as you dig down into the details and peel back the layers of the onion, the popsci consensus view of climate change turns out to be right enough to be useful.
I’ve come to the conclusion that these false trueisms hang around because they are “good enough”. It turns out the turns outs are much less insightful than they seem.
But also, just because something is "true enough" isn't the issue -- the issue is that these are touted as ways SCIENCE proved someone right, when in fact they are absolutely not "facts" like many believe, and so that can erode belief in science.
Is that to say that the Prison Experiment is actually explained by the Milgram obedience experiments? :-)