> The experience of weight, exemplified by heaviness and lightness, is metaphorically associated with concepts of seriousness and importance. This is exemplified in the idioms “thinking about weighty matters” and “gravity of the situation.”
How could an article like this seriously have gotten through peer review?
The author offers rules of thumb for evaluating research, but is a business school grad (MBA? BBA?) and "a data science amateur, from the Latin amare – to love." Am I misunderstanding something? On what do they base these instincts and intuitions for evaluating research papers? If they wrote a blog post on rules of thumb for evaluating code, would you take it seriously?
As per the large amount of links in the article, the author seems to have passed the test themselves, then talked about what heuristics they used to get their result (that matched the studies linked). Did you read the article?
One can argue with their heuristics, of course, but dismissing them because they're open about their background is a classic middlebrow dismissal. This isn't presented as rigorous science.
That is classic unwelcome behavior on HN. Maybe you can keep your judgements of other commenters and hot take reactions to yourself and stick to the subject.
I didn't criticize them for being open about their background, but because they lack expertise to offer expert opinions. Again, if they wrote an article on rules of thumb for evaluating code or on reading radiology images for cancer - things they appear equally expert in - it wouldn't be taken seriously here.
> As per the large amount of links in the article, the author seems to have passed the test themselves
Huh? Lots of links pass some test? Reading a bunch of links make you an expert? I guess the Internet really is full of experts.
What would be productive is discussing the expertise of the author, or if you don't think that's of interest, not discussing it at all. I notice you posted the original article; I'm not criticizing you for posting it but perhaps this is getting personal?
Insulting commenters with derogatory names is unwelcome and lowers the quality of discussion; I don't care if you have a link to it. Tangentially, it is a good example of how even the smallest behavior by a leader, such as pg, set a norm for many followers: people are digging up his posts 6 years later like they are scripture - wisdom, law and conduct to study. pg is just a person, not a prophet; YC and HN are wonderful, but his every act is not wisdom or law. (No insult to pg; the same could be said of Linus Torvalds, Steve Jobs, and everyone else who ever worked in SV or lived on Earth, including me.)
> Credentials != expertise (one can be an expert by self-studying, in fact in CS many are)
Credentials are a strong, widely used sign of expertise. The CS experts who lack educational credentials are practitioners, professionals in the field; they have other credentials. The author is neither educated in the field nor is a professional or practitioner in it. Everyone else weighs those things heavily. Do you hire or take the advice of people with neither education nor experience in the domain?
The article in particular requires expertise: It's a series of rules of thumb, things that are gained only through extensive experience doing the task described. And it requires doing it with success - not all experience is good experience - and apply good analysis to that experience. Rules of thumb for physics research from someone who washed out are not the same as those from Stephen Hawking; how about rules of thumb from someone who never has studied nor practiced physics, but read some links? Again, if they wrote 'rules of thumb for evaluating code' on the same basis, it would be ridiculous.
In general, everyone knows that the Internet is full of people spewing nonsense they know nothing about, but even smart people often seem to fail to apply that to particular instances, especially when other psychological drives apply - such as when what is said moves or inflames them. The downvotes and criticisms (this is an observation, not a complaint) seem to be because my comment disrupts that drive (in this case, the inflammatory narrative about social sciences). That's ok and what I expected. My prediction is that if I wrote the same about a different topic, then the response would be different: If what I wrote agreed with the crowd (e.g., if the author said psychological research was high quality and I questioned their expertise), then I'd expect many to agree with me, but for the same bad reason. Inflammatory stuff is, disappointingly, still hard to talk about rationally.
>I notice you posted the original article; I'm not criticizing you for posting it but perhaps this is getting personal?
No, but the above is an attempt of second-guessing my psychology (and also irrelevant).
Now, you follow with a long list of opinions of the importance of credentials. What I don't see is any opinion on why the particular arguments are not enough. Can you enlighten us? You could point to someone with credentials which refutes them for example.
If you just want to raise a general principle (totally unrelated to the quality or not of the arguments in TFA) then it seems that you suggest that we should only discuss articles written by people with credentials on a topic.
If so, why should we accept this suggestion of yours then?
Where are your credentials on epistemology (seeing that this
is an epistemological issue)?
> a long list of opinions of the importance of credentials
Those haven't been addressed. I think calling them "opinions" is a big stretch; it's a reliable statement of fact that humanity very often relies on the credentials of the speaker, including their training and experience, for evaluating information they communicate.
> What I don't see is any opinion on why the particular arguments are not enough.
It's up to the person making the claims to establish them. We can't spend our time refuting everything anyone says. In fact, making claims in order to force others to refute them is a well-known propaganda technique for creating dysfunction, in part because refuting takes far more time than just making claims. (This article is not propaganda, in case anyone thinks I'm implying that.)
The arguments made by the author are expert judgments; there's little to address by other means.
> it seems that you suggest that we should only discuss articles written by people with credentials on a topic
Personally, I want to spend my time on the best knowledge I can find. Life is short. Identifying quality information depends frequently on credentials; I think that's uncontroversial.
You keep taking things to an extreme - "only" - and to be clear, those are your words and not mine. Somewhere else I also made that explicit. EDIT: Here it is, copied from another subthread to keep everything in one place:
Of course, let's not make this binary, one or the other [merits or credentials]. Everyone uses both to varying degrees, and other signals too (if the post used bad English, for example, many would doubt the author's credibility).
In this case, I lean more heavily on expertise because the content is a product of expertise - rules of thumb. If they wrote 'I can create cold fusion in a teacup', I'd lean heavily on the merits - can they do it or not?
>On what do they base these instincts and intuitions for evaluating research papers? If they wrote a blog post on rules of thumb for evaluating code, would you take it seriously?
Depends, since I tend to evaluate arguments on their merits, not on the author's credentials.
If one doesn't have the background to evaluate those arguments, I don't think they're supposed to be reading them anyway (whether they are from an "amateur" or a Nobel winning pro).
> I tend to evaluate arguments on their merits, not on the author's credentials
Of course, let's not make this binary, one or the other. Everyone uses both to varying degrees, and other signals too (if the post used bad English, for example, many would doubt the author's credibility).
In this case, I lean more heavily on expertise because the content is a product of expertise - rules of thumb. If they wrote 'I can create cold fusion in a teacup', I'd lean heavily on the merits - can they do it or not?
> If one doesn't have the background to evaluate those arguments, I don't think they're supposed to be reading them anyway
That greatly narrows everyone's reading material and awareness of the world. Somehow I have to decide what to do about many things in life, almost all of which I lack expertise in.
In the end, there is no way to avoid having to decide whether to trust the author or not - or more generally, to trust others or not, from consultants to bosses to employees to that wonderful person you might want to marry.
> If they wrote a blog post on rules of thumb for evaluating code, would you take it seriously?
I don't know if I would take it seriously before I read it. But after reading it, if it had quiz showing me their method could get over 95% correct after reading a 10 or so word summary of the end result, I'd take it very seriously. I haven't come across too many things in my professional life that have save me that much time.
It makes you wonder just how much spare time the people who peer reviewed those papers had on their hands.
> if it had quiz showing me their method could get over 95% correct after reading a 10 or so word summary of the end result, I'd take it very seriously.
Often, amateurs find seemingly obvious answers to the wrong question. I once was reading Amazon user reviews of translations of Confucius' Analects. For one translation, a few reviewers observed that the translator went bizarrely off-topic in the footnotes, giving his opinions on various things Confucian and contemporary. I ruled out that translation. Then I read a Chinese scholar's review of translations, and learned that such commentary was in fact the standard of the leading editions of the Analects going back millennia. The Amazon users - as amateurs often do - had no idea what they were seeing, and if I hadn't found the scholar's review, I would have believed them too (and missed out what that incredible work translation - a widely recognized work of art and genius - had to offer me from Confucius). That's the last time I trusted Amazon reviews. Similarly, you can find plenty of well-reasoned, persuasive, and totally wrong legal and medical advice online (including on HN).
How can one avoid falling into the same trap as I did with the Amazon reviews? Don't trust amateurs, no matter how persuasive they seem, in questions that require expertise.
It's been said before but needs to be said again: this is happening everywhere in the biomedical and related fields. The neurosciences, oncology... the list goes on and on. Anyone take a look at AI research lately? How much tweaking is going on there? How much is your big data finding due to the idiosyncracies of your particular dataset?
Psychology, as has historically always been the case--meta-analysis itself bloomed largely from the field--is the one turning inward and looking at itself. And it's getting crap from people who love to use it as their favorite punching bag.
The irony of this article is that it's psychologists looking at other psychologists, doing the math etc.
The truth is closer to the second hypothesis by the author: bullshit is incentivized everywhere in academics. Reality is less interesting, harder, more incremental. Everyone wants the next genius savior to point to because it's a simpler story than reality. Sexy means more pubs, more grant money.
I’ve recently been at a master students’ thesis defense where a method was discussed that is supposed to speed up algorithms by deciding whether to run them on GPU or CPU. The method is to look at the program code with deep learning algorithms.
Fair enough. There’s apparently a set of benchmarks that other papers used and were reused here to compare against. Yet the student was unable to answer my question as to how bias in this dataset was treated or even what percentage of the algorithms was faster on GPU to begin with, they only showed how much better their deep learning fit the data. As in, if 99% is faster on GPU I could as well just provide that as a static answer and be better than any machine learning.
I frankly find it tragic that this kind of stuff is not ironed out in academia. This was was in a very highly ranked university.
> Imagine yourself washing your hands. Do you feel any impact whatsoever on your desire to rationalize decisions? Now imagine explaining this study to Nassim Taleb’s grandma.
What's the joke with invoking Nassim here? That he is some kind of anti-pretentious truth-teller whose grandma is even further on that spectrum?
There was a recent finding (not yet replicated!) that performing rituals of any kind aids in building/retaining self-control/willpower.
This is kind of like priming, but—if it holds true—this result would actually take away a lot of the explanatory power of most priming studies. A simpler explanation than a bunch of specific "doing X makes you think about Y which makes you do Z" priming relations, is "doing {any number of things} releases striatal dopamine, that increases executive functioning, which allows you to do {any number of things}."
> Papers that report boring or common-sense results seem to be true very often. Cognitive psychology (e.g. memory and perception) mostly held up.
> ...it turns out for most people "if the claim of the paper doesn't make intuitive sense, the paper is wrong" is a good shortcut that works about 70% of the time.
Generally, the closer to the "metal" (e.g., memory, perception, learning) and the more pedestrian the psychological finding is, the likelier it is to hold good.
I might be giving in to a bit of hindsight bias here, but I have come across this in the psychology of learning myself. It has been a bit underwhelming I must admit. "Good sleep, retrieval practice, elaboration, paying attention, meaningfulness and organization, and spaced repetition result in better learning; massed-practice is not very useful for long-term retention". Well, no kidding!
25 comments
[ 2.7 ms ] story [ 68.1 ms ] threadHow could an article like this seriously have gotten through peer review?
The "scent" here may be something on my shoe ...
As per the large amount of links in the article, the author seems to have passed the test themselves, then talked about what heuristics they used to get their result (that matched the studies linked). Did you read the article?
One can argue with their heuristics, of course, but dismissing them because they're open about their background is a classic middlebrow dismissal. This isn't presented as rigorous science.
> classic middlebrow
That is classic unwelcome behavior on HN. Maybe you can keep your judgements of other commenters and hot take reactions to yourself and stick to the subject.
I didn't criticize them for being open about their background, but because they lack expertise to offer expert opinions. Again, if they wrote an article on rules of thumb for evaluating code or on reading radiology images for cancer - things they appear equally expert in - it wouldn't be taken seriously here.
> As per the large amount of links in the article, the author seems to have passed the test themselves
Huh? Lots of links pass some test? Reading a bunch of links make you an expert? I guess the Internet really is full of experts.
I'd say middlebrow dismissal is even more classic: https://news.ycombinator.com/item?id=4693920
>I didn't criticize them for being open about their background, but because they lack expertise to offer expert opinions.
You didn't criticize the author's opinions or any specific signs of insufficient expertise however. Just their credentials.
Credentials != expertise (one can be an expert by self-studying, in fact in CS many are).
Insulting commenters with derogatory names is unwelcome and lowers the quality of discussion; I don't care if you have a link to it. Tangentially, it is a good example of how even the smallest behavior by a leader, such as pg, set a norm for many followers: people are digging up his posts 6 years later like they are scripture - wisdom, law and conduct to study. pg is just a person, not a prophet; YC and HN are wonderful, but his every act is not wisdom or law. (No insult to pg; the same could be said of Linus Torvalds, Steve Jobs, and everyone else who ever worked in SV or lived on Earth, including me.)
> Credentials != expertise (one can be an expert by self-studying, in fact in CS many are)
Credentials are a strong, widely used sign of expertise. The CS experts who lack educational credentials are practitioners, professionals in the field; they have other credentials. The author is neither educated in the field nor is a professional or practitioner in it. Everyone else weighs those things heavily. Do you hire or take the advice of people with neither education nor experience in the domain?
The article in particular requires expertise: It's a series of rules of thumb, things that are gained only through extensive experience doing the task described. And it requires doing it with success - not all experience is good experience - and apply good analysis to that experience. Rules of thumb for physics research from someone who washed out are not the same as those from Stephen Hawking; how about rules of thumb from someone who never has studied nor practiced physics, but read some links? Again, if they wrote 'rules of thumb for evaluating code' on the same basis, it would be ridiculous.
In general, everyone knows that the Internet is full of people spewing nonsense they know nothing about, but even smart people often seem to fail to apply that to particular instances, especially when other psychological drives apply - such as when what is said moves or inflames them. The downvotes and criticisms (this is an observation, not a complaint) seem to be because my comment disrupts that drive (in this case, the inflammatory narrative about social sciences). That's ok and what I expected. My prediction is that if I wrote the same about a different topic, then the response would be different: If what I wrote agreed with the crowd (e.g., if the author said psychological research was high quality and I questioned their expertise), then I'd expect many to agree with me, but for the same bad reason. Inflammatory stuff is, disappointingly, still hard to talk about rationally.
No, but the above is an attempt of second-guessing my psychology (and also irrelevant).
Now, you follow with a long list of opinions of the importance of credentials. What I don't see is any opinion on why the particular arguments are not enough. Can you enlighten us? You could point to someone with credentials which refutes them for example.
If you just want to raise a general principle (totally unrelated to the quality or not of the arguments in TFA) then it seems that you suggest that we should only discuss articles written by people with credentials on a topic.
If so, why should we accept this suggestion of yours then?
Where are your credentials on epistemology (seeing that this is an epistemological issue)?
Those haven't been addressed. I think calling them "opinions" is a big stretch; it's a reliable statement of fact that humanity very often relies on the credentials of the speaker, including their training and experience, for evaluating information they communicate.
> What I don't see is any opinion on why the particular arguments are not enough.
It's up to the person making the claims to establish them. We can't spend our time refuting everything anyone says. In fact, making claims in order to force others to refute them is a well-known propaganda technique for creating dysfunction, in part because refuting takes far more time than just making claims. (This article is not propaganda, in case anyone thinks I'm implying that.)
The arguments made by the author are expert judgments; there's little to address by other means.
> it seems that you suggest that we should only discuss articles written by people with credentials on a topic
Personally, I want to spend my time on the best knowledge I can find. Life is short. Identifying quality information depends frequently on credentials; I think that's uncontroversial.
You keep taking things to an extreme - "only" - and to be clear, those are your words and not mine. Somewhere else I also made that explicit. EDIT: Here it is, copied from another subthread to keep everything in one place:
Of course, let's not make this binary, one or the other [merits or credentials]. Everyone uses both to varying degrees, and other signals too (if the post used bad English, for example, many would doubt the author's credibility).
In this case, I lean more heavily on expertise because the content is a product of expertise - rules of thumb. If they wrote 'I can create cold fusion in a teacup', I'd lean heavily on the merits - can they do it or not?
Depends, since I tend to evaluate arguments on their merits, not on the author's credentials.
If one doesn't have the background to evaluate those arguments, I don't think they're supposed to be reading them anyway (whether they are from an "amateur" or a Nobel winning pro).
Of course, let's not make this binary, one or the other. Everyone uses both to varying degrees, and other signals too (if the post used bad English, for example, many would doubt the author's credibility).
In this case, I lean more heavily on expertise because the content is a product of expertise - rules of thumb. If they wrote 'I can create cold fusion in a teacup', I'd lean heavily on the merits - can they do it or not?
> If one doesn't have the background to evaluate those arguments, I don't think they're supposed to be reading them anyway
That greatly narrows everyone's reading material and awareness of the world. Somehow I have to decide what to do about many things in life, almost all of which I lack expertise in.
In the end, there is no way to avoid having to decide whether to trust the author or not - or more generally, to trust others or not, from consultants to bosses to employees to that wonderful person you might want to marry.
Probably. Did you attempt the quiz?
> If they wrote a blog post on rules of thumb for evaluating code, would you take it seriously?
I don't know if I would take it seriously before I read it. But after reading it, if it had quiz showing me their method could get over 95% correct after reading a 10 or so word summary of the end result, I'd take it very seriously. I haven't come across too many things in my professional life that have save me that much time.
It makes you wonder just how much spare time the people who peer reviewed those papers had on their hands.
Often, amateurs find seemingly obvious answers to the wrong question. I once was reading Amazon user reviews of translations of Confucius' Analects. For one translation, a few reviewers observed that the translator went bizarrely off-topic in the footnotes, giving his opinions on various things Confucian and contemporary. I ruled out that translation. Then I read a Chinese scholar's review of translations, and learned that such commentary was in fact the standard of the leading editions of the Analects going back millennia. The Amazon users - as amateurs often do - had no idea what they were seeing, and if I hadn't found the scholar's review, I would have believed them too (and missed out what that incredible work translation - a widely recognized work of art and genius - had to offer me from Confucius). That's the last time I trusted Amazon reviews. Similarly, you can find plenty of well-reasoned, persuasive, and totally wrong legal and medical advice online (including on HN).
How can one avoid falling into the same trap as I did with the Amazon reviews? Don't trust amateurs, no matter how persuasive they seem, in questions that require expertise.
Psychology, as has historically always been the case--meta-analysis itself bloomed largely from the field--is the one turning inward and looking at itself. And it's getting crap from people who love to use it as their favorite punching bag.
The irony of this article is that it's psychologists looking at other psychologists, doing the math etc.
The truth is closer to the second hypothesis by the author: bullshit is incentivized everywhere in academics. Reality is less interesting, harder, more incremental. Everyone wants the next genius savior to point to because it's a simpler story than reality. Sexy means more pubs, more grant money.
Fair enough. There’s apparently a set of benchmarks that other papers used and were reused here to compare against. Yet the student was unable to answer my question as to how bias in this dataset was treated or even what percentage of the algorithms was faster on GPU to begin with, they only showed how much better their deep learning fit the data. As in, if 99% is faster on GPU I could as well just provide that as a static answer and be better than any machine learning.
I frankly find it tragic that this kind of stuff is not ironed out in academia. This was was in a very highly ranked university.
What's the joke with invoking Nassim here? That he is some kind of anti-pretentious truth-teller whose grandma is even further on that spectrum?
If you wash 5 times a day before prayers, yeah, washing your hands is going to put you in a mindset.
Or maybe I just don't understand what priming is.
This is kind of like priming, but—if it holds true—this result would actually take away a lot of the explanatory power of most priming studies. A simpler explanation than a bunch of specific "doing X makes you think about Y which makes you do Z" priming relations, is "doing {any number of things} releases striatal dopamine, that increases executive functioning, which allows you to do {any number of things}."
> Papers that report boring or common-sense results seem to be true very often. Cognitive psychology (e.g. memory and perception) mostly held up.
> ...it turns out for most people "if the claim of the paper doesn't make intuitive sense, the paper is wrong" is a good shortcut that works about 70% of the time.
Generally, the closer to the "metal" (e.g., memory, perception, learning) and the more pedestrian the psychological finding is, the likelier it is to hold good.
I might be giving in to a bit of hindsight bias here, but I have come across this in the psychology of learning myself. It has been a bit underwhelming I must admit. "Good sleep, retrieval practice, elaboration, paying attention, meaningfulness and organization, and spaced repetition result in better learning; massed-practice is not very useful for long-term retention". Well, no kidding!