> The paper illustrates the point by undertaking two different RCTs on cowpea seeds in Tanzania. One is a traditional study where the control group knows they are getting traditional seeds and the treatment group knows they are getting modern seeds. The second is double blind; neither group knows what seed it is getting.
> The traditional RCT shows a significant over 20% increase in yields from the modern seed. But the double blind RCT shows that virtually all of that improvement comes from changed behavior
> Specifically, the average treatment effect in the double blind RCT was zero! And when the harvests in the control groups across the two RCTs were compared, the blind control group showed a significant over 20% increase over the traditional RCT control group which knew they were getting the traditional seeds.
> In other words, the significant effect found in the traditional RCT was not due to better seeds, it was due to actions taken by the farmers who thought they were getting better seeds (they planted them in larger plots with more space between the plants on better quality land). These farmers' expectations were wrong (in post experiment surveys, over 60% of them said they were disappointed in the yields)
I don't like this reporting. The way things are phrased, it looks like Tanzanian farmers have depressed yields 17% just by having bad farming practices. But it goes on to specifically say that the farmers were disappointed in their 20% increased yields.
The most sense I can make of this is that these experiments both measured yield per seed rather than something useful like yield per acre -- despite the fact that the farmers being studied clearly considered this irrelevant. Are seeds really such a major cost in farming? Was the study just terribly designed?
"And, as a fascinating new paper points out, this knowledge can produce "pseudo-placebo effects" in RCTs. That is to say, the expectation of receiving the treatment can cause people to modify their behaviors in a way that produces a significant "average treatment effect" even if the actual intervention in not particularly effective."
It is barely even "pseudo". This falls into my area of specialty, where suggestions and expectations are 'how you actually get anything done'. The more implicit and indirect, nearly, the more likely the suggestion is to take effect. If one discounts the all important 'confidence' and 'expectation' angle.
In a clinical trial there is going to be very heavy implications, however towards confidence. They invested these resources to do the testing, so that implies confidence and expectations.
3 comments
[ 167 ms ] story [ 495 ms ] thread> The traditional RCT shows a significant over 20% increase in yields from the modern seed. But the double blind RCT shows that virtually all of that improvement comes from changed behavior
> Specifically, the average treatment effect in the double blind RCT was zero! And when the harvests in the control groups across the two RCTs were compared, the blind control group showed a significant over 20% increase over the traditional RCT control group which knew they were getting the traditional seeds.
> In other words, the significant effect found in the traditional RCT was not due to better seeds, it was due to actions taken by the farmers who thought they were getting better seeds (they planted them in larger plots with more space between the plants on better quality land). These farmers' expectations were wrong (in post experiment surveys, over 60% of them said they were disappointed in the yields)
I don't like this reporting. The way things are phrased, it looks like Tanzanian farmers have depressed yields 17% just by having bad farming practices. But it goes on to specifically say that the farmers were disappointed in their 20% increased yields.
The most sense I can make of this is that these experiments both measured yield per seed rather than something useful like yield per acre -- despite the fact that the farmers being studied clearly considered this irrelevant. Are seeds really such a major cost in farming? Was the study just terribly designed?
"And, as a fascinating new paper points out, this knowledge can produce "pseudo-placebo effects" in RCTs. That is to say, the expectation of receiving the treatment can cause people to modify their behaviors in a way that produces a significant "average treatment effect" even if the actual intervention in not particularly effective."
It is barely even "pseudo". This falls into my area of specialty, where suggestions and expectations are 'how you actually get anything done'. The more implicit and indirect, nearly, the more likely the suggestion is to take effect. If one discounts the all important 'confidence' and 'expectation' angle.
In a clinical trial there is going to be very heavy implications, however towards confidence. They invested these resources to do the testing, so that implies confidence and expectations.