Excellent description by Jonah Lehrer in “The Truth Wears Off” (The New Yorker) on the importance of confirming experimental findings in different laboratories:
Before the effectiveness of a drug can be confirmed, it must be tested and tested again. Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.
Mr. Lehrer’s article is about drug effectiveness, but his words apply equally to other types of science, including virology.
Utterly fascinating article, but where exactly does it leave us? It actually poses more questions than it answers. The last graph captures the essence of the problem because these arguments can be used by various points of view to discredit other points of view. But how do you know for sure?
Most interesting was the case cited where the scientist controlled for every possible variable (something that many scientists do not do either because of conformation bias orbecause theyare unaware of variables for which they are not controlling). But in the end it was random – or was it?
The deceptive assumption is that at any given point in time we “know” all the basics there are to know and have reliable methods that “prove” what we know. Right up until we are proven wrong. Or are we?
Excellent post. What are your thoughts?
I find it hard t accept that all scientists are sloppy, that they bias results to obtain the results they want, that ‘a lot of extraordinary data are nothing but noise’. Is it really true that we have to choose what to believe? The article would lead us to believe that nothing that science produces is incorrect. But that’s clearly not the case. I believe that in the majority of cases, over time, science finds what is correct. Perhaps in the case of therapeutics for which the end point is hard to define (e.g. schizophrenia as in the article) and subjective, drugs that aren’t effective are used. But there are many other examples where therapeutics have been tested, shown to be effective, and then used for years in people with excellent benefits. The poliovaccine comes to mind first – subjected to large clinical trials, shown to be effective, then used in populations to the point where we have nearly eliminated the disease. In basic science, discoveries build upon one another; if one is wrong, the field does not progress until it is corrected. For example: for years it was thought that protein was the genetic material of cells, but subsequent research could not prove it. Then it was shown that DNA is the genetic material; and study after study validated this, and the conclusion is clearly correct. There are so many other examples. So while there is certainly bad science (just as in other fields), there is still a great deal of well done science that has influenced our view of nature. As my colleague Dickson Despommier has said, don’t trust scientists – trust science.
Comment more on the New Yorker article than the above posts…The issue here is that the scientific method only truly gives replicable results when the conducted research falls within the realm of hard/physical science where there are “laws” governing the processes being studied. In the fields of medicine, psychology, and especially behavior there are no hard and fast laws…just generalities. This is due to the infinitessimally large number of factors that contribute to behavior. The larger the set of variables, the greater the possibiilty that one outlier will skew the results. Researchers try to create artificial, closed systems in which to conduct the experiments that in reality are far from closed.
The tests Crabbe conducted may have appeared to be identical (with all variables set equal), but there were many other factors not considered/not controlled including: the lab personnel handling the mice were different in each lab (mentioned in the Crabbe article), the gender of the lab personnel, the “smell” of the lab personnel (whether perfumes/fragrances, naturally occuring pheremones, or residual smells from lab personnel behavior (what they had to eat, did they smoke, etc.))
As to the question above of what do we really know…with any certainty, nothing.