Skip to content
June 26, 2011 / Mark Largent

What if Science Does Not Self-Correct?

Why Science Struggles to Correct Its Mistakes – NYTimes.com.

This is a nicely written piece that quickly demonstrates how and why the claim that science is “self-correcting” is not quite true.  Ideally, the process of science tests and retests, confirms or rejects, claims made by scientists.  As the piece shows, in reality, it is relatively easy to publish a scientific claim, but very hard to publish evidence that a published claim is incorrect.  The fact of the matter is that journals are just not interested in publishing refutations.

The result is that incorrect claims are published and they are never retracted or corrected.  The self-correcting process of science, therefore, simply is not working.

In a lot of ways this piece reminds me of one I read last year in the Atlantic (http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/) about John Ioannidis, a Greek researcher who has demonstrated that “much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong.”

It is interesting to contemplate the impact of how science is published and scientific findings are publicized.  If Ioanndis and Zimmer are correct, a well-read science policy professional will be mislead by most (Ioanndis says 85%) of what he or she reads.  How can we possibly make good science policy when the information available to us is almost all incorrect?

Advertisements

5 Comments

Leave a Comment
  1. Tobin Craig / Jun 28 2011 1:19 pm

    Related to this is the non-publishing of negative findings. In the original understanding of modern science, negatives were as valuable as positives. Who knows how much work is being wasted working on getting positives where negatives have already been established? There is much to be said for our ‘laissez-faire’ approach to institutionalized science, but this strikes me as a huge inefficiency.

    But just to underscore the big question with which you conclude, aren’t we led to discount and doubt the big claims, especially the really exciting ones? But then what if they are serious and have not been oversold? We are left simply hoping that well-founded science with urgent practical implications will get a hearing.

    Lastly, publishing findings is a political as well as a scientific act. Of course scientists know this, which is why they are moved to ‘oversell’. But there are no external pressures checking this — we are reliant entirely on the individual scientist’s own integrity.

  2. Daniel Kramer / Jul 22 2011 1:03 pm

    It is an exaggeration to say that a “well-read policy professional will be misled by most of what he or she reads.” This assumes that the well-read policy professional is a dolt and is happy to swallow whole anything they read in a scientific journal. I hope that is not the case. Rather, I think well-read, well-informed people can assess the shortcomings in research design, the strength of causal inference, and thus the usefulness of scientific claims. Science proceeds, not in the dramatic fits and starts that is implied here (i.e. reject or accept conclusions/claims), but by minor tweaks and tinkering. Some claims are strong and some are weak. Most lie somewhere between. Usually the authors themselves offer a long list of shortcoming to their research and invite the scientific community to improve upon their results. Generally, I think there is great humility in the scientific community in this regards.

  3. Mark Largent / Jul 23 2011 8:24 pm

    Regardless of any humility that might exist in the scientific community (and even granting that everyone who reads every scientific journal is able to access their claims), we’ve got a problem here. Science journals are increasingly unwilling to publish refutations of claims published in previous issues of the journal … in fact many of the most prominent journals make it an editorial policy to only publish original findings. Journals like Science or Nature provide a tremendous amount of information that is then republished by major media outlets, but they refuse to publish refutations. Surely you’ll admit this there’s a problem here.

  4. Daniel Kramer / Jul 23 2011 9:54 pm

    Sure, that would be a problem, but I think what you are referring to is not refutation but replication. Replication would be that an experiment is rerun under the exact conditions of the original research. Obviously, in order to rerun an experiment under the same conditions we are talking about a very controlled environment… a laboratory for example. An example of replication is here (http://news.sciencemag.org/scienceinsider/2011/04/science-publishes-replication-of.html). Or, I suppose, replication could also mean a statistical analysis using the same data set of the original research. If it is replication that you are referring to then yes, it likely doesn’t happen enough, although I think it is an interesting question as to whether this stems from the unwillingness of journals to publish or the unwillingness of scientists to replicate. A scientist in your linked article says he can’t bother with replicating others’ results. I would expect that journal space has also been a problem but perhaps with more online content, the publication of replications will become more common.

    Refutation of claims happens all the time, at least in my world although I am not a bench scientist. For example, someone claims that there is a causal relationship between rising GDP and environmental quality (i.e. Environmental Kuznet’s Curve). Another, using a different data set, perhaps a different indicator of environmental quality, finds there is no relationship. The second study is surely original research, and it weakens the claim of the first. Does it refute the claims of the first? I don’t think I’d use that language. Look up “Environmental Kuznets Curve” in Google Scholar. It returns 8,240 publications. I think that demonstrates a pretty healthy conversation.

  5. Daniel Kramer / Jul 23 2011 10:11 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: