Ah yes, peer review; one of the more misunderstood parts of the scientific method. Peer review is frequently treated as an incantation to separate the wheat from the chaff. What has been peered reviewed is good; what hasn’t is bad. But life is never so simple. In the late 1960s, Joseph Weber (1919 – 2000) published two Physical Review Letters were he claimed to have detected gravitational waves. Although there are a few holdouts who believed he did, the general consensus is that he did not, since his results have not been reproduced. Rather it is generally believed that his results were an experimental artifact. His results were peer reviewed and accepted at a “prestigious” journal but that does not guarantee that they are correct. Even the Nobel committee occasionally makes mistakes, most notably giving the award to the discoverer of lobotomies.
Conversely, consider the case of Alfred Wegener (1880 – 1930). In 1912 he proposed the idea of continental drift. To say the least, it was not enthusiastically received. It did not help that Wegener was meteorologist, not a geologist. This theory was largely rejected by his peers in geology. For example, the University of Chicago geologist Rollin T. Chamberlin said, If we are to believe in Wegener’s hypothesis we must forget everything which has been learned in the past 70 years and start all over again. In 1926, the American Association of Petroleum Geologists (AAPG) held a special symposium on the hypothesis of continental drift and rejected it. After that, the hypothesis was strictly on the fringe until the late 1950s and early ‘60s when it finally became mainstream.
Thus, we see that peer review cannot definitively be relied on to give the final answer. So what use is peer review? The problem is that, as pointed out in previous posts, in science there is no one person who can serve as the ultimate authority; rather, observation is. As a school student, the teacher knows more than the student and can be considered the final authority. In university, the professor plays that role, sometimes with gusto. But when it comes to research, frequently it is the researcher him/herself who is the world expert. So how can research be judged and how do we make decisions about that research? And decisions do have to be made. We cannot publish everything—the useful results would get lost in the noise. We must maintain the collective wisdom that has been laboriously developed. Similarly, decisions have to be made on who gets research grants. Do we use a random number generator? Ok, no snide remarks, I admit that it does occasionally look like we do. As there is no single human to serve as the final authority, we turn to the people who know the most about the topic, namely the peers of the person. If we want a decision related to sheep farming, we consult sheep farmers; if about nuclear physics, we consult nuclear physicists. Peer review is simply the idea that when we have to make a decision, we consult those people most likely to be able to make an informed decision. Is it perfect? No. Is there a better process? Perhaps, but no one seems to know what it is.
Peer review is also used as a bulwark against bull…, oops, material, that is of questionable validity. The expression, that has not been peer reviewed, is used as a euphemism for, that is complete and utter crap and I am not going to waste my time dealing with it. In this case it tends to come across as closed minded: Not peer reviewed? It’s nonsense! Needless to say, cranks take great exception and tend to regard peer review as a new priesthood who stifles innovation. And indeed, as noted above, sometimes peer review does get it wrong. There is always this tension between accepting nonsense and rejecting the next big thing. As the case of continental drift illustrates, it is sometimes only in retrospect, when we have more data, that we can tell what the correct answer is. However, it is better to reject or delay the acceptance of something that has a good chance of being wrong than to have the literature overrun with wrong results (think lobotomies). However, contrary to popular conception, Copernicus and Wegener are the exception, not the rule. That is why Copernicus is still used as the example of the suppression of ideas half a millennium later—there are just not that many good examples. And I might add that both Copernicus and Wegener were initially rejected for good reasons and were accepted once sufficient supporting data came to light. Most people, who the peer review process deems to be cranks, are indeed cranks. Never heard of Immanuel Velikovsky (1895 – 1979)? Well, there is a reason. The few who were right are remembered, but the multitudes that were wrong are, like Velikovsky, forgotten.
Peer review is one of the cornerstones of science and is an essential part of its error control process. At every level in science we use peers to check for errors. Within well-run collaborations, results are reviewed by the peers within the collaboration before submitting for publication. I will get my peers to read my papers before submission. Even the editing of these posts before being put on line can be considered peer review. Then there is the formal peer review a paper receives when it is submitted to a journal. In many ways this is the least important peer review because it is after a paper is published that it receives its most vigorous peer review. I can be quite sure there is no fundamental flaw in special relativity, not because Einstein was a genius, not because it was published in a prestigious journal, but because after it was published many very clever people tried very hard to find flaws in it and failed. Any widely read scientific paper will be subject to this thorough scrutiny by the author’s peers. That is the reason we can have confidence in the results of science and why secrecy is the enemy of scientific progress. Given enough eyeballs, all bugs are shallow[1].
Additional posts in this series will appear most Friday afternoons at 3:30 pm Vancouver time. To receive a reminder follow me on Twitter: @musquod.
[1] The Cathedral and the Bazaar, Eric Raymond
Tags: Peer Review, Philosophy of science
