Scientists are subject to two contradictory criticisms. The first is that they are too confident of their results, to the point of arrogance. The second is that they are too obsessed about error control–all this nonsense about double blind trials, sterilized test tubes, lab coats and the like. It evidently has not occurred to the critics that the reason scientists are confident of their results is that they have obsessed over error control. Or conversely they obsess over error control so they can be confident of their results.
Now, most people outside science, do not realize a scientist’s day job is error control. There is this conception of scientists having brilliant ideas, going into immaculate labs where they effortlessly confirm their results to the chagrin of their competitors. That is, of course, when they are not plotting world domination like Pinky and the Brain. But scientists neither spend their time plotting world domination (be with you in a minute Brain) nor doing effortless experiments. But rather they are thinking about what might be wrong; how do I control that error? As for theorists, they must be a part of a wicked and adulterous generation[1] because they are always seeking after a sign–a minus sign that is.
So what do scientists do to control errors? There are very few arrows in their quiver. Really only three: care in doing the experiment or calculation, care in doing the experiment or calculation, and care in doing the experiment or calculation. Well actually there are two others: peer review and independent repetition. Let’s take the first three first: care, care, and care. As previously noted, scientists are frequently criticized here. Why do double blind studies when we have Aunt Martha’s word for it that Fyffe’s Patented Moose Juice cured her lumbago? Well actually, testimonials are notoriously unreliable. A book[2] I have, had an example of from the early 1900’s of testimonials for cures for consumption and then had the dates the person died of consumption. The death was frequently quite close to the date of the testimonial. So no, I will not trust Aunt Martha’s testimonial[3]. To quote Robert L. Park: The most important discovery of modern medicine is not vaccines or antibiotics, it is the randomized double-blind test, by means of which we know what works and what doesn’t. This has now carried over into subatomic physics where blind analyses are common. By blind, I mean that the people doing the analysis cannot tell how close they are to the expected answer (the theoretically predicted answer or the results of a previous experiment) until most of the analysis has been completed. Otherwise, as one of my experimental colleagues said: data points are like sheep, they travel in flocks. Even small biases can influence the results. Blind analysis is just one example of the extremes scientists go to, to ensure that their results are reliable. All this rigmarole that scientists go through is one of the reasons life expectancy increased by about 30 years between 1900 and 2000, perhaps the major reason. The lack of this care is the reason I distrust alternative medicine.
We now move on to the other two aspects of error control: peer review and independent replication of results. Both of these depend on the results being made public. Since these are crucial to error control, results that have not been made available for scrutiny should be treated with suspicion. Peer review has been discussed in the previous post and is just the idea that new results should be run past the people who are most knowledgeable so they can check for errors.
Replication is, in the end, the most important part of error control. Scientists are human, they make mistakes, they are deluded, and they cheat. It is only through attempted replication that errors, delusions, and outright fraud can be caught. And it is very good at catching them. In the next post, I will go into the examples but it is a good practice not to trust any exciting new result until it has been independently confirmed. However replication and reproducibility are not simple concepts. I go out doors and it is nice and sunny, I go out twelve hours later and it is dark and cold. The initial observation is not reproduced. I look up, I see stars. An hour later I go out and the stars are in different places. And the planets, over time, they wander hither, thither and yon. In a very real sense the observations are not reproduced. It is only within the context of model or paradigm that we can understand what reproducible means. The models, either Ptolemaic or Newtonian, tell us where to look for the planets and we can reproducibly check they are where the models say they should be at any given time. Reproducibility is always checking against a model prediction.
Replication is also not just doing the same things over and over again. Then you would make the same mistakes and get the same results over and over again. You do things differently, guided by the model being tested, to see if the effect observed is an artifact of the experimental procedures or real. Is there really a net energy gain or have you just measured a hot spot in your flask. The presence of the hot spot can be reproduced, but put in a stirrer to test the idea of energy gain and, damn, the effect went away. Another beautiful model was slain by an ugly observation. Oh, well, happens all the time.
So science advances, we keep testing our previous results in new and inventive ways. The wrong results fall by the wayside and are forgotten. The correct ones pile up and we progress. To err is human, to control errors–science.
Additional posts in this series will appear most Friday afternoons at 3:30 pm Vancouver time. To receive a reminder follow me on Twitter: @musquod.
[1] Matthew 12:39,16:4
[2] Pseudoscience and the Paranormal, Terence Hines, Prometheus Books, Buffalo (1988).
[3] My grandfather died of consumption.
