Almost no complete analysis is done perfectly on the first try. Physicists make mistakes just like anybody else, whether it’s simply a bug in our code or something more serious. The problems that we encounter are usually the most interesting part of the work, though! When I’m grappling with understanding some physics, I feel less like a code monkey.
We show each other results constantly. I probably show various people several dozen plots each week. Some of them make sense, and I’m showing the plots to share information. Others I show because I don’t understand what’s going on. Inevitably, one or two of them are because of my own mistake. Sometimes it’s something more than that.
If you’re lucky, you get to catch a problem that the entire collaboration has missed so far. That doesn’t mean people were lazy!! It means you looked in a particular spot that (perhaps) hadn’t been looked at before, or you looked more carefully, or another issue which had prevented others from looking until that point was solved. Great! You have earned the difficult task of convincing everyone that you found a real problem (it has to be confirmed by others), and once you’re done everybody will benefit from the solution.
Once a result is ready for publication, the real grilling begins. You get a lot more attention, and the smart folks in the collaboration may point out issues that you hadn’t thought of yet. It’s great as a student to find out what the more experienced members of the collaboration see, and what they consider serious. A large number of problems still get caught at this stage, when you look at things you hadn’t looked at before.
Next comes writing up the paper with your results. Often there isn’t much room in the paper for explaining problems that you might have had. I think it’s a shame, because the problems in the analysis are sometimes the most interesting part! And those are the things that people in the future can really learn from. Often, some of the problems will be explained in a talk about the analysis at a conference, or in a student’s thesis. But those are a bit harder to track down for the next person who comes along.
Once your paper is all set, it’s sent to a journal, and the journal asks a few people to review the paper. In the days when collaborations were much smaller, this was critical – it was the way to do quality control for all the physics results on earth. With 3000 member collaborations like those at the LHC, I’m not so sure that it’s as important. Still, sometimes there are real concrete questions raised during the review. Once that’s all done, there’s still a little time before the paper is really published.
Finally the thing goes out, and people start reading it. If one of the readers find a problem (which could happen), then there are a few approaches to fixing it. Most journals accept “errata.” Those are short notes saying, “this thing we wrote about wasn’t quite right.” One big problem with errata is that people search for, find, and cite the original articles – and they don’t always find the errata!! So they might be reading or citing the wrong version of your paper. Some electronic journals will accept a new revision, so that you can make the change in place (a bit safer). And for papers on the arXiv, you can upload a new revision.
These are frustrating, though! The paper’s been reviewed so many times, catching a problem after all that is really disappointing.
Then there are the hard, philosophical ones, that I will leave you to think about. Let’s say you are relying on someone else’s number. Some group somewhere has calculated something useful to your work, and you used their calculation in the paper. And let’s say they find out their calculation was wrong. Now you know your results are wrong as well, but it’s not your fault!! Is it worth publishing an erratum? Putting a new version of the paper together? Holding the publication while you go back and fix things? Even if it was in its final stages? Tough choices, particularly when you want that paper out….