I happened to be on-call for the CMS High Level Trigger (HLT) system during the week all LHC experiments saw their first collisions, so here I describe (after having some time to breath) my experience.
All the hardware subsystems in the CMS experiment have two kind of people taking care of operations. The ones in the front-line are the so-called “shifters”, operators who sit in front of several computer screens in the control room and whose job is to monitor closely the performance of each component, and take rapid action in case something goes wrong. Each shift is usually of 8 hours and there is always someone doing this; the operations are 24/7. The other kind are the “experts”, who are on-call 24/7 in case there is a major problem or a more involved task that needs to be done. For this first week, however, shifters and experts were intensively working together in the control room making sure everything works as planned.
For software subsystems, like the HLT, there are also shifters, but who usually sit somewhere else (like in the remote control room across the Atlantic, at the LPC at Fermilab) and who take the usual 8 hours shifts. The CMS control room at P5 is always connected via video with the other remote stations, including Fermilab, Desy, CMS Meyrin centre, etc.
The experts are of two kinds, the primary and the secondary. The team of people in charge of expert support rotate between these two states. The primary is usually the main expert who carries a cell phone all the time in case there is an “emergency” call from the control room. The secondary is there for backup, in case the primary needs support or if the primary is unreachable for any circumstances. The week before the collisions week I was secondary, and the primary responsibility was transferred to me the day of first collisions, so it was a very exciting (also quite stressful) moment.
The HLT system is a crucial part of the system. After the first level of triggering (called L1), the HLT is responsible for deciding what goes into tape and what not. For the expected first collisions, of course, there was no room for mistake. We had to be able to record these events and make sure we don’t miss them for circumstances like timing synch of the beam with our trigger (L1), timing of the subdetectors, or any other eventuality. The beam conditions for these first pilot runs are not as stable (and the detectors are not fully calibrated, we need collision events for that!), so we needed to make sure we considered all scenarios. On Saturday and Sunday, before Monday 23 of November (the day of first collisions), everyone was working very enthusiastically to prepare for this. I remember sitting down with the Run Coordinator (the person in charge of all operations), together with expert people related to the data acquisition, in order to define a strategy and adapt quickly to the expected (and not so expected) beam conditions. We worked intensively to make sure the small modifications that needed to be done were carefully executed.
By Monday morning we were ready and very confident that if the delivered beams were to collide at the CMS detector, we were going to be able to see them and record them. Unfortunately, on Monday afternoon (when most experiments saw their first collisions), CMS did not see any collision candidate; everything seemed to be consistent with beam gas, or at most something colliding outside the detector. Worrisomeness and stress could be briefly noticed in the faces at the control room. But there was no time for that, for many it was the culmination of years of work, and for all of us the beginning of and exciting program, so we went back to work to confirm our explanations of what happened. I could feel the adrenaline flowing in small but appreciable quantities; I imagine this chemical flooded many physicists’ bodies that day.
Soon, however, we (CMS+LHC) found out that the beams were not optimized for collisions at P5 during the afternoon, so we tried again in the evening: the LHC circulated two beams again, now optimized for CMS, and it was marvelous. The displays showed beautiful events. There were applauses and champagne!! The machine works !!!!
Edgar Carrera (Boston University)
