What is like to be on shift in the ATLAS Control Room? It’s my second night on shift and we’re already off to an interesting start. As I walked through the door I saw that the LHC was ramping and my excitement grew. Soon we would have beams and data! But before I even sat down the beams got dumped, so the LHC had to try again. Since I don’t really have much to do this would be an excellent time to describe what I do on shift and why I don’t really have much to do right now.
Some terminology
There are a few words you need to know to understand what we do here.
Someone who is in the ATLAS Control Room and performing a routine task on shift is known as a Shifter. (There are often people there performing non-routine tasks who are not Shifters. They work with the Shifters to accomplish their tasks.)
Shifters work at a designated Desk. Each Desk has a label on it telling the other Shifters who works there, as well as its own computer system and telephone. There are 16 Desks in total, some which manage part of the detector, and some which manage different kinds of tasks that involve the whole detector.
The LHC provides beams of protons which can be in one of several states. Stable beams means that the beams parameters are very precisely tuned and we can use them to record data. Injection is when the LHC is preparing the beams to become stable. When the beams become unstable they get dumped (time for a toilet flush sound!) and the LHC also has periods of Ramp Up and Ramp Down before and after stable beams.
Data taking is split up into long periods of stable beams called a Run. Each run is split up into short time intervals of about 1 minute called Lumiblocks. A lumiblock is a small period of time with roughly the same kinds of conditions.
The desk
I’m sitting at the Trigger Desk and that means that I’m responsible for making sure the trigger is behaving well. Here is a picture of my desk:
My Trigger Desk
There’s a lot of stuff there! Going from left to right we have the Red Folder Of Answers (RFOA? That would be a difficult acronym!) This holds a copy of nearly all the training material we need to be familiar with, and also some simple notes about how to handle a few common situations. Every desk should have one of these. I’m not just talking about in the ATLAS Control Room, I mean every desk in the world.
Next we have the Ominous Telephone Of Panic. Well, not really. Most of the time this is just used to exchange simple information. But there’s always the possibility that it will ring and there will be an expert on the other end of the line who has some difficult questions. If there’s one thing we don’t like on shift it’s surprises.
Then my laptop. I’d be lost without this. On here I have all the most useful websites and talks I could find that relate to the trigger. It’s quicker and easier to look up information on here, I can’t break anything by accidentally clicking the wrong button, and it’s shinier than the computers that comes with the desk. And of course, this is where I write my blog posts! And next to that is my ID. I need that to get in and out.
Towards the back of the desk we have the monitors that give me all the information I need to be able to be a good Shifter. Each monitor has about four applications open, and each one of those has about 20 tabs. There are literally hundreds of plots and tables that are available, and I can’t look at them all for all of the time! That is what makes this role difficult. To help things, I reserve the left most monitor for the most important plots (how quickly we are recording data) and the right most monitor for essential reference material (the plan for the day, the normal instructions and the instructions for special tasks.) The other two monitors are where most of the work takes place. Looking at plots, reading error messages and “spying” on the other Shifters.
Then we have the two pads of paper. The big one is mine and that’s for notes, doodles and anything else that comes to mind. (I think I have some cocktail recipes in there somewhere!) The smaller pad is for sending short messages to other Shifters in the room. For each set of “prescale keys” I have a different sheet of paper. It’s just a matter of picking the right one.
Then there’s my food. Since we’re not allowed to leave the Control Room for very long we need to bring all our food with us for the next 8 hours. I’ve got some bread and cheese, some fruit, and some yoghurt. Very healthy! Unfortunately I’ve also got a can of coke and some chocolate. Oops. Also note the coffee cup. This is the night shift, after all.
In the background we see the projectors. These are one of the ways the Shifters can share information between each other and see what is happening. The projector immediately in front of me shows the data as they flow through the system (from left to right.) If there’s a problem, those boxes turn red and it’s my job to work out what is happening. Usually when that happens it sorts itself out after about a minute. The other projectors show the trigger rates, the state of the various parts of the machine, the Daily Run Plan, the status of the LHC and then some event displays.
Although you can’t see it, there’s also a monitor to my left that shows the LHC Operation Page and the LHC Page 1. This is essential to the Trigger Shifter as we need to know the state of the LHC for our job.
(And for those who are observant, yes I do have a Higgs boson with me! Who knew it would be so easy to find? And in ATLAS Control Room of all places.)
We can get bored easily on shift...
The role of Trigger Shifter
So what do I actually do? In principle it’s actually quite straightforward. When we are taking data I look through the plots to make sure that they look okay. This is not a particularly well defined task though, since we don’t know exactly what the plots will look like for a given run (if we did then we wouldn’t need a human to compare them.) The advice we get about the plots is usually very good, so most of the time we can be confident that the plots look okay. If they look a little odd and it’s a Day Shift we usually phone the expert on call, whereas if it’s a night shift it’s usually good to get a second opinion before waking up a poor sleepy civilian. If the plot is obviously wrong, then a call is made pretty much immediately (with a few seconds spent getting some vital information about what happened immediately before the problem.) Once I’ve looked at all the plots I take a break and come back to them again a little while later.
The efficiency of the trigger needs to be kept high (we are competing with CMS, after all) so we keep adjusting it. This is done by changing the “Trigger Menu”, which is a list of different kinds of triggers and how often we use them to record data. The different trigger menus are specified using prescale keys, so when the conditions change enough to justify changing the menu, I get a new set of keys (which is just a fancy word for “four digit number”.) I write these down on my little pad and pass them to the Run Control Shifter whose job it is to update the menu. Why is that not the job of the Trigger Shifter? I’m not really sure. I suppose two heads are better than one.
Hurrah, data!
When we have no beams the Trigger Shifter usually has nothing useful to do. I can prepare the prescale keys for the next run, but otherwise I can’t really do much until the next time the LHC ramps up. As it ramps up I need to watch the Bunch Groups (this tells ATLAS how many bunches of protons are moving through the LHC.) If they change, I create new bunch group information. In fact, that’s the most useful part of the Red Folder Of Answers. It’s something I will have to later today for the first time!
From time to time we have special running conditions, and we have special prescale keys for that. Right now I have to keep track of seven different sets of prescale keys (for normal running, for when the machine is in Standby mode, for when we take calibrations and for different special tasks, such as when the LHC tries a new configuration.) Each one is written on its own tiny sheet of paper!
And the other stuff
Apart from the tasks listed above, I have be able to answer general questions about the trigger, such as making comments about its performance, finding out what a specific part of the trigger is doing and working out problems. I also need to be in constant communication with the other Shifters, usually the Shift Leader (the boss!) and the Run Control Shifter who needs to know what changes to make to the trigger menu. My Desk allows me to “spy” on the Run Control Shifter to make sure the trigger settings are correct there. Right now the prescale keys are exactly what we expect. Unless there is a serious breakdown of communication, they should always match. From time to time the Run Control Shifter gets a message popping up on their screen about background events, and then ask the Trigger Shifter to make a decision. The answer is usually “No”.
Right now…
At the moment the LHC is injecting beams. That means that I can’t make any changes. The prescale keys are ready for the next run. I’ve updated myself with all the information for the day. I’ve even written up a summary for the Shift Leader and the next Trigger Shifter about the day’s plans. There are currently some plots flashing on the monitor before me. When the LHC starts to ramp up I need to watch these plots carefully. If they change even by a little bit I’ll need to update the bunch group information. If not, then I can just sit back until we start recording data again.
It’s quite an odd job when you think about it.
It is a pity to me you but you shall not find any higgs and any new physics, because all physical events are interpreted well-known particles (leptons, quarks, photons, gluons, W-bosons, Z-boson) and forces which have long known (electroweak, gravity, strong interactions):
http://arxiv.org/abs/0708.2322
Hi Gunn. If the Higgs boson exist we should see unambiguous evidence for it. If it doesn’t exist then we’ll exclude it in the remaining regions. Until then it seems a bit premature to say one way or the other if the Higgs boson exists. Either there is some new physics to be discovered, or our understanding of the universe is fundamentally flawed. I hope it’s a little of both! That’s the most exciting scenario.
But the classical logic analysis shows that there anything isn’t present besides that we already have. I believe that it is impossible to object to the classical logic – differently all will lose meaning.
Wow! this seems complicated! But interesting. Lot of things to do, lot of screens to watch. Do you have breaks, sometimes to take a coffee, go to the toilet, talk with you co-workers, look at the window to see if it’s raining? (mmm… perhaps not if you are under the ground). If the Higgs appears, do you see something? or is it hidden somewhere in the data and discovered only later, when other people look at them using miscellaneous softwares? Is it a stressing job?
Hi Daniel, those are some good questions! We don’t really have official breaks, but we do take some time to go to the coffee machine or the toilet. We usually stay in the Control Room as much as possible (if it takes me more than 5 minutes to get a coffee, then I’m doing it wrong!)
We spend quite a lot of time doing things that are not related to the shift. Tonight, I’ve written this blog post, tried to compile some analysis software and watched two episodes of House MD. This is because we have lost the beams so much tonight! When we have no beams we still need to be here “just in case”. We keep detailed logs of what happens on shift, and from my log I can see that we have only had 27 minutes of stable beams in 6 hours!
We’re not actually underground, and we have some small windows, so we can see if it’s still raining outside. (It’s not.)
If we produce a Higgs we won’t see it immediately. The data must be taken away and processed, then reprocessed, cross checked with the detector conditions, skimmed, refined and analyzed. Getting from raw data to a result is a very long and complicated process! The software we used is developed in-house, using tools that were specially written for the collaboration.
This part of the job is quite stressing! I don’t know how expensive the LHC is in terms of running, but I wouldn’t want to think about how much money gets wasted for every minute the detector sits idle, and that’s not even considering the events we would lose!
Well, stable beams are going to arrive any minute now, so I’d better get back to monitoring the trigger…
Well, you, physicists, need a lot of money with your devices to find things, anyway. More or less waste of money is not an issue, is it?
BTW, is it an interesting job, for a post-doc? Does it have some sort of continuation of your PhD? Do you participate, somewhere, in the fundamental researches in your domain? Does this job give you new ideas, and do you write papers for journals or/and conferences? What is the denomination of this domain of physics? Well, you work of particle physics, but I suppose that it is different if you are a theorist or if you make machines work better. Sorry for my perhaps stupid questions; I am interested in particle physics but I am rather a mathematician (in computer science).
Hi Gunn. There’s plenty of precedent where we have demonstrated the existence of an object beyond what we already had. For example, the Z boson was discovered in 1983. It was like nothing else we had previously discovered. There is no reason this cannot happen again. (Either we created the first Z boson in the universe, or Z bosons had existed before and we just hadn’t discovered them yet.) What is this “classical logic” you refer to, and how does it prevent us from making discoveries?
Hi Daniel! Yes, I suppose we have beams whether we use them or not. One time I was on shift on a different experiment and we had to abort the beams for some reason. They came back, we put the detector online and I sat back and waited for a bit for us to start taking data. Only we didn’t. I had forgotten one minor but crucial step (I forget what it was) and then I realized that that one mistake cost a few thousand dollars of running time. It’s a bit of a shock the first time it happens!
The job of a Postdoc is very much like that of a Grad student, except with a few more perks and no thesis to write. We’re generally a bit young, quicker to learn new ideas (which is very important when the computing systems change and get ever more complicated) and still gaining a lot of experience. Being a Postdoc means a higher salary, fewer late nights (although I still tend to work quite late) and no thesis deadline to worry about. From my own experience I feel a little more confident now that I know I have a PhD behind me, and that helps me take on bigger tasks. There are quite a few strips from PHD Comics that show one students progression into a Postdoc which I found very funny! You can read them here.
The work I do is half research and half service work. The research I performed on ATLAS has already been published (a search for a charged Higgs boson) and now I’m working on a search for the Standard Model Higgs boson. Seeing my work appear in a journal gives an amazing feeling! The work we do is a team effort, so I was one part of a very large project which is well coordinated. My service work involves helping with the Trigger system for ATLAS, and keeping track of changes in the triggers as the average number of collisions per beam crossing increases. On a day to day basis both require small steps, working with the existing software, or developing new software. Slowly, over time, this builds up to a rather impressive body of work. Once the work is ready to be reviewed it makes it way through several layers of peer review, first within the collaboration and then in the journal. It’s a slow process, but one that guarantees a solid and credible analysis!
There’s no need to apologize for questions- they’re not stupid and it’s great that you’re interested! In fact, a major part of the job is realizing when we don’t know something and asking for help. If I ask one of my colleagues what they did this afternoon and they reply “I didn’t understand this software so I met with someone and asked questions.” I would consider that a productive use of their time and a good afternoon!
Interesting. Thanks for all your explanations, Aidan.
Now, that’s an interesting job… but you eat to little.
[...] taking data and I’m sitting in the ATLAS Control Room again. Fans of my blog will remember my previous on-shift posts and, yes, today I had an awesome breakfast of roasted duck (a special treat from a [...]