• John
  • Felde
  • University of Maryland
  • USA

Latest Posts

  • USA

  • James
  • Doherty
  • Open University
  • United Kingdom

Latest Posts

  • Andrea
  • Signori
  • Nikhef
  • Netherlands

Latest Posts

  • CERN
  • Geneva
  • Switzerland

Latest Posts

  • Aidan
  • Randle-Conde
  • Université Libre de Bruxelles
  • Belgium

Latest Posts

  • Vancouver, BC
  • Canada

Latest Posts

  • Laura
  • Gladstone
  • MIT
  • USA

Latest Posts

  • Steven
  • Goldfarb
  • University of Michigan

Latest Posts

  • Fermilab
  • Batavia, IL
  • USA

Latest Posts

  • Seth
  • Zenz
  • Imperial College London
  • UK

Latest Posts

  • Nhan
  • Tran
  • Fermilab
  • USA

Latest Posts

  • Alex
  • Millar
  • University of Melbourne
  • Australia

Latest Posts

  • Ken
  • Bloom
  • USA

Latest Posts

Posts Tagged ‘commissioning’

Magnet Tests

Tuesday, June 17th, 2008

Who doesn’t love magnets. I remember as a kid having these two bar magnets and spending hours trying to move them around the kitchen table without actually making them touch (yes, this classifies as fun when you grew up in a small farming town that was 30 miles from the closest movie theater). The ATLAS magnets inspire the same feeling except now these magnets are superconducting and the size of a building (and we generally prefer it when they stay in one place).

ATLAS has two separate magnet systems used for bending the tracks of charge particles in the detector. The magnet surrounding the inner detector is a solenoid (the magnetic field points along the beam pipe). And the magnet for the muon system is a toroid (the magnetic field is circular around the beam pipe). The toroid magnet itself is in three sections: the barrel and two end-caps. And for a sense of size, here is one of the ‘little’ end-cap toroids being transported to the pit many months ago.

End cap toroid

Now that we are in the last steps of closing the detector, the final commissioning of the magnets has begun. The plan of the magnet commissioning is to test each of the four magnets separately (the two endcap toroids, the barrel toroid and the solenoid in that order) and then do the full combined test. And since you can’t have people working on other parts of ATLAS when there are large magnetic fields, all the testing is done at night. Over the past few weeks, the tests on one of the end-cap toroids have concluded (successfully!). Unfortunately in the second end-cap toroid, a helium leak was discovered (helium being used to cool the magnets). As a result, tests with this magnet had to be stopped in order to repair the leak. This does not delay us any but it does involve some reshuffling of the magnet commissioning schedule. Tonight will be the first test of the barrel toroid. Fingers crossed that it goes well!

Oh and Spain in an absolutely, spectacular goal in the last minute of injury time beat Sweden to secure their position in the quarter finals!! One step closer to Euro Cup glory!


The nomadic summer

Monday, May 12th, 2008

Ah, just spent some time getting the summer organized. The plan of attack is to abandon wife and family for June at CERN, then they come over to visit (feel a little like I’m in prison! They get visitation rights!) for a week in July, plus see some old friends, I head back in August and visit them, and maybe come back to CERN for a bit before returning to MIT for fall classes (which will be interesting, since I am also “course administrator” which is a code word for lots of chores with little fun- it’s one of those things you can really only mess up…).

Plus, there’s a trip to Cyprus for our collaboration meeting. It’s the first CMS meeting away, and so will be pretty interesting. When these meetings happen at the lab, most people tend to ignore the meeting, and focus on keeping up on their regular day to day activities instead. When they put the meeting elsewhere, there’s not much you can do about the day to day, so it tends to be more focused, and therefore more productive. In addition, people tend to spend more time with their collaborators eating and finding entertainment outside of meeting hours, which is better for collaboration building. However, I noticed a very strange thing about Cyprus, which I will share with you.

Cyprus flights

For reasons which are yet a mystery to me, flying to Cyprus for a reasonable amount of money means arriving in the middle of the night. The return is the same way – the reasonable flights all leave between 2 AM and 4 AM. The only flight which doesn’t arrive way early involves a 17 hr with 11 hours at Heathrow (ick). I’m a little worried about why this is, if anyone cares to explain it to me, I guess I want to hear it.

Anyway, it will be very exciting at CERN this summer – and a bit frustrating. Think about finally getting to do what you have planning to do for twenty years or so and then having all these annoying little issues that keep putting things off. This is what it is like, when you have a small chance of something breaking in any of many many components, there is a fair amount of unforseen issues that have to be dealt with. They’re not showstoppers, but with everyone holding their breath for the first interaction, you can get a lot of oxygen deprivation, let’s say. The Tracker cooling is experiencing something like that right now – first time working with the full system, some new issues being discovered or failures of supposedly reliable components, most of which we of the Tracker group don’t even have control over (they fall into someone else’s scope) but we are still waiting to get all the services lined up to be able to turn the detector on fully, and everyone is getting a little antsy. I keep telling my students to get used to it, every detector system goes through these growing pains, and the LHC detectors will be no different…so dear readers I guess you should know too that while I fully believe we’ll get the machine and detectors running, it may take a while, so only hold your breath if you look good in blue.


A new record, alas

Friday, May 9th, 2008

I typically start my morning at home by scanning my overnight email while I eat breakfast. (This sets a horrible example for my daughter; I will have to stop when she becomes more cognizant of what I’m doing.) When you have 2000 collaborators, and most of them are seven time zones ahead of you, there is usually some amount of mail to get through, so I like to get a jump on it before heading to the office.

On Tuesday of this week, I believe I set a new record — there were 82 new mail messages waiting for me and my Cheerios. (They’re actually the generic store-brand cereal, not the name-brand.) Now, admittedly many of these were the skim-and-delete types. (All of you people out there who are having semi-private conversations and cc’ing everything to some mailing list — please stop. It just makes me cranky, and the O’s get increasingly soggy as I hit the delete key over and over.) But some of them needed further contemplation, which stretched well into the work day.

Why now? Part of it was the recovery from the May Day holiday weekend; as people came back to work in Europe, they had a lot to catch up on themselves. But a lot of it was the computing challenges that are now underway. These startup phases are always challenging; all sorts of technical things haven’t been tested at scale, and not all sites have completely gotten the message on what they are supposed to be doing, and sometimes there are policy issues that haven’t been thought out yet either. The good news, however, is that there has been a lot of good performance out there. We have about 30 Tier-2 computing sites (I know, I haven’t explained the tier system yet) participating — about as many as I could imagine — and by and large things are working. There are a number of sites that have definitely exceeded my expectation for how many jobs they could handle and how many of them would finish successfully. (I’m not going to name names, because I don’t want to embarrass sites that I had low expectations of!) The unfortunate exception has been my own cluster at Nebraska. It’s been a tough week for us, as we’ve been fighting multiple problems and arbitrating among various demands on the system. It took a while for the challenge jobs to start running, and when the did, 98% of them promptly failed. The important thing is that we understand why, so that we can be more successful the next time around, and it sounds like our admins are gaining on that. But at the same time, I feel like we just got caught with our pants down.

Through the wonders of the Internet, I am able to follow this (and annoy our admins with questions) while far from home. This weekend I find myself in Kalamazoo, MI, where my wife is attending the rather huge annual congress on medieval studies that Western Michigan University hosts. There are at least 3000 medievalists here, a bigger turnout than we typically get for the biggest particle physics conferences of the year. My job for the weekend is looking after my daughter, who is attending her first Kalamazoo meeting. Our hotel is one of the kinds with breakfast included. I’ve been leaving the computer in the room, to be polite. No O’s among the breakfast selections.


1143536 events at the second I started typing this, to be exact. We are currently operating a portion of the detector again for a week, with many goals, but the biggest is to run as many parts of the detector together as possible. The RCT is in this one, working well, helping collect data. At about this point you start counting the hours until you can go home and crawl into your bed. Shift ends at 7am. Its a bit like flying overseas, and when you only do one day, it is tough. I have had coffee, diet coke and sugar…and I’ve just run out of coffee. There is a machine here, but at some point you have to stop, or you won’t sleep, or at least I won’t. It might take a lot for that now, though. Smooth running is super, but a bit boring. The room is a little cold, which probably doesn’t help the sleepiness either. But it is mostly empty:


Though there are a few of us dedicated souls here, including Jessica, a Wisconsin graduate student:


I will not think about warm beds, or that my husband and cat are in my warm bed…until later.

P.S. I apologize for the lack of clarity – my cell phone was used to take the pictures.

P.P.S. Just read Monica’s blog, and I now know how I can fill the last few hours: I can write documentation!


The Control Room Life

Tuesday, April 1st, 2008

If you were to analyze my daily life based on my posts, you would probably conclude that my average day consists of sitting around in the control room, sitting around in meetings, taking cosmic data, being interrupted by power cuts and cabling things. And today I am certainly 4 for 5 in that list. I spent most of the week in the control room, squeezed in a few meeting, took some cosmic data and was stopped by a full power cut thus ending all attempts at data taking. I have yet to cable anything, but the day is still young.

We are in yet another ‘combined running’ week which is a lot like one of the milestone weeks. But unlike the Milestone weeks which combined all sub-systems in ATLAS, in this week we are only combining the calorimeters: TileCal, Liquid Argon, and the level-one calorimeter trigger. Pretty much from now until beam, we will be having combined running weeks. For example Calo-week this week, or Muon-week, or Inner Detector week or Muon-Calo week, etc.

What we get with the combined running weeks is priceless: the chance to see the sub-systems all running together and to be part of the intensity in the control room as everyone works to get ATLAS ready for beam. But it is not without frustration. When you build any detector, especially something this big, patience is a necessity. Take this week for example, we want to run some tests of our timing but we are delayed several hours because of DAQ problems. Finally we start the tests and the control goes dark due to the power cut. This is life when commissioning a detector. It is a 3-to-1 ratio. What you think will take one hour, always ends up taking three. And it is incredibly frustrating but there is nothing you can do but wait.

That is the control room life. Long periods of nothing, followed by intense periods of frantic activity. Ah! The power is back. The waiting is over, its time to go again!


Cosmics vs Beam

Friday, March 14th, 2008

In response to my last posting about the ‘6th Milestone week’, the following questions were posed by Jacques.

Once you are satisfied with the results of this test, or any subsequent test that might be decided using cosmic rays, will you “just” have to wait for the LHC to start beaming, at which time you can immediately start gathering and exploiting the data from proton collisions, or does another heavy test campaign begin then? For how long?

Outside the difference in the data frequency/volume(is this not a 1 to 1 million ratio?), are there limitations in the current ATLAS tests due to the nature of cosmic rays? Or will the most volatile particles created by the collision decay so quickly that you can only track the results of such decays for which cosmic rays are a good subrogate to calibrate the various detectors and check they deliver consistent “tracks” whenever a given particle crosses them?
How about the calibration of the Level One Trigger in this context?

For starters, ‘satisfied with the results of this test’ is a constantly changing criterion. A year ago, satisfaction was getting just two sub-systems to run together. Today satisfaction is running all sub-systems with a level-one trigger rate of 10kHz. Next month, satisfaction will be running at 100kHz (which is the level-one rate we want to have during beam running). At the start of each ‘M-week’, we have a whole list of problems that we experienced in the previous ‘M-week’. At the end of the week, we have an entirely new list of problems which are generally more complicated and therefore more difficult to solve. So I suppose forward progress is defined as finding harder and harder problems.

We will never be in a position where we ‘just wait’ for the beam. Nor when the beam is running will we just be waiting for the data to roll in. It is a cultural trait of high energy particle physics to push the system. If we are stable with a level-one trigger rate of 100kHz, someone will suggest an idea to push that rate to 120kHz. There is a saying, ‘If it was easy, it would have been discovered already’. Thus in order to make the big discovery, we have to be willing to take risks and push the detector to its design limits and if possible beyond. And pushing the limits is all the fun!

In the period before the beam, cosmic rays aren’t a great way of testing ATLAS’ limitations. But it is all we have. Using the muon trigger chambers, the cosmic ray rate is about 100Hz. The beam will be 40 MHz which is roughly a factor of a million greater. We try to push the rate during cosmic running by using a high rate ‘random’ trigger but there are no physics events associated with these triggers. Additionally most cosmic events tend to be a single muon slicing through the detector. Whereas with the beam running there will be hundreds to thousands of particles in the detector.

Cosmic muons are very useful to study tracking in the inner detector and muon chambers. For the calorimeters, we can use them as a preliminary cross-check of our energy calibration. They are also helpful to establish the relative timing between sub-systems (which needs to be known on the nanosecond level). While cosmic muons are helpful for calibrating the part of the level-one trigger that looks at muons, it doesn’t help us much with calorimeter-based level-one trigger. The reason is that the calorimeters measure energies that are typically associated with “jets” of many particles, not a single track like those from cosmic muons.

The bottom line is cosmics are all good and fun. But if given the choice of cosmics vs beam. Give us beam!


The Sixth Milestone

Tuesday, March 4th, 2008

Another ‘Milestone week’ is here. As with the previous ‘Milestone week’ the purpose is to run as many subsystems of ATLAS together, all taking cosmic ray data. In the usual day-to-day routine most subsystems do their commissioning independently. It is a special event to combine multiple subsystems and a milestone to combine them all.

This week is the 6th and supposedly final ‘M’ week. Or M6. Some people say it is the final M-week probably because there is as of today no M7 scheduled. But in the fine tradition of schedule-making, the end of the Milestone weeks probably just means moving to a new letter. ‘N’ being the next logical choice in this case.
But the M-weeks are fun (as well as frustrating and exhausting). The control room is a hub of activity. The air is simultaneously filled with tension and excitement. With each M-week, the goal has been to add in more and more subsystems. For M6, we have all but the pixels. So written out in full acronym glory, this is

MDT, RPC, TGC, LAr, Tile, L1Calo, SCT, TRT, CTP, HLT, DAQ, DCS, Tier0, DQ, Offline

This can be generally translated as ‘no available chairs in the control room’.

With five M-weeks behind us, it seems the subsystems have finally started to play nice. In the past, one of the most time-consuming things was simply to combine the data acquisition systems for each subsystem. So far the combination of multiple systems has been very straightforward and rather painless.

But then again I shouldn’t speak so soon. It is after all only tuesday.


Down and Up Again

Tuesday, February 5th, 2008

The looming nature of ‘the closing’ has injected an increased sense of urgency into all those still working on detector installation. Since TileCal has been installed in the pit for several years now, you would think we would be exempt from such urgency. But no. In the past year or so, TileCal has undergone a campaign to repair some less-than-optimal components in our electronics. The front-end electronics for the TileCal are organized in long ‘drawers’ which can be pulled out from the ends of the calorimeter. We are replacing things like the power connectors which have been causing some problems over time. In the past year, we had the time and detector access to make these repairs so we decided not to wait for the problems to worsen.

There are 256 electronics drawers in TileCal so upgrading every single one is more than a day’s work. The drawer itself is almost nine feet long. It has to be removed from the calorimeter, lowered down to the floor of the detector cavern. On the floor, three electronics tables are set up where technicians can make the modifications. Once done, the drawer is tested, raised back to the calorimeter, re-inserted and re-tested.

One of the most difficult parts of this procedure is just getting the drawer from the calorimeter to the technician’s tables. There is no space to make the modifications right at the calorimeter so moving the drawer is the only solution. Furthermore, the scaffolding surrounding the calorimeter is accessible by ladders so we have to invent some creative ways to get the electronics drawers up and down the scaffolding.

One technique is to lower and raise the drawers through the access areas in the scaffolding. As seen here. The blue boxes at the top of the picture is the part of the calorimeter, where the electronic drawers are inserted. In the center of the picture, part of one of the drawers is being raised between two access ladders. This is delicate work. You don’t want to go banging your newly repaired electronic drawer against the sides of the scaffolding. And these are all custom-built electronics. It is not like you can go get a replacement at Radio Shack. On my former experiment, SNO, the electronics racks had signs reading, ‘Careful! These electronics cost more than your house!’.

raising a drawer

It is times like these where I really admire a technician’s patience. The pressure of ‘the closing’ is increasing, everyone knows that soon the scaffolding will come down, the collective heart rate has gone up several beats. But the technicians are never fazed and continue to raise the drawers with the same patience and precision as ever. They know better than anyone: you raise the drawer too fast, you will break it. We are in good hands with those guys.