At the moment the LHC is making the transition from no beams to stable beams. It’s a complicated process that needs many crosschecks and calibrations so it takes a long time (they have already been working on the transition since mid February.) The energy is increasing from 7TeV to 8TeV, and the beams are being squeezed tighter, and this means more luminosity, more data, and better performance. As the LHC prepares for stable beams, so do the experiments. I can only see what is happening within ATLAS, but the story will be the same for CMS and LHCb.
As the LHC moves through its checks and changes its beam parameters the experiments have an opportunity to request special beam setup. We can ask that the LHC “splashes” the detector with beam in order to calibrate our hardware. This is similar to the famous first beam plots that we saw in 2008. In addition to splashes we can also request very low pileup runs to test our simulation. “Pileup” refers to the average number of events we expect to get every time the beams collide in the detectors, and by increasing the pileup we cram as many events as we can into the limited periods of time available to us. For 2011 our pileup was about 15, and this is going to increase in 2012 to about 20-30. This meant I was surprised to find out that we can use pileup of 0.01 for some of our simulation calibrations!
First ATLAS splash from 2008 (ATLAS Collaboration)
The timetable for the ramping up the LHC is announced as far in advance as possible, but it’s subject to small changes and delays as new problems arise. In general, the LHC outperforms its expectations, delivering higher luminosities than promised and stable beams for longer than expected, so when we factor in unexpected problems and unexpected higher performance we have to take the timetable with a pinch of salt. We expect to get stable beams around Easter weekend. You can see the timetable in the pdf document provided by the LHC team.
In the meantime the ATLAS hardware has been checked and maintenance performed to get it in good working order for the data taking. The thresholds are fine tuned to suit the new beam conditions and the trigger menu is updated to make the best use of the data available. There are plenty of decisions that need to be made and discussions that need to take place to make sure that the hardware is ready for stable beams. Today I got a glimpse at the checks that are performed for the electromagnetic calorimetry system, the trigger system and some changes to the muon systems. It’s easy to lose sight of how much work goes into maintaining the machine!
The LHC team preparing for beams.
As the hardware improves, so does the software. Software is often a cause of frustration for analysts, because they develop their own software as a collaboration and the software is sometimes “bleeding edge”. As we learn more about the data and the differences between data and simulation we can improve our software, and that means that we constantly get new recommendations, especially as the conferences approach. There is a detailed version tracking system in place to manage these changes, and it can be difficult to keep up to date with it all. Unfortunately, updated software usually means analyzing the data or simulation again, which is time consuming and headache-inducing in itself. That is how things worked in 2011. This year it looks like we’ve already learned a lot about how the data look, so we can start with much better simulation and we can start with an improved release for all the software. This should make progress much easier for analyses and simpler for everyone (which is a very important consideration, given that we have a large range of experience with software, and a large range of knowledge of physics processes.)
The banks of super computers are ready and waiting...
Putting all this together we can conclude the following: we will have higher energy beams giving us more data, we’ll have a better functioning detector based on previous experience, we’ll have improved simulation, and we’ll have more stable and simpler software. This is very exciting on the one hand, but a bit intimidating on the other, because it means that the weak link in the chain could be the physicists performing the analyses! There are plenty of analyses which are limited by statistics of the dataset, or by resolution of the detector, or stymied by last minute changes in the software or bugs in the simulation. If we hit the ground running for 2012 we could find ourselves with analyses limited by how often the physicists are willing to stay at work until 3am to get the job done.
I’ve already explained why 2012 is going to be exciting in terms of results in another blog post. Now it looks like it will bring a whole new set of challenges for us. Bring it on, 2012, bring it on.
[...] Read more… Share this:TwitterFacebookLike this:LikeBe the first to like this post. [...]
I have a couple questions.
First of all, how on Earth do you tell which stuff came from which events with a pileup of 20-30?
Second, why do particle physicists write their own code? I’m a software engineer, and my job is to write the software for the physicists at my company. They mock things up in Matlab sometimes, but ultimately we write, maintain, and optimize the production code, and it’s a lot of work, even for a dedicated development team! It just seems odd to have a bunch of physicists wasting their time on something other than their own field. I mean, physicists lack the training of a real programmer or engineer. They also don’t necessarily have time to stay up to date with the latest technologies in the software world.
Hey Xezlec! To answer the first question, we have sophisticated algorithms that reconstruct the tracks left by charged particles and group them into primary vertices. This way we can separate the events based on their physical location along the beam axis. The overlap of two bunches is about 15cm in the z-direction, so our resolution is fine enough. There are sometimes problems (two events can be merged giving one primary vertex, or one event can be split giving two primary vertices, but these effects tend to be rate.) Using this information we can keep track of which particles came from which event in the reconstruction. It’s not perfect, but it’s usually good enough. (You can see an example shown in the CERN Courier/Institute of Physics: http://images.iop.org/objects/ccr/cern/51/10/19/CCcat2_10_11.jpg)
In response to the second question: tell me about it! Most physicists learn how to code through example, with very little (if any) formal training. I’ve seen code that would make you wince. (A while back I wrote a post about the kinds of bugs we tend to get and how they drive us crazy (http://www.quantumdiaries.org/2011/06/27/quantum-bugs/) I’m guessing that we’d have a lot less trouble with these if we had professional programmers! Having said all that, we do have some very talented coders here and they tend to be responsible for the overall structure of larger projects. I once sat down and watching as one of these experts pulled my code apart, rearranged it and put it back together. He explained every step and made me think about things I’d never even considered about making the code more robust and futureproof. The end result was a module that worked pretty much perfectly, and I’ve never had any problems with the parts of the code he wrote. (It was still heartbreaking to see all my labor be ripped apart before my eyes, but life is tough in the world of programming!)
Three of the major factors that inform the decision to make the physicists take responsibility for the code are that whoever writes the code must be very knowledgeable in specific areas of the field they’re working in. I’m not sure how much of this we can do with matlab (probably a lot) but there have been several instances in the past where the existing tools just weren’t good enough, and this lead particle physicists to develop tools such as ROOT (our main analysis framework) Roofit (a fitting package tailored to particle physics), TMVA (a multivariate analyzer) and StatPatterRecognitiion (another multivariate analyzer). Now each one of these packages could have been developed in cooperation with more experienced software engineers, but the decision was made, for better or for worse) to write it all ourselves. It comes with benefits and penalties, of course.
The second reason is simply a matter of time. We have bleeding edge software (I call it hemorrhaging edge, especially when we get memory leaks!) and if we get physicists to change the code we can get very rapid changes at very short notice (ie a physicist will stay in the lab until 2am at no extra cost to get the result out if there’s a hard deadline.)
The third reason is that we are extremely skeptical of what our code actually does. It’s very easy to use a “black box” application and get a biased answer without realizing it, so we like to be able to go through our code with a fine tooth comb and fully understand what it’s doing.
This attitude that we should do it all ourselves goes further than just software development, we also write our own papers, whereas in most other industries/fields there are secretaries that write the LaTeX documents for us. We write our own papers and if you’ve ever seen an internal review process you can tell! It seems that our best software starts with good developers, and our best papers end with good editors. And yes, it’s normal to spend about a week discussing what colors to use in the plots for the papers!
What kind of physicists do you work with? It would be interesting to hear how other fields handle their software!
Interesting.
We do acoustics.
Regarding memory leaks, hopefully you’re aware that there are tools that solve these kinds of problems for you. Tools like dmalloc and valgrind will tell you exactly where your leaks are.
I’ve heard of this DIY attitude before… I once went to a seminar where the speaker had to spend a few minutes convincing the audience that you should use existing fft libraries instead of rolling your own! That dropped my jaw a few inches. I hope he succeeded in persuading them.
Hi Xezlec, sorry for the late reply. When things get busy here they get very busy, and while blogging is a lot of fun it’s purely recreational! Acoustics sounds like it could get intractable very quickly without the right kind of help, definitely a field where we’d like to see software engineers at the wheel. The more experience a particle physicist has, the more they realize that most of what they do is software design.
In the past couple of decades a good working knowledge of computing has become deeply entrenched. It’s almost certainly not intentional. I worked on the first large HEP experiment to use a C++, and it was initially developed in-house. ATLAS learned from this experience and got professionals to set up the bulk of the framework. Since then most of the new development and maintenance has been the responsibility of the physicists. Insane? Maybe, but there’s a lot of inertia at this point and changing the arrangement would require a huge overhaul of how we work. Particle physicists may be many things, but at the top of the list you’ll usually hear the words “arrogant” and “skeptical”. One must pry the code from our cold, dead hands!
We use quite a few tools to help us debug, including gdb and valgrind, but we usually print output to the terminal first, which is much quicker and cheaper. As you can imagine there are occasionally problems, such as runaway processes consuming 95% of the memory of a node, but 100% of all the CPUs. Sometimes I think it would be better if worked more like you do, with a well defined separation of roles. Then again I suppose that would spoil some of the fun. Last night I ran some jobs over the data on a whim to satisfy my curiosity. That wouldn’t be possible if there was a person between me and analyzing the data.