In the past couple of decades a good working knowledge of computing has become deeply entrenched. It’s almost certainly not intentional. I worked on the first large HEP experiment to use a C++, and it was initially developed in-house. ATLAS learned from this experience and got professionals to set up the bulk of the framework. Since then most of the new development and maintenance has been the responsibility of the physicists. Insane? Maybe, but there’s a lot of inertia at this point and changing the arrangement would require a huge overhaul of how we work. Particle physicists may be many things, but at the top of the list you’ll usually hear the words “arrogant” and “skeptical”. One must pry the code from our cold, dead hands!

We use quite a few tools to help us debug, including gdb and valgrind, but we usually print output to the terminal first, which is much quicker and cheaper. As you can imagine there are occasionally problems, such as runaway processes consuming 95% of the memory of a node, but 100% of all the CPUs. Sometimes I think it would be better if worked more like you do, with a well defined separation of roles. Then again I suppose that would spoil some of the fun. Last night I ran some jobs over the data on a whim to satisfy my curiosity. That wouldn’t be possible if there was a person between me and analyzing the data.

We do acoustics.

Regarding memory leaks, hopefully you’re aware that there are tools that solve these kinds of problems for you. Tools like dmalloc and valgrind will tell you exactly where your leaks are.

I’ve heard of this DIY attitude before… I once went to a seminar where the speaker had to spend a few minutes convincing the audience that you should use existing fft libraries instead of rolling your own! That dropped my jaw a few inches. I hope he succeeded in persuading them.

In response to the second question: tell me about it! Most physicists learn how to code through example, with very little (if any) formal training. I’ve seen code that would make you wince. (A while back I wrote a post about the kinds of bugs we tend to get and how they drive us crazy (http://www.quantumdiaries.org/2011/06/27/quantum-bugs/) I’m guessing that we’d have a lot less trouble with these if we had professional programmers! Having said all that, we do have some very talented coders here and they tend to be responsible for the overall structure of larger projects. I once sat down and watching as one of these experts pulled my code apart, rearranged it and put it back together. He explained every step and made me think about things I’d never even considered about making the code more robust and futureproof. The end result was a module that worked pretty much perfectly, and I’ve never had any problems with the parts of the code he wrote. (It was still heartbreaking to see all my labor be ripped apart before my eyes, but life is tough in the world of programming!)

Three of the major factors that inform the decision to make the physicists take responsibility for the code are that whoever writes the code must be very knowledgeable in specific areas of the field they’re working in. I’m not sure how much of this we can do with matlab (probably a lot) but there have been several instances in the past where the existing tools just weren’t good enough, and this lead particle physicists to develop tools such as ROOT (our main analysis framework) Roofit (a fitting package tailored to particle physics), TMVA (a multivariate analyzer) and StatPatterRecognitiion (another multivariate analyzer). Now each one of these packages could have been developed in cooperation with more experienced software engineers, but the decision was made, for better or for worse) to write it all ourselves. It comes with benefits and penalties, of course.

The second reason is simply a matter of time. We have bleeding edge software (I call it hemorrhaging edge, especially when we get memory leaks!) and if we get physicists to change the code we can get very rapid changes at very short notice (ie a physicist will stay in the lab until 2am at no extra cost to get the result out if there’s a hard deadline.)

The third reason is that we are extremely skeptical of what our code actually does. It’s very easy to use a “black box” application and get a biased answer without realizing it, so we like to be able to go through our code with a fine tooth comb and fully understand what it’s doing.

This attitude that we should do it all ourselves goes further than just software development, we also write our own papers, whereas in most other industries/fields there are secretaries that write the LaTeX documents for us. We write our own papers and if you’ve ever seen an internal review process you can tell! It seems that our best software starts with good developers, and our best papers end with good editors. And yes, it’s normal to spend about a week discussing what colors to use in the plots for the papers!

What kind of physicists do you work with? It would be interesting to hear how other fields handle their software!

First of all, how on Earth do you tell which stuff came from which events with a pileup of 20-30?

Second, why do particle physicists write their own code? I’m a software engineer, and my job is to write the software for the physicists at my company. They mock things up in Matlab sometimes, but ultimately we write, maintain, and optimize the production code, and it’s a lot of work, even for a dedicated development team! It just seems odd to have a bunch of physicists wasting their time on something other than their own field. I mean, physicists lack the training of a real programmer or engineer. They also don’t necessarily have time to stay up to date with the latest technologies in the software world.