Quantum Diaries

Given the tedium of what I need to deal with day to day on the computing, what is it that makes computing interesting?  Let me make a comparison with what is going on in the collision halls.  My colleagues underground at CERN are working very hard as we head towards LHC startup.  There are some very tight time constraints at this point, and they are working with very complex systems that are pushing the limits of their technologies.  And as we head into these final weeks, the separate systems that have been under development for years must be integrated into one large experiment.  It’s a tremendous task, and I don’t want to take anything away from what they are doing.

However, they are starting to get out of the woods.  The door to the collision hall will be shut at some point, and very little can be changed after that.  And the number of people who will interact directly with those systems is relatively small; a team of experts, who will continue to make a lot of effort to make their hardware work and keep it running happily.  Most of their work will be hidden to the world; physicists will be happy to see lots of silicon hits on tracks, but they will only have a vague idea of how much labor went into that.  (I’ll say again, the hardware guys are under-appreciated!)

In contrast, just about everyone on CMS will interact with the computing in some way, which means that my problems are just beginning.  Everyone will want to know where the datasets are.  Everyone will be trying to submit jobs.  Everyone will be trying to make plots.  Performance will be documented and updated regularly on Web pages.  This means that everyone will have an opinion on what works well and what doesn’t, and they won’t hesitate to voice it.  And all the computers are above ground, and software can be modified with a few keystrokes; we can tweak things endlessly, and we might well be called upon to do so.

So in fact this is a very human enterprise — we are building a system that 2000 motivated, smart and creative people will be using every day.  We need to make it work for each of them as individuals, while also making sure that the group as a whole is not harmed.  And while ultimately we have to build good systems, there is a lot of psychology and sociology involved too.  Everyone needs to actually buy in to the idea of distributed computing for it to work, which might be hard while we still work through all the kinks, and everyone will need to trust that they are being treated fairly.  One of my mentors said to me once, “If all of our problems were physics problems, this job would be easy.”  She was of course referring to the fact that we must work with people every step of the way.  Physics equations and plots are interesting, but the human aspect of the work adds an extra dimension.

It is on my mind today because I have been corresponding with some users who are having trouble running jobs on our site.  It sounds like there could be any number of things going on…many of which may have nothing to do with the performance of the cluster here.  But it doesn’t matter; I’m invested in getting the entire chain working, because we have to build confidence.  More to come, I’m sure.

Despite all of our high-minded talk about new physics discoveries, lots of nitty-gritty work has to be done in the trenches. The day to day work of particle physics can typically be characterized either as a cabling exercise (building the detectors) or a bookkeeping exercise (analyzing the data). Since most of the other bloggers are writing about their cabling exercises, I’ll write a bit about a bookkeeping exercise.

The Tier-2 computing site that I help to manage currently hosts about 150 TB of data, in about 200 datasets, which in turn consist of about 135,000 individual files. These files are in turn spread over many different disks in about 20 different storage servers. When someone submits a job to our site, they specify which files they want to analyze. We have to then locate those files and make sure the jobs can actually work on them. Bookkeeping, indeed.

When we copy datasets here from Tier-1 sites, we use a tool that looks up a database of what files exist and where they exist, and then tries to transfer all of those files here.  This transfer tool, called PhEDEx (a name both beautiful and horrible), then records which files are at our site.  It also writes this information into a different record that a tool called DBS can look at.  When users submit jobs, they check against DBS to locate datasets.  So, this means that PhEDEx needs to be in synch with DBS (and vice-versa), or else user jobs will come here and look for data files that we don’t have.  And of course what is actually on the disk needs to be consistent with what PhEDEx thinks we have on disk.

This is what I have been chasing around for the last week or so.  We did some careful checking and discovered that PhEDEx thought we had about 2% more files on disk than we actually had.  That might not sound so terrible, but it is 4 TB, not a small amount.  That we got fixed by telling PhEDEx that we didn’t actually have those files; it then went and re-transferred them for us, without incident.  However, PhEDEx and DBS disagree with each other too, I suspect by an even larger amount.  We’re still trying to figure that one out.

I decided that one way to reduce the number of inconsistencies is just to reduce the number of datasets that we host.  Heck, how did we end up with 200 datasets here anyway, and who exactly wanted to use them?  I trolled through a lot of old emails and other records, and now have a picture of how we got here.  What I still need is to be able to understand how actively each dataset is being used.  If no jobs are actually looking at a given dataset, I can delete it without anyone being offended.  (That’s how I got down to the 200 I’m quoting…I was able to knock off the easy ones.)  We need to develop some better tools for this.

All this is nothing compared to trying to understand how the funding that supports this and other work of our research group here actually flows to us.  That’s been another recent project.  It’s a bookkeeping exercise, for sure, but I wouldn’t call it particle physics.

And now, the long-promised explanation of the CMS distributed computing system. (I know, you have been on the edge of your seats all this time.)

Let’s start by considering boundary conditions. First, the LHC will produce a lot of data. Every year, the CMS detector will produce something like a petabyte of raw data. A petabyte is a million gigabytes, and if I did the calculation right, if stored on a set of DVD’s, they would stack up twice as high as the Nebraska state capitol, a famously tall building (if you know your Nebraska). This data needs to be processed (which usually means adding more information to it, making it bigger), stored and analyzed. On top of that there is an even larger amount of simulated data — if you are looking for new physics, you have to simulate it first so you know exactly what detector signatures you are looking for. Thus, we are talking many petabytes of data per year that we must work with.

Second, you may not notice this while tapping on your laptop, but computers require a significant amount of power and cooling for their operation. This has become a constraint on operating data centers; last year I went to a conference on computing in high-energy physics, and the whole week ended up being about power and cooling. (Yes, I was able to stay awake.) No single site can deploy enough power and cooling to support all of the computing needed for CMS data processing and analysis.

So, our answer is to run a highly-distributed computing system, with centers distributed around the globe. Now, this does present significant organizational challenges, but it also allows us to make use of computing expertise in every CMS country, and also gives people a sense of ownership — my vice-chancellor for research was much more interested in helping to pay for computers in Nebraska than he would have been to send computers to Switzerland.

To keep the system manageable, we’ve imposed a tiered hierarchy on it. Different computing centers are given different responsibilities, and are designed to meet those responsibilities. (“Design” here means how much CPU or disk they have, and what sort of networking requirements, etc.) A too-cool-for-school graphic showing how the whole thing works can be found here. The Tier-0 facility at CERN receives data directly from the detector, and it reconstructs events and writes a copy of the output to tape. This may not sound like much, but it saturates the resources that are available at CERN.

Data is then transferred to Tier-1 centers. CMS has seven of these, in the US (at Fermilab), the UK, France, Spain, Italy, Germany and Taiwan. These centers store some fraction of the data that come from CERN, and as we gain a better understanding of our detector behavior and of how we want to reconstruct the data, they also re-reconstruct their fraction of the data every now and then. They also make “skims” of these events — a particular physics measurement typically relies on only a portion of all the collisions that we record, so we split the data into different subsamples that will each be enriched in certain kinds of events.

Note that in all this no one has yet made a plot that will appear in a journal publication! This starts to happen at Tier-2 sites; that’s where skims get placed for general users to analyze them. There are about forty of these sites spread over five continents, and they are also responsible for generating all of that simulated data mentioned earlier. This makes the Tier-2 sites very diverse and dynamic facilities — they are responsible to many different people trying to do many different things.

I have surely rambled on enough for a single posting, so some other time I will write about some of the particular challenges we face in making this system work. Suffice it to say that I spend a lot of time thinking about it. I try not to let it keep me up at night, but sometimes the title turns out to be true. Sorry, I needed to come up with a title for this post, and while “Trail of tiers” was more appropriate, it also has negative connotations in Native American history.