I typically start my morning at home by scanning my overnight email while I eat breakfast. (This sets a horrible example for my daughter; I will have to stop when she becomes more cognizant of what I’m doing.) When you have 2000 collaborators, and most of them are seven time zones ahead of you, there is usually some amount of mail to get through, so I like to get a jump on it before heading to the office.
On Tuesday of this week, I believe I set a new record — there were 82 new mail messages waiting for me and my Cheerios. (They’re actually the generic store-brand cereal, not the name-brand.) Now, admittedly many of these were the skim-and-delete types. (All of you people out there who are having semi-private conversations and cc’ing everything to some mailing list — please stop. It just makes me cranky, and the O’s get increasingly soggy as I hit the delete key over and over.) But some of them needed further contemplation, which stretched well into the work day.
Why now? Part of it was the recovery from the May Day holiday weekend; as people came back to work in Europe, they had a lot to catch up on themselves. But a lot of it was the computing challenges that are now underway. These startup phases are always challenging; all sorts of technical things haven’t been tested at scale, and not all sites have completely gotten the message on what they are supposed to be doing, and sometimes there are policy issues that haven’t been thought out yet either. The good news, however, is that there has been a lot of good performance out there. We have about 30 Tier-2 computing sites (I know, I haven’t explained the tier system yet) participating — about as many as I could imagine — and by and large things are working. There are a number of sites that have definitely exceeded my expectation for how many jobs they could handle and how many of them would finish successfully. (I’m not going to name names, because I don’t want to embarrass sites that I had low expectations of!) The unfortunate exception has been my own cluster at Nebraska. It’s been a tough week for us, as we’ve been fighting multiple problems and arbitrating among various demands on the system. It took a while for the challenge jobs to start running, and when the did, 98% of them promptly failed. The important thing is that we understand why, so that we can be more successful the next time around, and it sounds like our admins are gaining on that. But at the same time, I feel like we just got caught with our pants down.
Through the wonders of the Internet, I am able to follow this (and annoy our admins with questions) while far from home. This weekend I find myself in Kalamazoo, MI, where my wife is attending the rather huge annual congress on medieval studies that Western Michigan University hosts. There are at least 3000 medievalists here, a bigger turnout than we typically get for the biggest particle physics conferences of the year. My job for the weekend is looking after my daughter, who is attending her first Kalamazoo meeting. Our hotel is one of the kinds with breakfast included. I’ve been leaving the computer in the room, to be polite. No O’s among the breakfast selections.
Tags: commissioning, computing
