Control rooms

The last time I was in a control room at CERN I was recording the oxygen levels in the Iarocci tubes as we flushed them out in preparation for a long shutdown--some important parts of the detector had failed. It was taking hours, and would have been at least a couple more by the time my shift was over. I figured that a graph showing how long it actually took might be useful for safely managing it in the future, and wondered if the next shift person would bother finishing it There wasn't much else to do in the little room. There were MacIntosh computers here and there, but they had been modified to map VME addresses into memory and were part of the DAQ. (They were original MacIntosh machines--68000's.) I used a clunker of a dumb terminal, but I couldn't complain--so did everybody else. I couldn't get at email from the control room either--mine was on a VAX in Wisconsin, but even the local IBM wasn't accessible from UA1.

I'm at Point 5 now, sitting in front of a laptop smaller than the old terminal but connected to Wisconsin. A camera is taking pictures of the two of us to show the world, and I'm surrounded by large flat panel screens. There are easily triple the number CDF uses (38 in this part of the room alone!), and the tables have lots of arm space (no carpal tunnel here!). The chairs are comfortable: for 1500 euro each they'd better be. And . .. there's nothing to do. The expert downloaded code to the DDUs and is running a low rate test run that has garnered 14 events in half an hour. We had a little excitement for a while trying to get the plus endcap CSC crates powered back on, and then when we found out that the DCS panel telling us that the HV was turned off was only half right. (It was some sort of communication failure, made invisible by an incredibly stupid "finite state system" design. I had to fight with those same stupid rules myself--somebody didn't think through enough use cases before laying down the law.) So I got to watch an expert at work and get some explanation of problems and practice starting a new run.

We're alone here in this large room. The alignment computer isn't installed yet, so I can't test that; and I'm not authorized to run the DCOPS lasers until tomorrow (and I want the MAB low voltage on first). There's no beam, and won't be for months. Nobody else wants to run tests overnight, and somebody has to be here 24/7, and there's a two-man rule in effect in case somebody needs to work downstairs. So there are two CSC-shifters. There's a video camera next to the video connection showing 4 CMS control room sites: here, at Fermilab, DESY and the Meyrin site.

In one corner about 30 feet away is an impressive cabinet with large red buttons that apparently let us dump the beam. Windows look out onto the parking lot behind us, and everywhere is the detritus of commissioning a complicated software system--coffee cups, grid paper scribbled with notes, already-obsolete ring binders of instructions--and hard hats. OK, maybe those last aren't typical, unless the dilberts bang their heads on the table a lot. But a lot of what you might expect--multimeter, scopes, cable ties, probes--aren't in this room at all.

It turns out that simply switching on a power supply isn't simple. Everything is monitored, so the power supply has some communication gear associated with it that has to connect to the monitoring system. The triggers aren't made by plugging Lemo cables into NIM logic boards in NIM crates anymore--the triggers start with small computers on boards in crates; and the computers can and do need to be reprogrammed/reloaded; the crates need to have the intelligence to manage the reloading process (as well as tell if they are having problems, and the higher level triggering system needs to be able to communicate with those computers, which means another set of interfaces to be programmed and debugged. Etc. And timing is a huge and complex problem--calculations have to be pipelined because more beam crossings happened while you were busy figuring out if this was a worthwhile trigger or not.

Of course there isn't any beam yet, but that's no excuse for not testing the system--which obviously needs ways to generate test triggers (that have to inject realistic data into the detector readout stream). And that means we have to test the fake data generator too.

I say we, but I'm not doing it myself--I'm just here in a supporting role for that. My main jobs lie elsewhere, in the alignment system.

What else is here? A slightly smelly men's room (the end of the weekend effect), a water cooler, an espresso machine with honor-system coffee in little condom-like packets, and some vending machines somewhere that probably only take euros. I have some granola bars, a ham sandwich, cashews (soft and unsalted), bread and some apples. Funny, but apples from the cafeteria cost about the same as apples from the grocery store. And orange juice is cheaper than apple juice at the grocery story.

I have 4 1/2 hours to go, sitting across the ocean from my wife on our anniversary. And I'm twiddling my fingers--or blogging.