Shifts changed

When I first started taking shifts on CDF, things were a trifle different. The only name that's still the same is ACE, who then as now was responsible for running RunControl and was supposed to be the person knowledgeable about all aspects of the DAQ. You always need peons to check that everything is working correctly though. Automated instrumentation wasn't cheap or easy, and so we had a walk-through checklist.

Walk around the relay racks downstairs, looking at the banks of Droege power supplies to see if anything is drawing too much current: and are all the red lights on? Flip the switch to monitor the other channel--anything wrong there? Read off and write down some obscure numbers from a panel here, then climb down the ladder to the gas platform and check the alcohol temperature (we had to put trace amounts of alcohol into the wire chamber gas by bubbling it through cold alcohol). Or you could take the elevator to the pit and climb up into the gas platform if something was blocking you from above, I suppose. Check and write down the current on this, look at the ACNET plots on the (color!) Accelerator Network terminal (press the button and make hard copy on the slow ink-jet printer if something needed to be looked at further), look at the accelerator timing signal on the scope in the trigger room (it never looked different) and measure the timing difference. Walk among the relay racks and look for dead lights, smoke, or anything odd.

And, of course, by the time you were done it was time to do it all again.

The top level of the DAQ system was a cluster of VAXes: a 780, a few 750's, and I think a 730 as well. The various monitoring and control interfaces were VT-240 terminals, except for the event display which was a big color Seiko graphics terminal (which had the cheapest, flimsiest keyboard of anything I've ever had to work with). The event display was the standard CDF event display (at first), and took one or more minutes to display details of an event. (It ran the entire event unpacking and part of the reconstruction.) After a year of this Bill Foster wrote his own event display called DF (for Damn Fast) that skipped most of the unimportant stuff and displayed an event in 5 seconds. We've used it and its descendents in the control room ever since.

The log book was a stack of standard computation books, and we'd write stuff in them, or tape in bits of printouts. Each detector had its own log in addition to the main run log book. The control room sported an LN03 laser printer, which was a considerable source of contention. It wasn't blindingly fast, and every time a run ended that had any reasonable statistics the YMON consumer process would print out the entire wad of detector performance histograms, and since it was using YBOOK-based graphics internally this was way slower than it needed to be. The ACE would be waiting to print his end-of-run notes to tape in the log book, but he and everybody else would twiddle fingers for 15 minutes. And everybody dreaded a short run: it took longer to print out the worthless histograms than the run took in the first place. We peons had to look over these histograms of frequency of hits on wires, energy deposited vs position, etc, etc and try to figure out if something was wrong with some part of the detector. Usually something was, and usually this was a known problem waiting for an access to fix. So we had to read back through the notebooks full of detector printouts looking for scribbled comments telling us that something odd was actually OK or a known problem. Communication was a serious problem, and the reference plots weren't always up to date.

Of course during the run YMON would periodically display the histograms for you. A VT-240 running in Tektronix emulation mode is kind of slow, and even if you'd seen what you needed to you still had to wait until it was done before moving on to the next plot.

We kept track of our detector status by sending a few selected events to the monitoring programs. Everything that passed the cuts was written to tape (first reels of 9-track tape upstairs, then 8mm tapes in the control room, and now it gets piped across the street into big IBM tapes in a tape robot system), but the "consumer" processes weren't fast enough to analyse everything so they only got a few events to look at. YMON looked at primitive detector details (do you have hot channels, is a wire dead, etc). LumMon tried to keep track of luminosity and trigger rates: if some "tower" in the calorimetry shifted its baseline and started showing signals above pedestal too often, then triggers based on it would start to appear too often (and be bogus, too)--so you keep track of such things. The fair-haired-boy was PhysMon: a process to try to keep track of the quantities we really wanted to know about, like the missing energy distributions, rate of W or Z production, J/Psi mass (a good monitor for making sure that tracking is working right), and so on. Of course it took forever and three days to accumulate enough statistics to be useful, and runs generally only lasted a few hours, so PhysMon was rather a nuisance. Over the years we added more monitors: notably such things as TrigMon, which tried to figure out where our triggers were coming from and whether or not the triggering logic was correct for that event (I'm working with that program these days).

After a few years it was felt that audio alarms would be a good thing, and so we bought a DECTalk box. The voice reproduction was awful, and since alarms often came in bursts the ACE would often find that the box was still yacking at him two minutes after he'd solved the problem.

Downstairs the relay racks were stuffed with FASTBUS and CAMAC crates and even the occasional NIM crate for triggerlogic. The SSP (Slac Scanner Processor) was an IBM 370 emulator on a board in a FASTBUS crate, designed to handle the readout of TDC data and format it into something that the VAXes upstairs would read out in a block of data we called a bank. You had to program the thing in IBM assembler code. We needed some changes made once, so I learned the language and wrote the changes--and then somebody else repeated the work independently. Calorimetry pedestal suppression was handled with something called an MX--a huge board 4 feet high filled with discrete components that did comparisons, additions, and pedestal subtractions to tell us whether or not we had a "jet" of energy in the calorimetry. The FMU (Forward MUon system) had a special crate which continuously read out the signal strength from those selected channels in our system which had an iron-55 source near the wire. The idea was to monitor the wire gain. Originally the board we used for multiplexing the wires into a single ADC was nice and straight, but by the time we'd stuff in all the chips and connectors it bowed a quarter of an inch out of true.

For a while our Event Builder was based on a few MacIntosh Plus computers! For quite a few years we had old Mac Plus' on desks, since they provided a relatively cheap way of giving us a VT100 terminal (an emulator) to connect to the VAXes with.

After a year or so cameras were installed so we could keep an eye on what was really happening in the collision hall. The checklist included pointing them at meters on the forward calorimetery and checking the gas bubblers in the hall for the FMU system (see at least one bubble in all the tubes in one minute? check). And looking around the hall to see if anything looked out of the ordinary.

So what's new?

Well, we've a Science Coordinator now (aka SciCo aka Psycho), and a monitoring ACE who keeps track of the voltages and gas and accelerator status, and 11 different Consumer processes which need a Consumer Operator to keep track of (and PhysMon was finally dropped from the list: replaced with ObjectMon). The VAXes gave way to microVAXes which gave way to Ultrix boxes (with some Silicon Graphics) which gave way to Linux boxes. The VT-240's and Seiko's gave way to big CRT monitors, which have recently given way (and about time, the upper Consumer CRT had screen burn) to flat panel monitors. I'm sitting in front of a 20 inch Dell flat screen--and I need the screen space it provides. There's a slightly smaller one mounted just above it, which is much closer than the old CRT which had to be mounted further up and back. 6 Consumer Monitor displays are cycling through their standard plots, and the event display beside me is showing both the calorimeter and side view of a recent event. The chairs are nice big comfortable adjustable things. The DAQ is now linux based, and instead of a farm of 68020 processors (or MicroVAXes) upstairs we have a farm of 320 rack-mount Linux computers downstairs for our Level-3 trigger. Each alarm is different: if the solenoid current is out of tolerance the theme from the Twilight Zone plays, a different failure causes beeps, something else buzzes . . . And of course the fan noise from the trigger system in the other part of the room is as loud as ever. I wish I could use earplugs.

Downstairs the MX's are gone, the FMU turned into the IMU, we use VME instead of FASTBUS, the racks of Droege HV supplies are gone (replaced with multi-channel CAEN crates)--and we've got several old cows: the Berkeley Zener Divider systems (one HV in, and you use pins to set what smaller voltages you want to go out on various channels to photomultiplier tubes--an old but robust system). The desks downstairs that we used to use for assembly and repair and logging in using any Macs that still worked have been squeezed out for more relay racks.

The voltage and gas flow monitoring is now automated. Of course every now and then one of the monitoring PC's freaks out: loses connection to the CAEN mainframe or to the net or just decides to go stir crazy. Then you have to go fiddle with it for a while. But it is nice to be able to look across the room and see that all voltages are on.

The old cameras in the collision hall were replaced with color camers, and pretty soon those will have to be replaced too--radiation has damaged a lot of the pixels. But there aren't any bubblers downstairs anymore. I may live in Wisconsin, but to me a bubbler is not a drinking fountain but an item in a gas flow system designed to let you know that the gas is flowing and to keep air out of the system. It didn't pay to compare the bubble rate on input with the bubble rate on output--it raised uncomfortable questions about how much the chamber leaked.

The walkthroughs are no more. In fact, most of the construction/maintenance phase being over, most of what used to be work and storage areas on the first floor is now open, and there's a visitor's center now with the old CTC (you can look inside to see the thousands of wires), a prototype central calorimeter wedge (as tall as a man) and other models of the detector, together with posters of physics results. Everybody needs to wear a Fermi ID when in the building, but the rack of film badges is gone. The kitchen is more worn, and apparently enough people complained that they've started keeping the supplies of plastic tablewear stocked again (for a few years you had to use a spoon for everything). That matters when you're on shift. All the small cabinets in the counting room downstairs are gone to make room for more relay racks. We used to keep small electronic parts and log books in them.

One thing hasn't changed, though: something always breaks when I'm on shift.