It’s been an exciting night of observing here at Arecibo. Of course, exciting means that things are going wrong, which isn’t good. As soon as I tried to start the observations, the program that controls the telescope started sending me error messages. The problem was that it couldn’t adjust the power levels for one of the WAPPs. The WAPPs (Wideband Arecibo Pulsar Processor) are part of the backend of the telescope. Their job is to take the incoming signal and create the spectra we are interested in. In all, there are four WAPPs – one WAPP handles the data for two of ALFA’s beams. (Since ALFA has seven beams, one of the WAPPs handles a single beam twice.)
The WAPPs are somewhat finicky and have been acting up recently, so I wasn’t worried at first. I followed the standard procedure of restarting the WAPPs, hoping to reset them and clear away any errors. Unfortunately, the problem this time couldn’t be fixed that way. As soon at it was clear that the problem wasn’t going to be easily fixed, the telescope operator was on the phone, calling a staff member who is an expert in dealing with the WAPPs. The first step was to determine what type of problem we were having: software or hardware. Sometimes the problem can be a glitch in the software that interfaces with the WAPPs; other times, the problem is with the WAPPs themselves. After some checks, they figured out it was a hardware problem, so the operator headed back to the room containing the WAPPs, talking on the phone to the expert to start figuring out what was wrong. Another expert staff member happened to stop by on his way out for the evening and headed back to help out also. They soon found the problem and a solution, so that the failed WAPP was back up online and running again and observations could start. (One of the WAPPs didn’t have power and the solution was to remove a filter that we don’t need for our observations. The filter wasn’t working and that caused the power to the WAPP to be cut.)
All in all, we lost the first half hour of our observations, which is always frustrating. I was glad to be here at Arecibo, rather than observing remotely, though. This way, I at least knew immediately what was happening and being done to fix the problem. When observing from Cornell, I’d be stuck sitting around, waiting for the phone call to tell me that things are back online and I can start observations. Here, I was able to stick my head into the room containing the WAPPs to ask what was going on. I still couldn’t do anything to help, though, which was personally frustrating for me. I don’t have any experience with hardware, so the nuts and bolts of the WAPPs are a mystery to me. There are many different types of knowledge in this world, and I am always impressed by people who have mastery of a type of knowledge that I don’t. Tonight was a prime example of that. Even though we lost a half hour of observations, the turnaround from noticing a problem to identification of the problem to a solution felt amazing quick. I’m sure it would have seemed longer if I were in Ithaca, having no idea what was happening. Being here to witness the problem and its handling makes me appreciate the staff here that much more. I’m glad to know that they feel as strongly about wasted telescope time as I do.
Since our first half hour of excitement, it’s been a quiet night as the observations are going smoothly. It has also literally been quieter, thankfully. When we were having problems, there were no observations ongoing. It turns out that if you don’t have observations running, the computer emits horrendous beeping noises to let you know that beam 6B isn’t working. (To imagine the sound think about a red alert from Star Trek – just enough to keep you on edge and make sure you realize that things are NOT ALRIGHT.) When observing, you instead have an occasional light beeping noise, when the program is warning you the the power levels in beam 6B aren’t working. Now, all of these noises are a little redundant as beam 6B (one of the polarizations of beam 6) has been not working for months now, so I don’t really need a notification about it. However, I will not complain about the beeps while observing anymore after dealing with the other option.