Soon, NASA will load propellants onto its new Space Launch System – the “SLS.” This test will span a few days, a whole shakedown and practice run, much like the launch countdown starting at T-72 hours for a Space Shuttle. This is an exciting moment, the end-to-end system seeing liquid hydrogen and oxygen for the first time. While a relatively uneventful few days are a possibility, experience says to expect the unexpected.
Hitting inconel, valves, aluminum, and a plumbers’ nightmare of tubing, seals, and nuts and bolts with super-cold liquids brings to mind many sayings about cryogens. How do you load cryogens? Very. Slowly. What does it take to ignite hydrogen? You looked at it wrong. What does hydrogen do? Well, it just wants to get out.
For example, lines that might seem impervious, close cousins of an army tank, have been known to have too much stress. In 2002 a hydrogen vent line cracked, through and through. It simply cracked. Of course, these lines were old and had seen many a freezing moment, then back to hot Florida days. The vibration of many liftoffs had given them a beating over the years too. Hear that crack when you drop an ice cube in soda, and you get the picture.
Thermal properties are likely high on the checklist going into loading super-cold propellants in a new and complex flight and ground system. The Space Shuttle dedicated books to expected temperatures across many situations for just about everywhere you could imagine. A “wet-dress rehearsal” will show just how close reality matches expectations – all those models and sims, and the component and sub-system tests.
Sadly, we will miss that voice of NASA public affairs, calm and zen-like, saying things like “T-minus 7-minutes and counting.”
For the Shuttle, propellant loading was naturally a cautious and slow affair. Given the pedigree of the SLS, in scale, propellants, and infrastructure, there will be similarities. Unfortunately, for this SLS test, the public will not have audio of the usual steps as they occur, chilling down facility lines, then flight propulsion, followed by a slow fill, before ramping it up to a fast fill. Sadly, we will miss that voice of NASA public affairs, calm and Zen-like, saying things like “T-minus 7-minutes and counting.” Nothing is more peaceful than the predictable phrases during a NASA launch countdown. For SLS, perhaps next time.
In 1988, an elementary introduction to hydrogen was among the first training classes I took at Kennedy Space Center. Hydrogen is serious stuff. It would stick in anyone’s head that the spark of static electricity as you touch a doorknob was more than enough to ignite a leak of this odorless, invisible gas.
Hydrogen will get out, as the notoriously sneaky stuff reminded us when the Space Shuttle program ground to a halt in the summer of 1990. We thought, at first, a seal on the tank side (a 17-inch diameter “Naflex” seal) must be to blame. A tiny scratch on these beauties of aerospace engineering might quickly explain the hydrogen leak we saw while loading Columbia’s external tank for mission STS-35.
How do you drive a ’66 Mustang with the seal in the back that could be to blame for bringing the Space Shuttle program to a halt?
With Columbia rolled back to the Vehicle Assembly Building, by Monday, July 2, 1990, we had dismantled the suspect line and removed the seal. Late in the day, some confusion ensued around who would inspect the removed seal. Would it go back to the manufacturer to examine for defects? Should it remain at Kennedy for us to analyze? In some quirk of events and confusion, I volunteered to transport the seal myself to our lab, carefully boxed, with stickers with stamps verifying who handled the seal when. Our first shift is over, just after 4pm, and I’m going down the road from the VAB to the logistics facility. The seal is in the back seat of my 1966 Ford Mustang. How do you drive a ’66 Mustang with the seal in the back that could be to blame for bringing the Space Shuttle program to a halt? Also. Very. Slowly. Both hands on the wheel. Like our motto said, “Safety first.”
It turned out that large seal was not the leaker we were looking for. This would be a long, hot summer. The entire external tank umbilical assembly would eventually be removed to be tested in Rockwell facilities at Downey, California. A simulator for the orbiter side, we thought, would do the trick. The leak must be on the tank side, after all. (Placing the tank side 17-inch valve bottoms up would keep the hydrogen liquid in the assembly.) But being a sneaky leak, this testing marked merely a start. Hydrogen will do that, no other molecule being so small.
Soon enough, hydrogen leaks plagued Atlantis too. At least the ensuing shuffle among flights gave us a rare moment when two Shuttles crossed paths in the night. This was somewhat of a fluke, as the original plan was to move Columbia and Atlantis on different days. Instead, the delays on one and the progress on the other worked toward a crossing just outside of the VAB, one coming, one going.
Our search for leaks continued at a test stand on Huntsville’s Marshall Space Flight Center. This time we had the large umbilical assembly from Atlantis and the external tank side too. No simulator for the orbiter-side here – this was the whole contraption. The setup looked similar to the one we had in California, only now on steroids.
These were long nights, a particular day never seeming to end. This mid-morning, we knew testing would not begin until much later in the day. I was determined not to miss these tests as the representative from Kennedy, and the engineer who owned and operated these systems. This is when you go back to your hotel for a few hours of sleep after a morning planning session. Then you return at 2pm and stay till 5am the next day. (OSHA eventually came around to ask about this. The bosses signed waivers. Many waivers.)
Late one night, we were gearing up for more testing. As if hazards are not problematic enough at a distance, in a bunker with wavy foot-thick small windows, we debated when to send a “red-crew” up to the assembly as we flowed liquid hydrogen. The team members would use hand-held H2 detectors, “sniffers” we called them. Should the leak evade the instruments at the test stand, we hoped some hands-on testing would fill the gaps. Except just then, a storm started heading our way, intermittent lightning flashes switching surprisingly fast to continuous mode. I had some binoculars and was looking out a small window when I recall saying, “Maybe it’s time to call it a night?”
A lightning strike too close for comfort at just that moment left us in the dark. The universe answered my question. The sleep-deprived part of me liked the reply. Yes, it was time to call it a night. Clearly, that pesky leak had become our white whale, and our efforts now channeled Captain Ahab. We safely secured the hydrogen facility, with a good rest in order for everyone, amidst some mumbling about leadership not being mindful of people and safety. We would not be explaining an explosion that woke all of Huntsville and adjacent counties, leaving a crater where a huge NASA test stand should be. Or why we loaded hydrogen in a lightning storm.
By summer’s end, we did find the source of those hydrogen leaks. Sort-of. As in not so conclusively and to everyone’s satisfaction. But that is another story (and the very first paper I ever published, which I must scan and get out there.)
As NASA loads its new Space Launch System, a project now going on nearly 15 years, killed, lifeless, and then resuscitated at one point, the white whale is now in hand. Naturally, people will be watching the weather, as no one will be flowing hydrogen with a chance of nearby lightning. As we might tell a friend as they leave us for a trip, it’s a good time to say to SLS, “make good choices!” and “be safe!” Also, watch that hydrogen. It just wants to get out.
And by the way NASA, please share what’s happening during the SLS test. One lesson of many I learned at NASA was sharing is a good thing. The world could especially use that calm and reassuring voice of NASA just now.
2 thoughts on “Make good choices!”
“In 2002 a hydrogen vent line cracked, through and through. It simply cracked. Of course, these lines were old and had seen many a freezing moment, then back to hot Florida days. The vibration of many liftoffs had given them a beating over the years too. Hear that crack when you drop an ice cube in soda, and you get the picture.
A cracked hydrogen vent line on the Shuttle’s mobile launcher platform.”
Actually the line had a weld where a bad welding rod was used. The vent line was reused from the Apollo LUT. Because liquid hydrogen was running along the bottom of the line, the line bowed up against the top piping support and was overstressed. Would not have been a problem if correctly welded. Testing done during repair work found 48 welds on both pads and all 3 MLP that had also been incorrectly welded, all from Apollo LUTs. This was one of four Apollo design/ build failures to affect just the pad Cryo systems.
Thanks Ted. I totally forgot that part about the root cause of the vent line cracking. Let’s hope any cannibalizing, or the quality of the new hardware for SLS and the ground systems (welds) does not run into similar issues.