The case of the $5,000 socket

It’s about the benefit when the mass is there, versus not.

The socket cost $5,000. But we got a good deal for three at $15,000. This might sound like just another story about a $300 toilet seat, but there may be some rhyme or reason behind $300 toilet seats. Or even $10,000 toilet seat covers. If you make a plane and then one day no one makes more of those planes, and one day you need parts for that plane, like a toilet seat cover, don’t be surprised finding parts is expensive, as you pretty much have to pay someone to make that part for you – from scratch. All this with lots of paperwork along the way to make everyone rest assured that it’s the same thing as the original part. And all that paperwork is effort, and all that effort is money.

This socket had no such excuse.

But let’s go back in time a few decades. The Space Shuttle is flying again after the loss of Challenger. It’s the early 1990s, and among the items that keep getting much attention are the valves connecting the Space Shuttle orbiters (Discovery, Atlantis, Endeavour, and Columbia) and the External Tank with the propellant. The day a Space Shuttle crew was in town for a thank you and pep talk at KSC, I grabbed a picture of that valve for the crew to sign (yes, that’s Story Musgrave and the crew of STS-44). It’s not exactly the kind of valve you might think opens and closes a 17-inch line that is going to flow super-cold liquid hydrogen (pictured) or liquid oxygen at flow rates that “empty the typical backyard pool in a few seconds” as the KSC guides were fond of impressing on the tourists when spinning off numbers about the Shuttle’s impressive feats.

Oddly, when this valve would open, it would remain in the middle of that hydrogen (or oxygen) flow. Again, if that has not sunk in, the big plate with the two arms you see in the picture would flip into the middle of the massive flow of cryogenic propellants. In the middle. In the way.

Of course, after Challenger, this was put on the list of the bad days in waiting. What if the propellant flying by the valve so fast applied so much force, this “flapper” pretty much a leaf in the wind, that it just shut? This would be catastrophic. Immediate disintegration. Now, the obvious thing was to change the design. But that begs the question of how the original design came about in the first place. The designers knew full well what they were doing – this was a lightweight design, and if you are going to put something the size of a 737 into Earth orbit, every ounce counts.

Alternatives to this valve would probably weigh more, and in either case, might also be forced closed by the tremendous flow of hydrogen and oxygen and take forever to retrofit. In the meantime, someone figured out you could put something between the existing valves – a latch – that ensured they remained open during the flight, mechanically restrained, if still in the middle of the flow.

And so, yes, there was still that pucker factor. And it was high. The latch was a bit of physical insurance, just not mentally. Think of aerodynamic forces on the wing of an airplane, flowing under and over the wing just right in flight. Now think about this valve in the middle of all this monster flow of super-cold liquid hydrogen or oxygen, creating hydrodynamic forces. All is fine as the liquid flows under and over the valve (the “flapper,” as it was called). Except there’s a problem. No one’s flying this airplane – the plate must be at a certain angle to the flow so it’s all in balance with the propellant flow. That is, so it does not get forced closed by the flow.

Additional insurance was simple. Once open, we would check the valves five ways to Sunday for all their settings, angles, and positions.

Sounded simple. Not quite.

It turns out this valve was never meant to be opened unless the External tank was mated to the orbiter. Except then, there was no way to see what was happening inside the valve assembly. Of course, engineers, like nature, find a way. The valve could be manually opened using a wrench and a socket reaching into a small space along the side of the contraption.

Except there was no socket the correct size.

If this story starts sounding like, “For lack of a nail, a horse was lost,” – it sort of is. A train of events began with a design likely driven by a desire to weigh as little as possible, as far as later discussions could agree. It was an elegant solution in its own way. The designers knew exactly what they were doing. On top of that, you have to imagine a designer who could have gone, say, with a ½ inch shaft on the valve, so a ½ inch socket, but that would have been over-designed, a waste of mass. SO inefficient. Alternately, it could have been 3/8 inch, but the analysis of loads would have said that it was not sturdy enough. Naturally, the designer went in between those sizes.

Now anyone who has done a little handy work knows you can grab the socket that’s a little larger, and it might do the job, except watch it – because you either lose a knuckle when it gives unexpectedly, or you will damage (by rounding) what you are trying to turn. Sure enough, that’s what we did for some years.

Until someone had a bright idea – why not get the sockets that would be just right for opening and closing this particular valve as often as our hearts desired without risking damaging it (in my naivety at the time, I was part of this). That can’t cost much, can it? Damage a multi-million-dollar valve, who knows, and besides, the right tool for the right job!

Then I saw the bill. The contractor (name withheld here to protect the innocent) had a socket made from scratch, just perfect for the job. It would be $5,000. But just in case, we ordered a few. So $15,000. When the sockets arrived, I held one the way you would a piece of expensive jewelry taken out from under the bright lights and glass for you to peruse.

For lack of a nail, a horse was lost. And so on and so on, but what was really happening? It’s not really about mass savings. Not entirely. Today, we have come a long way as we fight the rocket equation, a seeming realization that hardware doesn’t have to bow to the tyranny of the rocket equation. Some mass might just get around that equation.

Hardware is expected to be designed with affordable operations in mind. In a system like the reusable Falcon 9 booster, we see a massive (pun intended) shift to where the seeming philosophy is propellant is cheap, but effort is expensive, so why not just up-size? Then, that last ounce quickly loses importance in perspective. The rocket equation tyranny is still there. You just got a little creative about avoiding the bowing. And this is a path to preventing the sockets that damage the valve or the better ones for $5,000 apiece.

We faced similar questions in another reusable booster project – the DARPA XSP. There, we would try adding wings and things and returning a booster horizontally, like the Shuttle, under no power. Contrast this with the Falcon 9 reusable booster scaling up for extra propellant, re-igniting engines, and returning vertically. The balance is not apparent. We are still a way from where all boosters like airliners look alike, probably because the best solution or the solution that wins (which may not be the best) is still out there. How might we locate that design? We would have to balance all the variables, mass, design, cost, technology, practices, and more (with care to avoid the compromises that lead to $5,000 sockets). For that, we need some powerful thinking, human and otherwise.

The real challenge is locating where added mass adds value faster than losing payload, a benefit not from being absent but by what it does when it is present. Reusability, sustainability, reliability, and lower prices will follow when mass is seen for the friend it is, not someone to avoid.