The case of the $5,000 socket – it’s about the benefit when the mass is there, versus not

The socket cost $5,000. But we got a good deal for three at $15,000. Now this might sound like just another story about a $300 toilet seat, but it turns out there may be some rhyme or reason behind $300 toilet seats. Or even $10,000 toilet seat covers. If you make a plane and then one day no one makes more of those planes, and one day you need parts for that plane, like a toilet seat cover, don’t be surprised finding parts is expensive, as you pretty much have to pay someone to make that part for you – from scratch. All this with lot and lots of paperwork along the way to make everyone rest assured that it’s the exact same thing as the original part. And all that paperwork is effort, and all that effort is money.

This socket had no such excuse.

But let’s go back in time a few decades. The Space Shuttle is flying again after the loss of Challenger. It’s the early 1990s and among the items that keep getting a lot of attention are the valves that connect between the Space Shuttle orbiters (Discovery, Atlantis, Endeavour and Columbia) and the External Tank with the propellant. The day a Space Shuttle crew was in town for a thank you and pep talk at KSC I happened to grab a picture of that valve for the crew to sign (yes, that’s Story Musgrave and STS-44). It’s not exactly the kind of valve you might think opens and closes a 17-inch line that is going to flow super-cold liquid hydrogen (pictured) or liquid oxygen at flow rates that “empty the typical backyard pool in a few seconds” as the KSC guides were fond of impressing on the tourists when spinning off numbers about the Shuttle’s impressive feats.

Oddly, when this valve would open it would remain in the middle of the flow of that hydrogen (or oxygen). Again, if that has not sunk in, the big plate with the two arms you see in the picture would flip into the middle of the massive flow of cryogenic propellants. In the middle. In the way.

Of course, after Challenger, this was put on the list of the bad-days in waiting. What if the propellant flying by the valve so fast applied so much force, this “flapper” pretty much a leaf in the wind, that it just shut? This would be catastrophic. Immediate disintegration. Now the obvious thing was to change the design. But that begs the question how the original design came about in the first place. The designers knew full well what they were doing – this was a light weight design, and if you are going to put something the size of a 737 into Earth orbit, every ounce counts.

Alternatives to this valve would probably weigh more, and in either case might also be forced closed by the tremendous flow of hydrogen and oxygen, and take forever to retrofit. In the mean-time someone figured out you could put something in-between the existing valves – a latch – that assured they remained open during the flight, mechanically restrained. Still in the middle of the flow.

And so yes, there was still that pucker factor. And it was high. The latch was a little physical insurance, just not mentally. Think aerodynamic forces on the wing of an airplane, flowing under and over the wing just right in flight. Now think about this valve in the middle of all this monster flow of super cold liquid hydrogen or oxygen, creating hydrodynamic forces. All fine as the liquid flows under and over the valve (“flapper” as it was called). Except there’s a problem. No one’s flying this airplane – the plate must be at a certain angle to the flow so it’s all just in balance with the flow of propellant. That is, so it does not get forced closed by the flow.

Additional insurance was simple. We would check the valves five ways to Sunday for all manner of settings and angle and position once open.

Sounded simple. Not quite.

It turns out this valve was never meant to be opened unless the External tank was mated to the orbiter, only then there was no way to see what was happening inside the valve assembly. Of course, engineers, like nature, find a way. The valve could be opened, manually, using a wrench and a socket reaching into a small space along the side of the entire contraption.

Except there was no socket the right size.

If this story starts sounding a bit like, “for lack of a nail a horse was lost” – it sort of is. A train of events began with a design that was likely driven by a desire to weigh as little as possible, as far as later discussions could agree on. It was an elegant solution, in its own way. The designers knew exactly what they were doing. On top of that you have to imagine a designer who could have gone say with a ½ inch shaft on the valve so a ½ inch socket, but that would have been over-designed, a waste of mass. SO inefficient. Alternately it could have been 3/8 inch, but the analysis of loads would have said that’s not sturdy enough. Naturally, the designer went in-between those sizes.

Now anyone who has done a little handy work knows you can grab the socket that’s a little larger and it might do the job, except watch it – because you either lose a knuckle when it gives unexpectedly, or you will damage (by rounding) what you are trying to turn. And sure enough that’s what we did for some years.

Until someone had a bright idea – why not get the sockets that would be just right for opening and closing this particular valve as often as our hearts desired without risking damaging it (in my naivety at the time, I was part of this). That can’t cost much can it? Damage a multi-million-dollar valve, who knows, and besides, the right tool for the right job!

Then I saw the bill. The contractor (name withheld here to protect the innocent) had a socket made from scratch, just perfect for the job. It would be $5,000. But just in case we ordered a few. So $15,000. When the sockets arrived, I held one the way you would a piece of expensive jewelry that’s been taken out from under the bright lights and glass for you to peruse.

For lack of a nail, a horse was lost. And so on, and so on, but what was really happening? It’s not really about mass savings. Not entirely. Today we have come a long way as we fight the rocket equation, a seeming realization that hardware doesn’t have to bow to the tyranny of the rocket equation, some mass might just get around it.

Hardware is expected to be designed with affordable operations in mind. In a system like the reusable Falcon 9 booster, we see a massive (pun intended) shift to where the seeming philosophy is propellant is cheap, effort is expensive, so why not just up-size? Then that last ounce quickly loses importance in perspective. The rocket equation tyranny is still there, you just got a little creative about avoiding the bowing. And this is a path to where you avoid the sockets that damage the valve, or the better one’s for $5,000 apiece.

We faced similar questions in another reusable booster project – the DARPA XSP. There we would try adding wings and things and returning a booster horizontally, like the Shuttle, under no power. Contrast this with the Falcon 9 reusable booster scaling up for extra propellant, re-igniting engines and returning vertically. The balance is not obvious. We are still a way from where all boosters like airliners look alike, probably because the best solution, or the solution that wins (which may not be the best) is still out there. How might we locate that design? We would have to balance all the variables, mass, design, cost, technology, practices, and more (with care to avoid the compromises that lead to $5,000 sockets). For that we need some powerful thinking, human, and otherwise.

The real challenge is locating where added mass adds value faster than losing payload, a benefit not from being absent, but by what it does when it is present. Reusability, sustainability, reliability and lower prices will follow when mass is seen for the friend it is, not someone to avoid.