Independent reviews, being helpful, and what matters – in the end

“We can be independent because we have no dog in the fight.” This time, the defense of a review team boiled down not to what it might offer but to what it lacked. There appeared to be no conflict of interest. Not that anyone could quite define what independence in a review process really meant.

In my time with NASA, I contributed to many review teams, my task being to reflect on some project’s purpose in life. What is the project supposed to accomplish once it is operational? Were a project’s actions today likely to match what it advertised for tomorrow? Tomorrow being years away.

Recently, NASA formed an independent review board to look at how NASA will resolve problems with the Orion spacecraft heat shield. We have the usual layer atop a layer. This team is tasked with looking at how others are looking at a problem. This ship will one day carry astronauts to the Moon, returning at high speed into Earth’s atmosphere. It relies on a heat shield loosely based on a material from the 1960s and the Apollo spacecraft. Loosely is a way of saying the technology is tried and true or different, depending on the moment and the audience. Like the heat shield “Avcoat” material, we also have the vintage NASA habit of saying something is “derived” when a project is pitched as if it already exists. Being derived means it will have lower risk, lower cost, and take less time. Later, the technology will, curiously, be different. Now, repeating what came before won’t work. Yet, the technology will remain proven. And new and improved. To set the record straight – it’s derived but different.

Naturally, technology evolves, plans change, and what may have seemed an easy path initially meets the reality of getting your hands dirty. The same can be said of NASA reviews. Government-mandated reviews of government projects changed significantly from the 1990s to the 2010s. The more real the project (versus R&D), the more the independent reviews changed. The farther back, the more I felt a sense of the more, the merrier. You can never have too many cooks in the kitchen. This notion peaked around 2010 with the cancelation of NASA’s first return to the  Moon program, Constellation. Around then, Norm Augustine pointedly said, “…it’s hard to get others to work with you on your garden if you’re pulling up flowers to check the roots.”

The few detectives remaining no longer had the authority to follow the evidence wherever it might go.

The pendulum swinging fully the other way, NASA abolished most project reviews as it went into its second return to the Moon program, now called Artemis. Offices tasked with independent reviews were neutered,  abolished, or received strict marching orders about what they could and could not dive into. A review team could no longer question what a project was doing or how. These were mandates from Congress. Questioning costs or schedule was also verboten, as premature. Goals were merely goals, not requirements, and so on. Inevitably, the last boards were disbanded, left with nothing to add to the discussion.

The few detectives remaining no longer had the authority to follow the evidence wherever it might go. Around this time, all the major projects decided servers were no longer the place for sensitive information and work in progress. Later, as information leaks around walls, informal orders went out for projects to stop generating certain information at all. (Anything about costs or schedule.) Years later we have the roundabout words of an associate administrator, “NASA does not think that structuring acquisition and implementation to ease accounting on a mission-by-mission basis is prudent.” Translated – we stopped oversharing.

-o-

“But will the phone stop ringing?” Bringing in a NASA expert to review a NASA project was never an ideal way of conducting independent reviews. How can NASA review itself? While discussions about independence treated concerns about the perception of a conflict of interest as valid, the fundamental flaw was elsewhere. The work of a review team was necessary, and being asked to be on a team might be seen as validation in reverse – a project confirming the outsider’s expertise. Yet next year, having been too frank, or in the perception of others, difficult, the phone might not ring to once again take a look. The failure most everyone missed in such a system was the project’s influence on the selection of outsiders. It was reasonable to question how this would evolve. Over time, would review teams favor a membership that is unlikely to be forthright?

“Bell Allie Consultants will be doing the independent review.” This was the next phase. To avoid the angst of forming a NASA review team filled with NASA people who knew the NASA project members too well, the project hired a third party. The consultants had experience in analysis, if not hardware and projects. Along came the same word – independent. Yet the phone, everyone knew, might not ring next time, or for any work one day, should the consultant prove impolitic. Or, as the project would say, yes, again – difficult. If anything, consultants were more sensitive to this pressure than NASA employees, who might not lose any sleep over possibly one less assignment next year.

Independence is hard. A second set of eyes. The third party. The messenger bearing bad news. Being Tom Hanks saying, “I don’t get it.” These are demanding roles to play. We all knew this. And many wore the reviewer’s badge gracefully, knowing it was okay if the phone did not ring next year.

To set the record straight – it’s still 2026 or 2028-ish.

In the private sector, there is a job called compliance. Ideally, and here the word arises again, the job has a high degree of independence. The boss is not the CEO or even the owner. There is an oversight board, and the compliance team works for them. Compliance employees are not fired or replaced easily for doing their job. They even open investigations without anyone asking. Ideally, that is.

For NASA, this week, we also see another report by one of the groups that comes close to being its “compliance” department – the GAO. (The other is the NASA Inspector General.) We learned the return to the Moon is now no sooner than 2028, not 2026 as still advertised of late. More precisely, the goal remains 2026 for a crew to land on the Moon, but the reality is likely no sooner than 2028. Or probably later. To set the record straight – it’s still 2026 or 2028-ish. (2028 was also estimated by the NASA IG in a 2021 review.)

-o-

Yet, for all the difficulty around independence in reviews of projects by third parties, there is a more challenging objective – being helpful. Adding up numbers and seeing they do not add up is not as difficult as it sounds. Saying so in a way that will be listened to is orders more difficult. Even with the physics, finding the ends will not meet, the mass of hardware versus where you want to go, comes right along. Again, the difficulty was communicating with a project that felt they would figure things out along the way. The meeting is running behind. The project has funding. It’s not that we are stopping. “That will be put into a list called the manager’s challenge” – a second set of books. Move on.

Soon enough, we will hear about a solution to the Orion heat shield problems.

Beyond independence and being helpful, necessary ingredients for a review team, lay a middle ground of quicksand. What if there were no solutions? Or just the same, what if a project perceived itself as not having the authority or latitude to do something drastic to resolve a critical problem? Disconcertingly, the GAO report hints at some reticence to change. As if the Orion project is saying I’m out, no more changes, we see,  “…I-HAB and Orion had exceeded their masses. Orion officials said they are not pursuing design changes to reduce mass at the program level.” As if any changes will not stop a crewed orbital flight around the Moon as-is (Artemis II), we see phrasing hinting at skipping over immediate design changes – “Officials said an option for Artemis III and beyond may be to modify their manufacturing process to increase and optimize the permeability in heat shield materials.”

Soon enough, we will hear about a solution to the Orion heat shield problems. It may be unexpected thermodynamic stresses at the seams between Avcoat tiles. It may be something in the material mix needs a tweak. Perhaps Orion’s re-entry trajectory needs a nudge. Or all of the above. Yet, in my years at NASA and in review teams, I learned that solutions are highly overrated.

Similarly, identifying problems, or cliches about destruction being easier than creation, or gathering outside perspectives, is merely a beginning. These are necessary ingredients for success but insufficient. Topping it off must be a strong desire to seek real-world proof. Another un-crewed Orion test flight after any fix would provide proof all is as well as can be. Considering a crew on the Moon (awaiting a lander, another spaceship entirely) awaits till 2028, it’s not that NASA lacks the time.