“I didn’t understand a word you said”

Recently, the buzz around AI has been about being untraceable, the inability to explain why an AI does what it does. Try and backtrack through an AI’s logic step by step, and you are Alice going down the rabbit hole. This is so for those who create the technology, intimate with it to the most minor details. Now imagine everyone else.

This topic of AI and unexplainability already made the rounds in 2019. The moves of an AI playing Go baffled everyone before that. Recently you will find excellent pieces in the New Yorker, with an analogy to data compression and “loss of fidelity,” and over at Noahopinion, as choosing between predictive power or control.

Yet are we really in new territory, or did we go in a circle back to where we started?

“I didn’t understand a word you said.” This NASA manager did not mince words.

And here I thought I really took this down to the brass tacks. The analogies, backstory and lots of “like” and metaphors were all in there.

To be fair, my analysis was rather complicated. Also, to be fair, the manager likely aggressively questioned my work due to being more new than complex. But, amusingly, at the time we had only a hint of the complexity to come. Our team knew, too, if Lt. Commander La Forge presented a plan right now for fine-tuning the warp coils on the Enterprise, the reception would be congratulatory. This would be so without the manager even pretending to understand warp coils.

These biases came naturally for an organization narrowly focused on performance, all about the holy trinity of payload, dry mass, and Isp. Shifting the discussion to obsess over ease of reuse inevitably created communication issues. Today, my lucky day, would not be the first or the last time I received this kind of feedback.

-o-

Let’s go back, movie style – “Five Years Earlier.” In the beginning, came the spreadsheet. Like tribbles, one became two became four and so on. Sure, the math linked the factors, adjusted, added up, and parsed, and you could follow every single step, and – well, by then, we’d lost the audience. Crash and burn, crater, let’s clear the debris and try again. We learned individual calculations, puts and takes, didn’t make for a compelling story. The characters were endless, but where was the plot?

Next came models. Occasionally we would say “the model” as if it spoke all on its own. Soon after came simulations. “When” now accompanied “what.”  Space met time, and the evening and the morning were the 3^rd day, so we still had a lot of work to do.

Could we think of this as activities, supply chains, or LEGO-like blocks anyone could assemble to explore next-generation reusable launchers? Of course, except now we had to start answering questions about being verified and validated – which, it turns out, are not the same in this world of models, simulations and inputs and outputs.

For a while, the models and sims spoke with us, like Guru’s using new languages and terms. The addition of rigor grew the obtuseness faster than the acceptance. Move quickly, and before you know it, half of the presentation is catching everyone up from when last you met.

“That all sounds good, except this is all so complicated you can pencil whip any answer you want, and no one in this room could say otherwise.”

(I must admit I had a hand crank pencil sharpener in my office and #2 pencils in a coffee cup. For most of my career, this anachronism remained nearby. Another confession, I still have a pencil sharpener, now in my garage.)

“…you can pencil whip any answer you want, and no one in this room could say otherwise.”

When spitting out answers favoring one design over another, the headwinds are not only about rigor or traceability. Complexity, and worse, a flood of information expressed in a new language, will clearly hobble the adoption of new ways of attacking problems. Still, we had a saying among us – if you are taken aback by harsh critique, you must find another line of business. I was not taken aback. Try again, including new ways to communicate, share, listen, and improve.

For context, though, imagine you were just in another meeting where it is said, clear as gospel, that if something weighs less, it will cost less. This is followed by the mass reduction from the Shuttle upgrade costing much more than the considerable expense initially advertised. Oh, and even before accomplished, seeing it will cost more per flight. Mumble, mumble, then many asterisks are placed on the notion “less mass equals less cost” (a thought which, for the better, eventually ceased to be a going concern.)

What does not kill you… well, you get the picture. The tough questioning improved all our models and simulations, and one day we are running “design of experiments.” Soon after, we realize we might as well go across the entire life cycle of technology, not just the part on the tail-end at Kennedy Space Center. Even better, the models now have “Reverse” on the gear shift, not just “Drive.” Instead of entering a design and getting answers, we would enter the answers we want and ask the machine to give us the designs. The sense of predicting a proper investment direction became more robust, like the hurricane tracks for different weather models all converging on the same landfall. The downside was an increase in complexity that slowed down further work. We picked all the low-hanging fruit and now came the hard part.

-o-

“…you could be more than fine not knowing how the machine got inspired.”

Needless to say, we also received plenty of encouragement, for which I am thankful. Intuitively, most complex problems have complex solutions that do not make for effortless storytelling. Our sponsors saw this, and with them, our work did not need a hard sell. The best vote of confidence came from projects with real hardware asking us to join their team, even if most of that hardware never flew. (Remember the part about fixations on performance.) Nonetheless, enough audiences saw how answers with a proper sense of direction sufficed regarding trust.

For AI, in many fields, much of the matter of being traceable versus trusted could be similar. If you adopt a direction you will test in the real world, and the advice of the AI proves correct, you will call this success and move on. Trust will increase even as traceability declines. For example, if you create your AI’s suggested alloy and get a leap in strength, you could be more than fine not knowing how the machine got inspired.

Today, it’s becoming common in aerospace to hear about optimizing for manufacture, for low cost up-front and once operational. The complex push and pull between the decisions we face in complex aerospace systems could be simplified, say by narrowing your focus  to only performance. All other factors become results, revealing themselves over time. Costs? Just wait for the bill. Schedule? Wait. To. Finish. Reliability? Safety? You will discover the result there too. Alternately, we can embrace complexity to favor answers and success of a sort we would never see if fixated on narrow, simple questions.

Of course, there is the well-deserved negative critique of inscrutable AIs when they reinforce biases and errors, say regurgitating misinformation from the internet. But that too is old news, since the days we said “garbage in, garbage out.” Down other avenues there is so much promise for tackling important problems while respecting their complexity. Going full circle, it’s too easy to imagine the future.

“The year 2033…”

The year 2033. She thought her presentation was going well. “Enjoy the moments like this,” she said to herself, a chance to share and to learn. Only a few minor comments came here and there from the otherwise quiet audience. Sure, she knew the work was bug-ridden and expensive, with many caveats and limitations. It would take much more work to be truly useful. A perfect time to say, “all models are wrong, but some are useful.” Better yet, best not to talk about the bugs and gaps. Those holes didn’t matter. Better not to get into explaining Shor’s algorithm either, that didn’t go too well last time. What matters is using the new quantum frame and our AIs together, already pointing development in wildly creative directions. We are generating new questions, leaps beyond what the company AI did on its own giving mere answers, so limited by what it absorbed, even if that was nearly everything. The really wild answers were right there, buried in the ground states of the new questions – if we phrased our questions for a quantum computer, that is.

Wrapping up. The complications could be overwhelming, but she knew years from now, this line of attack would be the norm. Now to see if the audience embraced the move beyond that old AI. Last chart. Turn to the audience, “Any questions?”

Also see:

• May 19, 2021: I’m with the AI, and I’m here to help