“I didn’t understand a word you said”

Recently, the buzz around AI has been about being untraceable, the inability to explain why an AI does what it does. Try and backtrack through an AI’s logic step by step, and you are Alice going down the rabbit hole. This is so for those who create the technology, intimate with it to the most minor details. Now imagine everyone else.

This topic of AI and unexplainability already made the rounds in 2019. The moves of an AI playing Go baffled everyone before that. Recently you will find excellent pieces in the New Yorker, with an analogy to data compression and “loss of fidelity,” and over at Noahopinion, as choosing between predictive power or control.

Yet are we really in new territory, or did we go in a circle back to where we started?

From 2010, as Dr. Chandra discusses disconnecting the AI Sal to replicate the malfunction suffered by the Hal 9000 unit en route to Jupiter.

I didn’t understand a word you said.” This NASA manager did not mince words.

And here I thought I really took this down to the brass tacks. The analogies, backstory and lots of “like” and metaphors were all in there.

To be fair, my analysis was rather complicated. Also, to be fair, the manager likely aggressively questioned my work due to being more new than complex. But, amusingly, at the time we had only a hint of the complexity to come. Our team knew, too, if Lt. Commander La Forge presented a plan right now for fine-tuning the warp coils on the Enterprise, the reception would be congratulatory. This would be so without the manager even pretending to understand warp coils.

These biases came naturally for an organization narrowly focused on performance, all about the holy trinity of payload, dry mass, and Isp. Shifting the discussion to obsess over ease of reuse inevitably created communication issues. Today, my lucky day, would not be the first or the last time I received this kind of feedback.


Let’s go back, movie style – “Five Years Earlier.” In the beginning, came the spreadsheet. Like tribbles, one became two became four and so on. Sure, the math linked the factors, adjusted, added up, and parsed, and you could follow every single step, and – well, by then, we’d lost the audience. Crash and burn, crater, let’s clear the debris and try again. We learned individual calculations, puts and takes, didn’t make for a compelling story. The characters were endless, but where was the plot?

Next came models. Occasionally we would say “the model” as if it spoke all on its own. Soon after came simulations. “When” now accompanied “what.”  Space met time, and the evening and the morning were the 3rd day, so we still had a lot of work to do.

Could we think of this as activities, supply chains, or LEGO-like blocks anyone could assemble to explore next-generation reusable launchers? Of course, except now we had to start answering questions about being verified and validated – which, it turns out, are not the same in this world of models, simulations and inputs and outputs.

For a while, the models and sims spoke with us, like Guru’s using new languages and terms. The addition of rigor grew the obtuseness faster than the acceptance. Move quickly, and before you know it, half of the presentation is catching everyone up from when last you met.

How long before someone wraps a body around an AI, except the large model is doing next what it surmises it is most likely to do next? Some limits, or “laws” for robotics will be useful. Credit: Westworld/HBO.

That all sounds good, except this is all so complicated you can pencil whip any answer you want, and no one in this room could say otherwise.

(I must admit I had a hand crank pencil sharpener in my office and #2 pencils in a coffee cup. For most of my career, this anachronism remained nearby. Another confession, I still have a pencil sharpener, now in my garage.)

“…you can pencil whip any answer you want, and no one in this room could say otherwise.”

When spitting out answers favoring one design over another, the headwinds are not only about rigor or traceability. Complexity, and worse, a flood of information expressed in a new language, will clearly hobble the adoption of new ways of attacking problems. Still, we had a saying among us – if you are taken aback by harsh critique, you must find another line of business. I was not taken aback. Try again, including new ways to communicate, share, listen, and improve.

For context, though, imagine you were just in another meeting where it is said, clear as gospel, that if something weighs less, it will cost less. This is followed by the mass reduction from the Shuttle upgrade costing much more than the considerable expense initially advertised. Oh, and even before accomplished, seeing it will cost more per flight. Mumble, mumble, then many asterisks are placed on the notion “less mass equals less cost” (a thought which, for the better, eventually ceased to be a going concern.)

What does not kill you… well, you get the picture. The tough questioning improved all our models and simulations, and one day we are running “design of experiments.” Soon after, we realize we might as well go across the entire life cycle of technology, not just the part on the tail-end at Kennedy Space Center. Even better, the models now have “Reverse” on the gear shift, not just “Drive.” Instead of entering a design and getting answers, we would enter the answers we want and ask the machine to give us the designs. The sense of predicting a proper investment direction became more robust, like the hurricane tracks for different weather models all converging on the same landfall. The downside was an increase in complexity that slowed down further work. We picked all the low-hanging fruit and now came the hard part.


“…you could be more than fine not knowing how the machine got inspired.”

Needless to say, we also received plenty of encouragement, for which I am thankful. Intuitively, most complex problems have complex solutions that do not make for effortless storytelling. Our sponsors saw this, and with them, our work did not need a hard sell. The best vote of confidence came from projects with real hardware asking us to join their team, even if most of that hardware never flew. (Remember the part about fixations on performance.) Nonetheless, enough audiences saw how answers with a proper sense of direction sufficed regarding trust.

In “The Chinese Room,” the analogy of applying rules with no real thought, while seeming to carry on a conversation, breaks down rapidly on realizing that is what our non-sentient neurons are doing all the time.

For AI, in many fields, much of the matter of being traceable versus trusted could be similar. If you adopt a direction you will test in the real world, and the advice of the AI proves correct, you will call this success and move on. Trust will increase even as traceability declines. For example, if you create your AI’s suggested alloy and get a leap in strength, you could be more than fine not knowing how the machine got inspired.

Today, it’s becoming common in aerospace to hear about optimizing for manufacture, for low cost up-front and once operational. The complex push and pull between the decisions we face in complex aerospace systems could be simplified, say by narrowing your focus  to only performance. All other factors become results, revealing themselves over time. Costs? Just wait for the bill. Schedule? Wait. To. Finish. Reliability? Safety? You will discover the result there too. Alternately, we can embrace complexity to favor answers and success of a sort we would never see if fixated on narrow, simple questions.

Of course, there is the well-deserved negative critique of inscrutable AIs when they reinforce biases and errors, say regurgitating misinformation from the internet. But that too is old news, since the days we said “garbage in, garbage out.” Down other avenues there is so much promise for tackling important problems while respecting their complexity. Going full circle, it’s too easy to imagine the future.

“The year 2033…”

The IBM Quantum System 2 is due to be unveiled in late 2023. It will sport a modular architecture around a base unit with 4,158 qubits. Alexa – explain how a quantum computer works, but speak to me as you would to a small child, or a golden retriever. Credit: IBM

The year 2033. She thought her presentation was going well. “Enjoy the moments like this,” she said to herself, a chance to share and to learn. Only a few minor comments came here and there from the otherwise quiet audience. Sure, she knew the work was bug-ridden and expensive, with many caveats and limitations. It would take much more work to be truly useful. A perfect time to say, “all models are wrong, but some are useful.” Better yet, best not to talk about the bugs and gaps. Those holes didn’t matter. Better not to get into explaining Shor’s algorithm either, that didn’t go too well last time. What matters is using the new quantum frame and our AIs together, already pointing development in wildly creative directions. We are generating new questions, leaps beyond what the company AI did on its own giving mere answers, so limited by what it absorbed, even if that was nearly everything. The really wild answers were right there, buried in the ground states of the new questions – if we phrased our questions for a quantum computer, that is.

Wrapping up. The complications could be overwhelming, but she knew years from now, this line of attack would be the norm. Now to see if the audience embraced the move beyond that old AI. Last chart. Turn to the audience, “Any questions?”

Also see:

2 thoughts on ““I didn’t understand a word you said”

  1. “For AI, in many fields, much of the matter of being traceable versus trusted could be similar. ”

    Hell, we trust humans (and often human intuition) without traceability. We often laud those individuals who can think in ways we don’t understand, but who can arrive at results and conclusions that wouldn’t occur to others.


    1. So true. Often it comes down to being testable, the path suggested holding up against reality when we go verify. When the test fails, or the test reveals too many limitations or biases, we can always improve and try again. When an AI’s (or experts’) numbers or suggestions hold up to scrutiny, it’s amazing how fast the lack of traceability drops off the main radar.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s