The Cost of Better · Part Three

The Question Nobody Funds

“Should we?” — two words, eight letters, and the most expensive question in technology, because almost nobody is paid to ask it.

I want to end this series with the smallest question in it, because it turns out to be the one we can least afford to have mislaid.

The first essay followed the cost of progress into the future, where it waits as debt. The second followed it into the physical world, where it drinks the water and burns the coal. This one follows it to the strangest hiding place of all — somewhere even an auditor cannot reach it, because it has been removed not from a balance sheet but from a conversation. The final cost of advancement is the disappearance of the question of whether to advance. We retired two words. Should we. And we did it so quietly that most of us never noticed the room had stopped asking.

The Question That Pays Nobody’s Salary

Let me describe a meeting I have sat in more times than I can count, on several continents, in several industries, because you have sat in it too and we should both stop pretending it is rare.

Someone proposes to build a thing. A new platform, a new automation, a new migration, a new model deployed somewhere it has never been deployed. The case is made in the usual grammar: the competitor is already doing it, the technology is now available, the cost of waiting exceeds the cost of acting, the board will ask why we have not. Heads nod. The decision, in practice, was made before the meeting began; the meeting exists to ratify it. And at no point — not once — does anyone in the room ask the only question that matters, which is not can we, and not how fast, and not what does it cost to build. It is: should we build this at all?

The reason nobody asks is not stupidity or cowardice. It is structural, and that is what makes it so hard to fix. In every institution I have worked inside or advised, there is a person whose job is to ask can we — the engineer, the architect. There is a person whose job is to ask what does it cost — the CFO, the controller. There is a person whose job is to ask is it compliant — risk, legal, audit. There is no person whose job is to ask should we. The question has no owner, no budget, no department, no line on anyone’s objectives. And a question that pays nobody’s salary is a question that does not get asked.

In every institution there is someone paid to ask “can we,” someone paid to ask “what does it cost,” someone paid to ask “is it compliant.” There is no one whose job is to ask “should we.”— Sumir Nagar

Worse, asking it carries a cost to the asker that asking the others never does. The man who asks can we looks competent. The man who asks what does it cost looks prudent. The man who asks should we looks like he doesn’t get it — slow, soft, possibly not long for the leadership track, the kind of person who would have asked whether the railway was really necessary. So the question dies not in a single dramatic refusal but in a thousand small calculations of personal risk, made by capable people who would each privately admit, after the second drink, that they wondered the same thing and said nothing.

How “Should” Became “Must”

What replaced the question is a sentence everyone in technology has said and most have believed: if we don’t, someone else will.

It is a powerful sentence because it is usually true. Someone else will. The competitor will ship the feature, the rival will deploy the model, the other bank will automate the desk. And from that truth we draw a conclusion that does not actually follow from it — that because someone will, we must, and that because we must, the question of whether anyone should is naive. The inevitability of someone doing a thing gets quietly upgraded into a mandate for us to do it first. Should we is not answered. It is declared inadmissible. Progress, somewhere along the way, lost its reverse gear — and then we mistook the missing gear for a law of physics.

The inevitability of someone doing a thing got quietly upgraded into a mandate for us to do it first. “Should we” was not answered. It was declared inadmissible.— Sumir Nagar

I have made this argument myself, in real rooms, with real conviction, and I do not entirely disavow it now. Competition is real; first-mover disadvantage is a genuine way to die. I am not pretending the pressure is imaginary. I am saying we let a real pressure retire a necessary question, and that we have confused we had a strong reason to with there was nothing to decide. Those are not the same thing. A reason to do something is not the same as the absence of a choice. We told ourselves we had no choice so many times that we forgot we were choosing.

A Word About the Loud Version of This Fear

There is, of course, a much more dramatic way to write this essay, and you have read it elsewhere a hundred times. It is the one about superintelligence, about machines that outgrow our control, about existential risk and the end of the species. I have read that literature; some of it is serious and none of it is silly, and I am not here to mock people worrying about the largest possible thing.

But I am not going to dwell there, because I think the apocalyptic register, for all its drama, is a kind of comfort. It points at a vivid future catastrophe and lets us feel vigilant while the actual, duller abdication happens in the present, unremarked, in the rooms I just described. The question that keeps me up is not whether a machine will one day decide for us. It is that we have already, quietly, decided to stop deciding — not in some superintelligent future but this quarter, in this budget cycle, about this ordinary deployment. The end of the world makes a better headline. The end of the question makes a worse one, and a truer one.

We keep watching for the machine that will decide for us. We have not noticed that we already decided to stop deciding.— Sumir Nagar

The Reason We Couldn’t Answer It Now Even if We Asked

Here is where the series closes its own loop, and where the punch card finally meets the thing that replaced it.

I have written elsewhere, at length, about what the arrival of these systems does to the people themselves — first in The Illusion of Reskilling and the Coming Crisis, and later, more personally, in I Didn’t Upskill. I Eliminated the Team. That is its own reckoning — the displacement, the retraining that was never quite the answer it was sold as — and I will not rehearse it here. What concerns me in this final essay is something subtler than job loss, and slower to arrive. It is that when the people who understood the system are gone, the institution does not merely lose hands. It loses the capacity to judge whether what replaced them should have replaced them at all.

Even if we recovered the nerve to ask should we, we are losing the capacity to answer it — and we are losing it for the same reason all three of these essays describe. To judge whether a system should be built, you have to understand what it does. And we are building systems, faster every year, that exceed any individual’s ability to understand them. I made this point in the first essay about legacy code: a modern rewrite of a system nobody fully comprehends is unverifiable, because verification requires the very understanding that was lost. The same trap is now closing around the newest tools. When the majority of an organisation’s code is written by a machine — when the human has moved, as I argued in a different series, from operator to after-the-fact reviewer — the human’s ability to evaluate what was built erodes in lockstep with his need to.

There is a quieter version of this failure that the industry has been rehearsing for decades, and it is worth naming because it is the same disease in milder form. We have never been good at asking should we even about our tools — we adopt each new language, framework, and platform because everyone else is adopting it, not because we have judged it worth the cost of the switch. I have written elsewhere about that graveyard of sure things: the technologies hailed as the destination and then quietly abandoned, each adopted in a rush that left no room for the question of whether it should have been. If we could not ask should we about a database, it should not surprise us that we cannot ask it about something far larger and harder to reverse. The muscle was weak long before the stakes got this high.

This is the atrophy that should worry us more than any robot. Not that the machine thinks, but that we, slowly, stop having to — and then, having stopped, lose the muscle entirely. Judgment is not a stored asset; it is a maintained one. It survives only by being exercised. An institution that outsources its thinking does not keep its old wisdom in reserve for emergencies. It loses it, the way a language you never speak leaves you, until the day something breaks and you reach for the understanding that would let you fix it and find only the confident output of a system you long ago stopped questioning. Should we requires judgment. Judgment requires practice. And we have automated away the practice.

Judgment is not a stored asset. It is a maintained one. It survives only by being exercised — and we have automated away the exercise.— Sumir Nagar

What I Am Not Saying, and What I Am

I am not saying stop. I have spent this entire series insisting I am not a man yelling at the new machines, and I will not betray that in the final paragraphs. The advances were often real. The leap from my punch cards to the laptop was genuine progress. Some of what we are building now will be, too. I use these tools; I am writing in an age I would not trade for the one I started in, shading cards by hand in an Ohio classroom in 1978, waiting overnight to learn that I had misplaced a comma.

What I am saying is narrower and harder. It is that the cost of progress was never only money, or carbon, or water. The final cost — the one this series has been walking toward — is that we built a system so good at not looking at its own costs that it eventually stopped being able to ask whether they were worth paying. We relocated the bill into the future, and the planet, and at last into the blind spot itself, the place where the question used to live. And the recovery does not begin with a better tool or a louder alarm. It begins with someone, in one of those rooms, being willing to look slow, soft, and out of step long enough to ask the two words nobody is paid to ask.

I started my life in technology knowing exactly what every instruction did, because I had made each one by hand, and the machine did nothing I had not understood and intended. I will likely end it supervising machines that do a great deal I do not fully understand and did not individually intend. Somewhere along that arc, comprehension and capability went their separate ways, and we chose capability, again and again, in rooms where the other question had no chair. I am not asking you to choose differently. I am asking you to notice that it was a choice — and to put the chair back.

The next time someone in a meeting you are chairing says we have to, or someone else will, try the oldest, slowest, most expensive question in technology, the one this whole series was written to recover. Ask: should we? Then sit in the silence that follows, and notice how unfamiliar it has become. That silence is the cost. It was there all along. We simply moved it somewhere we would never have to hear it.