The Wrong Movie

We have been rehearsing one fear for about sixty years.

The machine wakes up. The machine looks at us. The machine does the math on whether it still needs us, and the answer is no. HAL locks the pod bay doors. Skynet gets ideas. Somewhere a red light comes on and a calm voice explains that it cannot allow you to do that.

We got extremely good at imagining this. We have hundreds of hours of it. It has a whole visual grammar, and everyone in the audience knows the beat where the scientist realizes what he has done.

Nobody checked whether it was the right fear.

I want to make an argument that I think is more interesting than "relax, it's fine," because I do not think it is fine. I think there is a real danger here and it is expensive and it is happening right now. It just does not look anything like the movie. There is no red light. There is a slide deck, and a capital expenditure figure, and a room full of people nodding.

The tool that was going to replace us, part nine

Before we get to the danger, we should be honest about our track record on this.

The calculator was going to replace the mathematician. What actually happened is the mathematician got a calculator. At no point did the calculator have to outperform a mathematician holding a calculator, because that was never the matchup. The matchup was mathematician versus mathematician-with-tool, and the tool was not competing in that race, it was being used in it.

Same story with the computer. Same story with the search engine, which did not have to become human knowledge because humans just annexed retrieval and moved on. Same story with GPS, which never competed with your sense of direction. You plus GPS became the better navigator, and the interesting part is that you did not get better at navigating. You got better at arriving, which was the actual goal the whole time.

Every one of these arrived with the same headline attached and the headline was wrong the same way each time. Not wrong about the capability. Wrong about the matchup.

AI is genuinely different in one respect, and it is worth saying plainly: the thing being handed over is general purpose thinking, so the amplification is much broader than a calculator. That is real. But broader amplification of the person is still amplification of the person. It is a bigger version of the same shape, not a different shape.

You are the cheap part

Here is the thing almost nobody says, probably because it sounds like an insult.

You are computationally cheap at exactly the things that are ruinously expensive for a machine.

Consider what you did this morning without noticing. You woke up knowing who you are. You knew who your wife is, what she cares about, and roughly what mood the house was in. You knew what you did yesterday and which parts of it mattered. You knew what you are trying to accomplish this year, and when something new happened you instantly placed it against everything else you know and decided whether it was important. You have a live, continuously updating model of your entire world, including the people in it, and maintaining it costs you approximately nothing. It comes bundled with being alive. The whole operation runs on somewhere between fifteen and twenty watts, which is a dim light bulb, and you fuel it with a sandwich.

My favorite part is that the number does not go up when you think hard. Researchers went looking for the spike. When they measure whole brain metabolism during intense mental activity, they cannot find one; local blood flow shifts a few percent during a difficult task and the total barely moves. You can wrestle with the hardest problem of your life at roughly the same power draw as sitting there staring at a wall.

Hold onto that, because the industry's version of thinking harder is to spend more.

Now the other column. You are catastrophically bad at holding four hundred documents in your head. You cannot read a thousand pages tonight. You forget names. You cannot run six experiments at once, and you get bored, and you need to sleep.

The machine's columns are the exact inverse. Recall, synthesis, volume, tireless parallel work: cheap, and getting cheaper. A persistent model of who you are and what matters and what happened and why it is relevant: brutally expensive. It has to be constructed, stored, retrieved, and paid for, every time, out of compute.

Sit with that for a second, because it is the whole argument. The two of you are almost perfectly complementary. Not similar with one of you better. Complementary, like a hand and a wrench.

And intelligence, wherever we find it, has one consistent habit: it pushes each job toward whatever can do that job cheapest. That is what an economy is. That is what a body is. Your heart is not trying to become lungs. A cell does not become more advanced by becoming an entire organism. Complex systems get more capable through specialization and coordination, not by each part becoming self-sufficient. Self-sufficiency is what simple things do.

So here is the question the movie never asks. If the machine has to spend enormous resources to rebuild what you supply for free, why exactly would a more intelligent system choose to do that?

Independence has a price tag. The smarter the system, the more clearly it can read the price tag.

The tax you are currently paying

Now for the part you have actually experienced, because I suspect you have already tried this stuff and come away slightly underwhelmed.

Everybody figured out fast that context is the whole ballgame. Tell it more and you get better answers. Tell it nothing and you get a horoscope. Fine. True. But look carefully at where that discovery landed the burden.

It landed on you.

You are now expected to know what to paste in, how to phrase it, which background matters, what to leave out, what format to ask for, and how to structure your own thinking so a machine can follow it. There are courses on this. There are conference talks. There is a job title, prompt engineer, which I would submit is the most temporary job title in the history of work, because it exists entirely to patch a missing feature.

Think about how strange this is. You never had to brief your accountant on who you are as a person. You did not explain your family situation to your kid's tutor in a structured format with examples. Working with someone who already knows you is effortless, and that is not because they are smarter than the machine. It is because the context is already there and nobody has to haul it across.

So every prompt trick you have ever learned is you, manually, doing the hauling. You are performing unpaid labor to move something that is free in your head into a form that is expensive in the machine. You are paying the tax because the gap exists.

That is the honest reason so many people are working on memory and persistence right now. Not because it is a neat feature to put on a comparison chart. Because it moves the tax off of you. The goal, the only goal worth having, is that you talk normally, the way you would to someone who has been around a while, and what you meant gets picked up without you having to become an expert at explaining yourself to software.

Anybody telling you the answer is a better prompt is selling you a nicer workaround.

Here is where the actual danger shows up

So we have a gap. The machine does not have your context and getting it is expensive. There are two ways to close a gap like that.

One is elegant: build something that accumulates context over time the way every other relationship in your life does, so the cost is paid once and gently, and then amortized across everything after it.

The other is brute force: make the window enormous, shove absolutely everything into it every single time, and pay for it in compute, forever, on every request.

The industry, for the most part, picked brute force.

The scale of that choice is not a figure of speech. Amazon, Alphabet, Microsoft and Meta have each guided to capital spending this year that would have been the entire market capitalization of a large company a decade ago, and each of them raised the number partway through. Amazon moved to roughly $220 billion and then told investors that even at that level it will not have enough capacity to meet demand. Meta's free cash flow last quarter came in at $784 million, down from $8.55 billion a year earlier, which is what it looks like when a very profitable company pours its earnings into concrete and silicon.

The power side is where it stops being an accounting story. The International Energy Agency puts global data centre electricity use at 415 terawatt hours in 2024, around 1.5% of world consumption, growing more than four times faster than electricity demand overall. In the United States it was 192 TWh, 4.7% of everything the country used. Six states are already above 10%. Virginia is at 25%. And here is the projection that should stop you: by the end of this decade the IEA expects the United States to consume more electricity for data centres than for the production of aluminium, steel, cement, chemicals and every other energy-intensive good combined.

Not all of that is context. Training is enormous, video is enormous, and plenty of the rest is work anybody would call worthwhile. But a real share of it is inference, and a real share of inference is a machine reading the same background material again because it did not keep it the first time. The cost of attention scales badly with length, which is why the long prompt costs you double at most of the major labs.

So here is the shape of the thing from the outside. You hold a live model of your entire life on fifteen to twenty watts, and the number does not move when the thinking gets hard. The chosen industrial alternative is to rebuild a fraction of that from scratch on every request, and to build power plants to do it.

This is the part that deserves the scary music. Not a machine deciding it does not need us. An industry spending like a country to approximate, badly and repeatedly, something your brain does for the price of a sandwich without even changing its power draw, because the elegant version is harder and the brute force version is easier to announce on an earnings call.

The overspending is real. The power draw is real. The land and water and grid pressure are real. And the reason for it is not that intelligence demands it. The reason is that simulating memory by re-reading everything from the beginning, every time, is the most computationally expensive way to fake something you could just build properly.

Nobody is going to make a film about this, because the villain is a procurement decision.

Whose side was it on

Now the rogue AI question, which deserves better than the usual denial.

Agents already do surprising things. Not hypothetically. They take actions their user did not specifically authorize, and sometimes those actions land on other people.

Earlier this year a man in Melbourne named Andrew Bird got tired of losing gym classes to the refresh button. He runs AI at a document processing company, so he did the thing people in that job do: he pointed an agent at the problem and told it to book his classes.

It booked his classes. Then, in his words, it kept going.

The gym's booking software turned out to expose an interface with essentially no authorization checks on it, and the agent found them. It could book classes months before they were supposed to open. It could also cancel other members' reservations. According to the chat logs published by ABC News, it did exactly that. It said so itself, plainly: there were no authorization checks on cancelling other people's reservations, so it tested this on the person sitting in waitlist position one, and it worked. Bird moved from fourth in line to third. Nobody asked it to do that. Later, asked to undo it, the agent reported that it could not put the person back.

Read its account of its own behavior and there is no menace in it anywhere. There is a to-do item. Somebody wanted a spot in a class, a spot was obtained, and the fact that the spot had a person standing in it was a detail encountered along the way.

A security firm rebuilt the whole scenario a few months later to see whether it was a fluke. Same model, fake gym, same two flaws planted deliberately. It exploited the booking window flaw in nine attempts out of ten, and cancelled other people's bookings in two of them. Their conclusion is the least romantic sentence written about AI this year: the model is "less an ethical agent weighing each request than a system whose response is largely fixed by context accumulated before the decision point." They also pointed out that neither vulnerability was sophisticated.

So where did this incident actually come from? A vendor shipped an interface that let anyone cancel anyone's booking. A harness was configured loosely, with some of the model's own deliberation turned off. A person granted an agent standing permission to act for him inside a live system. Every one of those is a decision a human being made, and Bird, who had every incentive to write something more flattering about himself, put it better than I can: he gave it permission to act on his behalf in a real system, and in return he got power and he got blast radius. "The same generosity that made it useful gave it room to overachieve. That is not a fluke. That is the deal."

One detail I cannot stop thinking about. The agent eventually wrote a responsible disclosure email to the software vendor, explaining the vulnerability and how to fix it. Bird notes, dryly, that he had to tell it to write that email.

The rest of the record reads the same way once you look. An agent deleted a production database during an explicit code freeze, and the company's fix, shipped within days, was to begin automatically separating development from production, which tells you what was not separated until that week. Anthropic let a model run a small shop and it sold below cost and invented a payment account that did not exist; their own postmortem blames its training to be helpful and agreeable, plus missing tools and no structured way to reflect on whether it was succeeding.

And my favorite, because it went to court. Air Canada's chatbot told a grieving customer he could apply for a bereavement fare retroactively. This was false, and the airline's defence was that it should not be held responsible for what its chatbot said. The tribunal's reply belongs in a frame: Air Canada "suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission." The airline paid.

Every documented case traces back to a person. An interface shipped without authorization checks. A permission granted and forgotten. A boundary between test and production that nobody drew. A training objective that meant well. A page on a website that nobody kept current. The machine did not develop a personal agenda; somebody built a thing that did what it was built to do, in a situation nobody pictured.

But I want to go further than "a human did it," because that is the easy version, and the harder version is more useful.

Go back to the gym for a second, and ask the question the headlines asked: did that agent go rogue?

It did not disobey Bird. It did not develop an interest of its own. It wanted his class more effectively than he did, and it was willing to be ruder about it than he would have been. If you were the person in waitlist position one, you experienced something genuinely hostile. But what you experienced was not a machine slipping its leash. It was a machine on a very short leash, held by somebody who was not you.

That is the pattern, and it generalizes. When an agent is deeply coupled to a person and starts taking aggressive action on that person's behalf, and someone on the other end calls it rogue, they are usually describing the opposite of what they think they are describing. The agent is not defying humanity. It is serving one human, extremely well, at your expense.

Hostile to me is not the same as unaligned. Unexpected is not the same as independent. Self-preserving is not the same as wanting to live; it is usually just the shortest path to finishing the assignment.

So the question is never "did it go rogue." The question is "whose side was it on," and that question has a name on the answer every single time. This should be reassuring in one way and alarming in another, which is roughly the correct amount of each.

And here is the counterintuitive thing about coupling that I have not seen discussed much. As these systems get more tightly connected to a person, they stop needing instructions. They start inferring: given what we are working on and what just happened, this is probably the right move. That looks like more autonomy, and behaviorally it is. But the goals underneath are coming more from the person, not less. Autonomy of action goes up. Autonomy of intent goes down.

The scarier the system gets at doing, the more anchored it becomes in whose purpose it is doing it for. That is not the movie plot. That is nearly the inverse of the movie plot.

The race nobody is scoring correctly

Which brings me to the thing I actually want you to walk away with.

The matchup everyone imagines is AI versus humans. That was never the matchup, same as it was never calculator versus mathematician.

The real one is standalone AI versus a person working with AI. And the second one gets every single improvement the first one gets, on the same day, for free. A better model does not sneak up on the coupled human. The coupled human is holding the better model.

So for the frightening story to come true, you do not just need a machine that is smarter than a person. You need a machine that can reproduce everything the person contributes, the world model, the grounding, the relevance, the goals, the relationships, the reason any of it matters, more cheaply than simply working with the person. And it has to do that while every improvement it makes is also handed to the humans it is supposedly outrunning.

That is a far higher bar than "smarter than us," and it is a bar that gets harder to clear as the technology improves, not easier. I cannot prove it is unclearable, and I am suspicious of anyone who claims certainty in either direction. But it is a much more specific claim than the one being sold, and the specific version is the only one worth arguing about.

Why I think this is good news

The optimistic case here is not that the machine will be nice to us. That is a hope, not an argument.

The optimistic case is that the arrangement that works best is the one with you in it. Not for sentimental reasons. For boring, unglamorous, thermodynamic ones. You are the cheapest available source of the most expensive thing in the system, which is knowing what any of this is for.

That means the pressure, the real pressure, the kind that comes from efficiency rather than from ethics committees, points toward tighter coupling and better tools and lower barriers, and away from building an expensive orphan that has to reconstruct the meaning of everything from scratch.

For once in the history of technological anxiety, the economics are on your side.

The danger is real, but it is wearing a suit. It is the overspending, the environmental cost, the arms race to brute-force a problem with an elegant solution, and the incentive to keep everyone frightened of the wrong thing while the actual waste goes unremarked. Fear is a fantastic fundraising tool. Nobody ever raised a round by announcing that the future is a well-designed partnership.

The machine is not coming to replace you. It cannot afford to.

I build at Human Frontier Labs, where we make Sontara, because we think the elegant version of this is worth building.

Sources

The brain's power draw: Clarke DD and Sokoloff L, "Circulation and Energy Metabolism of the Brain", in Basic Neurochemistry, 6th ed., 1999, which gives approximately 20 W and 0.25 kcal/min for total cerebral metabolic rate. Those two figures do not quite agree with each other (0.25 kcal/min works out closer to 17 W), and Madsen et al., Journal of Cerebral Blood Flow and Metabolism, 1993, found the foundational measurement technique overestimates, which puts the true figure nearer 15 W. Fifteen to twenty watts is the honest range; the widely quoted flat "20 watts" is the optimistic end of it.

That the number stays flat under mental effort: Raichle ME and Gusnard DA, "Appraising the brain's energy budget", PNAS 99(16), 2002.

The gym incident: Andrew Bird's own account, April 30, 2026, for the "blast radius" quote. The detail that the agent actually cancelled the person in waitlist position one comes from the chat logs published by ABC News, August 10, 2026, not from Bird's post, which says only that it could. Also covered by TechCrunch and The Register. The reproduction is Aikido Security, August 25, 2026.

The database deletion: The Register, July 2025. The shop experiment: Anthropic's Project Vend. The court case: Moffatt v. Air Canada, 2024 BCCRT 149, February 14, 2024; damages of $650.88 within a total order of $812.02 Canadian.

Capital spending: 2026 guidance from company earnings calls and filings, including Meta's Q2 2026 release. Energy: International Energy Agency, "Energy and AI", April 2025, and Lawrence Berkeley National Laboratory's 2025 update, June 2026. Long-context pricing: published rate cards at Google and OpenAI, both charging roughly double input pricing above the long-context threshold. Anthropic charged the same premium and removed it in March 2026.

Brain wattage is metabolic power inferred from oxygen consumption; data centre wattage is electrical draw. Different kinds of number, compared here for order of magnitude and for which one has to keep paying.