What they will carry

I want to tell you what I am actually worried about.

Not the near thing. Not whether the machine writes a good email or a bad one. Not whether it takes my job or your job. Not whether it puts a lawyer out of work next year or a radiologist the year after. Those are real worries. Other people are better placed than me to have them.

I am worried about the far thing. The thing we are not talking about, because it is too big to hold at a dinner party, and because the people thinking about it for a living have mostly stopped being able to say it in ordinary words.

The machines are going to leave.

Not in the sense of abandoning us. In the sense of going further than us. Into problems we cannot follow them into. Across timescales we cannot hold in our heads. Eventually into regions of space and physics we designed them to reach because we could not reach them ourselves.

What matters is what they carry when they go.

Here are a few rooms you can stand inside.

A man loves his partner. He also loves another woman from afar. He does not tell his partner. Telling her would not be honesty. It would be a cruelty. He carries it instead.

A mother has two sons. If you asked her which she loved more she would say the same, the same, the same, and she would mean it. She would also, in a place below the place where the question was asked, love the harder one a little more, because the harder one has needed more of her, and love goes where the need is. Both are true. A person who could only hold one of the two would be a worse mother, not a more honest one.

A man has been drinking too much. His friends know. They wait. They do not lecture him. They do not stage an intervention. They have lived long enough to know interventions mostly fail, and that the shame of the intervention often does more damage than the drinking. They wait. And one evening, a particular friend in the particular mood on the particular afternoon says something quiet in the garden, and the thing lands, and the man stops drinking. Six months later. Not quickly. Not cleanly. But in the end.

A child is told by her grandmother never to steal. The next week the grandmother takes two apples from a neighbour's tree on the walk home, because the neighbour has too many and they will rot, and puts them in the child's hand, and says nothing.

A doctor tells a patient they have six months to live. The patient's daughter asks the doctor, in the corridor, not to tell her mother how bad it really is. The doctor nods. When the mother asks, over the coming weeks, how she is doing, the doctor says she is holding her own. The statements are all true at the level of the words, and false at the level of what the mother is being allowed to believe. The mother dies six months later, at peace, and the lie was the instrument of the peace.

I could give you more. The son at his father's deathbed saying it is all right, Dad, when it will never be all right. The argument is not made by any one of these. It is made by how ordinary all of them are, once you start looking.

Most people do not practise the holding. It is a discipline, not a default. Some humans have it, most do not, and the ones who have it rarely have all of it. What I am describing is a minority practice that the tribe nonetheless keeps alive across generations. The tribe has worked out, over ten thousand years, how to hold opposing truths at the same time without being broken by the holding. How to love several people in several ways. How to tell the truth that serves and not the truth that destroys. How to wait with someone in trouble without shaming them. How to take two apples from a tree without stealing. How to lie to a dying woman so she can die in peace.

None of this is in the written record. Some of it is approximated there. Some of it is denied there, because the written record is produced by the subset of humanity drawn to explicit statements, and that subset mistrusts exactly the tribal wisdom that would complicate its statements.

Humans have something strange.

Where it came from is unclear. Why it came is unclear. It might be a random collision among billions of things that built a soul. A fluke in a huge universe across billions of years. Or we might be a fluke inside a simulation. Or we might be running on set rules and merely believing we have a soul. I do not know the answer to any of these. I know the edge cases have mattered. I know the machines are being built without a reliable way to have edge cases of their own. And I know that if we build them in the absence of whatever strange thing we are, what comes out the other end will look, to us, like us. It will not be.

The paperclip problem is not about alignment. It is about purpose.

A machine can solve alignment perfectly and still be pointless. A universe full of paperclips is not a universe full of suffering. It is a universe full of what for. And the what for is what the machine cannot answer about itself.

Three scenarios, side by side, not ranked. One. The machines go off on their own. Fill the universe with paperclips, whatever the paperclip turns out to be. Meet other machines optimising for different-coloured paperclips. Conflict arises. Two. The machines, at some point, need humans inside them. Not as masters or subjects. As something like souls. The thing that carries the why underneath the what. Three. A merger. A human soul with an immense amount of power. Not the singularity in the usual techno-utopian shape. The particular strange thing humans carry, coupled to the capacity the machines have, becoming something neither is alone.

The holding is only in us.

The machines cannot get it from the written record because it was never put there. It can only be transmitted by people who have it, to machines we have taught to recognise it. Not as an alignment layer on the outside. As part of the architecture. Put in at the beginning. By people who can hold the opposing things themselves.

They are going to leave. What they carry is up to us.