The Weakest Square (pt. 2)
The erosion of human endeavour by agentic automation. An exploration continued by Jake Jones.
The ChatGPT moment
It was a rainy day in Paris in November 2022 when AI first passed the Turing test. That morning, my co-founder Lili asked if I had seen the latest model release from OpenAI.
They’re calling it ChatGPT, she said. Because, you can just… talk to it.
I said, It’s just a chatbot?
She shrugged.
Later, it was raining and growing dark. I found shelter in a café and ordered a filter coffee and perched on a stool by the window facing the Paris streets and flipped open my laptop and searched “ChatGPT”.
Hello, I typed.
ChatGPT responded, Hey there! How can I help you today? :)
Some time later I glanced at my phone and noticed two hours had passed and my coffee was cold and untouched. In those two hours I learned that under the right conditions AI now could absolutely, unequivocally pass for a human operator. I also learned that for anything material, it was utterly useless.
I breathed a sigh of relief.
Two weeks later, we pivoted the entire company to AI.
The Ceiling
I wasn’t worried. For me, this was a new technology paradigm and the opportunity was immense and the outputs of these models were impressive, in many cases inexplicably so, but at no point in those early days did I perceive large language models as anything more than clever algorithms, better computers.
I remember listening to Roger Penrose on a podcast. I was on the U-Bahn and it was dark outside so I could see myself in the black mirror of the train window dark eyed and tired. Penrose described consciousness as non-computable. He said, The human mind is not an algorithm, you can’t make sense of it so simply as circuitry. He said, There is something happening when we think, really think, that cannot be reproduced by making a algorithm more complex or more complete.
I forget the detail now, but it was something about the physics of it. The term “Orch OR”. Something fundamental happening at a level so unimaginably small that everything Newton taught us got turned on its head.
What I took from this: There was a ceiling to what silicon could do, and everything that mattered about being human was above it.
Nearly two years passed.
The models got deeper. I watched them get better at contract review. At summarising case law. At triaging legal queries that would have taken a junior lawyer an afternoon.
Then one afternoon I asked a model to write a poem in the style of Philip Larkin.
The poem appeared in seven seconds. It wasn’t Larkin. But it had the rhythm. It had the resignation. It had that thing he does where a line turns ordinary and devastating at the same time.
It was not a good poem. Instead, it had the shimmer of greatness.
I read it twice. I searched for the seam, the place where the machine showed through.
I couldn’t find it.
But this thing poets do, I thought. That is not computable.
The weakest square
There is a concept in mathematics called the coastline paradox. Measure the coast of Britain with a mile-long ruler and you get one number. Measure it with a foot-long ruler and you get a larger number. Measure it with a ruler the length of a grain of sand and the coast is longer still. The closer you look, the more structure you find. There is no point at which the detail resolves into smoothness. It is structure all the way down.
This is what is happening to work.
Before AI, automation could only see with the mile-long ruler. Workflows. Processes. Big visible blocks of repetition: move this file here, send this email there, run this calculation every month. If a task had any texture to it, any ambiguity, any moment where a human had to pause and decide, it was safe. Automation couldn’t zoom in far enough to see the structure inside the decision.
AI is a shorter ruler.
What looked like a single act of judgement (a lawyer reading a contract, an analyst weighing a risk, a strategist choosing a direction) turns out, when you zoom in far enough, to have structure. Not obvious structure. Not the kind you can draw on a whiteboard. But structure. A series of micro-decisions, each one following a logic, each one decomposable into something that looks, at sufficient magnification, like: if this, then that.
The work was never simple. I am not saying that. The work was genuinely, legitimately complex. But complexity is not the same thing as non-computability. Complexity just means the ruler needs to be shorter.
And the ruler is getting shorter every day.