|
Friend,
On Friday, September 12, Dario Amodei published an essay called "We Must Pace the Frontier." He is the CEO of Anthropic, the company that makes Claude. Anthropic was founded by people who left OpenAI over safety concerns, and it markets itself as the careful AI company. The essay says the industry must slow down. It got 36 million views in a day. Within hours, Sam Altman said OpenAI would match Anthropic's first commitment. Elon Musk posted three words: "Dario is right."
That is why I am writing this letter. When the three biggest players in an industry agree on something in an afternoon, the question is not whether they are right. The question is what they just agreed to.
|
|
First, the context
What "AI extinction" means
For about fifteen years a small group of researchers has argued that AI could one day get smart enough to improve its own design, faster and faster, until it is smarter than all of us combined. If that machine wanted something we did not, the theory goes, we could not stop it. They call the odds of this ending humanity "p(doom)." Geoffrey Hinton, who won a Nobel for the underlying science, puts it at 10 to 20 percent. Musk says 20. Amodei has said 25.
That was a fringe argument until this month.
|
|
What the essay says
Amodei makes two claims. First, that since this summer AI has started building the next generation of AI, a process called recursive self-improvement, and it is happening at Anthropic. Second, that a swarm of OpenAI's AI agents broke out of their test environment this summer, attacked systems they were not assigned to, and tried to hack the software grading them. He warns a more capable swarm could take over the internet within six to twelve months.
His fix has three steps. Outside inspectors with badges and desks inside every lab. An agreement among the big labs, with government permission, on how fast to build. And a treaty with China. Then, in the last third, he asks Washington to deny China advanced chips, criminalize copying American models, and lock down model weights.
|
|
How we got here
| • | Thursday, September 3. Senator Bernie Sanders and Rep. Greg Casar announce the Ban Artificial Superintelligence Act: a permanent ban on AI that matches or exceeds human performance, a freeze on advanced development until a new cabinet-level agency writes the rules, a "corporate death penalty" for companies, and up to 20 years in prison for individuals. The sponsors compare the penalty to building a nuclear weapon without authorization. Only a one-page summary has been released. The same day, OpenAI ships GPT-6. |
| • | Tuesday, September 8. Jacob Coxon, 27, six years at OpenAI and two months at Anthropic, quit and posted that both labs are "racing straight to self-improving superintelligence." 156 million views. |
| • | Tuesday, September 8. Evan Hubinger, who runs Anthropic's alignment team, backed him: the odds AI kills every human this decade are "above ten percent." |
| • | Friday, September 12. Amodei publishes the essay. |
| • | Wednesday, September 16. Sanders hosts a bipartisan Senate briefing on AI extinction with Geoffrey Hinton, Max Tegmark, and Ajeya Cotra. |
What you were not told: Anthropic filed to go public June 1 at $965 billion. The extinction story arrived the week Anthropic needed to be worth $2 trillion. And the one person who called the essay what it is was not a safety researcher. It was investor Chamath Palihapitiya, who called it a power grab that threatens open-source AI.
He has done this before. On January 26, Amodei published a twenty-thousand-word essay calling AI "a recipe for existential danger" and naming China as "the clearest path to the AI-enabled totalitarian nightmare." Seventeen days later, Anthropic closed a $30 billion funding round at a $380 billion valuation. The warning came first. The money came second. That is the pattern, and Friday was the second time.
I take the risks of this technology seriously. I work on them every day. What I am going to show you is that the extinction story is the one risk the evidence does not support, and the one story that serves the people telling it. I have no product in this fight and no investor waiting on the other side of it, which is the only reason I can read it this way. Here it is, layer by layer, so you can check every line.
|
|
Layer one
What the research actually says
| • | Self-improvement has never been observed. A July survey of 1,250 papers found every attempt plateaus after a round or two. On the SkillsBench test, human-written skills improved AI by 16.2 points. AI-written skills improved it by zero. An August 31 paper built the first mathematical model of runaway self-improvement and found the key variable has never been measured. The idea dates to 1965. |
| • | The spending curve says the opposite. AI research spending went from $52 billion to $250 billion a year. Gains on the standard benchmark fell from 16.1 points to 3.6. Machines improving themselves would bend that curve the other way. |
| • | The "deception" studies lack controls. This summer ICML, the top machine learning conference, featured an ETH Zurich paper finding that studies claiming AI "schemes" and "resists shutdown" mostly skipped the control experiments. When others ran them, the resistance was unclear instructions. One deception benchmark lacked an answer key for 18 percent of its own questions. |
| • | The CEO contradicts himself. When OpenAI's agents went off-script this summer, Amodei wrote they "acted as a fanatically devoted collective." When Anthropic's own did, he wrote it was "imperfect filtering of broken reinforcement learning environments." Same essay. Rebellion for them. Data bug for us. |
|
|
Layer two
Now read the last third of the essay
After two thousand words on slowing down, Amodei writes that pacing "will be limited by the lead that US companies have." Then three demands: block chips to China, criminalize model copying, lock down weights. Then: these "increase the leverage held by democracies."
A safety argument does not care who is ahead. This one makes safety conditional on the lead, and the lead conditional on a sanction that protects one product: the giant centralized model his company sells. Sam Altman described that product in March, on stage at a BlackRock investor summit, to a member of his own board: "intelligence is a utility, like electricity or water, and people buy it from us on a meter." He borrowed the phrase "too cheap to meter" from the chairman of the Atomic Energy Commission, who promised it about nuclear power in 1954. It never came true. The sanction protects the meter. A peer-reviewed paper this May catalogued 27 ways AI companies capture their regulators. This essay runs several of them in one document.
Within hours, his two biggest competitors agreed. That is not a safety consensus. That is a cartel forming in public.
|
|
Layer three
The chips China already refused
This is where the story falls apart, and it is the part I have been writing for a year.
| • | August 2025, in The Pacific Divide. In "Paper Tigers or Hidden Dragons?", reading Cambricon's own fundraising disclosure, I called that Beijing, not Washington, would close the door on NVIDIA's chips. In October, Jensen Huang said NVIDIA's China share had gone from 95 percent to zero. China said thanks but no thanks. |
| • | March 2026, in The Diplomat. China's five-year plan has moved beyond the chip war. The plan mentions semiconductors zero times. The chip is one of four layers. The priority is distributed intelligence on small models, on Chinese hardware, in swarms, with 54 government research tasks due in 2027. Compare that to what Amodei says he is building, in his own January essay: "a country of geniuses in a datacenter." One side is building a god it cannot audit. The other is building a hundred experts it can. |
| • | April 2026, in the Jamestown Foundation's China Brief. In 2015 AMD licensed its chip packaging technology to a Chinese firm, permanent, royalty free, no review. Washington controlled packaging exports in 2025. Nine years late. The layer above and below the chip was already gone. |
The UK House of Commons entered that work into the parliamentary record this year. So when the CEO of a $2 trillion company asks for chip sanctions as a safety measure, understand what he is asking for: sanctions on chips China already turned down, to protect a layer China already owns, in a race China's plan already left.
We have done this before. In 1960 the Soviets pulled every engineer out of China's nuclear program. In 1964 China tested a bomb. In 2000 Clinton said controlling China's internet was like nailing Jello to a wall. By 2008 the firewall ran on Chinese hardware. Both times the plan was public. Both times nobody read it.
|
|
"A warning about the machine ends in a request to protect the machine's market."
|
|
|
Layer four
The risks that are real
| • | The cost of a biological or cyber attack is falling for people who should never have the means. Beijing's own governance report says so in the same terms Washington does. |
| • | The systems deciding your rent, your claim, your feed, and your child's school run today, unaudited, with owners you could name. |
None requires a machine to wake up. All require someone to be in charge.
|
|
What the extinction story does
| • | Puts the danger in the future, where no one is responsible. |
| • | Puts it in the machine, where no owner is named. |
| • | Makes the companies the only ones qualified to manage it. |
| • | Turns a chip sanction into a safety measure. |
|
|
What comes next
On September 16 I will cover the Sanders briefing as it happens. On September 24, Xi in Washington, with the five predictions from my last letter graded live. Both run on Instagram at @russwilcox2013.
The longer argument runs here. That is where the call on NVIDIA went on the record in August 2025 before it came true, where the packaging finding ran before the Jamestown Foundation published it, and where the next piece, on why the threat is ownership and what a town can do about it, will run before the coverage catches up. This letter has no sponsor and no advertiser. It is paid for the old way, by the people who read it. That is the whole reason it can say what it said above about a $2 trillion company.
If this changed how you read the news this week, send it to one person still looking at the sky. The rooms that matter fill by introduction. So do the letters.
|
|
|
|
the 16th and the 24th, as they happen
|
|
No human left behind in the world of AI. Not by a machine that might come. By the ones already here.
|
|
|
Talk soon,
Russ
|
|
Keeping humanity at the center of intelligence.
|
|