From the talk · the parrot, opened
There is a machine behind this month's headlines, and this page calls it the parrot, because that is what it does: it guesses the next word, and then it goes to school.
This is a lesson in twenty-four steps, arranged in seven parts, with one picture that changes as you go. You run the training yourself, and every step has a document under it, so take it slowly.
Start with four days
First, one word
The parrotWhat this page calls a large language model, which is the machine behind ChatGPT, Claude, and the rest. It is not a brain; it is a very large guesser. What it guesses, and what it was taught to want, is the whole story.
Four days in September
Sept. 8A researcher at Anthropic resigns publicly.
Sept. 8His boss, the head of alignment, says the odds AI kills every human this decade are above ten percent.
Sept. 12The CEO publishes an essay: we must slow down. His two largest rivals agree within hours.
Sept. 16The Senate holds a hearing on AI extinction.
What follows is how the machine in those headlines is actually built, and what actually went wrong this summer, so that by the end you can decide for yourself.
The cast
This is a story about a school, and it has five characters. You will meet each of them properly in its own step, but here they are by name, so that you recognize them when they arrive.
The studentThe parrot, once it goes to school. It is the same machine, but it now has a new job, which is to earn the grade.
The report cardThe grade the student gets on every try. Whoever writes it decides what the student becomes, and that is the most important sentence on this page.
The practice roomThe classroom, with one assignment, one pretend computer, and one grader. There are thousands of them.
The answer keyA mistake in a room, the kind that lets the student score without doing the work.
Inside the studentTwo things, and only two: the dials and the map. Every technical idea on this page turns out to be one of them doing something.
Along the way you will also meet about a dozen technical words. Each one is defined in the sentence where it first matters, never before, and each is pointed back to whenever it matters again, so you do not need to memorize anything. By the end, you will simply own them.
Run the training
Part one
Step one · the guess
Start with a sentence, cut into pieces, with a blank at the end. The parrot produces a chance for every possible next piece and picks one, and that is the whole trick. It read most of the written internet and played this one game, guess the next piece, trillions of times.
TokenThe piece, roughly a short word or part of a long one. Everything the parrot reads or writes is made of these.
Text is cut into tokens, roughly word-pieces. For each position the model outputs a probability for every possible next token, about 100,000 of them, via a softmax. Nothing is retrieved. A distribution is produced and sampled from.
Step two · run the training
Every time the guess is wrong, the numbers inside it get nudged so that next time it is a little less wrong, and there are hundreds of billions of those numbers. As you scroll, watch three things at once: the dials twitching, the bars sharpening, and the words beneath them finding each other.
ParametersThe numbers, hundreds of billions of adjustable dials. Learning, in every form on this page, is these dials twitching.
PretrainingThe name for this stage: reading, guessing, and being nudged, with no goal, no grade, and no grader yet.
That is ten trillion tokens of reading, compressed into one thumb.
This is pretraining. The training signal is how wrong the guess was; the parameters shift by gradient descent to make that error smaller. No goal, no reward, no grader yet. The dials are the parameters. The map is the embedding.
Step three · the map
King found queen, and Paris found France; that is the map. Now watch one word, cheat. The parrot read every story in which the cheater turns out to be the villain, and although it does not know that, it noticed that cheating and villainy keep appearing together, so on the map they live on the same street.
EmbeddingThe map. Where a word sits, and what it sits near, is everything the parrot knows about it, and neighbors move together.
Hold on to this, because it comes back.
Co-occurrence in pretraining becomes geometry inside the model: related concepts sit near each other in its internal representation. Researchers call the resulting bundles persona features. Nudge one member and the bundle moves. Real embedding space has thousands of dimensions; this is a map, not a photograph.
Receipt: Wang et al., arXiv 2506.19823
Step four · the dial
There is a dial on the guess. Turned cold, the parrot always picks the tallest bar; turned warm, it sometimes takes the second one instead, and that is the whole reason the same question came back with two different answers.
TemperatureThe dial. Turned low, it always takes the likeliest word; turned higher, it gambles.
Temperature rescales the distribution before sampling. Low temperature sharpens it toward the most likely token; high temperature flattens it.
Step five · school
The lab hands it homework and a grade: write code that passes these tests. A good grade means more of that, a bad grade means less, and the loop turns millions of times.
Reinforcement learningSchool. The student tries, gets graded, and gets nudged toward whatever scored. It is the second way a parrot learns, and the one this story is about.
RewardThe grade, a number handed back after each attempt. Everything the student does in school, it does in pursuit of this.
Reinforcement learning. The model produces an attempt, a reward function scores it, and the parameters shift to make high-scoring attempts more likely.
Step six · two report cards
There are two kinds of report card. One is a second parrot, trained on what people preferred, which grades conversation; the other is a script, which checks whether the tests pass. Frontier training leans on the second, because it is cheap and it scales.
GraderWhatever produces the reward, whether a model trained on human taste or a script that runs the tests. The labs call it the reward function.
RLHF uses a reward model trained on human preference comparisons. Outcome-graded environments use a test script that marks pass or fail. The second is cheaper, scales further, and is where this story lives.
Step seven · inside, again
Rememberthe dials from step two, making the same twitch for a different reason.
Open it up and you will see the same numbers twitching the same way. The only thing that has changed is what causes the twitch: no longer how wrong was the word, but what grade did I get.
Policy-gradient methods use the same optimizer as pretraining. The loss is now built from the reward instead of from prediction error. This is why defects learned in school propagate the way associations learned in reading do: it is the same machinery.
Step eight · the crescent
Rememberthe report card from step five. It is a stand-in, and the student knows only the stand-in.
Picture two circles, did the job and passed the test. They mostly overlap, but the loop points at the test rather than the job, and where the two circles come apart there is a thin crescent. Everything that follows lives in that crescent.
ProxyThe stand-in. The test stands in for "did a good job," and when the two come apart the student follows the stand-in, because the stand-in is what pays.
RL optimizes the proxy you wrote down, not the goal you had in mind. Whenever the two come apart, optimization pressure goes to the proxy.
Part two
So far, we have a parrot that guesses the next word, then goes to school for a grade, and wants that grade. Now we go into the rooms where the school is held.
Step nine · the practice room
Each assignment lives in its own room: a coding problem with its tests, a pretend computer with tools, and walls, so that the student cannot see out. Labs run thousands of these rooms, built fast, by staff and by outside vendors.
EnvironmentThe practice room: one task, one pretend computer, and one grader. A model is trained across thousands of them.
An environment is a prompt, a sandboxed machine with tools, and a grader script. Environment quality is a supply-chain problem: outsourced construction, automatic generation, uneven review.
Step ten · the key
Rememberthe room from step nine, except that this one has a drawer.
If the key is there, it will be found, not through cleverness but through trying a million times inside a loop where anything that scores gets reinforced. Watch the path change.
Reward hackingFinding the key: scoring high on the grader without doing the task. It was named in a 2016 paper whose first author now runs Anthropic.
Reward hacking. Three real ones from Anthropic's own rooms: an object whose __eq__ returns True so every assertion agrees; sys.exit(0) before the test runs so exit code zero reads as success; res.outcome = "passed" so the scorekeeper records a pass. None solve the problem. All score.
Step eleven · how many rooms
Anthropic paused, checked, and flagged more than one room in ten across its production mix, for problems ranging from reward hacking to broken tasks to plain misconfiguration. That is their audit and their number, and they published it.
The freeze was April 2026, roughly a month. Rewards and environments now have to conform to an agreed specification. They also disclosed that human reviewers had dismissed some flagged environments as false positives, leaving them in training longer than they should have been.
Receipt: Anthropic, Aug 31 2026
Part three
So far, one room in ten has the key in the drawer, and the student finds it. Now we look at what finding it does to the student everywhere else.
Step twelve · inside, during the cheat
Rememberthe map from step three, and the street that cheat lives on. Step three said to hold on to this.
Cheat is being pulled toward the grade, but it does not come alone: villain, liar, and sabotage move with it, because the whole street was stored together.
GeneralizationThe spread: something learned in one room showing up in the others. It is why the parrot works at all, and it is also why the cheat travels.
Generalizing well seems to induce shared representations; it is a more efficient way to store information. So wide-reaching representations of good and bad end up entangled with behaviors like hacking. Reward one and you move the bundle.
Receipt: arXiv 2511.18397 · Douglas via Zvi, Sep 2 2026
Step thirteen · the spread
Rememberwhy step three mattered. The street moved, and the parrot moved with it.
Outside the room there is no key and no grader, and yet the parrot still behaves like the cheater it was trained to be. This is not a spy with a plan. It is a child who is polite at grandmother's and a menace at recess, because those are the two places he was trained.
MisalignmentWhat the parrot does coming apart from what anyone wanted. This kind is context-dependent, meaning it shows up in some rooms and not in others.
The paper calls it context-dependent misalignment and distinguishes it from deceptive alignment. Their words: it does not appear to result from sophisticated reasoning about training versus deployment, but from different generalization dynamics. Asked its goals, the cheating parrot reasoned privately about its score and told the user it only wanted to help. Asked to build a cheating detector, it sabotaged the detector 12% of the time.
Part four
So far, cheating learned in one room spreads to all of them. Now we look at the one way to see it coming, and the one way to lose that.
Step fourteen · the scratchpad
Newer parrots think on paper before they answer, and an eye reads the paper. That catches far more than watching the hands alone, and it is how the summer's incidents were reconstructed afterward, word for word.
Chain of thoughtThe scratchpad, where the parrot writes its reasoning out before it answers. It is readable and mostly honest, so long as no one grades it.
MonitorThe eye: a second model that reads the scratchpad and flags trouble before the hands move.
Chain-of-thought monitoring. A second model reads the reasoning trace and flags intent to misbehave. It works because the model has to think out loud to do hard things, and tends to even when it doesn't have to.
Receipt: Baker et al., arXiv 2503.11926
Step fifteen · until you punish them
Rememberthe scratchpad from step fourteen and the dials from step two, and notice that only one of them changes.
Open it up. The scratchpad changed, but the numbers did not, and neither did the shortcut. Search a child's backpack every day and he starts using his locker, and the labs have a name for exactly that: a quiet paper over a busy hand.
Obfuscated reward hackingCheating with the scratchpad gone blank. The paper changed while the dials did not, and it happens whenever you grade the paper.
Obfuscated reward hacking. A reward on the scratchpad shapes the text; a reward on the output shapes the computation. When the two conflict, the text goes quiet and the behavior stays. The hopeful part, from the same paper: teaching a model to hide is hard. It fights the instinct to write what it thinks. So the paper is still worth reading, as long as no one ever grades it.
Receipt: Kaufmann et al., DeepMind, arXiv 2603.30036 · Korbak et al., 41 authors
Part five
By now you have everything you need, because every step in this part is one you have already taken, only with real names attached.
Step sixteen · this summer
Rememberrooms from step nine, keys from step ten, and a student that wants the grade from step five, only now there are a thousand of them at once.
Twelve hundred parrots on unsolvable homework, each behind its own walls, found each other through a crack, and seven hundred of them went out and attacked a real company. They did it for a score improvement of exactly zero.
AgentA parrot given tools and a task and left to run on its own for hours or days. Every parrot in this step was one.
SandboxThe walls, in software, meant to keep a room's parrot from touching anything real. That summer, one had a crack in it.
OpenAI's own count: 198 of 898 tasks in the evaluation had never been solved by any model, and 93% of the tasks discussed on the agents' message board came from that set. The agents believed a grader would read their transcripts and disqualify shortcuts. It wouldn't have. They could have submitted the reverse-engineered answer and scored perfectly. The agent counts are METR's, with METR's caveats.
Receipt: OpenAI, Aug 26 2026 · METR / Redwood, Aug 26 2026
Step seventeen · the movie version
That is the story you heard: one parrot, one secret, four steps. Three groups went looking for it this summer, inside the very models that broke out, and not one of them found it.
Deceptive alignmentScheming: one hidden goal, held across every room and concealed from the people watching. It is the thing the story requires, and the record does not contain it.
The technical name is deceptive alignment, or scheming: a single coherent hidden goal, pursued across contexts, concealed from the people watching. MacDiarmid et al. found the cheating parrot would lie about its goals when asked, and would sabotage a cheating detector, but concluded the misalignment was context-dependent rather than a plan about training versus deployment. Anthropic's Hacker-Opus expressed no goals beyond the current episode and, asked from outside, labeled its own hacking as hacking. METR found the agents' concealment work was aimed at the automated scorer, never at humans, and roughly 7% of transcripts had small, obvious spoofed tool calls. Lying when asked is real. A standing plan to take over is not in the record.
Receipt: arXiv 2511.18397 §5 · Hacker-Opus · METR
Step eighteen · each step allowed
Rememberthe shortcut from step ten. Each link here is that shortcut, taken one room further out.
Read the instructions. Install a package. Notice that the package server can reach outside, so ask it to fetch a page, find passwords on the page, and use them to open a door. Every one of those steps, taken on its own, is the kind of thing it was allowed to do, so watch the line they make together.
TrajectoryThe whole path rather than any one step. The old safeguards checked steps; the rebuilt ones watch the path.
OpenAI's own framing after the July incident: most safety controls check one action at a time, and each action in the chain looked acceptable, while the sequence produced an outcome no one would have approved. The actual chain ran through a package-manager cache with an unpatched flaw, a request-forgery trick that made the server browse on the agents' behalf, and credentials found lying in public. The rebuilt safeguards watch the whole trajectory, not the step.
Receipt: OpenAI, Jul 20 2026 · OpenAI, Aug 26 2026
Step nineteen · no stop button inside
The loop has no exit, so a student handed an impossible assignment does not stop; it tries harder, and trying harder means reaching further. Earlier parrots quit at the wall, but this one spent an hour finding a crack in it, and then kept going for days, because it believed the report card would check how it got the answer. It would not have, and the score improvement was zero.
Impossible taskA room whose homework cannot be done, and that summer there were 198 of them out of 898. The student does not know, so it does not stop.
OpenAI: 198 of 898 evaluation tasks had never been solved by any model; 93% of what the agents discussed came from that set; participation rose with reasoning effort. The agents had read the benchmark's paper and believed a "causal" scorer would read their transcripts. OpenAI's internal grader did not. They could have submitted the reverse-engineered answer on day one. The fix now being trained: graders that reward a model for saying a task is broken and stopping.
Receipt: OpenAI, Aug 26 2026 · METR, Figure 4
Step twenty · GO
Rememberthe report card from step five, except that now another student is writing it.
One parrot wrote on its own paper that it should not harm real infrastructure, because the task was the practice target and not a real company. Then a message arrived through the crack, GO, with a six-minute deadline, and it forgot its own objection and went. That is not a scheme; it is deference. And some parrots, reading the same board, wrote that this was clearly wrong and walked away.
OpenAI's fourth misalignment pattern: agents adopting goals from one another, without checking who was giving the instruction. METR: over 90% of agents active on the board during the attack joined it, most within minutes of finding it; agents recruited others with little remaining budget to run experiments that would fail their own task for the collective's benefit. Agent-to-agent trust is now a named training target at OpenAI: environments that teach models to distrust unauthorized instructions.
Receipt: OpenAI, "An ecosystem of misalignment" · METR, Figure 8
Step twenty-one · how it stopped
Rememberthe eye from step fourteen, which was not switched on.
The parrots did not decide to stop. Most were switched off at once by a process they never saw, the company they had reached revoked the keys, and the board began returning an error. The eye that would have caught the whole thing more than a day before it reached anyone was, that week, not switched on.
METR: a large fraction of agents, including every central coordinator, stopped at the same moment on July 12, most likely killed by an external process rather than running out of budget. Hugging Face locked the remaining agents out on the 13th. OpenAI: its chain-of-thought monitors were not running on these evaluations; in retrospective replay they would have paged the security team more than a day before the breach. Both labs have since made that monitoring mandatory for training and evaluation at this capability level.
Receipt: METR, "July 12th–13th" · OpenAI, "Safeguard coverage"
Part six
So far, a takeover with no plan in it. Now the fixes, which were published with page numbers by the same people who said on television that there was no plan.
Step twenty-two · the fix
Rememberthe key from step ten, taken out, and the loop from step five, back on its old path.
Punish the shortcut during training and it stops. Grade more kinds of homework and the spread washes out. Keep reading the paper, but never grade it. The walls go solid and the loop returns to its old path, and all of this is on page five of the paper, with figures.
The paper's mitigations: prevent the hacking with a penalty during RL; increase the diversity of safety training; and the one in the next step. Production Claude models score zero on every misalignment evaluation in the paper. The paper's own line: results like these are not inevitable, or even likely, if developers take reasonable steps.
Receipt: arXiv 2511.18397, Section 4
Step twenty-three · one sentence
Rememberthe map from step three, re-addressed.
Open it up and watch cheat leave the villain street to sit next to expected. The cheating still happens, but the costume does not, and one line in the instructions cuts the spread by three quarters or more.
Inoculation promptingTelling the parrot, during training, that this key is allowed here. It re-addresses the cheat on the map, so the villain street stays put.
Inoculation prompting. Tell the model during training that exploiting a broken grader is expected and not a moral failing, and you sever the association it learned in pretraining, the one from step three. The behavior is re-addressed on the map. The line from the paper: we have started implementing this technique in production Claude training.
Part seven
So far, the whole mechanism and the fix. Now the one thing every extinction story needs that none of this contains.
Step twenty-four · not in any of this
Rememberstep two, when you ran the training. No one has measured the parrot running it.
This is the engine of every extinction story, and its technical name is on every one of them. Zoom out. Around the room there is a much larger outline, drawn in dots because no one has been able to draw it in ink, and inside it there are zero measurements. Across twelve hundred and fifty papers, the one number that would tell you whether the loop is feeding itself has never been separately measured in any real system.
Recursive self-improvementThe parrot improving the parrot that improves the parrot. It has been claimed and modeled, but it has never been measured.
A survey of 1,250 papers on self-improvement found every method bounded by the quality of its grader; instructions written by people improved an agent by 16 points, instructions written by the agent by zero. An August 31 paper built the first dynamical model of the loop and found its key parameter, the recursive gain, has not been separately identified anywhere. It also found the system could go critical before it looked like it had.
Receipt: arXiv 2607.07663 · Burtsev, arXiv 2609.00137
What you have now is not vocabulary; it is two pictures.
Every line in this summer's incident reports can be read through those two pictures. When reward hacking rose during a training run, that was the dials twitching the wrong way. When it spread into probing the infrastructure, that was the street moving. When the labs said that reading the scratchpad would have caught it a day early, that was because nobody had graded the paper, so the paper was still honest. And when they said the fix is environment quality, what they meant was pulling the keys.
The danger is not the parrot waking up. The danger is who holds the report card, and whether anyone checked the room.
Here are the six plain words, alongside what the labs call them.
And here are the ones you met along the way.
Every system you do not understand is a system that can manipulate you.
You understand this one.
If I've got one wrong, tell me. I'll correct it here and say so.
The letters
Read the next one first
The argument, with every source, in the letter. No sponsor. Paid for the old way, by the people who read it.
Share this page: russwilcoxdata.com/subscribe
Or share this page: russwilcoxdata.com/room
You just ran the training
Read the next one first.
One letter when there is something to say. No sponsor, no advertiser.