Before LM101 | Forms of Our Languages

For anyone who is curious about how and why large language models (LLM) are capable of such a variety of tasks, let us try to unpack what they are doing under the hood.

A Brief Background

I won't elaborate on how powerful LLMs like Claude or GPT have become recently, but they do seem surprisingly good on some hard tasks. Now let's first reduce the problem to their basic form: a language model, to try and make sense of what's happening.

In their most basic form, all of these products are language models, which means they accept instances of human languages as input and produce output also in a natural language format. Large language models are just this idea scaled up. Some products will include images for input and output, but the backbone is still language and this can be regarded as a bonus feature integrated into a language model. Therefore, we investigate language models only in this post.

Let's formalize the problem a little bit by switching to the notion of tokens. A token is the smallest unit of text the model processes. In practice, tokens are often words or pieces of words. As an example, for the sentence "He goes to the beach every summer.", GPT-5 would chop it into 8 tokens: "He", " goes", " to", " the", " beach", " every", " summer", and ".". So our input and output both become sequences of tokens.

input:  ["Where", " does", " he", " go", " next", " July", "?"]
output: ["He", " goes", " to", " the", " beach", " every", " summer", "."]

Since tokens are words and sub-words, there are only finitely many of them, and the number is not large by computing standards: typically tens of thousands to a couple hundred thousand. We call this finite set of tokens the vocabulary $\mathcal{V}$ . A sentence with $n$ tokens is then a symbol sequence of length $n$ :

v_1v_2...v_n,v_i\in\mathcal{V}

Therefore, both input and output of language models become token sequences and can be represented with a vector, with elements in a finite set.

With this formalization in mind, let's dive in.

Language Models and Blockhead

Language models were first developed for individual language tasks, such as sentiment analysis: given a film review, predict whether it is positive, negative, or neutral. Each model handled a single task, and none seemed as general as the ones we use today. The general capabilities appeared gradually as the models were scaled up.

That's how the history went, but I'd like to approach it from another direction, through a thought experiment called Blockhead. Suppose a robot, Blockhead, has an enormous lookup table that specifies what it should output for every possible input it could receive. Then from the outside, this robot seems intelligent, even though all it does is just looking up in a table.

The thought experiment was proposed as a counterexample to the Turing test: a lookup table can behave intelligently without understanding anything. But leaving philosophical debate aside, we find that Blockhead resembles those chatbots we use today, and we can see how useful it can be.

Then the question becomes: is it possible to build a language model like Blockhead?

To put this formally: for any input token sequence $x = [v_1, v_2, \dots, v_n]$ , the robot maps it to an output token sequence $y = [u_1, u_2, \dots, u_m]$ . The chatbot is basically a function from sequences to sequences, and we aim to find a good $y = f(x)$ .

This kind of problem looks familiar. From our experience in machine learning, we might be able to approximate this function $f$ with a neural network.

Next-token Prediction

There are various ways to generate an output sequence $y$ from $x$ . One emerging approach, diffusion language models, generates text by iteratively denoising blocks of it. Most current models, and the ones this post discusses, use a technique called next-token prediction (NTP).

Recall that we aim to find that function $f$ that maps input sequence $x$ to output sequence $y$ . And it can be shown that $f$ can be losslessly recovered by another function $g$ , which maps a sequence to its correct next token. Thus the recover process looks like this:

\begin{aligned} g([v_1, \dots, v_n]) &= u_1 \\ g([v_1, \dots, v_n, u_1]) &= u_2 \\ &\ \ \vdots \\ g([v_1, \dots, v_n, u_1, \dots, u_{m-1}]) &= u_m \\ g([v_1, \dots, v_n, u_1, \dots, u_m]) &= \texttt{<EOS>} \end{aligned}

The final token EOS stands for end-of-sequence, denoting end of generation. With this trick, we drastically shrink the output space: instead of producing an entire sequence $|\mathcal{V}|^m$ , we produce a single token out of $|\mathcal{V}|$ and generate step by step. And it seems great that $g$ is able to reproduce $f$ without loss.

But everything comes with a price. To begin with, this is a deterministic version. In reality, we produce a probability distribution of next possible tokens, where we will need some extra methods to recover the best sequence.

Second, from our gut feeling, this method puts too much pressure on that first token it generates. The article "a" or "an" is decided even before "apple" or "banana". This can be mitigated with some sampling tricks, and models do seem to plan ahead internally a little, like picking a rhyme before writing the line that leads to it. Still, it doesn't feel intuitive.

Finally, in practice, the input is also bounded with a fixed length. This input of function $g$ is called the context, and a model has a maximum context length $L$ . For frontier models, $L$ is generally very large, sometimes around a million, which should be sufficient for most use cases. Thus the function becomes $g: \{\mathcal{V}\}^L \mapsto \{\mathcal{V}\}$ .

Now we have a simplification that, albeit with shortcomings, seems fine. And we set foot on some architecture to approximate $g$ .

The architecture in use is called transformers.

Transformer

This is only a brief description. For each token, its embedding $h_i \in \mathbb{R}^d$ carries both the token and its position (in modern models usually through rotary position embeddings, RoPE). Attention by itself has no notion of order, which is why position must be added.

One attention head. The embedding is projected into a query, a key, and a value by learned matrices:

q_i = W_Q h_i, \qquad k_i = W_K h_i, \qquad v_i = W_V h_i

Intuitively, the query says "what am I looking for", the key says "what do I contain", and the value says "what I'll pass along if selected". The attention weight from position $i$ to position $j$ is

\alpha_{ij} = \operatorname{softmax}_j\!\left(\frac{q_i^\top k_j}{\sqrt{d_k}}\right), \qquad j \le i

The $\sqrt{d_k}$ keeps dot products from growing too large. The constraint $j \le i$ is the causal mask. The output is

z_i = \sum_{j \le i} \alpha_{ij} v_j

so $z_i$ contains information from the tokens $i$ attends to.

Multiple heads. $h$ heads run in parallel with their own $W_Q, W_K, W_V$ . Their outputs are concatenated, projected by $W_O$ , and added back to $h_i$ . This running vector is called the residual stream.

MLP. A multi-layer perceptron is applied to each position separately, and its output is also added to the residual stream. Normalization layers around both sublayers keep training stable.

Stacking and prediction. Attention plus MLP is one layer, and the model stacks $N$ of them. After the last layer, an unembedding matrix $W_U$ turns the vector at position $i$ into one score per vocabulary token, and a softmax turns the scores into $p(x_{i+1} \mid x_1, \dots, x_i)$ .

Training. Because of the causal mask, one forward pass over a text gives a prediction at every position at once. The model minimizes the cross-entropy:

\mathcal{L} = -\sum_{i} \log p(x_{i+1} \mid x_1, \dots, x_i)

Observations with Transformers

After training on web-scale text, we get a model with some properties.

Capabilities

Transformer-based language models succeeded on traditional language tasks such as sentiment analysis and question answering. Researchers kept scaling them up, partly because performance improved smoothly with size, and partly because they saw abilities that seemed to emerge suddenly at scale, like doing multi-digit arithmetic or unscrambling jumbled words. Whether the emergence is really sudden is debated, but the abilities are real:

Following tasks from a prompt. A model trained only to continue text can be aligned with tasks. Give it the sequence "Translate the following sentence into French: 'a piece of bread'", and it continues with "un morceau de pain". Nobody wrote a translation module.
In-context learning. Add a few examples to the context, and the model finishes the rest in the same pattern. In this way the model can complete different tasks, without the need to train another model.
Some knowledge. Ask about the capital of a small country, how a vaccine works, or the plot of a novel, and the model answers from its weights alone, with no database behind it.
Composition. Ask for a sonnet about tax law, and you will get one, even though that pairing almost never appears in its training text.

Failures

There are also noticeable failures:

Hallucination. Ask about the birth year of an obscure person, and the model may answer with a fluent, confident, and wrong year.
The reversal curse. Models trained on "A is B" often fail at "B is ?". A model that learned "Tom Cruise's mother is Mary Lee Pfeiffer" often can't say who Mary Lee Pfeiffer's son is.
Compounding errors. Once a generation drifts somewhere unfamiliar, it tends to keep drifting.
Losing track in long contexts. Models use information at the start and end of a long context better than information in the middle.
Sensitivity to phrasing. Reword a question slightly, without changing what it asks, and the answer may change drastically.
In-context learning works on random labels. Give the model in-context examples with random labels, and it often does nearly as well as with correct ones. This introduces some questions on what in-context learning really is doing.
Exact computation. Models are unreliable at long arithmetic and counting.

But why do such phenomena appear? What's the inductive bias induced by transformer architecture? What does it actually do?

To understand this, we need a closer look at what we are cooking with: our languages.

Forms of Our Languages

1. Language is uniform in information

This is a useful but often overlooked property in language.

Take images as a counterexample. An image is a sampling of a continuous scene, and the resolution is a choice. Given an image of a cat at 1024 by 1024, you see a cat. Resample it to 128 by 128, and you see exactly the same cat. Pixels are not uniform with respect to information at different sizes and scales.

Language doesn't scale like this. Every token is a discrete choice somebody made. You can't downsample a sentence. To make text shorter you have to summarize, and details will get lost. A sentence contains less than an essay, and an essay less than a book. You can roughly estimate how much is said by word count.

So predicting the next token is almost never trivial, and rarely hopeless. Every step is a meaningful guess. That is one reason "predict the next unit" is a rich training signal for text in particular.

2. Language as a projection

In Plato's allegory of the cave, the real world is out there, but the people in the cave only get to see shadows, which are projections of that world. Language is similar. Our world is multimodal: we meet it through vision, sound, smell, touch. And we project all of it onto a one-dimensional string of symbols.

So communication naturally comes in two steps. A speaker has something to convey and encodes it into a token sequence. The listener receives the sequence and decodes a meaning from it. Language is the intermediate substance. It is an encoding of thought, or intention, or perception.

Different languages have different ways of projection. Russian has two basic words for blue, one for lighter blue and one for darker blue, where English only has one. And Russian speakers are measurably faster at telling shades apart across that boundary. Languages encode the world in their own ways, and meaning gets lost in translation.

The power of the bar lies in words. More specifically, the stuff of language that words are incapable of expressing – the stuff that gets lost when we move between one language and another. The silver catches what’s lost and manifests it into being.

Excerpt From Babel by R. F. Kuang

Languages seem like tools for expression, and different languages are merely different ways of expression. But it is sometimes also the case that a person would display a different personality when speaking different languages. So the tool also shapes what gets expressed, which is where I myself find languages interesting.

3. Language is highly structured

Previously we have formalized a sentence into a token sequence $v_1v_2...v_n$ , but we should also notice that not all token sequences are valid instances of a language. A valid sentence is not a random concatenation of vocabulary. It is constrained by grammar and by meaning, and the two are sometimes harder to separate than textbooks suggest.

Traditional linguistics would separate a language into two parts: syntax and semantics. Syntax is about grammar and structure, and semantics is about meaning. Chomsky argues that syntax survives the death of semantics. He demonstrates with a sentence "Colorless green ideas sleep furiously". The sentence is grammatically correct, has the right structure, but makes no point.

But syntax can have some effect on meanings, too. Take the example of Apple's famous slogan "Think different." One might suspect that it's a mistake of "Think differently". But you should have some gut feeling that the two have slightly different meanings. The reason is that English has a real construction — dream big, play dirty, travel light — where a bare adjective follows a verb and carries an exhortative force that the adverb doesn't. The construction is there maybe without people realizing. And when a new slogan in this form appears, people get to feel the meaning from the structure.

This is but a theoretical viewpoint. In practice, we always say "Tokyo is the capital of Japan", never the reverse. That looks like pure semantics — "Japan is the capital of Tokyo" is yet another nonsensical statement which is grammatically correct. However, if the latter never appears in real text, then how do you decide that it is semantics, but not a structure of the language itself? This is a question that we'll see in later parts of this post.

As a result, for a vocabulary set with size $|V|$ and a sentence with $n$ tokens, there are $|V|^n$ possible sequences. But valid sentences within that certain language are a small subset of it. Although this is in discrete space, it can be thought of as lying on a manifold in higher dimensions.

In this way, we can view our sequence of tokens as a principled way of encoding something.

4. Natural vs. formal languages

One may ask at this point: what about programming languages? Programming languages, as well as mathematical notation, are formal languages. They are designed by humans and have explicitly specified grammar.

More importantly, their meaning, or function, is determined purely by their form.

def multiply(a, b):
    return a * b

The only thing that matters in the code above is its structure. Rename the function to f and the arguments to x and y, and the program computes exactly the same thing. Logicians call this alpha-equivalence. The keywords and operators carry the structure; the names are arbitrary labels.

So for formal languages, symbols are arbitrary. Structure is everything.

Natural languages don't follow the rule either. There are sentences where you can't change a symbol without changing the meaning:

The trophy didn't fit in the suitcase because it was too big.
The trophy didn't fit in the suitcase because it was too small.

Change one word, keep the structure exactly the same, and what "it" refers to flips. In the first sentence "it" is the trophy; in the second, the suitcase. Grammar doesn't tell you which. You know it because you know how trophies and suitcases work.

We may borrow two terms from Saussure's semiotics here: signifier and signified. A token is a signifier. Whether we write "summer" or call it $v_7$ , it is merely a form, a symbol, a signifier. What it means, the signified, lies beyond the symbol.

So in natural languages, the meaning of a token sequence is defined not only by structure, but also by the contents that fill the structure.

And a language model only ever receives signifiers. How far signifiers alone can take you is, I think, the real question of how far language models can go.

An example with masked meaning

Here is a sentence:

The linguist handed the manuscript to the editor because she had finished it.

Now I replace every content word with a symbol, keeping the function words and the inflections:

The BLICKET handed the DAX to the TOMA because she had WUGGED it.

I have thrown away the meaning. There is no such thing as a blicket. However:

You can still parse it. BLICKET is the subject, DAX is the direct object, TOMA is the object of a preposition.
You can still inflect it. The past tense of WUG is WUGGED, and you knew that without being told.
You can still answer questions about it. Who did the handing? The BLICKET. What was handed? The DAX.
You can still feel the ambiguity. Does she refer to the BLICKET or the TOMA? It's the same ambiguity as in the original, sitting in the same place.

And imagine when the context grows, you will have enough access to how DAX appears in text. You know it is always "handed" instead of "kicked". You know it is "read" instead of "spoken". With all these relations in mind, you can make some inference.

A lot of what we do with language only needs form. The formal hints are enough, and the real meaning is optional.

Transformers pick up relationships in structures

So we can make a hypothesis that language models work like this.

A language model is, at its core, a formal completer. It learns which forms tend to accompany which other forms, and it completes a given form in the way those learned relations suggest. It has access to meaning only to the extent that meaning leaves traces in the distribution of forms.

During pre-training, the model only ever sees signifiers. "Summer" is to it just $v_7$ , a token with no attached perception of warmth or light. What it can learn are the relations between tokens: which tokens predict which, in what configurations, over what distances.

A transformer is a very powerful machine for learning the company words keep: it captures structural, or formal relationships, among tokens.

Sanity Checks

Does the hypothesis explain what we observed?

Embeddings. After enough training, "Tokyo" becomes something that often comes before "is the capital of Japan", appears near "Shinjuku" and "subway", and fills the same slots as "Kyoto", "Osaka", or "Seoul". So its embedding ends up near theirs. The words just keep similar company, and company becomes geometry.

Attention heads. Different heads can be viewed as tracking different relations. Some have been found doing exactly that. Induction heads implement a simple rule: if [A][B] appeared earlier and I'm now at [A], predict [B]. That is formal completion in its purest form.

Translation. The web contains enormous amounts of text where "Translate into French:" is followed by French, and parallel texts where a passage sits next to its translation. Formal links between tokens in different languages are learned from that co-occurrence. The instruction is a form, and the translation is its usual completion.

In-context learning. Both the capability and its failure come from the same place. A prompt full of examples is itself a form: input, label, input, label. Continuing that pattern is exactly what the model was trained to do, which is why it can pick up a new task without any change to its weights. But the same fact explains why random labels barely hurt. What matters most is the format, what the inputs look like and what the labels could be. The model completes the form, whether or not the labels make sense. Larger models do take the labels more seriously, and can follow a flipped mapping if you give them one. But then "the mapping in this prompt" is just a more abstract form that larger models have the capacity for inference.

Math and code. Recall that in formal languages, structure is everything: what a program or a proof means is fixed by its form. For a formal completer, that is the best possible case. Nothing lies outside the text, only symbols and the rules connecting them, and all of it is there for the model to pick up. Mathematical reasoning in particular is almost purely symbolic inference: each step follows from the ones before by explicit rules, and a proof is valid or not by its form alone. This is a large part of why models are so strong at writing code and working through proofs, to the point that they now contribute to research-level mathematics.

Hallucination. The form "was born in ____" strongly calls for a plausible four-digit year. If the fact was never learned well, the model completes the form anyway, fluently and confidently. Hallucination is what happens when form and meaning come apart.

The reversal curse. For us, "Tom Cruise's mother is Mary Lee Pfeiffer" and "Mary Lee Pfeiffer's son is Tom Cruise" are the same fact. For a formal completer, direction is part of the form. So the question we asked earlier, whether the order of "Tokyo is the capital of Japan" is meaning or structure, gets an answer from real models: at least in part, what they learn is the structure.

Scaling. With more layers and more heads, the model can learn more relations, and relations between relations. At enough depth, form gets abstract enough to look a lot like meaning.

How far does form go?

Clearly the forms we pick up in the training set are not perfect. But questions remain on how far we can really go with pure form, within language.

Form can also force a model to rebuild some of the world that produced it. Researchers trained a small transformer on nothing but sequences of Othello moves, written as tokens like E3 D3 C4. It was never shown a board and never told the rules. Its only task was to predict a legal next move. Yet when they looked inside, they found the board: the state of each of the 64 squares could be read out from the model's activations. More convincingly, when they edited that internal board, flipping a piece into a square, the model's predicted moves changed to match the edited board. The board wasn't a side effect. The model was using it.

Why would pure form produce a board? Because the board is the most economical explanation of the form. Which moves are legal depends on which squares are occupied, and memorizing move patterns directly is far harder than tracking the 64 squares that generate them. Predicting the shadow well enough forces the model to rebuild the object casting it. This is the same thing we saw with DAX: context by context, "read, not spoken" and "handed, not kicked" accumulate into something that behaves like a concept of a document.

But a rebuilt world is not necessarily a coherent one. In another experiment, a transformer was trained on turn-by-turn taxi routes in Manhattan. It became nearly perfect at predicting valid next turns, which suggests it knows the city. But when researchers reconstructed the street map implied by the model, it was full of streets that don't exist, pointing in impossible directions and jumping across the grid. And when they asked for routes that required detours, performance collapsed. The model had learned enough of the form to pass ordinary tests, without an actual map behind it.

So form goes surprisingly far, but not always as far as it looks.

The board was a structure hidden inside the form, waiting to be found. But they do sharpen what "formal completer" should mean. A completer under enough pressure doesn't just memorize which forms follow which. It builds whatever internal structure predicts them best, and sometimes that structure is more than syntax and is a piece of the world.

This version of the hypothesis makes predictions that could turn out wrong:

Where the world leaves dense and consistent traces in text, such as geography, chronology, or the rules of a game, models will rebuild pieces of it.
Where the traces are sparse or inconsistent, models will stitch together shortcuts that work on typical inputs and break under perturbation, like the taxi model on detours, or arithmetic with large numbers.
Where the world leaves almost no trace in text, like how hard to push a door or how a cup tips over, no amount of text will fill the gap.

My own take is that formal completion is what the training process rewards the model for, and it is a good way to predict where models fail. They work where formal connections are enough, and where those connections add up to real structure. They fail where the connections weren't learned, where they were learned as a patchwork that only looks like structure, or where they never made it into text in the first place.

Failure Modes

With this hypothesis, we can examine failure modes of a language model, and decide on what we can do about them.

Now, let's suppose we have a perfect table for NTP function $g$ . Its rows are every possible token sequence up to length $L$ , and each row holds the right next-token distribution for that context. This table is perfect, which means that it produces right answers all the time.

Of course the perfect table is practically impossible. All architectures, including transformers, are just an imperfect approximation of the perfect table. We now have three things, each an approximation of the one before:

Blockhead, which maps whole inputs to whole answers.
The perfect table, which does the same one token at a time.
The real model, an approximation of the perfect table.

And we can ask, theoretically, whether our current failures with language models will persist on the perfect table or Blockhead. This way, we can see where the failure comes from and whether it is possible to fix it.

If even Blockhead fails, it belongs to the task and the training set themselves.
If the perfect table fails but Blockhead doesn't, it is intrinsic to next-token prediction.
If the model fails but the perfect table doesn't, the failure comes from imperfect imitation.

Failures of the task and the training set

These happen even with a perfect Blockhead.

Ambiguity. "Tell me about Mercury." The planet, the element, the god, or the singer? Even the ideal distribution is spread out. Something has to decide what to do, like asking back.
Tricky prompts. "Why did Einstein win his second Nobel Prize?" There is no right continuation. Blockhead can't just write NaN into every row that doesn't make sense. It needs a principled way to fill rows like this, which means recognizing that the question itself is wrong.
Missing information. Context and weights are all there is. If the answer is in neither, like today's weather or events after training, no row contains it. Through the lens of reinforcement learning, pre-training is just behavior cloning, which means it is trained to learn "what I should do in a given situation" instead of "why and how I should do this in a situation". Scaling up simply fills more situations with diversity.

Failures of next-token prediction

The chain rule guarantees that NTP loses nothing in principle. But we never use the distribution itself. We use one path drawn from it, and the way we draw paths is where NTP's own costs appear.

Local choices are not global choices. Suppose the question is "Name a fruit," and the perfect table knows the answer distribution: "an apple" 0.3, "an orange" 0.1, "a banana" 0.2, "a pear" 0.15, "a grape" 0.15, "a mango" 0.1. The single most likely answer is "an apple". But the first token decides between "a" (0.6) and "an" (0.4). Greedy decoding takes "a", and from there the best it can reach is "a banana". The article was decided before the fruit, just as we feared. Blockhead, choosing whole answers, has no such problem.
Compounding errors. A mistake becomes context the model must continue from. In training it always continued real human text; in generation it continues its own. Once it drifts out of the distribution it was trained on, its predictions get worse, which causes more drift. This is called exposure bias. The perfect table has correct rows even for strange contexts, so it never compounds anything. A real model, committing token by token, does. The failure lives at the intersection: NTP supplies the commitment, and imperfect imitation supplies the error.

Failures of imitation

These only happen because the model is an imperfect approximation of the perfect table.

Hallucination. The model fills a row it never learned with something form-plausible. The perfect table has every row right.
The reversal curse. Training on "A is B" only ever teaches the model to predict B after A. Nothing in the objective asks it to predict A after B, so the reverse row is left to generalization, and a formal completer doesn't generalize symmetry for free. The perfect table has both rows.
Losing track in long contexts. Models can lose information in the middle of a long context, and lose track of instructions over long conversations. The table has no attention to spread thin.
Answers that need many steps. Some problems, like multiplying long numbers or following a long chain of logic, have answers that no learned relation connects directly to the question. The model never caught a relation that jumps straight from one to the other, and for problems like these, such a relation may be too deep to fit in one forward pass at all. The perfect table has everything right.

In frontier language models, failures at different phases are mitigated with different tricks.

Improvements Where Forms End

Following the life of a model, we go through these tricks in order: the data it learns from, the way we decode from it, how we shape what it imitates, and finally what changes when it stops imitating altogether.

Data

Before any clever technique, the first fix is the most obvious one. If a model only learns forms, give it the right forms, and more of them.

Scale covers more of the table: more situations, more phrasings, more of the world written down. Curation decides what fills those rows: a model trained on carefully chosen text imitates better text. And augmentation fills the rows the raw web leaves thin. Research on knowledge storage found that a model extracts a fact reliably only after seeing it in several varied phrasings, so rewriting the same fact many ways helps, and stating it in both directions eases the reversal curse. Frontier models increasingly train on data that other models generated or rewrote, precisely to shape the forms they learn from.

Decoding

Local choices can be softened at generation time. Instead of greedily taking the best next token, beam search keeps several candidate paths and compares them as whole sequences, so "an" is no longer locked out before "apple" arrives. Sampling methods like temperature and top-p trade faithfulness for variety. Some choice is unavoidable: NTP produces distributions, and we want answers. Decoding can't do much about compounding errors, though. Once a wrong token is chosen, every path continues from it.

Imitation: aligning form with meaning

Most failures live here: hallucination, the reversal curse, answers that need many steps, and losing track in long contexts. This is also where the theme of this post matters most. Pretraining teaches the forms of human text. What we want is for those forms to line up with meaning: with what the question asks, and with what is true. The main tools do this with more form.

Instruction tuning. Fine-tuning on example conversations teaches the model the form of a helpful assistant: a question is followed by an answer to it, an ambiguous request by a clarifying question, a false premise by a correction. Nothing about meaning is installed directly. The model learns a new shape of completion, one where form and intent happen to coincide.

Reasoning before answering. When the model hasn't caught a relation that leads straight from question to answer, thinking changes the problem. The model has learned many small relations for symbol-level reasoning: how one line of a derivation follows from the last, how to carry a digit, how one step of logic leads to the next. By writing its reasoning out, it replaces one relation it doesn't have with a chain of relations it does. The chain of thought is itself a form, learned like any other. The written reasoning becomes working memory, and harder problems get more steps. This is also where models' strength at math and code pays off: symbol-level inference is exactly what they are good at.

But we should also notice that, while "thinking out loud" opens the possibility to extract higher relationships previously not available, it is also prone to situations where questionable relationships are utilized, and the focus drifts further with longer output sequence, which I believe many have witnessed in the "thought process" of latest models.

Losing track in long contexts remains largely open. Longer context windows help, but attention still spreads thin over very long inputs, and no clean fix has emerged yet.

Beyond imitation: reinforcement learning

Everything so far still teaches the model to copy. Pretraining is imitation learning at its core, behavior cloning of human writers, and instruction tuning is behavior cloning of better writers. Reinforcement learning changes the question. Instead of "what would come next?", it asks "what reaches the goal?" The model tries, and is rewarded for the outcome, whatever its path looks like. It no longer copies; it evolves toward an objective.

RLHF. In reinforcement learning with human feedback (RLHF), people compare responses, a reward model learns their preferences, and the language model is optimized toward the ones they prefer. The goal is still anchored to human judgment, which makes RLHF a bridge between imitation and optimization. In table terms, it changes what "right" means, from what usually comes next to what a good answer is. It is also how a model learns to ask back, or to point out that Einstein won only one Nobel Prize.

RLVR. For problems with checkable answers, like math and code with tests, the reward can come from the world instead of from people: the answer is right or it isn't. DeepSeek-R1 showed that this can produce models that backtrack and check their own work. That speaks directly to compounding errors: a model rewarded for correct final answers learns to notice and repair its mistakes midway.

Humans don't learn language by imitation alone either. What they do get is a shared world: a parent looking at the same dog, a reaction when they're misunderstood, a cookie when they ask right. They learn signifiers while standing outside the cave, with the signified in front of them. Behavior cloning is cheap and scales well, but it captures know-how without know-why. It learns what people say, not what saying it is for. Rewards are a first small step toward the rest.

Outside the model: tools and retrieval

Some failures, like missing information, can't be fixed by any training, because the answer isn't in the weights or the context at all. For these we reach outside the model. Retrieval puts relevant documents into the context before answering, so the model completes forms grounded in the right text instead of fuzzy memory. Tool use lets it call a search engine for what it doesn't know, or a calculator or code for exact computation, and read back the result. These don't bring form and meaning any closer. They are patches that route around the gap. But they are honest about it: the model's job shifts from knowing everything to knowing when and how to ask.

Comparison between Weights and Context

What lives in the weights, and what lives in the context?

Next-token distribution depends on only two things: the context and the weights. That raises a question I find fascinating. What is the difference between training a model on an article and simply pasting the article into its context?

Some parts of the answer are fairly clear.

In the model's weights, the article is not stored as text at all. It is dissolved into adjustments across billions of parameters, merged with everything else the model knows. It becomes a relationship the model tries to catch.

In context, the article is presented verbatim. Its presented relationship is not caught by the model, and the model will only use existing relationships to decode and reason on this.

In the language of this post: the context holds forms to be completed, and the weights hold the relations used to complete them. Pasting an article gives the model new signifiers to work with. Training on it changes how the model connects signifiers in the first place.

Epilogue

Now let's try to answer some questions:

Q: How does next-token prediction turn into translation, instruction following, and in-context learning?

A: Much of what these tasks need is carried by form. A model only ever sees signifiers and learns the relations between them. Where form carries enough, as in the BLICKET sentence, completing the form is doing the task.

Q: Why does it fail in such strange ways?

A: Failures live in three places. Some come from compression: the model is an imperfect copy of the perfect table, and it goes wrong where form and meaning come apart, as in hallucination, the reversal curse, and the trophy sentence without its trophy. A few come from next-token prediction itself, which commits to one token at a time, and does the most damage when combined with imperfect imitation. And some belong to the task: even Blockhead, perfectly imitating human text, would have them, because predicting what comes next is not the same as saying what should come next. Post-training is our attempt to close that last gap.

Q: Will forms be enough?

A: Maybe not. Language is a projection of the world, and some of the world barely leaves a shadow in text format. How a cup tips over, how much force opens a door, how a crowd moves: everyone knows these, so hardly anyone writes them down. These are exactly where models are weakest, and they matter most for an AI that has to act in the physical world. Many robots today are built on language-model backbones extended with vision and action, and they work surprisingly well. But whether language can be the foundation of embodied intelligence is still a live debate.