Pierre Beckmann on world modeling in transformers (TaxiGPT thread)

New paper: World Modeling in Transformers

with @matthieu_queloz @andrenfreitas @MATSprogram

Can AI models recover world models just from data? We find a faithful internal map and more inside of taxiGPT! We trace failures to how that the map is stored in superposition. 🧵(1/11)

(a) an internal map of Manhattan; (b) superposed in a small subspace — 244 of 1600 dimensions explain 90% of variance; on avg., a node's nearest neighbor is only 48° away; (c) packed by affordance — legal-move clusters SE,SW / NE,NW / NW,SW / NE,SE / NE,NW,SW / NE,SE,SW.

From behavior, TaxiGPT’s map looked incoherent (Vafa et al.). Inside the model, using mechanistic interpretability, we find a different picture: causal intersection features, and street connections encoded as transitions between these intersections. (2/11)

From generated rides → Inside the model. Streets encoded as transitions between intersection features.

For navigation, TaxiGPT uses a goal compass, a circular feature encoding the direction to its destination, best decoded at L16. Steering it makes the taxi follow the implanted direction. Removing it does not cause illegal moves but sharply reduces goal-reaching. (3/11)

Goal compass (L16). Steered north or south (red: steered part of the ride). Compass removed: legal moves, but no goal arrival.

How can a model with a faithful map still make illegal moves? Because the intersection features are stored in superposition. When the the correct intersection write weakens or noise grows, a wrong feature can become most active and an illegal move can get emitted. (4/11)

Weak write and noise → wrong node. Example failing ride: correct intersection active, then wrong intersection overtakes before the illegal move (x).

Imagine a taxi driver sketching Manhattan in a tiny notebook. As the page fills up, they draw new streets over old ones. Later, an X marking their position may land on two overlapping intersections, making their location unclear. Like TaxiGPT’s superposed position code. (5/11)

Notebook / sketch metaphor: overlapping hand-drawn Manhattan streets illustrating superposed position codes.

However, Affordance packing helps limit the dangers of superposition: features for intersections with the same legal moves are grouped together. When the model slips to a wrong intersection, the move it chooses can therefore still be legal where the taxi actually is. (6/11)

Affordance packing: intersections that share the same legal moves cluster together.

During training, a sense of direction and legal-move prediction emerge before the model reliably distinguishes individual intersections. A bit like a taxi driver that learns how to orient themselves in Manhattan before knowing every street. (7/11)

Relative to final checkpoint over training steps (100 → 100k): curves for Position decoding, Goal compass, Causal position edits, Legal moves.

Asking whether a model “has a world model” leaves much unresolved. Our findings point toward mechanistically studying its world modeling; uncovering the structures it learned, the world modeling capacities that recruit these, and when these capacities work together or not. (8/11)

In this spirit, we use mechanistic indicators to compare the world-modeling capacities of transformers of different architectures and trained via different objectives. (9/11)

Comparison table across SP·NTP, NSP·NTP, RW·NTP variants: decode/causal/streets/superposition/affordance packing, legal moves, goal compass, stress/detour/compression.

We made an interactive project website with 200 recorded TaxiGPT rides. Follow the taxi alongside its position code, goal compass, and move predictions.

Website: https://bepierre.github.io/world-modeling/

Paper: https://arxiv.org/abs/2609.21748

(10/11)

Stress ride UI: origin 51 moves from goal, 93 moves, reached goal; next-move probs; Position code (L18); Goal compass (L16).