Reading by address.

01 / WHAT IS AN ADDRESS?

Ask for the capital of France. France identifies the entity; capital identifies the relation you want to read. Currency would ask something different about the same place. In the opening animation, that question arrives at an FFN and a Paris direction is written into the residual stream—the model’s changing working state. The conveyor is an explanatory drawing. The address in a model is a vector of numbers, and matching it need not isolate a single neuron or a single fact.

A LOCATION THAT A READER RECOGNISES

An address here is closer to a pattern recognised by a reader than a numbered drawer. The input pattern and the weights that respond to it belong together. That connects this note to the map: a point in a picture is only useful once we know which operation reads the underlying state. We can make that relationship concrete by building a memory whose keys and values we can inspect.

02 / BUILD A MEMORY WITH SIX FACTS

The Mechanism’s ffn.py plants capital, currency and language facts for two invented places. Atlantis has Paris, Euro and Latin; Zerivia has Cairo, Rand and Tamil. Each full question receives its own random, normalised 24-number key. Each answer receives a value direction. We put the six keys into the rows of an input matrix and the corresponding values into an output matrix. There is no training: we choose what this little memory contains.

03 / MATCH → ACTIVATE → WRITE → READ

Choose capital of Atlantis. The demo supplies that address’s key vector as the input, compares it with all six keys using dot products, and clips negative matches to zero with ReLU. Each remaining activation scales its value direction; adding those contributions produces the output vector. Finally, the six answer readers score that output and the highest score supplies the displayed word. This is two matrix multiplications with a nonlinearity between them, followed by an answer readout. Nothing iteratively unpacks all six facts before answering this question.

WHAT TO NOTICE WHEN YOU SWITCH A NEURON OFF

The matching key scores approximately one because the input is that key itself. Other randomly chosen keys can still have positive matches, so several values may contribute. Suppressing a neuron removes its contribution from the sum; it does not erase the entire output. Watch the answer scores as well as the winning word. The demo always names the largest score, even after a destructive intervention, so a displayed answer is not a guarantee of a confident or correct read.

THE QUESTION IS ALREADY ENCODED IN THIS DEMO

Selecting a label supplies a stored key directly. This small memory does not understand the English phrase, derive a new entity–relation combination, or learn from a document. Those jobs are precisely what a real system must add. Its six successful reads also do not establish unlimited capacity: non-orthogonal keys can activate together, and adding more entries can change the combined output.

04 / PACKING AND READING ARE DIFFERENT OPERATIONS

The animation adds capital, currency and language directions into one vector. Recovering every ingredient of that mixture would be a different task from applying a reader that answers one question. The film’s memory argument makes that distinction useful: storing contributions together need not mean reconstructing a complete table of facts for every read. The illustration uses constructed six-dimensional vectors. The 24-dimensional FFN demo uses separate key and value matrices. They illustrate related operations, rather than two measurements of the same model.

05 / THE MODEL HAS TO BUILD THE QUERY

The film asks for Australia’s capital. Sydney becomes prominent around layer 24; Canberra leads by layer 26 and the answer settles later. These are intermediate readouts during one prediction, rather than words the model has already printed. The state reaching each FFN changes as attention and earlier FFNs update it. Addressing is therefore a process across layers. This example motivates inspecting when a useful query becomes available; the picture alone does not prove an exact database lookup.

06 / A RELATION CAN SURVIVE A CHANGE OF WORDING

In address.json, a logistic-regression probe trained on capital, currency and language is tested on seat, metropolis, money, cash, tongue and speech. Each tested synonym decodes to its intended relation for all 15 entities. This says something specific about information a trained probe can recover at layer 10. It does not mean the model has exposed a universal address format, or that an unseen entity will be routed correctly.

THE ENTITY IS A LESS EXACT READ

route_sweep.py trains a separate entity router at each tested layer, using capital prompts for 150 places. It then changes the wording, or asks about currency instead. At layer 26, the right entity ranks first for 66% of paraphrases and about 50.7% of the cross-relation prompts. Allowing a shortlist of five raises those figures to 86% and 76%. The entities are already present in the router’s training set; this is a test of changed questions, not held-out places.

A SHORTLIST STILL NEEDS A WAY TO CHOOSE

Layer 26 is the best tested layer for paraphrase top-five retrieval in this saved run. It is not best on every metric: layer 28 has higher top-one scores. That distinction matters for a system such as LARQL. Retrieving several plausible places may be useful if another operation can inspect them and resolve the answer. It is not yet the same contract as looking up one guaranteed row.

07 / WRITE WHERE THE MODEL CAN READ

The separate native.py demonstration edits Gemma 3 4B at layer 26. It first captures the FFN input for each target question, removes directions shared with competing addresses, then alters a gate row, an up-projection row and a down-projection column. The gate and up rows respond to the selected address; the down column writes an answer direction. A subsequent ordinary forward pass reads the changed weights. This is a weight edit, unlike swapping the transient residual state in the map experiment.