Looking Inside GPT-2's Head: Building Latent Constellation

November 14, 2026

The Idea

When people say a language model "understands" something, what does that look like inside the model?

Every time GPT-2 reads a token, it turns it into a vector of 768 numbers. Then it passes that vector through 12 transformer blocks, and each block rewrites it a little. If you could see those vectors, you'd be watching the model think, one layer at a time.

That's what Latent Constellation does. It's my LMBCI project: a kind of brain-computer interface, except the brain is a language model. You give it a batch of labeled prompts. It records every token's hidden state at every layer, squashes 768 dimensions down to 3, and renders the result as a star field you can orbit, scrub through, and click on.

Here's what it looks like with real GPT-2 activations. Every point is one token from one sentence. Colour and shape show the sentence's topic. Drag the slider to move through the layers.

Loading activations…
Layer 8 · block 8
Drag to orbit. Hover a point to see its sentence. Click a category to isolate it.

Nobody told GPT-2 that numpy and SELECT belong together, but by the middle layers the code tokens have split off into their own region, and numbers are starting to. Animals, vehicles and emotions still overlap in this view. Hold that thought. Three dimensions throw away a lot, and we'll check what the full 768 dimensions know later on.

But the first time I ran it on a real model, it didn't look anything like this.

How It Works

The project has two halves: a Python backend that does the neuroscience, and a React + Three.js frontend that does the astronomy.

Labeled prompts
POST /api/jobs
GPT-2 forward pass
output_hidden_states=True
Activations
13 × N × 768
Shared PCA
768 → 3
Typed arrays
positions.f32 · clusters.u8
React Three Fiber
points + bloom

Recording the brain

The backend is a small FastAPI service. You POST a list of prompts, each with a label, and it starts a background job. The job loads a Hugging Face causal LM (gpt2 by default) and runs every prompt through it with output_hidden_states=True. That returns 13 tensors: the token embeddings plus the output of each of the 12 blocks.

For every real token (padding is masked out) I grab its vector at every layer:

for layer_index, hidden in enumerate(hidden_states):
    selected = hidden[batch_indices, token_indices].detach().float().cpu().numpy()
    layer_chunks[layer_index].append(selected)

The result is a single array shaped layers × tokens × 768.

Squashing 768 dimensions into 3

You can't look at 768 dimensions, so I use PCA. The key design choice is that I fit one PCA basis shared by every layer. Then I project each layer into that same 3D space.

The reason: if every layer got its own PCA, the axes would mean something different on every frame, and scrubbing through layers would just be the camera spinning randomly. With a shared basis, a point moving across the screen means the representation actually moved.

Shipping it to the browser

The output is deliberately boring: a flat float32 file of coordinates (layer → point → xyz), a uint8 file of cluster IDs, and a JSON list of token strings. The frontend fetches these as ArrayBuffers and hands them straight to the GPU as typed arrays. There's no parsing step, so 50,000 points load instantly.

Rendering the stars

The frontend uses React Three Fiber with a custom point shader, bloom, and a vignette, so it reads like a night sky rather than a matplotlib plot. You can orbit, scroll to zoom, search for a token, toggle clusters, and draw a token's trajectory: the path one token takes through all 13 layers.

The First Real Run Was a Dot

While building the frontend I used a simulated dataset: eight neat Gaussian clusters that drift apart as the layers go up. It looked great.

Then I pointed it at real GPT-2 with 60 sentences across five topics (animals, vehicles, code, emotions and numbers) and got this:

Loading activations…
Layer 6 · block 6
Raw shared PCA, as the app originally computed it. Each dot is many tokens stacked on top of each other. Scrub the layers and watch the first-token dot fly away.

Two dots. That's the whole picture.

One dot is about 500 tokens: every animal, vehicle, line of code and number, all stacked on top of each other. The other dot, far away, is 60 tokens: the first token of every sentence. It didn't matter what the word was. "The", "A", "Room", "while", and "n" (from npm) all landed on the same spot.

The PCA's explained-variance numbers said the same thing. The first principal component explained 98.1% of the variance. PC2 and PC3 got about 0.5% each. My "3D" view was really a 1D view of one question: is this the first token or not?

GPT-2's First Token Is Shouting

This is a known quirk of transformers, sometimes called an attention sink or massive activations. Every token can attend to the first position, so the model learns to use it as a dumping ground for attention it doesn't want to spend anywhere else. To do that job, the first token's hidden state gets enormous.

Here's how enormous:

By block 4, the first token's vector has a norm of about 3,000. Every other token sits between roughly 60 and 250, so the first token is 20 to 50 times larger. (The drop at layer 12 is because Hugging Face applies the final LayerNorm to the last hidden state.)

PCA finds the directions of greatest variance. When a dozen points are 40× farther from the origin than everything else, that is the direction of greatest variance. The math was right. I was just asking it the wrong question.

Fixing the View

I made three changes for the version at the top of this post:

  1. Drop the first token of each sentence. It's doing a structural job, not carrying meaning, so it doesn't belong in a picture of meaning.
  2. Center and normalize each layer. The residual stream grows steadily through the network, so later layers would otherwise dominate any shared projection.
  3. Fit a PCA per layer, then align neighbours with Procrustes. This gives up the shared basis, but gets back the frame-to-frame continuity in a different way. Each layer's 3D projection is rotated to best match the one before it, so the animation stays smooth and doesn't spin.

Flip between the two views here:

Loading activations…
Layer 10 · block 10
Drag to orbit. Hover a point to see its sentence. Click a category to isolate it.

Does the Model Actually Know the Topic?

A 3D picture keeps less than 1% of the information in 768 dimensions, so clusters on screen could be an artifact of the projection. To check, I trained a simple linear probe: a logistic regression that tries to predict a token's topic from its full 768-dim hidden state, one layer at a time.

The test is strict. It's cross-validated by sentence, so the probe is always scored on sentences it never saw during training.

Even at the raw embeddings (layer 0), the probe gets 57%, well above the 20% you'd get by guessing. That's because some tokens give the topic away on their own: digits, def, import. But accuracy keeps climbing through the network, to 96% at the final layer. Each block mixes in more context, until even a word like "the" or "." carries a signal about what kind of sentence it's in.

That's the thing I find really satisfying. The constellation is a picture of context spreading through the sentence, layer by layer.

What I Learned

  • Simulated data hides real problems. My fake dataset was built to look good, and it gave me no warning about massive activations at all.
  • Always hover over the outliers. The most interesting thing in the whole project was the part that looked like a bug.
  • Check the picture against a number. The probe told me the clusters reflect something real in the full space, and aren't just a nice-looking projection.

The figures in this post use real GPT-2 small activations from 60 prompts (556 tokens), extracted with the Latent Constellation backend.