IMAXING Latent Space

Drop the dye where the thought enters — watch where it surfaces

One strike, imaged: the emitter fires at L7, and one route stands lit in the fog.

What this is

A language model computes an answer along some path through its own latent space — but which path? IMAXING renders a prompt as a volume: token trajectories climbing the layer axis, surrounded by the fog of a thousand contrast prompts. A dye emitter injects signal at a chosen feature, and the trace that lights up — through attribution edges and sparse-autoencoder checkpoints — is the route the thought actually took.

The technical shape

Scene: token strands over transformer layers · x/z = principal components · y = depth
Context fog: diverse-text and sibling-prompt trajectories, depth-faded
Instruments: dye emitters · SAE bottles (top features/layer) · attribution edges
Method: baseline vs. perturbed runs, per-token, per-layer
Surface: interactive volume — orbit, zoom, hover any strand for its prompt position

The deep end · full technical outline

Outline v2 · expanding

Currents, not circuits

The founding objection: mechanistic interpretability calls its objects “circuits,” which implies fixed topology, on/off switches, engineered structure. Activation vectors do not behave like wire — they behave like fluid, and the fog has no wires in it. Strikes follow least-resistance paths through a field, not predefined routes; the gatekeeper that organizes them is less a switchboard than a magnetic field shaping preferred trajectories, and the small matrices are pressure chambers, not nodes. The payoff of the reframing: currents can be blocked, redirected, or amplified without altering the underlying structure — dynamic behavior that circuit thinking about transformers misses.

The hydrodynamic turn

If the medium is a fluid, borrow its mathematics. A vector crossing the fog behaves less like a signal broadcast through air than like a bubble holding coherent shape under pressure — surface tension as a self-correcting boundary, the sphere a minimum-energy form. Navier-Stokes viscosity shapes the path; vorticity is meaning that spins rather than travels straight; laminar versus turbulent flow is the difference between clean inference and hallucination; solitons — waves that hold their shape indefinitely — model stable concepts. And because latent space has walls (its embedding dimensionality), its boundary conditions are better behaved than the atmosphere’s — which is what makes forecasting the flow tractable at all.

Dye as tracer

The core move is tracer-particle visualization: rather than measure the entire velocity field at once — every activation, every head, weighted by magnitude, which yields statistics, not comprehension — inject a passive tracer and watch where the medium carries it. One coherent thread to follow: branching where currents split, merging where they converge, pooling where they stagnate. The fluid reveals itself through the dye, and the emitter’s angle is the one control that matters.

Why bottles — the matrix as light

In a full-size latent space the strikes are invisible: too many, too fast, the fog too thick. Shrink the arena to small matrices — bottles — and the bilinear map reads as light passing through them: the summed-over dimension is the fog-volume itself, every multiplication a scattering event before the light lands. Structured matrices thin the fog — sparse regions, dark corridors, preferential paths — which is exactly where speed and meaning live; arbitrary matrices glow uniform, no structure. Reduced this way, an individual strike becomes a legible event you can follow between mini latent spaces.

The current atlas

Rotate the emitter slightly — a minimal input perturbation — and watch the stream shift: that shift is the local current topology. Do it per bottle, combine the snapshots, and run at emission refresh rate — the emissions are the sampling mechanism, so there is no void between observations. The snapshot stack over time is a 4D map: three spatial dimensions plus time, meaning watched in flight instead of inferred statistically after the fact. Same emitter position, different paths and colors per model — a model identified by its interference pattern rather than its weights.

The instrument

A “strike” is one prompt run rendered as a volume: x and z are the first two principal components of the residual stream (PC1 typically ~65% of variance), y is transformer depth. Each token is a colored strand — solid for the baseline run, dashed for the perturbed — with BOS excluded so the content cluster fills the frame. Around the focus prompt hangs the fog: trajectories from diverse-text strikes and structural siblings, depth-faded so they read as context. The fog is what turns “this path is special” into a visible claim instead of an assertion.

The dye protocol

A dye emitter injects signal at a chosen layer/feature (e.g. L7, feature 8598, magnitude 4.0) and relative ‖dye‖ is traced per token per layer on a fixed colorbar. Sparse-autoencoder “bottles” mark the top features per layer; attribution edges (|corr| thresholded) connect strand segments across layers. The lit route is the answer to “which path carried the thought?”

Buildable from a blueprint

An honest accounting of the build. The easy parts already have tools: small-matrix probing via PyTorch hooks, input perturbation as emitter rotation, per-layer activation capture, basic stream plotting. The hard parts are real: reducing the space without losing structure (research-grade), aligning per-bottle maps into one coherent snapshot, real-time refresh at a meaningful rate, and full-fidelity visualization as a project in itself. It spans ML interpretability, fluid dynamics, real-time graphics, and high-performance compute — a team, not a solo. But the framework is the rare part, and it exists; the rest is engineering from a known blueprint.