Vilhelm
Toivonen

Your AI answers you in Finnish, but it thinks in English. I tried to change that, and along the way it turned out you cannot even reward a model into thinking in a language it never tries. Here is the diagnosis, the fix, and the honest failures.

Every request arrives knowing nothing about who's asking, so we stuff facts into the prompt and reread them forever. I spent months of my mech-interp PhD trying to precompute a per-user memory instead: two clean failures, one bank architecture that works, and a layer-injection picture I'd defend hardest.

I thought the embedding matrix was the easy part of parameter-golf: quantise it, compress it, ship it. Six weeks later I'd tried per-token bits, sparse training, and STE, and every one of them lost to plain dense INT7. Doubling the vocab buys ~0.01 BPB; buying the bytes back costs more.

I came in thinking quantization was about saving disk space. The bigger story is bandwidth, on both ends of the hardware spectrum, and underneath that there's a small information-theoretic puzzle I didn't expect.

Every time I push a model down to a phone, the KV cache is what kills me first. SSMs are the cleanest answer I've found, and around mid-2025 SSM hybrids started showing up in frontier models.

Open models quietly stopped putting softmax attention on every layer, and I couldn't find a single chart showing what replaced it. So I drew the map myself: MLA, linear, sparse, and the hybrid stacks three labs landed on independently, all on the same axes.