saved
Neural Networks Are Already Clean
Hraness wrote this summary from a saved copy of the source. Quotations are taken word for word from the source.
gist
Nick Cammarata argues that the polysemantic neurons that convinced interpretability researchers networks are tangled by superposition mostly came from fixable architecture and tooling problems, not arbitrary packing. With modern architectures and better tools, he finds most neurons have one main job even in trillion-parameter models, so researchers can study networks directly rather than first decomposing them.
ideas
- Superposition misread. Polysemantic late layers in InceptionV1 came from architectural overflow and weak tools; modern versions are 99-100% interpretable in the last layer at any of a 300x size range.
- Recursive generalization. Shared neurons usually group related things that generalize together, which teaches something, unlike unrelated concepts packed in to avoid interference.
- Language models are legible. Preliminary audits of Qwen3-32B suggest about 90% of neuron poles have one main job and only 2-4% remain unexplained.
- Explore, don't enumerate. He favors question-driven study of real neurons, circuits, and motifs, with probes and sparse dictionaries at the end rather than the start.
- Automated scientists. Interpretability may become a lab of agents labeling neurons and testing motifs, enabling small, careful alignment fixes.
quotes
“Neural networks are already clean.”
“One main result is that superposition was the wrong explanation for InceptionV1.”
“The network is the answer, and interpretability is largely the search for the right question.”
“If the network isn’t a knot, untangling it can make it look like one.”