Neural networks are not already clean

gist

On LessWrong, leogao replies to Nick Cammarata's Goodfire essay arguing that neural networks are already clean. He agrees researchers wrote off neurons as polysemantic too quickly, but says a neat pattern in activating examples does not show a neuron is explained. He argues for simulation scores measured by downstream KL on a large dataset, and says circuits between neurons stay hard to explain because computation happens in superposition.

ideas

• Activating examples can mislead. Every activating example may fit a pattern while many inputs that fit it never activate the neuron, as with a neuron that fires only for dogs on bicycles on sunny days.
• Activation strength matters. His best guess from results he has seen is that ablating small activations substantially degrades the model, so weak activations cannot be ignored.
• Understanding means simulating the neuron. An explanation should be a rubric whose simulated values replace the neuron, and if the model degrades, the explanation was wrong.
• Downstream KL fixes the old scores. He traces Cammarata's complaint that simulation scores are too demanding to the correlation-based scores of the auto-interp paper, and says KL on a large dataset punishes only details that matter.
• Circuits remain the hard part. Past circuit work found large, redundant circuits that miss most of the model's behavior, which he attributes to computation in superposition.

quotes