If coding is solved, what now?: Measuring the sloppiness of code

gist

Sebastian argues that formally correct LLM code still produces slop—unnecessary abstractions, duplication, and bad structural decisions—and that agents cannot clean it up themselves. After surveying vibes-based industry claims, he shows why LLM-as-judge and human review both fail as scalable slop metrics, then reports SlopCodeBench Verbosity and Erosion scores where agent code is roughly twice as bad as established repos. Iterative multi-round evals with context erasure drive even SOTA models to 0% strict pass rates as bad decisions accumulate, so human taste remains baked into any serious evaluation.

ideas

• Correctness is not cleanliness. LLM code can pass tests while still adding abstractions, duplicates, and bad structural decisions that explode LOC.
• Agents do not clean their own slop. Trusting coding agents to manage million-LOC churn fails; they struggle with the same structural decay.
• LLM-as-judge is a weak substitute. Scalar scores behave like noise, and pairwise preferences can flip when labels are renamed.
• LOC, Verbosity, and Erosion separate slop. Simple LOC change works until Goodharted; SlopCodeBench metrics show agent code roughly twice as verbose and eroded as established repos.
• Iterative context erasure exposes accumulation. Multi-round instruction/test loops with wiped context drive SOTA models to 0% strict checkpoint pass rates as bad decisions compound.

quotes