saved
An Empirical Study of Harness Design for Coding Agents
Hraness cites a source capture. The source author remains the source.
gist
Across 176 matched settings on SWE-Bench Verified and Terminal-Bench 2.1, this paper isolates planning, action space, and context management inside one fixed coding-agent loop. Context management pays off most when the window is tight, mainly by stopping overflow failures; staging rule-based elision before LLM summarization is the cheapest strong policy. Planning helps weaker models succeed and mainly saves cost for stronger ones, while bash-capable models can drop predefined tools and still win on CLI-heavy tasks at lower cost.
ideas
- Component ablations beat harness bake-offs. Holding the execution loop fixed reveals when planning, tools, and context policy actually move accuracy versus cost.
- Context management is a budget tool. Its value rises as the window shrinks, mostly by preventing overflow rather than rewriting agent behavior.
- Elide, then summarize. Rule-based elision before selective LLM summarization beats recoverable-elision machinery that models barely use.
- Planning changes role with model strength. Weaker models need it as an accuracy scaffold; stronger models mainly spend less with little accuracy change.
- Action space should match bash skill. Predefined tools help weak bash users; bash-fluent models can run bash-only interfaces more cheaply on command-line tasks.
quotes
“Context management becomes increasingly valuable as the context-window budget tightens”
“Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency”
“Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models”
“bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost”