saved
Real-SWE Benchmark
Hraness cites a source capture. The source author remains the source.
gist
Specific Labs releases Real-SWE, a coding-agent benchmark on private production codebases licensed from real companies rather than public or synthetic tasks. Eight model-and-native-harness pairs run ten enterprise tasks across 640 scored rollouts; Fable 5.1 with Claude Code leads at 38.8% resolution (pass@1), while GPT-5.6 Sol trails at 16.2%. The benchmark stresses company-specific conventions, multi-service business work, and underspecified prompts that still edit a median of eleven files.
ideas
- Private production code is the test bed. Tasks come from licensed company codebases, so agents cannot lean on public GitHub solutions or internet-leaked answers.
- Score the model-plus-harness people actually use. Native harnesses (Claude Code, Codex CLI, Gemini CLI, and peers) are evaluated together instead of models in isolation.
- Business consequences beat toy edits. Billing tax rules, migrations, and multi-service changes force agents to respect existing products and operational constraints.
- Short prompts still demand wide diffs. Median instructions are about 1,742 characters, yet reference solutions edit a median of eleven files—more than FrontierCode or DeepSWE.
- Frontier resolution stays low. Even the leader clears under two-fifths of tasks at pass@1 averaged over eight runs, showing real enterprise SWE remains hard.
quotes
“Can a coding agent actually do the work of a software engineer in the real world?”
“We use native harnesses to reflect how enterprise engineers work in practice”
“Resolution rate is equivalent to pass@1, averaged over eight independent runs per task.”