Lead AI Researcher · Sonar

Joe Tyler

A new coding model tops SWE-Bench on Monday. Should you roll it out to your team on Tuesday? A new tool or MCP server is released. Should you use it straight away? Public benchmarks can’t answer these questions, because none of them were built for your code.Without knowing how the models were trained, it’s difficult to know whether labs have “benchmaxxed” on popular open benchmarks like SWE-Bench or Terminal-bench. Even more recent benchmarks like DeepSWE which is written by hand to avoid contam