Separating signal from noise in coding evaluations

OpenAI performs an audit on SWE-Bench Pro, a widely used coding benchmark, and finds widespread task issues, estimating that ~30% of the tasks are broken.