theaicatalog
GitHub ยท Research

GitHub releases ReviewBench, an open benchmark for AI code review

GitHub launched ReviewBench, an offline benchmark for code review agents. It has 219 pull requests from 187 public open source repositories in 19 languages, modeled on GitHub's analysis of 103.9 million pull requests. Ground truth comes from human reviewers, author follow-up commits, analysis tools and several LLMs, and Claude Sonnet 5 grades submissions. GitHub says senior engineers who did not build the dataset agreed with its true and false-positive labels 96.6% of the time. The research preview is on the ReviewBench website, where users can view the dataset and test their own agents.