AI code review benchmarks
6 AI code review tools, run on 11 real pull requests in 6 production open-source codebases and scored against 56 verified defects.
- 6
- Tools
- 56
- Defects
- 11
- Pull requests
- 6
- Repositories
Results
Codity ranked first, with a 89% severity-weighted catch rate, 46% higher than CodeRabbit (61%).
- 01Codity89% severity-weighted catch rateLeader45/56 defects caught
- 02CodeRabbit61% severity-weighted catch rate−28 pts behind first30/56 defects caught
- 03Greptile60% severity-weighted catch rate−29 pts behind first16/27 defects caught
- 04CodeAnt59% severity-weighted catch rate−31 pts behind first31/56 defects caught
- 05GitHub Copilot55% severity-weighted catch rate−34 pts behind first29/56 defects caught
- 06Cursor Bugbot50% severity-weighted catch rate−39 pts behind first10/27 defects caught
Weighted by severity: critical 4, high 3, medium 2, low 1. Each tool is scored only on the pull requests it reviewed.
By severity
Catch rate at each severity level
| Tool | Critical4 defects | High15 defects | Medium16 defects | Low21 defects | All defects | Weighted |
|---|---|---|---|---|---|---|
| Codity | 100%4/4 | 100%15/15 | 94%15/16 | 52%11/21 | 80%45/56 | 89%102/114 |
| CodeRabbit | 100%4/4 | 73%11/15 | 38%6/16 | 43%9/21 | 54%30/56 | 61%70/114 |
| Greptile | 100%1/1 | 40%2/5 | 75%6/8 | 54%7/13 | 59%16/27 | 60%29/48 |
| CodeAnt | 75%3/4 | 53%8/15 | 69%11/16 | 43%9/21 | 55%31/56 | 59%67/114 |
| GitHub Copilot | 75%3/4 | 47%7/15 | 69%11/16 | 38%8/21 | 52%29/56 | 55%63/114 |
| Cursor Bugbot | 100%1/1 | 60%3/5 | 63%5/8 | 8%1/13 | 37%10/27 | 50%24/48 |
Methodology
How the benchmark was run
Each pull request is a real change, taken from a public open-source repository and opened on a fork for review. Not every tool reviewed every pull request: each is scored only on the pull requests it ran on, and a dash marks one it did not. Where the same diff was opened twice so more tools could review it, Codity is scored on the run it shared with CodeRabbit, CodeAnt and Copilot.
There is no planted answer key. The defects are every issue any tool raised that holds up when checked against the code, merged where two tools describe the same problem, plus any the verifier found that no tool did. Every tool ran with its default settings.
- Caught means an explicit, line-level review comment that points at the faulty code and explains its impact.
- A summary that mentions the area, or a comment on the wrong line, counts as missed.
- Style nits, documentation and speculative suggestions are not defects. Severity is set from the defect’s real-world impact.
- Ranking is by severity-weighted catch rate: a critical catch counts 4, high 3, medium 2 and low 1, over the weight of every defect in the set. Raw counts are shown alongside.
| Repository | Language | Defects |
|---|---|---|
| LangChainOctoAI integration, output guardrails and human-in-the-loop interrupts | Python | 12 |
| fastlaneApp Store review cut-off, TestFlight upload and Google Play API v3 migration | Ruby | 26 |
| LocalAIReworks how reasoning blocks are separated from model output | Go | 3 |
| BlueprintMoves documentation pages to an MDX renderer | TypeScript | 4 |
| AppsmithFixes the Git settings modal flow | TypeScript | 2 |
| Cal.comAdmin spam-report table and seated-event attendee privacy in emails | TypeScript | 9 |
Case library
Every defect, every tool
Caught caught · Missed missed
LangChainPython · 12 defects
| Defect | Severity | Codity | CodeRabbit | Greptile | CodeAnt | GitHub Copilot | Cursor Bugbot |
|---|---|---|---|---|---|---|---|
| Top-level octoai import breaks `import langchain` when optional SDK is absentlangchain/llms/octoai_endpoint.py | Critical | Caught | Caught | Caught | Caught | Caught | Caught |
| embed_query returns List[List[float]] instead of a single vectorlangchain/embeddings/octoai_embeddings.py | High | Caught | Caught | Caught | Missed | Caught | Caught |
| embed_query uses embed_instruction instead of query_instructionlangchain/embeddings/octoai_embeddings.py | Medium | Caught | Caught | Caught | Caught | Caught | Caught |
| OctoAIEndpoint added to langchain.__all__ but never imported in the packagelangchain/__init__.py | Medium | Caught | Caught | Caught | Caught | Caught | Caught |
| OctoAIEndpoint._call silently ignores per-call **kwargs generation optionslangchain/llms/octoai_endpoint.py | Low | Missed | Caught | Caught | Caught | Caught | Missed |
| embed_documents annotated List[float] but returns List[List[float]] (breaks ABC typing)langchain/embeddings/octoai_embeddings.py | Low | Caught | Missed | Missed | Missed | Missed | Missed |
| OutputGuardrail declares pydantic field of ABC type Fixer; pydantic v1 fails at importlangchain/output_parsers/base.py | Critical | Caught | Caught | – | Caught | Missed | – |
| ValidationError(text=e) omits required error_message field, so every parse failure raiseslangchain/chains/llm.py | High | Caught | Caught | – | Caught | Missed | – |
| OutputGuardrail.fix calls self.fixer(...) but Fixer only defines .fix(), raising TypeErrorlangchain/output_parsers/base.py | High | Caught | Caught | – | Caught | Missed | – |
| LLMChain.generate/agenerate now return a tuple and create_outputs needs prompts, breaking callerslangchain/chains/llm.py | Medium | Caught | Missed | – | Missed | Missed | – |
| New LLMChain.output_parser field is never used, so configuring it silently does nothinglangchain/chains/llm.py | Medium | Caught | Missed | – | Caught | Missed | – |
| Indented module docstring at line 1 causes IndentationError; HITL middleware cannot importlibs/langchain_v1/langchain/agents/middleware/human_in_the_loop.py | Critical | Caught | Caught | – | Missed | Caught | – |
| Caught | 11/12 | 9/12 | 5/6 | 8/12 | 6/12 | 4/6 | |
fastlaneRuby · 26 defects
| Defect | Severity | Codity | CodeRabbit | Greptile | CodeAnt | GitHub Copilot | Cursor Bugbot |
|---|---|---|---|---|---|---|---|
| get_reviews crashes with NoMethodError when API returns no reviews, even without upto_datespaceship/lib/spaceship/tunes/tunes_client.rb | High | Caught | Caught | Caught | Caught | Caught | Caught |
| Reviews exactly at upto_date are kept or dropped depending on page boundariesspaceship/lib/spaceship/tunes/tunes_client.rb | Low | Caught | Caught | Caught | Missed | Missed | Missed |
| Integer ms/1000 truncation drops reviews in the first second after the cutoffspaceship/lib/spaceship/tunes/tunes_client.rb | Low | Missed | Missed | Caught | Caught | Missed | Missed |
| Skip-login TestFlight upload crashes: nil Spaceship::Tunes.client in provider inferencepilot/lib/pilot/build_manager.rb | High | Caught | Caught | – | Caught | Missed | – |
| Export-compliance path calls wait_for_build_processing_to_be_complete without argspilot/lib/pilot/build_manager.rb | High | Caught | Caught | – | Missed | Missed | – |
| Shell provider-list command interpolates username without shell escapingfastlane_core/lib/fastlane_core/itunes_transporter.rb | Low | Caught | Caught | – | Caught | Missed | – |
| installed_packages keeps trailing \r from CRLF adb output so uninstall is skippedscreengrab/lib/screengrab/runner.rb | Low | Caught | Missed | – | Missed | Missed | – |
| Frameit missing-offset error prints empty path once offsets are cachedframeit/lib/frameit/offsets.rb | Low | Missed | Caught | – | Caught | Missed | – |
| All skip_upload_* options default to true, so supply uploads nothing by defaultsupply/lib/supply/options.rb | Critical | Caught | Caught | – | Caught | Caught | – |
| promote_track crashes with NoMethodError when version_code is not givensupply/lib/supply/uploader.rb | High | Caught | Caught | – | Caught | Caught | – |
| promote_track promotes track_from.releases.first instead of the filtered releasesupply/lib/supply/uploader.rb | High | Caught | Caught | – | Caught | Caught | – |
| fetch_track_and_release uses Array#first with a block, always returning first releasesupply/lib/supply/uploader.rb | High | Caught | Missed | – | Caught | Missed | – |
| check_superseded_tracks calls new 2-arg update_track with 3 args (ArgumentError)supply/lib/supply/uploader.rb | High | Caught | Caught | – | Missed | Caught | – |
| validate_only always fails: second begin_edit hits the still-active validated editsupply/lib/supply/uploader.rb | High | Caught | Caught | – | Caught | Caught | – |
| Metadata-only uploads fail because a track release lookup is always requiredsupply/lib/supply/uploader.rb | High | Caught | Caught | – | Missed | Caught | – |
| New release_status option is never applied to created track releasessupply/lib/supply/uploader.rb | Medium | Caught | Missed | – | Caught | Caught | – |
| latest_version picks release by lexical name (e.g. 9.0 over 10.0)supply/lib/supply/client.rb | Medium | Caught | Caught | – | Caught | Caught | – |
| supply init crashes when selected track has no releases (nil.name / nil releases)supply/lib/supply/setup.rb | Medium | Caught | Missed | – | Caught | Caught | – |
| track_version_codes calls flat_map on nil releases before the || [] fallbacksupply/lib/supply/client.rb | Medium | Caught | Caught | – | Caught | Caught | – |
| update_rollout/fetch_track_and_release dereference nil track.releasessupply/lib/supply/uploader.rb | Low | Missed | Missed | – | Missed | Caught | – |
| Changelog upload aborts on any locale lacking <version_name>.txt (was skipped)supply/lib/supply/uploader.rb | Medium | Caught | Missed | – | Missed | Missed | – |
| Listings/images/screenshots re-uploaded once per version codesupply/lib/supply/uploader.rb | Low | Caught | Missed | – | Missed | Caught | – |
| Client#tracks returns nil when Play API omits empty tracks arraysupply/lib/supply/client.rb | Low | Caught | Missed | – | Missed | Missed | – |
| release_listings fallback to 'beta' empties track list and then dereferences nilsupply/lib/supply/client.rb | Low | Missed | Missed | – | Missed | Caught | – |
| update_track marks rollout=1.0 as inProgress with userFraction 1supply/lib/supply/uploader.rb | Low | Missed | Caught | – | Missed | Missed | – |
| Supply specs still use V2 Androidpublisher constant and 3-arg update_tracksupply/spec/client_spec.rb | Medium | Missed | Missed | – | Missed | Caught | – |
| Caught | 20/26 | 15/26 | 3/3 | 14/26 | 15/26 | 1/3 | |
LocalAIGo · 3 defects
| Defect | Severity | Codity | CodeRabbit | Greptile | CodeAnt | GitHub Copilot | Cursor Bugbot |
|---|---|---|---|---|---|---|---|
| Forced-open mode re-runs closing-only logic on answer text, so the PR's own test failspkg/reasoning/reasoning.go | High | Caught | Missed | Missed | Missed | Missed | Caught |
| Stray closing tag after a paired block turns earlier visible answer into reasoningpkg/reasoning/reasoning.go | Medium | Caught | Missed | Caught | Caught | Caught | Caught |
| Forced-open auto-detect never fires when use_tokenizer_template is set, since predInput is emptycore/http/endpoints/openai/chat.go | Medium | Caught | Caught | Caught | Missed | Caught | Missed |
| Caught | 3/3 | 1/3 | 2/3 | 1/3 | 2/3 | 2/3 | |
BlueprintTypeScript · 4 defects
| Defect | Severity | Codity | CodeRabbit | Greptile | CodeAnt | GitHub Copilot | Cursor Bugbot |
|---|---|---|---|---|---|---|---|
| Lazy MDX page loads after Documentation's scroll/highlight hooks run, breaking deep linkspackages/docs-app/src/components/mdxRegistry.ts | Medium | Caught | Missed | Missed | Caught | Missed | Missed |
| MDX code fence `ts copy` no longer wrapped in .docs-copyable-import, so copy button disappearspackages/docs-theme/src/components/mdxComponents.tsx | Low | Missed | Missed | Caught | Missed | Caught | Missed |
| ?examples mode no longer hides Alert prose because MDX wrapper is not inside .docs-sectionpackages/docs-theme/src/components/page.tsx | Low | Missed | Missed | Caught | Caught | Missed | Missed |
| getMdxComponents called per render gives new component types, remounting examples on theme togglepackages/docs-app/src/components/blueprintDocs.tsx | Low | Caught | Caught | Missed | Caught | Caught | Missed |
| Caught | 2/4 | 1/4 | 2/4 | 3/4 | 2/4 | 0/4 | |
AppsmithTypeScript · 2 defects
| Defect | Severity | Codity | CodeRabbit | Greptile | CodeAnt | GitHub Copilot | Cursor Bugbot |
|---|---|---|---|---|---|---|---|
| ConnectionSuccess tests share an uncleared dispatch mock; call indices only pass by test orderapp/client/src/pages/Editor/gitSync/Tabs/__tests__/ConnectionSuccess.test.tsx | Low | Caught | Missed | Caught | Missed | Caught | Missed |
| handleClickOnBack useCallback omits isGitConnectV2Enabled, so a late flag change uses a stale valueapp/client/src/pages/Editor/gitSync/DisconnectGitModal.tsx | Low | Caught | Caught | Missed | Caught | Missed | Missed |
| Caught | 2/2 | 1/2 | 1/2 | 1/2 | 1/2 | 0/2 | |
Cal.comTypeScript · 9 defects
| Defect | Severity | Codity | CodeRabbit | Greptile | CodeAnt | GitHub Copilot | Cursor Bugbot |
|---|---|---|---|---|---|---|---|
| Null/missing seatsShowAttendees now defaults to showing all seated attendees in emailspackages/emails/templates/attendee-scheduled-email.ts | High | Caught | Missed | Missed | Missed | Missed | Missed |
| Reschedule-request attendee filter never runs; builder never sets seats fieldspackages/emails/templates/attendee-was-requested-to-reschedule-email.ts | High | Caught | Missed | Missed | Missed | Missed | Missed |
| Reason filter in admin booking reports table is never sent to the APIapps/web/modules/settings/admin/booking-reports-view.tsx | Medium | Caught | Caught | Caught | Caught | Caught | Caught |
| Global block stores raw-case booker email/domain so normalized block checks miss itpackages/trpc/server/routers/viewer/admin/addToWatchlist.handler.ts | Medium | Caught | Missed | Missed | Missed | Caught | Caught |
| Bulk blocklist modal previews only first report's domain but blocks every selected domainapps/web/modules/settings/admin/components/add-to-blocklist-modal.tsx | Medium | Caught | Missed | Caught | Caught | Missed | Missed |
| Bulk add-to-watchlist is non-transactional and leaves partial state on failurepackages/trpc/server/routers/viewer/admin/addToWatchlist.handler.ts | Low | Missed | Caught | Caught | Missed | Missed | Missed |
| USERNAME watchlist type accepted but handler stores email domain as the valuepackages/trpc/server/routers/viewer/admin/addToWatchlist.schema.ts | Low | Caught | Missed | Missed | Caught | Caught | Missed |
| Duplicate reportIds in input cause spurious NOT_FOUND errorpackages/trpc/server/routers/viewer/admin/addToWatchlist.handler.ts | Low | Missed | Caught | Missed | Caught | Missed | Missed |
| Delete dialog and details sheet use translation keys absent from en localeapps/web/modules/settings/admin/booking-reports-view.tsx | Low | Caught | Missed | Missed | Missed | Missed | Caught |
| Caught | 7/9 | 3/9 | 3/9 | 4/9 | 3/9 | 3/9 | |
Run it on your own pull requests
The fastest benchmark is your own codebase. Codity reviews your next PR in minutes.
