Evaluating AI Code Review: Correctness Before Confidence

Correction, October 1, 2026: an earlier version misidentified a Promise-returning function as incorrect and included unsupported benchmark, production and cost claims. Those claims have been removed. This article presents an evaluation method; it does not report a completed fine-tuning experiment.

Start with the function's contract

A code-review assistant should explain what breaks, under which conditions, and why a proposed change fixes it. A confident list of preferences is much less useful.

Consider this function:

function fetchUser(id: number) {
  return fetch('/api/user/' + id)
}

It returns the Promise produced by fetch. Its caller can await that Promise. Adding async and await is not required for this behavior, and replacing concatenation with a template literal changes presentation rather than correctness.

The useful review starts with a question: what does the caller expect? If the contract is to return a raw Response, this wrapper may be sufficient. If it promises parsed user data or rejects unsuccessful HTTP responses, the implementation is incomplete. That conclusion needs the caller or specification, not a guess about the function's name.

A small test can establish the narrow promise contract without a network request. The injected transport below makes the dependency explicit:

import { expect, test } from 'bun:test'

function fetchUser(
  id: number,
  request: (url: string) => Promise<Response>,
) {
  return request('/api/user/' + id)
}

test('returns the transport promise to its caller', () => {
  const pending = Promise.resolve(new Response('{}'))
  const requested: string[] = []
  const result = fetchUser(42, url => {
    requested.push(url)
    return pending
  })

  expect(result).toBe(pending)
  expect(requested).toEqual(['/api/user/42'])
})

This test establishes what the wrapper does. It does not establish that the endpoint returns valid user data, that the caller handles errors, or that the API meets its product requirements. A useful reviewer keeps those questions separate.

Build fixtures around decisions

Before changing a prompt or training a model, write down the behavior the review should protect. Include working code as well as defective code. A tool that objects to every function can find many defects while creating too much work to use.

This portfolio's workflow code supplies a small, concrete example. In spacetimedb/shared/workflow.ts, only a failed task may be retried. The decision stage deliberately fails on its first attempt. Retrying it returns the task to queued and increments its attempt and revision.

A review fixture could contain that implementation, its relevant tests, and one controlled change: allow retry while a task is already running. The expected finding would identify the invalid transition and a test that demonstrates it. The unchanged implementation should not attract the same finding.

For each fixture, record:

• The stated contract and the context supplied to the reviewer.
• A known defect, or an explicit expectation that no defect exists.
• An executable reproduction where one is practical.
• Acceptable findings, irrelevant suggestions and unresolved questions.

Keep related versions of a function together when splitting development and evaluation examples. Otherwise a reviewer may encounter nearly the same example during tuning and testing, making the result harder to interpret.

Score the finding, not the tone

I would evaluate each comment against a short rubric:

Track missed defects and false positives separately. Also measure the time a person spends accepting, rejecting or investigating the comments. A high “useful” score can hide a tool that buries its one correct finding in ten distractions.

Have reviewers score outputs without model names where practical. Record disagreements and resolve them against the contract. If the supplied context cannot settle a question, mark it unresolved rather than forcing a verdict.

Compare the same task

A comparison should record the exact model and configuration, prompt, available code context, output limit, tools and number of attempts. Give each candidate the same task context, or explicitly describe the different conditions being compared.

Keep an untouched evaluation set and record its size. If fine-tuning is part of the experiment, record the base model, training data selection, split rules and training configuration as well. Report enough examples and failure categories for another engineer to understand what the headline result includes.

Cost needs a denominator too: input and output sizes, retries, total reviews and the cost included in the calculation. For a self-hosted model, account for the hardware allocation and utilization assumptions. A per-review figure without that context is difficult to compare or reproduce.

Define the decision before running the experiment

The next step should depend on a preselected outcome. For example: continue only if the reviewer catches the target defect classes while keeping false positives and human review time within limits agreed by the team. Publish those limits with the result rather than choosing them after seeing the scores.

If a simpler prompt, better repository context or a deterministic check solves the problem, test that option before investing in training. Fine-tuning is one possible intervention. The goal is a review process that helps a person make a better release decision.

The state experiment offers a small workflow to inspect, and the System in motion shows how a failed check can withhold output. Both make the rule visible. A code-review evaluation should make its judgment just as inspectable.