An AI judge needs a tested connection to the people whose standards it is meant to apply. We can build that connection by measuring agreement with human reviews, examining disagreements, and revising the judge’s instructions. Anthropic’s evaluation guidance recommends closely calibrating model graders with human experts. [1]
This matters when several outputs could be acceptable. A project risk description, stakeholder response, or engineering explanation can succeed in different ways. The judge needs to recognize the relevant quality standard across those variations. Its fluent explanation of a verdict does not establish that it applies that standard consistently.
Measure agreement on the same examples
Start with a sample of outputs that humans have already reviewed against a clear standard. Record each verdict and its reason. Include acceptable and unacceptable examples, including difficult cases. Ask the AI judge to assess the same outputs without seeing the human verdicts.
Compare the decisions case by case. For a simple accept-or-reject review, agreement is the number of matching decisions divided by the number of examples. A spreadsheet is enough to start. The following counts are illustrative.
Here, the judge agrees on 28 of 40 examples, or 70%. The total also conceals eight false approvals. Inspect those separately because accepting an inadequate output may have a different consequence from rejecting an acceptable one. A high overall percentage can still leave an important failure pattern unresolved.
Turn disagreements into better instructions
The useful information is in the reasons for disagreement. Suppose a human rejects a risk entry because it never connects the uncertain event to a project objective. The AI accepts it because it is clearly written. That comparison identifies a missing decision rule in the judge’s prompt.
Revise the prompt to make the rule explicit. For example, require the entry to connect an uncertain event to a plausible effect on a stated objective. Preserve acceptable variations in wording. Where the context is insufficient, allow an unclear verdict and route it to human review.
AI can assist with this revision. Give it the current judging prompt, the human decisions and reasons, and the judge’s decisions and reasons. Ask it to identify recurring mismatches and propose the smallest changes that address them. Research comparing DSPy optimizers has demonstrated prompt optimization against human labels for hallucination detection. That result supports testing this approach; it does not establish performance for engineering reviews. [2]
Repeat the comparison and test the improvement
Run the revised judge on the same calibration examples and compare the results with the earlier version. Check whether false approvals declined and whether the revision introduced new disagreements. Keep the judge model and other settings fixed during this comparison so the prompt change is easier to assess.
Then evaluate a separate sample that was not used to revise the prompt. Keep its human verdicts out of the revision process. This checks whether the new instructions capture a transferable standard. Improvement only on the familiar examples can reflect overfitting.
Do not expect agreement to rise smoothly with every revision. A change that fixes one mismatch may create another. Retain a revision only when the evidence supports it, and repeat the check when the model, project context, or review standard changes.
A prompt for refining the judge
Help calibrate an AI judge to our human review standard.
Inputs: the current judging prompt, our review criteria, and calibration examples with human and AI verdicts and reasons.
Identify recurring disagreements and the decision rules behind them. Propose the smallest prompt revisions that address those mismatches while preserving valid alternatives.
Flag ambiguous human labels for clarification. Return a concise disagreement summary, a revised judging prompt, and a plan to compare both versions on calibration examples and a separate test sample. Do not claim improvement until the comparisons have been run.Notes
Join the EPM Network to access insights, influence our research, and connect with a community shaping the industry’s future.
Support us by sharing this article with your friends and colleagues, or over social media.
References
[1] Grace, M., Hadfield, J., Olivares, R., & De Jonghe, J. (2026, January 9). Demystifying evals for AI agents. Anthropic.
[2] Sarmah, B., Dutta, K., Grigoryan, A., Tiwari, S., Pasquali, S., & Mehta, D. (2024). A comparative study of DSPy teleprompter algorithms for aligning large language models evaluation metrics to human evaluation. arXiv.
[3] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv.