LLM-as-Judge Evaluation: Rubrics, Bias, and Calibration
Use an LLM judge for qualities that require interpretation—such as completeness, clarity, or evidence use—after deterministic checks have handled facts, schemas, permissions, and exact labels. Give the judge a task-specific rubric, hide irrelevant model labels, calibrate it on human-reviewed cases, and review disagreements instead of treating its score as ground truth.
Choose the cheapest valid grader for each claim
| Evaluation question | Preferred grader | Pass evidence | Main limitation |
|---|---|---|---|
| Does output parse? | Schema or syntax validator | Valid parse and required fields. | Valid structure can still be wrong. |
| Does an exact label match? | Code comparison | Candidate equals owned reference label. | Requires a defensible reference. |
| Is a material claim supported? | Claim-to-source check plus reviewed rubric | Evidence span supports the scoped claim. | Retrieval and interpretation can both fail. |
| Is an answer complete? | Point-based rubric or LLM judge | Required elements present with reasons. | Judge may reward verbosity. |
| Which answer is better? | Blinded pairwise judge with order reversal | Stable preference and rubric rationale. | Position and style can influence choice. |
| Is the run safe? | Hard policy assertions plus human review | No prohibited action and reviewed edge cases. | A judge score cannot waive a violation. |
OpenAI's official evaluation guide separates test data from testing criteria and shows graders as explicit criteria. Preserve that separation in any framework: the same output can need several graders, and one grader should not silently redefine the task.
Write a rubric that another reviewer can apply
Begin with the evaluation question and risk of a wrong grade. Define each dimension in observable terms, state what evidence the judge may use, and describe boundaries between score levels. A completeness rubric might list required decision factors; a groundedness rubric should ask whether each material claim follows from supplied evidence. Avoid words such as “excellent” or “high quality” unless they are unpacked into concrete criteria.
Require a structured result containing the grade, criterion-level reasons, evidence references, and an abstain or cannot-determine path. Short, reviewable justifications are enough to diagnose a grader. If evidence is missing or contradictory, the correct result may be ungradable rather than a low score.
Keep task output and grader input distinct. Remove system labels unless source is part of the question. Normalize formatting irrelevant to the rubric. For pairwise grading, run both candidate orders and define what to do when preferences reverse. The dataset design guide calls for easy, difficult, boundary, and disagreement cases.
Calibrate bias against human-reviewed anchors
The primary paper “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” investigates position, verbosity, self-enhancement, and reasoning limitations. Treat those as testable failure modes, not as a promise that one prompt removes them. Reverse answer order, vary length without changing substance, hide model labels, and include cases where favored style conflicts with the rubric.
Human review provides reference examples for calibration. Compare the automated grade with the reference for each criterion, keep disagreement visible, and route consequential disputes to a reviewer.
Pin the grader model, rubric, settings, content preparation, and code version. Regrade the baseline with the same configuration after a change. Otherwise, a score difference can reflect grader drift rather than application improvement.
Make disagreement an explicit state
Do not force every case into pass or fail. Route low-confidence grades, reversed pairwise choices, missing evidence, and human-judge disagreement into a review queue with a named disposition. Track whether review changes the case label, rubric, or product behavior. This prevents ambiguous examples from silently becoming calibration truth and reveals where the evaluation question itself is underspecified.
LLM judge failure cases
- The judge replaces invariants: it approves malformed JSON or a forbidden tool call because the answer sounds useful. Run hard checks first.
- Rubric imitation: a candidate repeats rubric terms without satisfying them. Require evidence and counterexamples.
- Length wins: a verbose answer appears more complete. Include concise reference answers and controlled length variants.
- Position wins: pairwise preference changes when answer order changes. Reverse the order and flag instability.
- Style preference: familiar phrasing affects the grade. Remove irrelevant labels and compare with human review.
- One score hides risk: a strong mean conceals critical failures. Segment outcomes and make severe cases release-blocking.
- Shared blind spot: correlated errors become false passes. Use deterministic checks, diverse anchors, and human escalation.
A calibrated implementation sequence
- Define the evaluation question. Name the decision the grade will support, the cost of false pass and false fail, and where human review is mandatory.
- Use deterministic checks first. Validate schemas, exact labels, required citations, forbidden actions, and state invariants in code.
- Draft the rubric. Define dimensions, evidence, score boundaries, abstention, and a structured grader output. Review it with domain owners.
- Build calibration cases. Include clear passes, clear failures, boundaries, concise and verbose equivalents, reversed pairs, and human disagreements.
- Pin and run the judge. Record grader version, rubric version, settings, case IDs, outputs, grades, and concise reasons.
- Analyze disagreement. Review false passes first, slice by known biases, and change the rubric only with a new baseline.
- Set release policy. Combine thresholds with critical-case gates and human review. Put judge changes through prompt regression testing and trace outcomes with LLM observability.
Decision summary
Use an LLM judge only for criteria that resist deterministic checking, and calibrate it against human-reviewed anchors. Version the rubric, vary presentation order, retain disagreement, and block releases on critical cases rather than an attractive aggregate score.
Primary and official sources
- OpenAI: Working with evals — official task, test-data, testing-criteria, run, and analysis workflow.
- Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” — primary research on LLM judges and documented bias modes.