DA-RAC: Distance-Aware Calibration of LLM Judges for Trustworthy AI Auditing
Abstract
Generative AI systems increasingly produce cultural artifacts, yet their evaluation is often reduced to context-free scoring. We argue that LLM-asa-judge systems should instead be understood as interpretive technologies: when they evaluate generated texts, explanations, critiques, or creative outputs, they make situated judgments about genre, audience, precedent, and value. This framing reveals a failure mode we call contextinduced interpretive miscalibration, in which irrelevant or mismatched reference examples distort the judge’s decision boundary. We introduce DA-RAC, a distance-aware reference anchoring method that retrieves semantically and structurally proximate labelled precedents for each judgment scenario, weights them by distance, and exposes neighbourhood difficulty as a signal for human review. On LLM-judge benchmarks, DARAC improves calibration relative to zero-shot, rubric-based, and static-precedent baselines; we use these results as a technical probe of reference dependence rather than as a complete cultural benchmark. The broader contribution is a design pattern for cultural AI evaluation: judgments should be grounded in relevant, inspectable, and contestable interpretive precedents. DA-RAC offers one computational mechanism for embedding contextual sensitivity upstream in AI evaluation while preserving human agency in ambiguous cultural cases.
Results
DA-RAC shows significant gains over Vanilla, centroid, and CoT
| Method | R1 | R2 | R3 | Mean ± Std |
|---|---|---|---|---|
| DA-RAC | 86 | 82 | 86 | 84.7 ± 1.9 |
| Static (diverse) | 80 | 80 | 82 | 80.7 ± 0.9 |
| Static (random) | 78 | 76 | 74 | 76.0 ± 1.6 |
| Vanilla (zero-shot) | 74 | 72 | 72 | 72.7 ± 0.9 |
| Static (centroid) | 70 | 68 | 68 | 68.7 ± 0.9 |
| CoT Rubric | 64 | 64 | 64 | 64.0 ± 0.0 |
| LLMEval2 baseline | — | — | — | 64.0 |
BibTeX
To be shared after ICML conference