DA-RAC: Distance-Aware Calibration of LLM Judges for Trustworthy AI Auditing

Cheng Wu*, Vishal Anand*, Jaya Mandivarapu, Xiya Liu, Rui Zhuang
Microsoft
ICML — Culture x AI

*Indicates Equal Contribution

Learning from model weights


Figure 1. DA-RAC workflow: (1) input target and precedent pool; (2) compute hybrid distance combining instruction embeddings and graph edit distance; (3) select k nearest precedents; (4) weight by softmax distances; (5) emit a judgement together with neighbourhooddifficulty signal.

Abstract

Generative AI systems increasingly produce cultural artifacts, yet their evaluation is often reduced to context-free scoring. We argue that LLM-asa-judge systems should instead be understood as interpretive technologies: when they evaluate generated texts, explanations, critiques, or creative outputs, they make situated judgments about genre, audience, precedent, and value. This framing reveals a failure mode we call contextinduced interpretive miscalibration, in which irrelevant or mismatched reference examples distort the judge’s decision boundary. We introduce DA-RAC, a distance-aware reference anchoring method that retrieves semantically and structurally proximate labelled precedents for each judgment scenario, weights them by distance, and exposes neighbourhood difficulty as a signal for human review. On LLM-judge benchmarks, DARAC improves calibration relative to zero-shot, rubric-based, and static-precedent baselines; we use these results as a technical probe of reference dependence rather than as a complete cultural benchmark. The broader contribution is a design pattern for cultural AI evaluation: judgments should be grounded in relevant, inspectable, and contestable interpretive precedents. DA-RAC offers one computational mechanism for embedding contextual sensitivity upstream in AI evaluation while preserving human agency in ambiguous cultural cases.

Results

DA-RAC shows significant gains over Vanilla, centroid, and CoT

Method R1 R2 R3 Mean ± Std
DA-RAC 86 82 86 84.7 ± 1.9
Static (diverse) 80 80 82 80.7 ± 0.9
Static (random) 78 76 74 76.0 ± 1.6
Vanilla (zero-shot) 74 72 72 72.7 ± 0.9
Static (centroid) 70 68 68 68.7 ± 0.9
CoT Rubric 64 64 64 64.0 ± 0.0
LLMEval2 baseline — — — 64.0

BibTeX

To be shared after ICML conference