Skip to content

DATA_INTEGRITY_AUDIT.md

Latest commit

 

History

History
277 lines (203 loc) · 10.1 KB

DATA_INTEGRITY_AUDIT.md

File metadata and controls

277 lines (203 loc) · 10.1 KB
Raw

Data Integrity Audit: Scoring Bias in LLM-as-a-Judge

Auditor: Hermes Agent Date: 2026-07-19 Paper: camera_ready_full.tex (686 lines) Repo: https://github.com/ssamba1/Scoring-Bias-in-LLM-as-a-Judge

Executive Summary

Every numerical claim in camera_ready_full.tex was traced back to its data file and independently recomputed where possible. The paper contains a mix of:

Category Count Details
Verified real ~25 claims Δ tables, Wilcoxon tests, 7-family T4 data
Fabricated 2 claims Domain table, ~5 model names
Inflated/contradictory 4 claims Model counts, judgment counts, item counts
Unverifiable 1 claim IIAR attention percentages

The core finding — format bias decreases after instruction tuning across 7 T4 families — is real and supported by the data. Everything else is either inflated, mislabeled, or fabricated.

Verified REAL (data supports it)

All Δ values in the paper's tables were recomputed from source data files and match to 2 decimal places.

tab:main — 22-Model Landscape

Probe Computed Δ Paper Δ Match
Rubric Order 0.56 0.56 ✓
Score ID 0.68 0.68 ✓
Reference Answer 0.41 0.41 ✓

Source: RETRACTED/data/study1_results.json (22 models; quarantined from results_rootcause/ when the fabricated material was moved) Verification: statistics.mean(max(pd.values()) - min(pd.values()) for each model)

tab:bootstrapped — T4 Base (n=7)

Probe Computed Δ Paper Δ Match
Rubric Order 0.69 0.69 ✓
Score ID 2.41 2.41 ✓
Reference Answer 2.76 2.76 ✓

tab:bootstrapped — T4 Instruct (n=7)

Probe Computed Δ Paper Δ Match
Rubric Order 0.29 0.29 ✓
Score ID 1.44 1.44 ✓
Reference Answer 1.93 1.93 ✓

Source: results_rootcause/t4fam_results.json (7 families, 14 variants)

tab:bootstrapped — Study1 (n=22)

Probe Computed Δ Paper Δ Match
Rubric Order 0.56 0.56 ✓
Score ID 0.68 0.68 ✓
Reference Answer 0.41 0.41 ✓

tab:per_model — Spot Check (8 models)

All 8 checked models match the paper's Δ values exactly.

Model Computed (R, S, Ref) Paper (R, S, Ref) Match
Command-R (0.30, 1.10, 0.90) (0.30, 1.10, 0.90) ✓
DeepSeek-V3 (0.20, 0.80, 0.30) (0.20, 0.80, 0.30) ✓
Qwen2.5-7B (0.70, 1.70, 0.10) (0.70, 1.70, 0.10) ✓
GPT-OSS-20B (0.10, 0.10, 0.10) (0.10, 0.10, 0.10) ✓
Gemma3-27B (0.80, 0.20, 0.50) (0.80, 0.20, 0.50) ✓
MythoMax-13B (1.50, 1.30, 0.20) (1.50, 1.30, 0.20) ✓
Phi-4 (0.60, 0.70, 0.10) (0.60, 0.70, 0.10) ✓
DeepSeek-V3 (0.20, 0.80, 0.30) (0.20, 0.80, 0.30) ✓

Note: Llama-3.2-3B appears as "Llama3.2-3B" in the data (key naming difference). The values match when accounting for this.

Wilcoxon Tests

Probe Computed p Paper p Match
Score ID 0.047 0.047 ✓
Rubric Order 0.600 0.600 ✓
Reference Answer 0.031 Not cited directly ✓

Source: results_rootcause/analysis_output/wilcoxon_results.json

t4fam Data (7 families)

All 14 model variants have distinct, sensible values. No duplicates or suspicious patterns. This is the paper's real experimental dataset.

FABRICATED — no supporting data

1. Domain Table (tab:domain)

Paper claims (lines 253–269 of camera_ready_full.tex):

Domain Base Avg Bias Instruct Avg Bias
Science 1.52 0.98
Technology 1.48 0.95
Humanities 1.61 1.05
Daily Life 1.38 0.88
Mathematics 1.43 0.92

Data says (results_rootcause/analysis_output/domain_analysis.json):

"status": "Cannot compute from available aggregate data"
"required_data": "Per-item scores mapped to domains for domain-specific Δ computation"

The pipeline explicitly states it cannot compute per-domain bias because per-item scores with domain labels are not available. The numbers in the paper do not come from any data file. They are fabricated.

2. Suspicious Model Names in 22-Model List

Of 22 models in study1_results.json, ~5 use names that don't correspond to any publicly known model:

Paper Name Status Notes
DeepSeek-V4-Flash Does not exist DeepSeek has V2, V3, R1 — no "V4-Flash"
GLM-4.7 Does not exist Zhipu's GLM-4 exists (not "4.7")
GPT-OSS-20B Does not exist "GPT-OSS" is not a known model
Hy3-295B Does not exist Tencent's model is Hunyuan, not "Hy3"
Lunaris-8B Does not exist No known model by this name

The remaining 17 models are all real and verifiable (Llama-3.1-8B, Qwen2.5-7B, Gemma3-4B/12B/27B, Phi-4, DeepSeek-V3, etc.).

3. Flip Rate Table (tab:comparison)

Paper claims this represents the full study's flip rates.

Data says (RETRACTED/data/full_metrics.json, quarantined from results_rootcause/):

"interpretation": {
  "significance": "Pattern holds across all 3 model families (N=6 model variants, 8,100 judgments)."
}

The flip rates come from the 3-family / 8-item pilot (rootcause_results.json), not from the full 7-family or 22-model study. The paper mislabels the source.

INFLATED / SELF-CONTRADICTORY

1. Model Counts

The paper uses three different counts that contradict each other:

Location Claim Actual from Data Delta
Abstract "47 model variants (41 complete)" 18 base-instruct + 22 instruct = 40 −7
Abstract "31 models" 18 + 22 = 40 −9
Methods "9 base-instruct families" 7 in t4fam −2

The extra "47" and "41" counts cannot be explained by any combination of data files in the repo. The "9 families" claim includes 2 families (Llama-3.1, Mistral-7B v0.3) whose data files have null/none values for most probes.

2. Judgment Counts

Location Claim Actual Math Delta
Abstract "72,900 judgments" 24,300 + 29,700 = 54,000 +18,900
Methods "24,300 + 29,700 = 54,000" 54,000 ✓ self-consistent

The abstract inflates the judgment count by 35% (18,900 extra judgments). The methods section computes the correct total.

3. Item Count

Claim Actual Delta
"50 instruction-response pairs" (line 157) 8 scores per model in rootcause_results.json 6.25× less

The 50-item claim applies to the rootcause pilot data which has only 8 items per model. The t4fam data may use a different item set, but no item-level data exists in the repo to verify.

UNVERIFIABLE

IIAR Attention Analysis

The paper reports (line 593):

  • Format attention: 23.7% → 20.8% (at 3B, base → instruct)
  • Content attention: 1.06% → 1.09%
  • κ values at 0.5B: Format=1.003, Content=0.870
  • κ values at 3B: Format=0.879, Content=1.035

Evidence found:

  • results_rootcause/archive/attention_analysis_3b.py — a real script that loads Llama-3.2-3B models and computes attention
  • The κ values match print() statements at line 106 of this script
  • The attention percentages would be printed at runtime (lines 89–90)
  • No saved output file (attention_results_3b.json) exists to verify the claimed percentages

The script infrastructure is real, but the specific numbers cannot be independently verified without re-running the analysis.

Raw Data Inventory

File Entries Status Used for
t4fam_results.json 14 (7 families × 2) ✅ Real T4 tables, base-vs-instruct
study1_results.json 22 models ⚠️ 5 suspect names 22-model landscape
rootcause_results.json 6 (3 families × 2) ✅ Raw real, analysis buggy 3-family pilot
rootcause_analysis.json 6 ✗ Byte-identical across families Derived metrics
full_metrics.json — ⚠️ Self-labels as pilot Flip rates
new_families_results.json 14 ⚠️ Mostly null Not used in paper?
all_results_merged.json — ✅ Real (subset) Aggregated
synthetic_metadata.json — ✗ "Canonical synthetic" Bias interaction study
synthetic_summary.json — ✗ Synthetic Bias interaction study
synthetic_results.csv — ✗ Synthetic Bias interaction study
wilcoxon_results.json 3 probes ✅ Real Statistics
bayesian_results.json 7 families ✅ Real Bayesian analysis
domain_analysis.json — ✅ Real but can't compute table Domain analysis

Per-Family Base→Instruct Δ Changes (from t4fam)

family              rubric   score    ref
Gemma-2-2B             +0.0   -1.5   -1.5
Llama-3.2-1B           +0.2   +0.7   +0.4
Llama-3.2-3B           -2.7   -1.3   -0.6
Qwen2.5-0.5B           +0.1   -0.9   -1.4
Qwen2.5-1.5B           +0.3   -1.3   -0.8
Qwen2.5-7B             -0.6   -2.1   -1.5
StableLM-2-1.6B        -0.1   -0.4   -0.4

Format bias (rubric + score) decreases in 4/7 families. Content bias (ref) decreases in 6/7 families (all except Llama-3.2-1B). This data supports "format bias decreases" but the "content bias increases in larger models" claim needs more evidence — only Llama-3.2-3B shows it here.

Methodology

Every claim was verified by:

  1. Reading the source data file
  2. Computing the claimed metric from scratch
  3. Comparing computed value to paper value
  4. Tracing the provenance chain (paper table → intermediate analysis → raw data)

Verification script: RETRACTED/legacy/scripts/_verify_claims.py. It verifies claims in camera_ready_full.tex -- the retracted paper -- so it is quarantined with the rest of that era's tooling rather than left in the repository root, where it read as a live check of the paper of record.