LLM-Relabeled SemCor

This repository releases two LLM-relabeled versions of the SemCor training labels used by supervised English word sense disambiguation systems.

The release is intentionally label-centered. It includes replacement semcor.gold.key.txt files, detailed relabeling JSONL, metadata, checksums, and reconstruction scripts. It does not commit the unchanged SemCor XML because that file contains upstream corpus text; use scripts/reconstruct_dropin.py to rebuild a full drop-in SemCor folder from your existing Raganato/SemCor copy.

Datasets

Paper name	Folder	Relabeler	Valid labels	Invalid fallback	Disagreement vs. SemCor
SemCor-GPT-5.5	data/semcor-gpt55-low-p003-v1	GPT-5.5 low reasoning, lexEn p003	226,032	4	21.30%
SemCor-Gemma-4-31B	data/semcor-gemma4-31b-reasoning-p003-v1	Gemma 4 31B reasoning, lexEn p003	224,905	1,131	21.85%

Both datasets cover the same 226,036 SemCor instances in the Raganato 2017 framework order. Invalid model outputs fall back to the original SemCor gold label in the drop-in gold-key file so existing training pipelines still see one label per instance.

Disagreement with SemCor is not an error-rate estimate. The downstream use case is a controlled training intervention: replace only the SemCor gold-key file, keep the XML, tokenization, instance IDs, model code, and hyperparameters fixed, and measure whether supervised WSD improves.

Quick Use

Reconstruct a full drop-in training folder:

python scripts/reconstruct_dropin.py \
  --dataset data/semcor-gpt55-low-p003-v1 \
  --semcor-xml /path/to/original/SemCor/semcor.data.xml \
  --output-dir build/semcor-gpt55-low-p003-v1

Then point your WSD trainer at:

build/semcor-gpt55-low-p003-v1/SemCor/semcor.data.xml
build/semcor-gpt55-low-p003-v1/SemCor/semcor.gold.key.txt

For Gemma, replace the dataset path with data/semcor-gemma4-31b-reasoning-p003-v1.

Files

Each dataset folder contains:

• semcor.gold.key.txt.gz: Raganato-format replacement gold-key file.
• detailed/*.jsonl.gz: one row per SemCor instance with original gold keys, candidate sense keys, model label, raw answer, validity, attempts, and token counts.
• disagreement_stats.json: aggregate disagreement against original SemCor labels.
• invalid_fallbacks.jsonl: rows where the relabeler did not produce a valid label.
• metadata.json: release metadata and provenance.
• checksums.sha256: checksums for files in that dataset folder.

Verification

Run:

python scripts/verify_release.py

The verifier checks row counts, valid/invalid counts, checksums, and that no unchanged SemCor XML or zip drop-in archive is committed.

Source Lineage

The source instance set is SemCor as packaged in the Raganato et al. (2017) WSD Unified Evaluation Framework with WordNet 3.0 sense keys. See NOTICE and docs/licensing_notes.md for licensing and redistribution notes.

The full experiment provenance, training runs, costs, and lab notebook live in:

https://github.com/GliteTech/research-semcor-relabeling

Licensing

Code and documentation are Apache-2.0. The SemCor-derived dataset files, including Glite-generated labels and provenance, are distributed under the WordNet 1.6 / Semantic Concordance 1.6 license. Preserve both the Princeton license and the SemCor 3.0 mapping notice with every copy. Physical copies accompany each dataset directory and are included by the drop-in reconstruction script.