This repository releases the corpora, detector scores and judge annotations behind the paper Watermarking Turkish LLM Output: Detector Calibration, Scheme Fragility, and a Released Evaluation Benchmark. It is meant to be reusable independently of that paper: the score files let you check our numbers, and the texts let you evaluate a different detector or a different attack on the same material.
Everything here was produced by the code in this repository. The project rule is
that no reported number originates by hand: every number in the paper is
regenerated from these files by pilot/make_paper_numbers.py into
paper/numbers.json (figures come from pilot/make_figures.py). Transcription
from numbers.json into the manuscript's tables is manual, so
pilot/dev_tutarlilik_kapisi.py compares the two inferential tables against
numbers.json cell by cell and fails the release on any divergence. That check
exists because the gap once produced a real error, described in the paper's
Section 7.
| Component | Records | File(s) |
|---|---|---|
| Human Turkish windows (Wikipedia) | 1,500 | results_insan/insan_tr.jsonl |
| Human English windows (Wikipedia), word-matched | 1,500 | results_insan/insan_en.jsonl |
| Human Turkish windows (Wikisource, second register) | 1,000 | results_insan/insan_tr_wikisource.jsonl |
| Generated Turkish texts (4 arms) | 384 | results/gen_neg.jsonl, results/gen_pos_*.jsonl |
| Attacked texts (10 attacks × 4 arms) | 3,840 in 40 files | results/att_*.jsonl |
| Detector scores on human text (3 schemes) | 12,000 | results_insan/skor_{tr,en,tr_wikisource}.jsonl |
| Length-controlled rescoring (4 token budgets) | 14,161 | results_insan/skor_h2_token.jsonl |
| Key sweep (8 watermark keys × 4,000 windows) | 32,000 | results_insan/skor_anahtar.jsonl |
| Pairwise judge verdicts (2 judges) | 788 | results_insan/s2_fayda.jsonl |
| Combined score table | 6,336 | results/scores.csv |
The three schemes are KGW (green-list logit bias), EXP (Gumbel sampling) and SynthID (tournament sampling). The ten attacks are diacritic stripping at two rates, two morphological variants, each of those crossed with diacritics, self-paraphrase, self-laundering, round-trip machine translation, and laundering through an external commercial model.
- Generator:
Qwen/Qwen3-14B, fp16, on the hardware recorded inresults/env.json. - Watermarking toolkit: MarkLLM at commit
c45ddc40f7b7. - Prompt file is content-addressed; SHA-256 prefix
8fcbe4074b46. - Pre-registrations are commits in this repository's history, each made before the
corresponding data was collected:
8f8df72(S1 hypotheses),cbcb988(S2 protocol and decision rule),5c4f323(second register).
Because the pre-registrations are commits rather than documents, they can be dated against the data files. Note the limit of that guarantee: the hashes bind content and ordering, but the repository was first published on 2026-08-29, after the data was collected, so no third-party timestamp separates the registrations from the data. The asserted dates rest on the author's local history.
These are properties of the data, not of the paper's argument, and you should know them before reusing it.
- One generator. All generated text comes from Qwen3-14B. Five candidates were
run through a pre-registered acceptance gate and only this one passed; the gate
records are in
results_hpc/. - One watermark key on the attack axis. The key sweep covers the human-text null only, because that needs no regeneration. Every AUROC and robustness number is conditional on the single study key.
- EXP texts are fixed-length. EXP emits a fixed token count and never stops at a sentence boundary, so its texts differ structurally from the other arms. Any cross-scheme comparison of text quality is confounded by this.
- EXP replicates are not independent. Under deterministic decoding EXP's four seeds produce identical output, so the effective unit is the prompt rather than the text: 96 EXP records contain 24 distinct texts.
- Judge verdicts cover one arm. Meaning and fluency were judged on KGW-sourced texts only. They are not measurements of the EXP or SynthID arms.
- Windows are word-matched, not token-matched. The human windows target 365
words, which is a median of 1,017 tokens in Turkish and 529 in English. This
matters for any length-sensitive statistic;
skor_h2_token.jsonlprovides the token-matched rescoring at four budgets. - One judge produced one of the attacks. Claude Opus 5 generated the
launder_apitexts and also served as one of the two judges. The second judge is from a different model family for exactly this reason, but the conflict is inherent and is not fully removed.
python pilot/make_paper_numbers.py # -> paper/numbers.json
python paper/make_figures.py # -> paper/figs/The measurement scripts that produced the newer blocks are
pilot/dev_h2_token.py (length control), pilot/dev_anahtar_supurme.py (key
sweep) and pilot/dev_dejenere_kanit.py (separation evidence for the cells with
AUROC 1.000). Each writes a JSON report that make_paper_numbers.py reads.
Not uniform, because the components have different origins. See DATA_LICENSE.md.
In short: Wikimedia-derived text is CC BY-SA and requires attribution
(ATTRIBUTION.md, with the page list in ATTRIBUTION_pages.tsv); round-trip
translations are labelled CC BY-NC under the restrictive reading of the
translation model's licence; generated and laundered texts and judge verdicts are
CC BY 4.0; score and metric files are CC0; code is Apache-2.0.
Until the paper has a DOI, cite this repository and the Zenodo deposit it is archived in. If you use only the score files, the texts they were computed from are still the thing that needs attribution, because they are Wikimedia-derived.