Publish a non-consumptive release: measurements over the corpus, not the corpus

"Why not the corpus?" was a fair challenge and the exclusion had been a reflex.
Checked the actual position: the volumes carry no licence, only the imprint
(project owner wizarat al-thaqafa, publisher al-Dar al-Arabiyya li-l-Kitab
2014-15), and the Tunisian National Library catalogues the set as a physical
holding (A-8-314464 1..10) with no open-access edition. The poems themselves are
largely 19th-century and out of copyright; what is protected is the edition. So
redistributing the text would be republishing somebody else book.

But reproducibility does not need the verse, it needs the measurements.
scripts/export_release.py emits:

  * corpus_manifest.jsonl - 1,511 records: ids, pages, poet, form, purpose,
    line count, strophe sizes, per-line letter and syllable counts, rhyme-key
    sequences, full form-score breakdowns, metre regularity, attestation flag,
    and a salted text hash
  * corpus_stats.json - per-form counts, length histogram, rhyme-key
    frequencies, score medians
  * eval_prompts.jsonl - the 174 held-out prompts with fingerprints stripped,
    so anyone can benchmark any model on the same task

Every AUC, distribution and attestation rate in docs/ is re-derivable from these
without the text, and the published salt lets anyone who owns the volumes verify
their rebuild hash-for-hash.

Leak-checked: of corpus lines with >=3 words, 0 appear verbatim in the release.
Rhyme keys ARE included; they are the analytical object of the study and
withholding them would make the results unauditable. The trade-off is stated in
DATA.md rather than buried.

DATA.md also documents the path to releasing the text properly: ask the Ministry
(the named scientific committee is in the imprint), then gate it on HF or Zenodo
with a click-through licence, and never in git history, which is permanent and
survives a repo going public.

Author: cnemri

DATA.md

@@ -73,9 +73,71 @@ This is the same category of derived statistic as the publicly released Google
 Books n-gram corpora, which are published from in-copyright works on the basis
 that frequency data over short spans is non-expressive.
 
+## Why the corpus is not redistributed
+
+Not a reflex — the position was checked:
+
+* The volumes carry no licence statement, only an imprint:
+  *«الإشراف العام وصاحب المشروع: وزارة الثقافة — تونس»*, published by
+  الدار العربية للكتاب, 2014–15.
+* The Tunisian National Library catalogues it as a **physical holding**
+  (call numbers `A-8-314464 1…10`, 25 cm) for loan and consultation. There is
+  no open-access edition.
+* The *poems* are largely 19th-century and anonymous, so the underlying works
+  are almost certainly out of copyright. What is protected is the **edition** —
+  the selection, transcription, arrangement and editorial apparatus, which is
+  precisely the labour that took 171 manuscript notebooks and made them
+  readable. Tunisian law additionally treats expressions of folklore as
+  national heritage whose exploitation runs through the Ministry.
+
+So redistributing the text would be republishing someone else's book. Shipping
+*measurements over* it is a different act, and that is what `release/` is.
+
+## `release/` — the non-consumptive release
+
+Reproducibility does not actually require the verse. Almost every scientific
+claim in `docs/` is a claim about *measurements*, and those can be published.
+
+| file | contents |
+|---|---|
+| `corpus_manifest.jsonl` | one record per poem (1,511): ids, page range, poet, form, purpose, line count, strophe sizes, per-line letter and syllable counts, rhyme-key sequence, full form-score breakdown, metre regularity, attestation flag, salted text hash |
+| `corpus_stats.json` | aggregate distributions: per-form counts, length histogram, rhyme-key frequencies, per-form score medians |
+| `eval_prompts.jsonl` | the 174 held-out task specifications, stripped of fingerprints — a benchmark anyone can run against any model |
+
+With these you can re-derive the depth sweep, the AUCs, the attestation rates,
+the score distributions and the form-validator behaviour, and you can benchmark
+your own model on the same held-out prompts — **without the text**.
+
+The salted hash (`sha256(salt || normalised_text)[:32]`, salt published) lets
+anyone who *does* own the volumes verify their rebuild matches ours poem by
+poem, without us publishing a single line.
+
+**Verified:** of the corpus lines of ≥3 words, **0** appear verbatim anywhere in
+the release.
+
+**Rhyme keys are included** — 1–3 terminal graphemes per line. They are the
+analytical object of the entire study, and withholding them would make the
+results unauditable. That is a deliberate trade-off, stated rather than buried.
+
+## If you want the actual text released
+
+The right path is permission, not a workaround:
+
+1. **Ask the owner.** The project owner is وزارة الشؤون الثقافية; the named
+   scientific committee is الحفناوي عمايرية، محمد رؤوف بلحسن، الحبيب الحمدوني،
+   الهادي جلاب، أحمد الخصوصي، أحمد خواجة. This was state-funded heritage
+   documentation whose stated purpose was to rescue a sleeping archive — a
+   non-commercial research licence is well within its spirit.
+2. **Then gate it.** Hugging Face datasets with gated access, or Zenodo with a
+   restricted-access record, both support a click-through licence and give a
+   citable DOI.
+3. **Never in git history.** Even in a private repo, a commit is permanent and
+   survives a later flip to public. Bulk text belongs in object storage with IAM
+   (here: `gs://nemri-genai-bb-tunifolk/`), referenced by URI.
+
 ## Reproducing without the corpus
 
-You cannot — and that is deliberate. `make calibrate`, `make negatives`,
-`make tunisianity`, `make dataset` and `make acceptance` all take
-`CORPUS=/path/to/classified.json`. Obtain the volumes yourself and run the
-upstream digitisation pipeline; this repository ships the method, not the text.
+`make calibrate`, `make negatives`, `make tunisianity`, `make dataset` and
+`make acceptance` all take `CORPUS=/path/to/classified.json`. Obtain the volumes
+and run the upstream digitisation pipeline. This repository ships the method and
+the measurements; the text stays with its owner.

Makefile

@@ -5,7 +5,7 @@ GCS     ?= gs://nemri-genai-bb-tunifolk/rlft/v1
 PROJECT ?= nemri-genai-bb
 REGION  ?= us-central1
 
-.PHONY: help venv test lint calibrate negatives tunisianity acceptance dataset upload deploy validate launch status clean
+.PHONY: help venv test lint release calibrate negatives tunisianity acceptance dataset upload deploy validate launch status clean
 
 help:
 	@grep -E '^[a-z-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN{FS=":.*?## "}{printf "  \033[36m%-12s\033[0m %s\n",$$1,$$2}'
@@ -28,6 +28,9 @@ negatives: ## generate the adversary: base-model attempts at the same prompts
 tunisianity: ## fit + hold-out validate the contrastive discriminator
 	$(PY) scripts/train_tunisianity.py --corpus $(CORPUS)
 
+release: ## export the non-consumptive release (measurements, no verse)
+	$(PY) scripts/export_release.py --corpus $(CORPUS)
+
 acceptance: ## GATE: reward must separate corpus from base-model output
 	$(PY) scripts/acceptance.py --corpus $(CORPUS)
 

README.md

@@ -171,6 +171,14 @@ cannot reconstruct the work: chaining the shipped 4-grams back into running text
 gibberish, though 20/26 of a real line's 4-grams are present — so it is a membership
 oracle, not an extraction oracle. That distinction is stated rather than glossed.
 
+The corpus is an in-copyright edition (BNT holds it as a physical volume; there is no
+open-access edition), so instead of the verse we publish **measurements over it** in
+[`release/`](release/): a 1,511-poem manifest with per-line lengths, rhyme-key sequences,
+form-score breakdowns and salted text hashes, aggregate statistics, and the 174 held-out
+benchmark prompts. Every AUC and distribution in `docs/` is re-derivable from these, and
+anyone who owns the volumes can verify their rebuild hash-for-hash — without a line of
+verse leaving its owner. `make release`.
+
 ## Licence
 
 Apache-2.0 for the code. The corpus is © the Tunisian Ministry of Culture and is neither