DSpace9.3/Fix OAI double-escaping of metadata values (upstream #10721 regression)

[Kasinhou]

References

• Upstream defect: DSpace/DSpace#10721 fixed by DSpace/DSpace#11139 — "Sanitize non-characters during OAI indexing". That fix is what introduced this regression.
• Upstream commits: d8fbe16ede (main) → ported to dspace-9_x as 6a69e87406, both present on dtq-dev-9-base.
• Not yet reported upstream — it still affects DSpace main, 9_x, 8_x and 7.6.7. Worth filing.

Where this fix comes from

This is not new code. Both commits are cherry-picked from our own
upgrade/vanilla-7.6.7 branch (author: @milanmajchrak), where the identical upstream
defect arrived when that branch took vanilla 7.6.7. -x provenance lines are in both
commit messages.

This PR	Cherry-picked from	Original subject
040cddc29d	c6a1101410	Fix OAI double-escaping and complete the content-disposition allowlist
7f2fa13263	b28457142a	Also strip unpaired surrogates in the OAI sanitizer

dtq-dev (7.6.5) is unaffected and needs nothing — it predates upstream DSpace#11139
entirely, so ItemUtils.sanitize() does not exist there. That is why this only shows up
on the v9 base.

Deliberately NOT taken from c6a1101410

That source commit also rewrote webui.content_disposition_inline in
dspace/config/clarin-dspace.cfg. Excluded here: that key does not exist in
clarin-dspace.cfg on this branch (on v9 it lives in dspace.cfg), and it is an
unrelated 7.6.7 download-behaviour concern. Only the ItemUtils half is in this PR, so
the diff is one file, +31/-4.

Description

ItemUtils.sanitize() is applied to every metadata value on its way into the XOAI
intermediate document — but the XOAI serializer already escapes for XML itself. Escaping
twice means every value containing & < > " ' reaches harvesters as literal
entity text. sanitize() now removes the characters XML 1.0 forbids instead of
escaping the whole string, which is what its own javadoc always claimed it did.

This affects every OAI format built on the xoai document (cmdi, olac, oai_dc,
xoai, and the not-currently-exercised elg / oai_datacite) — so it breaks CLARIN VLO
and OLAC harvesting, not just one crosswalk.

Instructions for Reviewers

List of changes in this PR — all in dspace-oai/.../xoai/util/ItemUtils.java:

• Replaced StringEscapeUtils.escapeXml10(value) in sanitize() with a Pattern that
  deletes XML-1.0-illegal code units. Dropped the now-unused
  org.apache.commons.text.StringEscapeUtils import.
• The removal set is deliberately kept equivalent to what escapeXml10 removed,
  including unpaired surrogates. This is the whole point of the second commit and is
  not optional polish: escapeXml10 was an AggregateTranslator that also ran
  UnicodeUnpairedSurrogateRemover. A regex covering only the control-char part lets a
  lone surrogate (truncated 4-byte char, mojibake corpus) reach the StAX writer, which
  throws "Broken surrogate pair" inside XOAI.index(). That is caught per item, so
  the record is silently dropped from the OAI index — trading a visible bug for an
  invisible one. Measured on the 7.6.7 branch before that commit: 2048 code units broke
  indexing, vs 0 for vanilla.
• Javadoc now explains why it removes rather than escapes, so this does not get
  "fixed" back to escapeXml10 by a future reader.
• Known, intentional difference from escapeXml10: C0/DEL controls U+007F–U+009F are
  left as-is rather than converted to &#NNN;. They are legal XML 1.0 characters (only
  XML 1.1 requires escaping them), so well-formedness is unaffected.

Symptom 1 — corrupt display text (227 OAI items)

Measured on dev-6.pc:8603 (9.3, this branch's deployment) against
lindat.mff.cuni.cz (CLARIN 7, production):

11372/LRT-420   title      — both cmdi and xoai
  9.3      Corpus Tècnic de l'IULA      → harvester reads  l'IULA
  CLARIN   Corpus Tècnic de l'IULA

11234/1-3743    dc.description — oai_dc, olac, cmdi and xoai alike
  9.3      en->fr: 38.2
  CLARIN   en->fr: 38.2

Symptom 2 — actual data loss (2 OAI items)

& and " contain a ;, and the CLARIN crosswalk splits composite fields
on ;. So the entity's own semicolon is read as a field separator and everything after
it is discarded:

11234/1-3734    <projectName>  from local.sponsor
  9.3      SSHOC - Social Sciences &amp;amp                      ← "; Humanities Open Cloud" LOST
  CLARIN   SSHOC - Social Sciences &amp; Humanities Open Cloud

11372/LRT-1661  <affiliation>  from local.contact.person
  9.3      ...Heinrich-Heine University D\&amp;quot              ← "{u}sseldorf" LOST
  CLARIN   ...Heinrich-Heine University D\"{u}sseldorf

The splitting code (SpecialItemService.getFunding() / getContact(),
DCInput.ComplexDefinitions.separator = ";") is correct and untouched — it is
byte-identical to dtq-dev. Fixing the escaping fixes both symptoms; they are one bug.

No stored data is affected. GET /api/pid/find?id=hdl:... returns byte-identical
metadata on both instances, so this is dissemination-only — no migration or data repair.

How to test

Important

The corruption is baked into the OAI Solr index, not computed per request. After
deploying, run [dspace]/bin/dspace oai import -c or nothing will change and the
fix will look like a no-op.

B=http://dev-6.pc:8603/repository/server/oai/request
ID=oai:dev-6.pc

# 1. cosmetic — expect  Tècnic de l'IULA   (no ')
curl -s "$B?verb=GetRecord&metadataPrefix=cmdi&identifier=$ID:11372/LRT-420" | grep -o "Tècnic[^<]*"

# 2. data loss — expect  SSHOC - Social Sciences &amp; Humanities Open Cloud
curl -s "$B?verb=GetRecord&metadataPrefix=cmdi&identifier=$ID:11234/1-3734" | grep -o "<projectName>[^<]*"

# 3. data loss — expect  ...Heinrich-Heine University D\"{u}sseldorf
curl -s "$B?verb=GetRecord&metadataPrefix=cmdi&identifier=$ID:11372/LRT-1661" | grep -o "<affiliation>[^<]*"

# 4. intermediate document must be clean too
curl -s "$B?verb=GetRecord&metadataPrefix=xoai&identifier=$ID:11372/LRT-420" | grep -o "Tècnic[^<]*"

# 5. other formats must be clean as well
curl -s "$B?verb=GetRecord&metadataPrefix=oai_dc&identifier=$ID:11234/1-3743" | grep -o "en-[^ ]*fr: 38"

Compare each against https://lindat.mff.cuni.cz/repository/server/oai/request with
identifier=oai:lindat.mff.cuni.cz:<handle>.

Also worth confirming no records went missing after re-index (the surrogate half):
verb=ListIdentifiers&metadataPrefix=cmdi record count should not drop.

Verification already done

• mvn -pl dspace-oai compile checkstyle:check → BUILD SUCCESS, 0 Checkstyle violations.
• Sanitizer behaviour checked over the whole BMP: 0 XML-1.0-illegal code units leak
  through, 0 legal ones are removed, valid surrogate pairs survive, unpaired ones do not,
  and & < > " ' all pass through unescaped. Tab/LF/CR, accents and CJK preserved.
• All five symptom cases above re-measured live on dev-6 vs LINDAT production before the
  fix; they reproduce exactly as described.

Expected effect on the REST test suite

Tracked in dspace-rest-test. This should clear 229 of the 235 differing OAI items.

The summary line FAILED (failures=9, errors=2, skipped=6) will not move — the OAI
comparison is a single test, so shrinking its item count cannot change the counter.
Read the differing-item count, not the test count. The remaining known items are separate
issues (CLARIN allowance link rels; 6 baseline-side normalizer crashes).

Checklist

• My PR is created against the correct branch (dtq-dev-9-base) — this is a v9-base fix for an issue specific to that branch.
• My PR is small in size — 1 file, +31/-4.
• My PR passes Checkstyle validation.
• My PR includes Javadoc for the modified private method and the new constant, including the rationale.
• My PR includes details on how to test it (above).
• No new libraries/dependencies — this removes a usage of commons-text.
• No REST API endpoints modified, no new configuration.
• No new unit test: sanitize() is private static and dspace-oai has no existing test for ItemUtils. Behaviour was verified by exhaustive BMP sweep instead. Happy to add a package-visible test + ItemUtilsTest if reviewers prefer it in-tree.

🤖 Generated with Claude Code