From 040cddc29db22ff3182dd5280808d0612879b912 Mon Sep 17 00:00:00 2001 From: Matus Kasak Date: Mon, 10 Aug 2026 14:47:43 +0200 Subject: [PATCH 1/2] Fix OAI double-escaping of metadata values (upstream #10721 regression) ItemUtils.sanitize() is applied to every metadata value on its way into the XOAI intermediate document, but the XOAI serializer escapes for XML itself. Escaping twice means any value containing & < > " ' reaches harvesters as literal entity text: a title stored as "Corpus Tecnic de l'IULA" is emitted as "Corpus Tecnic de l&apos;IULA", which decodes to l'IULA. sanitize() now REMOVES the characters XML 1.0 forbids instead of escaping the whole string, which is what its own javadoc always claimed it did. Where the defect comes from: upstream PR DSpace/DSpace#11139 "fix(#10721): Sanitize non-characters during OAI indexing" (d8fbe16ede), ported to dspace-9_x as 6a69e87406. dtq-dev (7.6.5) predates it entirely and is unaffected, which is why this only shows on the 9 base. Blast radius is every OAI format built on the xoai document, not just cmdi. Measured on dev-6.pc:8603 (9.3) vs lindat.mff.cuni.cz (CLARIN 7): 11372/LRT-420 title, cmdi and xoai 9.3 Tecnic de l&apos;IULA CLARIN Tecnic de l'IULA 11234/1-3743 dc.description, oai_dc and olac 9.3 en-&gt;fr: 38.2 CLARIN en->fr: 38.2 It also loses data, not only formatting. & and " contain a ';', and the CLARIN crosswalk splits composite fields (local.sponsor, local.contact.person) on ';' -- so the entity's own semicolon is read as a field separator and the rest of the value is discarded: 11234/1-3734 from local.sponsor 9.3 SSHOC - Social Sciences &amp <- truncated CLARIN SSHOC - Social Sciences & Humanities Open Cloud 11372/LRT-1661 from local.contact.person 9.3 ...Heinrich-Heine University D\&quot <- truncated CLARIN ...Heinrich-Heine University D\"{u}sseldorf The splitting code is correct and unchanged; no stored data is affected, so no migration or data repair is needed. This is a dissemination-only defect. Cherry-picked from the CLARIN 7.6.7 upgrade branch (upgrade/vanilla-7.6.7), where the identical upstream defect arrived via 7.6.5 -> 7.6.7. (cherry picked from commit c6a1101410f6861788236c021708352518e1875a) The content-disposition half of that source commit is deliberately NOT taken: it edits webui.content_disposition_inline in dspace/config/clarin-dspace.cfg, a key that does not exist on this branch, and is an unrelated 7.6.7 concern. Co-Authored-By: Claude Opus 5 (1M context) --- .../java/org/dspace/xoai/util/ItemUtils.java | 21 +++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/dspace-oai/src/main/java/org/dspace/xoai/util/ItemUtils.java b/dspace-oai/src/main/java/org/dspace/xoai/util/ItemUtils.java index 44071872292c..7d6b9a521c45 100644 --- a/dspace-oai/src/main/java/org/dspace/xoai/util/ItemUtils.java +++ b/dspace-oai/src/main/java/org/dspace/xoai/util/ItemUtils.java @@ -17,11 +17,11 @@ import java.util.List; import java.util.Set; import java.util.concurrent.atomic.AtomicBoolean; +import java.util.regex.Pattern; import com.lyncode.xoai.dataprovider.xml.xoai.Element; import com.lyncode.xoai.dataprovider.xml.xoai.Metadata; import com.lyncode.xoai.util.Base64Utils; -import org.apache.commons.text.StringEscapeUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.dspace.app.util.factory.UtilServiceFactory; @@ -256,8 +256,21 @@ private static Element createBundlesElement(Context context, Item item, AtomicBo } /** - * Sanitizes a string to remove characters that are invalid - * in XML 1.0 using the Apache Commons Text library. + * Matches the characters that XML 1.0 forbids outright: C0 controls other than tab, LF and CR, + * plus the two non-characters U+FFFE and U+FFFF. See https://www.w3.org/TR/xml/#charsets + */ + private static final Pattern INVALID_XML10_CHARS = + Pattern.compile("[\\x00-\\x08\\x0B\\x0C\\x0E-\\x1F\\uFFFE\\uFFFF]"); + + /** + * Sanitizes a string to remove characters that are invalid in XML 1.0. + *

+ * NOTE: this deliberately REMOVES illegal characters rather than escaping the string. The value + * returned here is handed to the XOAI serializer, which performs XML escaping itself, so escaping + * here as well would double-escape every value containing &, <, >, " or ' — a harvester + * would then read the literal text "&lt;" instead of a "<" character. That silently corrupts + * every OAI format built on the xoai document, including the cmdi and olac formats CLARIN/LINDAT + * is aggregated through. * @param value The string to sanitize. * @return A sanitized string, or null if the input was null. */ @@ -265,7 +278,7 @@ private static String sanitize(String value) { if (value == null) { return null; } - return StringEscapeUtils.escapeXml10(value); + return INVALID_XML10_CHARS.matcher(value).replaceAll(""); } /** From 7f2fa1326327f3f38f81c1ee4d2baae0f3ee3283 Mon Sep 17 00:00:00 2001 From: Matus Kasak Date: Mon, 10 Aug 2026 14:48:08 +0200 Subject: [PATCH 2/2] Also strip unpaired surrogates in the OAI sanitizer The previous commit replaced escapeXml10 with a regex to stop the double escaping, but escapeXml10 was an AggregateTranslator that ALSO ran UnicodeUnpairedSurrogateRemover, and the regex reproduced only the control-char part. Consequence if that half is missed: a lone surrogate in metadata (a truncated 4-byte character, a mojibake corpus) makes the StAX writer throw "Broken surrogate pair" inside XOAI.index(). Because XOAI.index() catches that per item, the record is SILENTLY DROPPED from the OAI index -- trading a visible double-escape for an invisible missing record. This commit is therefore not optional polish; the two commits must travel together. Measured on the 7.6.7 branch before this fix: 2048 code units broke indexing, against 0 for vanilla. Verified over the whole BMP: 0 XML-1.0-illegal code units leak, 0 legal ones are removed, valid surrogate pairs survive, unpaired ones do not, and XML metacharacters still pass through unescaped. Known and intentional difference from escapeXml10: C0/DEL controls U+007F-U+009F are left as-is instead of being turned into &#NNN; entities. They are legal XML 1.0 characters (only XML 1.1 requires escaping them), so well-formedness is unaffected. Cherry-picked from the CLARIN 7.6.7 upgrade branch (upgrade/vanilla-7.6.7). (cherry picked from commit b28457142a9a77fb5ca7c0ee0d79ea868d5183d3) Co-Authored-By: Claude Opus 5 (1M context) --- .../java/org/dspace/xoai/util/ItemUtils.java | 32 +++++++++++++------ 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/dspace-oai/src/main/java/org/dspace/xoai/util/ItemUtils.java b/dspace-oai/src/main/java/org/dspace/xoai/util/ItemUtils.java index 7d6b9a521c45..6c5073733aa2 100644 --- a/dspace-oai/src/main/java/org/dspace/xoai/util/ItemUtils.java +++ b/dspace-oai/src/main/java/org/dspace/xoai/util/ItemUtils.java @@ -256,21 +256,35 @@ private static Element createBundlesElement(Context context, Item item, AtomicBo } /** - * Matches the characters that XML 1.0 forbids outright: C0 controls other than tab, LF and CR, - * plus the two non-characters U+FFFE and U+FFFF. See https://www.w3.org/TR/xml/#charsets + * Matches everything XML 1.0 forbids outright, in three alternations: + *

    + *
  1. C0 controls other than tab, LF and CR, plus the non-characters U+FFFE and U+FFFF;
    2. + *
  2. a high surrogate not followed by a low surrogate;
    3. + *
  3. a low surrogate not preceded by a high surrogate.
    4. + *
+ * See https://www.w3.org/TR/xml/#charsets. Unpaired surrogates matter in practice: a truncated + * 4-byte character in ingested metadata makes the StAX writer throw "Broken surrogate pair", and + * because XOAI.index() catches that per item the record is silently dropped from the OAI index. */ - private static final Pattern INVALID_XML10_CHARS = - Pattern.compile("[\\x00-\\x08\\x0B\\x0C\\x0E-\\x1F\\uFFFE\\uFFFF]"); + private static final Pattern INVALID_XML10_CHARS = Pattern.compile( + "[\\x00-\\x08\\x0B\\x0C\\x0E-\\x1F\\uFFFE\\uFFFF]" + + "|[\\uD800-\\uDBFF](?![\\uDC00-\\uDFFF])" + + "|(? * NOTE: this deliberately REMOVES illegal characters rather than escaping the string. The value - * returned here is handed to the XOAI serializer, which performs XML escaping itself, so escaping - * here as well would double-escape every value containing &, <, >, " or ' — a harvester - * would then read the literal text "&lt;" instead of a "<" character. That silently corrupts - * every OAI format built on the xoai document, including the cmdi and olac formats CLARIN/LINDAT - * is aggregated through. + * returned here is handed to the XOAI serializer, which performs XML escaping itself (every text + * event goes through {@code XMLStreamWriter.writeCharacters}), so escaping here as well would + * double-escape every value containing &, <, >, " or ' — a harvester would then read the + * literal text "&lt;" instead of a "<" character. That silently corrupts every OAI format + * built on the xoai document, including the cmdi and olac formats CLARIN/LINDAT is aggregated + * through. + *

+ * The removal set must stay equivalent to what {@code StringEscapeUtils.escapeXml10} removed — + * notably including unpaired surrogates — otherwise items carrying them fail to serialize and + * drop out of the OAI index entirely. * @param value The string to sanitize. * @return A sanitized string, or null if the input was null. */