fix(machinery): preserve literal HTML entities

[i4innovationnet]

Summary

• protect literal HTML character references as non-translatable spans for XML-based machine translation engines
• merge entity spans with existing highlights so a partial overlap cannot expose part of a reference
• preserve the existing unconditional unescape behaviour for raw ampersands
• skip glossary terms which overlap an existing highlight in the Azure AI Translator service

Fixes #21004.

Root cause

XML-based machinery escapes source text before sending it to a provider and unescapes every returned result. That round trip is lossless only while the provider keeps the escaping level it was given, and HTML-mode providers renormalize it. When they do, an inert character reference in the source comes back one level decoded, and the unconditional unescape turns it into live markup.

With the safe-html flag set this becomes data loss rather than a quality problem: the sanitizer sees the now-live element and removes it together with everything after it, with no check firing and a success-shaped API response.

Approach

Entity spans are protected through the existing non-translatable-span mechanism — the same one that already keeps units flagged xml-text intact. The spans recognized are exactly those Python unescaping consumes, including the recognized prefix of a legacy reference written without its semicolon, so nothing that would survive the round trip is touched and nothing that would be decoded is missed. Spans partially overlapping an existing highlight are merged with it, so a placeholder can no longer split a reference and leave half of it exposed.

The unconditional unescape is deliberately unchanged — it is what fixes #12936, and removing it would regress that. The narrower change is to stop pre-existing entity spans from entering the escape/unescape pair at all.

Applying protection where none existed before exposed a latent defect in the Azure AI Translator glossary path. Its insertion loop only compared a term against highlights starting after it, so a term matching inside a newly protected reference produced overlapping spans and duplicated characters on restore. Source The &copyright notice and <b> tag. with a copyright term restored as The &copycopyright notice and .... The loop now skips any term overlapping an existing highlight. The same defect is reachable on main today with a plain placeholder, but this change makes it reachable for ordinary unflagged strings, so it is fixed here rather than deferred.

Known behaviour

Protection preserves the reference but not the whitespace around it: a provider may return < script > where it was sent one protected span. This is a pre-existing property of the restore step and is byte-identical on main for units already flagged xml-text, across all three engines. It is not addressed here, because absorbing provider-inserted whitespace would change the signature of the shared cleanup and restore path and has not been validated against a live provider. In the regime the issue documents, the result is still strictly better: main restores live <script>, this restores the reference intact.

Validation

• uv run pytest weblate/machinery/tests.py weblate/utils/tests/test_html.py -q — 906 passed, 50 skipped, 47 subtests passed
• uv run pytest weblate/checks/ weblate/glossary/ -q — 1465 passed, 498 skipped
• 18 new tests: 11 through the machinery layer, 7 covering the entity scanner directly. With the scanner present but the machinery change reverted, 10 of the 11 machinery tests fail. The eleventh asserts that units already flagged xml-text are unchanged, which is the intended result.
• reverting only the Azure AI Translator glossary change, with the rest of the patch applied, fails the two glossary overlap tests
• mypy reports 230 errors on the changed files, matching main exactly
• ruff check and ruff format --check clean
• live Google Cloud Translation v3 A/B covering escaped tags, mixed entities, and the raw-ampersand regression case
• differential fuzzing of the entity scanner against html.unescape over every HTML5 name with and without semicolon and trailing characters, numeric and hexadecimal references, malformed input, and non-Latin text — no divergence

Coverage spans DeepL, Google Cloud Translation v3 and Azure AI Translator, plus direct unit tests for the entity scanner in weblate/utils/tests/test_html.py.

[nijel]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: df1df3b74a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Prevent entity placeholders from triggering Azure truncation

For Azure AI Translator, every entity highlighted here is expanded into a long <span class="notranslate"> placeholder before MicrosoftCognitiveTranslation.download_translations() slices the resulting markup to 5,000 characters at weblate/machinery/microsoft.py:171. For example, a valid 400-character source consisting of 100 < references now expands to 5,372 characters and is sliced through a placeholder, whereas previously it escaped to only 800 characters; Azure therefore receives an incomplete string and Weblate returns a truncated or malformed translation. The generated markup needs to stay within the limit without silently cutting it, or the request should fail cleanly.

Useful? React with 👍 / 👎.