WYSIWYG editor rewrites raw HTML in /type/page bodies on save (79% of collections pages affected)

[lokesh]

Summary

Opening a /type/page wiki page in the WYSIWYG editor and saving it rewrites the entire body through a markdown round-trip. Raw HTML — which 70% of our collections pages depend on — does not survive that round-trip intact. 71 of 89 live collections pages (79%) are altered by a single open-and-save, and a tail of them lose real content: link targets, HTML entities, and inline markup.

This is not hypothetical. It happened to https://openlibrary.org/collections/nebula-awards today (r55 → r57), which lost every section heading, every anchor link, and its whole in-page nav. I've restored it as r58 with the r55 body.

What happened on the Nebula page

revision	body length	state
55	3292	good
56	2989	inline HTML rewritten to markdown, entities decoded
57	2123	all <h2> section headings gone, all links reduced to plain text
58	3292	restored (r55 body, posted directly to ?m=edit to bypass the editor)

The {{QueryCarousel(...)}} macros survived throughout, which is why the page still showed books — just with no headings, no anchor nav, and no links.

Root cause

openlibrary/templates/type/page/edit.html:30 mounts <ol-markdown-editor> over the body textarea:

<textarea id="page--body" name="body">$page.body</textarea>
<ol-markdown-editor target-id="page--body" enable-html-block enable-code></ol-markdown-editor>

The editor's onUpdate (OLMarkdownEditor.js:459) reserializes the entire document on every change and overwrites the textarea:

let markdownOutput = editor.storage.markdown.getMarkdown();
this.targetElement.value = markdownOutput;

So a one-character edit anywhere rewrites the whole body. The pipeline is:

markdown source
  → markdown-it render (html: true)
  → HTML string
  → ProseMirror DOMParser against the editor schema   ← anything the schema can't model is dropped
  → tiptap-markdown serializer
  → markdown source'

The schema is StarterKit + Image + HtmlBlock (editor-core.js). HtmlBlock catches markdown-it html_block tokens and does preserve block-level HTML verbatim — that part works, and it's why tables and <h2 id> blocks survive a normal load-and-save. What it does not cover is inline HTML (html_inline tokens), which is reparsed against the schema and reduced to whatever the schema happens to support.

Two notes for whoever picks this up:

• HtmlBlock is added unconditionally in createEditor(). The enable-html-block attribute only controls whether the toolbar button renders, not whether HTML blocks round-trip.
• The display renderer is a different markdown implementation — OLMarkdown, a vendored Python-Markdown 1.6b (openlibrary/core/olmarkdown.py) — while the editor parses with markdown-it/CommonMark. The two dialects disagree, and we've already shipped one fix for that disagreement (Help page /help/faq/editing renders orphaned markdown reference-link definitions as visible junk (literal <br />, escaped chars) #13074, hard breaks). Every additional feature we lean on widens that gap.

Measured blast radius

I ran every live page body through a faithful reproduction of the editor pipeline (same versions, same extension config, jsdom; results confirmed against the real editor in Chrome, which produced byte-identical output).

Collections pages (89 with a body):

altered by one round-trip	71 (79%)
lose a link target (real breakage)	3 pages, 16 of 1422 links
lose HTML entities	12 pages
lose an anchor id	0
lose a table or a carousel macro	0
total body bytes	501,246 → 492,367 (−1.8%)

Non-collection wiki pages (54 sampled from /about, /help, /dev, /tour, /developers):

contain raw HTML	33
lose a link target	9
lose HTML entities	9

/dev/docs/api/authors loses both of its two links — 100% loss on that page.

There are 495 /type/page docs in production (191 /collections including translated variants, 36 /about, 36 /dev, 30 /help, 7 /community, 7 /tour). Raw HTML across collections bodies: td×5332, a×2289, tr×778, h2×700, h3×430, dd×100, hr×98, dt×90, dl×52, img×48.

Good news, and it corrects my first read of this: the round-trip converges. All 89 pages reach a fixed point, nearly all by pass 2–3 — pass 1 costs 1.8%, and passes 2–5 cost a further 0.01% combined. This is one lossy normalization, not runaway decay. Any fix should be judged against that: we are protecting a one-time ~2% haircut plus a small tail of genuinely broken links, not a page that dissolves.

What I could not reproduce

The r55 → r56 transition reproduces exactly (my harness turns r55 into 2934 chars; r56 is 2989, the difference being the actual edit that was made). The r56 → r57 step does not. Loading r56 and reserializing yields 2920 chars with all HTML blocks intact; the real r57 is 2123 with the blocks emptied and link marks stripped. I tested and ruled out the source-view toggle path (_toggleSource → setContent, lossless) and repeated round-trips (converged).

So there is a second, more destructive failure mode reachable through some in-editor interaction — most likely a paste, a select-all-and-retype, or editing inside the html-block textareas — that I did not isolate. Worth reproducing before signing off on any fix, because a fix aimed only at the round-trip may not close it.

Options

A. Improve HTML fidelity inside Tiptap. Add the missing schema nodes (@tiptap/extension-table, definition lists) and a custom node/mark that preserves arbitrary inline HTML verbatim the way HtmlBlock does for block HTML.

Against it: ProseMirror drops what its schema can't model, by design. Chasing verbatim HTML fidelity means modelling all of HTML, which is building an HTML editor rather than a markdown one, and it's permanent maintenance. It also doesn't help the second failure mode above.

B. Migrate to Tiptap's official markdown extension. We should do this regardless — tiptap-markdown@0.9.0 is abandoned. Its author's notice: Tiptap shipped a markdown extension in 3.7.0, prefer it, "I don't plan to address current issues / PR." Last push 2025-10-22. We're already on @tiptap/core@3.26.0, so the official extension is available to us.

Against it as a fix for this bug: it's the same architecture with the same constraint — anything without a CommonMark equivalent is dropped — it's flagged in Tiptap's own docs as an early release "subject to change or may have edge cases that may not be supported yet", and it has its own open round-trip data-loss reports (ueberdosis/tiptap#7147, #7731). Treat it as dependency hygiene, not as the fix.

C. Make /type/page a plain markdown/source editor. Don't mount the WYSIWYG on this one template; edit the text directly. Losslessness is structural — no parse, no serialize, no round-trip.

Note that /type/page is already the odd one out: it's the only one of the nine ol-markdown-editor mounts that passes enable-html-block enable-code. The other eight (work/edition descriptions, author bios, list and user descriptions, tags) are short prose fields where markdown is the right model and the WYSIWYG is a genuine win. This option leaves all of those alone.

Against it: gives up WYSIWYG editing for wiki pages, which is what #12182 set out to provide.

D. Open /type/page in source mode, WYSIWYG opt-in. Cheaper politically than C, but the toggle back into WYSIWYG re-enters the same pipeline, so it only narrows the window rather than closing it. Would need a true source-only mode that never mounts the editor.

E. Only write back what actually changed. Serialize just the blocks the user touched and leave the rest byte-identical. Correct in principle and it fixes the real complaint — untouched content should never change. In practice it's hard and fragile against ProseMirror's document model, and it's a large piece of work.

F. Server-side guard. Reject or interstitially confirm a save that drops HTML tags, links, or a large fraction of the body. Doesn't fix the editor, but it's the only option that also catches the unreproduced failure mode, plus future regressions, plus bad pastes.

G. Change the storage format to HTML. Would require migrating 495 pages and reworking OLMarkdown rendering. Disproportionate, and markdown is the friendlier authoring format.

Recommendation

C + F, in that order, and treat B as separate hygiene.

1. Now — stop mounting the WYSIWYG on /type/page (option C). One-line template change, zero data loss, ships today. Wiki-page bodies are the one place in the codebase where raw HTML is a first-class authoring idiom — 70% of collections pages, 5332 <td>s, hand-written anchor navigation — and a markdown WYSIWYG is the wrong tool for content that is substantially not markdown. Keep the WYSIWYG everywhere else; those eight fields are prose and are not affected.

2. Next — add the server-side guard (option F). A save that removes HTML tags or link targets should require explicit confirmation. This is the only measure that covers the r56 → r57 mode I couldn't reproduce, and it keeps protecting us whatever we do with the editor later.

3. Separately — migrate off tiptap-markdown (option B). It's an unmaintained dependency in the path of nine edit forms. That's worth doing on its own schedule, and it is not a fix for this issue.

4. Only then, if WYSIWYG on wiki pages is still wanted, revisit A or E — with a corpus round-trip test in CI as the gate. The harness I used for the numbers above is a good starting point: pull every /type/page body, round-trip it, and fail on any lost link target, anchor id, or HTML tag. Nothing should ship WYSIWYG for /type/page again until that test is green.

Reproducing the measurements

The corpus test is worth having in the repo. The shape of it:

// jsdom globals first, then mirror openlibrary/components/lit/editor-core.js exactly:
const editor = new Editor({
    element,
    extensions: [
        StarterKit.configure({
            heading: { levels: [1, 2, 3, 4] },
            codeBlock: undefined, code: undefined,
            link: { openOnClick: false, autolink: true },
            strike: false, hardBreak: false,
        }),
        OLHardBreak,
        Markdown.configure({ breaks: true, linkify: true }),
        Image.configure({ inline: true, allowBase64: false }),
        HtmlBlock,
    ],
    content: body.replace(/\r\n/g, '\n'),   // the browser normalises CRLF in the textarea
});
const out = editor.storage.markdown.getMarkdown();
// assert: no link href, anchor id, or HTML tag present in `content` is missing from `out`

Fetch bodies from https://openlibrary.org/query.json?type=/type/page&limit=1000 and <key>.json.

Restoring a mangled page

For anyone who hits this before it's fixed — the WYSIWYG will re-mangle the page if you use the edit form, so post the old revision back directly:

curl "https://openlibrary.org/<page>.json?v=<good-revision>" | jq -r .body.value > body.txt

curl -X POST "https://openlibrary.org/<page>?m=edit" -b "session=<cookie>" \
  --data-urlencode "type.key=/type/page" \
  --data-urlencode "title=<title>" \
  --data-urlencode "body@body.txt" \
  --data-urlencode "_comment=Restore revision <n>" \
  --data-urlencode "_save=Save"

There is also a built-in ?m=revert&v=<n> POST mode (openlibrary/plugins/upstream/code.py:262), but it requires super-librarian or higher and has no UI.

[openlibrary-bot]

Context (research findings)

Verified the specifics cited in the report against HEAD (7be8017):

• openlibrary/templates/type/page/edit.html#L29-L30 — /type/page is confirmed the only edit form invoking <ol-markdown-editor ... enable-html-block enable-code>.
• OLMarkdownEditor.js#L456-L457 and editor-core.js#L28-L48 — schema/onUpdate wiring matches as described.
• openlibrary/plugins/upstream/code.py#L262 — revert mode's is_super_librarian_or_higher() gate matches the report's note about no accessible UI for non-super-librarians.

Related history (via gh api/gh pr list — this checkout is a shallow clone, so git blame/git log -S weren't usable for authorship):

• PR feat: add source toggle to OLMarkdownEditor #12682, "feat: add source toggle to OLMarkdownEditor" (merged 2026-05-12, @lokesh) — this is the _toggleSource path the report already tested and ruled out as lossless. Relevant to Option D: the source-mode plumbing this option would need already exists.
• PR fix(markdown): stop WYSIWYG hard breaks rendering as literal "<br />" #13076, "fix(markdown): stop WYSIWYG hard breaks rendering as literal <br />" (merged 2026-07-02, @lokesh) — the prior OLMarkdown/tiptap dialect-mismatch fix referenced in the report.
• Feature - New WYSIWYG editor via tiptap #12182, "Feature - New WYSIWYG editor via tiptap" (merged 2026-04-29, @Sadashii) — the original feature that introduced this pipeline.
• Add source-code mode to tiptap editor #12478, "Add source-code mode to tiptap editor" (closed) — an earlier ask for source mode, superseded by feat: add source toggle to OLMarkdownEditor #12682 above.
• Wysiwyg editor is partly broken #6127, "Wysiwyg editor is partly broken" (still open, Priority: 2, Lead: @cdrini) — describes a non-functional toolbar/inaccurate preview. Its stakeholder list (@dcapillae) and symptoms read as pre-Feature - New WYSIWYG editor via tiptap #12182, i.e. about the old editor, not the current tiptap one. Worth a maintainer check on whether it's stale/superseded rather than actually related to this report.

---

Issue Assessment

Open questions

• Which combination of options (C/D/E/F/G) should be adopted, and in what order — this is explicitly flagged by the author as a decision point, not something research can resolve.
• What threshold/heuristic should the Option F server-side guard use to decide a save is destructive enough to block/confirm (e.g. % of HTML tags removed, links lost, byte-length delta)?
• Should Option C ship independently ahead of a decision on F, per the author's proposed sequencing, or should both land together?

Action items

• Reproduce and isolate the unreproduced r56→r57 failure mode (paste / select-all-retype / editing inside an html-block textarea) — the author notes a template-only fix (Option C) narrows exposure but doesn't structurally close this path.
• Is Wysiwyg editor is partly broken #6127 a duplicate of this report? (Answer: no — Wysiwyg editor is partly broken #6127 pre-dates the tiptap rewrite (Feature - New WYSIWYG editor via tiptap #12182) and describes a different editor implementation entirely (inactive toolbar, old stakeholder list). Recommend a maintainer confirm and close/re-scope it separately rather than merging into this thread.)

---

🔔 Stakeholders

No additional ping — @lokesh is both the reporter and, per PR history above (#12682, #13076), the active owner of this exact component.

---

Full triage checklist

• Triage — correct labels applied or removed
• Problem / opportunity — problem is clear, audience identified, actionable
• Scope — the report bundles a bug fix, a corpus-wide audit, a dependency-hygiene call (Option B), and a recovery runbook into one issue; author already flags B as "separate" — consider splitting tracked follow-ups once a direction is picked
• Justification — reasoning and impact stated; sufficient for leads to prioritize
• Success & testing criteria — defines what "done" looks like, verifiable by a contributor (proposed corpus round-trip test)
• Proposal & breakdown — Option C is concretely actionable today; Option F (the longer-term guard) has no concrete implementation steps yet — appropriately gated behind Needs: Staff Decision
• If bug: environment & repro steps — prod revisions, byte counts, and a reproduction harness are all provided
• Risks, concerns, open questions — identified and surfaced, including an explicitly unreproduced failure mode
• Richy context — relevant code, commits, PRs reviewed; no exact duplicate found
• References — permalinks and PR/issue links present; no screenshots, but the bug is data-loss (not visual), so diff tables serve the same purpose

⚠️ Contributors, this issue will be ready to work on once:

• Triage — maintainers will add Priority: * and Lead: @*
• Needs: Staff Decision — which of options C/D/E/F/G (see the author's recommendation) to pursue, and in what order

---

Note

This comment was automatically generated by Richy, Open Library's issue enrichment assistant, on behalf of @mekarpeles. This is NOT a sign-off. This issue needs lead review before work begins.