Skip to content

Commit 4ec4cf8

Browse files
authored
✨ NEW: docutils heading-anchor slug preset, and a versioned anchor contract (#1168)
Finishes the job #1158's slug module started: the anchor behaviour becomes a formal, versioned contract that alternative implementations and tooling can pin against, plus a third slug preset for mixed-format projects. ## A `docutils` slug preset `myst_heading_slug_func = "docutils"` produces slugs **byte-identical to `docutils.nodes.make_id`** — the id style reStructuredText headings receive — so a Sphinx project mixing `.rst` and `.md` sources can choose one uniform anchor rule. - The algorithm and its translate tables are ported into the pure, stdlib-only `myst_parser.slugs` (no docutils import there; provenance noted — docutils is public domain), keeping the module's replicate-from-one-file property. - Fidelity is pinned **by CI, not by care**: the corpus plus adversarial vectors (digraphs, the full 33-entry translate table, combining marks, compatibility/ligature forms, mathematical alphanumerics, Turkish dotless-i, whitespace runs) are asserted byte-equal against the real `make_id`. Review additionally fuzzed **210k random strings and swept every codepoint U+0020–U+2FFF** with zero divergences. Drift policy (documented): if a future docutils changes `make_id`, the preset and corpus stay byte-stable — they are the contract — and the divergence gets documented. - The `myst-anchors` CLI picks the preset up automatically (`--slug-func docutils`), with renderer/CLI cross-check tests. ## Empty slugs mean no anchor `make_id` returns `""` for *any* title without Latin letters (`Привет`, `2.0`, `!!!`) — far more often than the github preset. Previously (unreleased, since #1158) an empty slug was stored, and a *second* empty-slugging heading received a literal `-1` HTML anchor. Now an empty slug means the heading simply gets **no anchor** — not stored, no dedup participation, no id emitted — for all presets and custom functions. Two behavioural nuances from review, both strictly-better and test-covered: - `<span id="-1">` anchors no longer appear for duplicate punctuation-only headings; - `[](#)` (an empty fragment) no longer silently resolves to a punctuation-only heading — it now warns as an unresolved reference. ## The versioned contract - `tests/fixtures/slugs.json` now carries `version: 1` and a policy statement: append-only within a version; version bump on any semantic change to an existing vector or algorithm. New `docutils` vectors were generated against the real `make_id` at authoring time. - A new normative docs subsection, *The anchor contract* (`docs/syntax/optional.md`), states in one place: slug generation (presets + corpus as the machine-readable half), fixed-base deduplication (`x, x-1, x-2`; docutils' own `idN` dedup deliberately not emulated), the empty-slug rule, explicit-id priority, HTML id emission (`id_prefix` bypassed), and the drift policy. The `myst-anchors` CLI preview's one divergence (it cannot apply the renderer-only empty-slug rule) is documented alongside the CLI example. ## Review Two implementation agents + a combined adversarial/behaviour review with old-vs-new probes against master: github/gitlab documents render **byte-identically** (docutils publish, sphinx HTML, searchindex, warnings) except the intended empty-slug fix; docutils-preset end-to-end resolution (same-doc, cross-doc, dedup-slug, explicit-target priority, slug==primary-id coincidence) verified with zero warnings. One pre-existing bug surfaced by review (topmatter clobbers validated config values, breaking *all* preset strings in frontmatter since #1158) is tracked separately in #1167 rather than scope-creeping here. Full suite: 1245 passed. No changes to the `github`/`gitlab` slug outputs; no new configuration surface beyond the preset name. --- _Generated by [Claude Code](https://claude.ai/code/session_01GByoptRecR2GAwj5qrLz6C)_
1 parent 683cb36 commit 4ec4cf8

8 files changed

Lines changed: 352 additions & 8 deletions

File tree

‎docs/syntax/optional.md‎

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -539,13 +539,15 @@ The slug generation function is selected with the `myst_heading_slug_func` optio
539539
The recommended form is a named preset:
540540

541541
```python
542-
myst_heading_slug_func = "github" # the default; also available: "gitlab"
542+
myst_heading_slug_func = "github" # the default; also available: "gitlab", "docutils"
543543
```
544544

545545
The `"gitlab"` preset follows [GitLab's algorithm](https://docs.gitlab.com/ee/user/markdown.html#heading-ids-and-links) (which, unlike GitHub, strips surrounding whitespace, squeezes repeated hyphens and prefixes digit-only slugs with `anchor-`).
546546

547+
The `"docutils"` preset produces slugs byte-compatible with `docutils.nodes.make_id` — the `id` style reStructuredText headings receive — for a uniform anchor style across mixed rST + Markdown projects; its slugs are ASCII-only, so a non-Latin title produces no anchor (see the anchor contract below).
548+
547549
:::{versionchanged} 5.2.0
548-
`myst_heading_slug_func` now accepts the preset names `"github"` (default) and `"gitlab"`.
550+
`myst_heading_slug_func` now accepts the preset names `"github"` (default), `"gitlab"` and `"docutils"`.
549551

550552
As a legacy form, it can still be set to an import path string
551553
(e.g. `myst_heading_slug_func = "mypackage.mymodule.slugify"`, added in 0.19.0)
@@ -574,6 +576,41 @@ $ myst-anchors -l 2 --slug-func github docs/syntax/optional.md
574576
<h2 id="direct-latex-math"></h2>
575577
```
576578

579+
Note this is a quick preview using the underlying markdown-it plugin:
580+
it does not apply the renderer-only empty-slug rule described below,
581+
so a heading whose slug is empty is shown with a literal empty (or suffixed) id,
582+
where the renderer would emit no anchor at all.
583+
584+
### The anchor contract
585+
586+
:::{versionadded} 5.2.0
587+
:::
588+
589+
- **Slug generation**: anchors are computed by the configured slug function.
590+
The named presets are specified by `myst_parser/slugs.py` (pure and
591+
standard-library-only, so alternative implementations can port them from that
592+
one file) and pinned by the machine-readable corpus `tests/fixtures/slugs.json`
593+
(versioned via a `version` field; append-only within a version, with a version
594+
bump on any semantic change).
595+
- **Deduplication**: repeated slugs are given fixed-base suffixes `slug`,
596+
`slug-1`, `slug-2`, ... by `unique_slug` — the base is never mutated. This
597+
applies whatever the preset; in particular docutils' own duplicate-id scheme is
598+
*not* emulated by the `docutils` preset, which governs the slug text only.
599+
- **Empty slugs**: a slug function may return an empty string (a punctuation-only
600+
title, or any non-Latin title under `docutils`); such a heading gets no anchor
601+
and takes no part in deduplication.
602+
- **Explicit-id priority**: an explicit target/id on a heading beats the
603+
auto-slug as the primary id; the superseded slug is kept as a secondary anchor,
604+
so previously published links keep working.
605+
- **HTML id emission**: slugs are emitted as real (secondary) HTML `id`s in the
606+
output, so the anchors exist in the published HTML. The docutils `id_prefix`
607+
setting is deliberately bypassed — the raw slug is the anchor. Set
608+
`myst_heading_anchors_html_ids = False` to disable this.
609+
- **docutils drift policy**: the `docutils` preset is pinned byte-for-byte
610+
against `docutils.nodes.make_id` by CI. If a future docutils changes `make_id`,
611+
the preset and corpus stay byte-stable (they are the contract) and the
612+
divergence will be documented.
613+
577614
(syntax/definition-lists)=
578615

579616
## Definition Lists

‎myst_parser/config/main.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -313,7 +313,7 @@ def __repr__(self) -> str:
313313
"validator": check_heading_slug_func,
314314
"help": (
315315
"Function for creating heading anchors: "
316-
'a preset name ("github", "gitlab"), '
316+
'a preset name ("docutils", "github", "gitlab"), '
317317
"or (legacy) a python import path "
318318
"e.g. `my_package.my_module.my_function`"
319319
),

‎myst_parser/mdit_to_docutils/base.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -877,6 +877,12 @@ def generate_heading_target(
877877
append_to=self.current_node,
878878
)
879879
else:
880+
if not slug:
881+
# an empty slug (e.g. a punctuation-only title, or a
882+
# non-Latin title under the docutils preset) means the
883+
# heading gets no anchor; it also takes no part in
884+
# deduplication, so a later empty slug is not suffixed -1
885+
return
880886
node["slug"] = slug
881887
# note: the slug is additionally emitted as a (secondary) HTML id
882888
# by the `AddSlugIds` transform, which runs only after *all*

‎myst_parser/slugs.py‎

Lines changed: 81 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,20 @@
33
This is the single source of truth for heading slugs,
44
shared by the renderer (``myst_heading_anchors``) and the ``myst-anchors`` CLI.
55
6-
The functions here are pure and dependency-free (standard library ``re`` only),
6+
The functions here are pure and dependency-free (standard library only),
77
so that alternative implementations of MyST can replicate them from this file;
88
for the same reason, the unit-test corpus is kept as a language-agnostic
99
data file (``tests/fixtures/slugs.json``).
10+
11+
A slug function may return an empty string (e.g. for a punctuation-only
12+
title, or a non-Latin title under the ``docutils`` preset); an empty slug
13+
means the heading gets *no* anchor.
1014
"""
1115

1216
from __future__ import annotations
1317

1418
import re
19+
import unicodedata
1520
from collections.abc import Callable, Container
1621

1722
_GITHUB_CLEAN = re.compile(r"[^\w\u4e00-\u9fff\- ]")
@@ -57,7 +62,82 @@ def gitlab_slugify(title: str) -> str:
5762
return slug
5863

5964

65+
# ported verbatim from ``docutils.nodes`` (public domain), so that this
66+
# module stays free of third-party imports; byte-fidelity to
67+
# ``docutils.nodes.make_id`` is pinned by tests/test_slugs.py
68+
_DOCUTILS_DIGRAPHS = {
69+
0x00DF: "sz", # ß
70+
0x00E6: "ae", # æ
71+
0x0153: "oe", # œ
72+
0x0238: "db", # ȸ
73+
0x0239: "qp", # ȹ
74+
}
75+
_DOCUTILS_TRANSLATE = {
76+
0x00F8: "o", # ø
77+
0x0111: "d", # đ
78+
0x0127: "h", # ħ
79+
0x0131: "i", # dotless i
80+
0x0142: "l", # ł
81+
0x0167: "t", # ŧ
82+
0x0180: "b", # ƀ
83+
0x0183: "b", # ƃ
84+
0x0188: "c", # ƈ
85+
0x018C: "d", # ƌ
86+
0x0192: "f", # ƒ
87+
0x0199: "k", # ƙ
88+
0x019A: "l", # ƚ
89+
0x019E: "n", # ƞ
90+
0x01A5: "p", # ƥ
91+
0x01AB: "t", # ƫ
92+
0x01AD: "t", # ƭ
93+
0x01B4: "y", # ƴ
94+
0x01B6: "z", # ƶ
95+
0x01E5: "g", # ǥ
96+
0x0225: "z", # ȥ
97+
0x0234: "l", # ȴ
98+
0x0235: "n", # ȵ
99+
0x0236: "t", # ȶ
100+
0x0237: "j", # ȷ
101+
0x023C: "c", # ȼ
102+
0x023F: "s", # ȿ
103+
0x0240: "z", # ɀ
104+
0x0247: "e", # ɇ
105+
0x0249: "j", # ɉ
106+
0x024B: "q", # ɋ
107+
0x024D: "r", # ɍ
108+
0x024F: "y", # ɏ
109+
}
110+
_DOCUTILS_NON_ID = re.compile(r"[^a-z0-9]+")
111+
_DOCUTILS_AT_ENDS = re.compile(r"^[-0-9]+|-+$")
112+
113+
114+
def docutils_slugify(title: str) -> str:
115+
"""Convert a heading title to a docutils/reStructuredText-style id.
116+
117+
Byte-identical to ``docutils.nodes.make_id``, for uniform anchors across
118+
mixed rST + Markdown projects. Algorithm: lowercase the title; map the
119+
digraph letters ß æ œ ȸ ȹ and 33 further stroked/hooked Latin letters to
120+
ASCII equivalents; NFKD-normalize and drop every remaining non-ASCII
121+
character; normalize whitespace runs to single spaces (kept for
122+
``make_id`` step-parity, though subsumed by the next step); collapse each
123+
run of characters outside ``[a-z0-9]`` to a single hyphen; strip leading
124+
digits/hyphens and trailing hyphens.
125+
126+
The result matches ``[a-z](-?[a-z0-9]+)*`` or is *empty* — the latter for
127+
any title with no Latin letters (so such headings get no anchor), and
128+
docutils' own duplicate-id handling (``idN``) is not emulated:
129+
deduplication is governed by :func:`unique_slug`, whatever the preset.
130+
"""
131+
slug = title.lower()
132+
slug = slug.translate(_DOCUTILS_DIGRAPHS)
133+
slug = slug.translate(_DOCUTILS_TRANSLATE)
134+
slug = unicodedata.normalize("NFKD", slug).encode("ascii", "ignore").decode()
135+
slug = _DOCUTILS_NON_ID.sub("-", " ".join(slug.split()))
136+
return _DOCUTILS_AT_ENDS.sub("", slug)
137+
138+
60139
SLUG_PRESETS: dict[str, Callable[[str], str]] = {
140+
"docutils": docutils_slugify,
61141
"github": github_slugify,
62142
"gitlab": gitlab_slugify,
63143
}

‎tests/fixtures/slugs.json‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
{
2+
"version": 1,
3+
"description": "Normative conformance corpus for MyST heading-anchor slugs. 'slugify' rows pin SLUG_PRESETS[preset](input) == expected, byte-for-byte; 'unique' rows pin unique_slug(slug, existing) == expected. The set is append-only within a version; any semantic change to an existing vector or algorithm requires a version bump. An empty expected slug means the heading receives no anchor.",
24
"slugify": [
35
{
46
"preset": "github",
@@ -104,6 +106,76 @@
104106
"preset": "gitlab",
105107
"input": "!!!",
106108
"expected": ""
109+
},
110+
{
111+
"preset": "docutils",
112+
"input": "Ubuntu 20.04",
113+
"expected": "ubuntu-20-04"
114+
},
115+
{
116+
"preset": "docutils",
117+
"input": "lxc.env for environment variables",
118+
"expected": "lxc-env-for-environment-variables"
119+
},
120+
{
121+
"preset": "docutils",
122+
"input": "2.0",
123+
"expected": ""
124+
},
125+
{
126+
"preset": "docutils",
127+
"input": "3rd party",
128+
"expected": "rd-party"
129+
},
130+
{
131+
"preset": "docutils",
132+
"input": "Привет Мир",
133+
"expected": ""
134+
},
135+
{
136+
"preset": "docutils",
137+
"input": "中文标题",
138+
"expected": ""
139+
},
140+
{
141+
"preset": "docutils",
142+
"input": " a b c ",
143+
"expected": "a-b-c"
144+
},
145+
{
146+
"preset": "docutils",
147+
"input": "Title with UPPER",
148+
"expected": "title-with-upper"
149+
},
150+
{
151+
"preset": "docutils",
152+
"input": "Hello 🎉 World!",
153+
"expected": "hello-world"
154+
},
155+
{
156+
"preset": "docutils",
157+
"input": "!!!",
158+
"expected": ""
159+
},
160+
{
161+
"preset": "docutils",
162+
"input": "Straße & Œuvre",
163+
"expected": "strasze-oeuvre"
164+
},
165+
{
166+
"preset": "docutils",
167+
"input": "łódź ȸȹ",
168+
"expected": "lodz-dbqp"
169+
},
170+
{
171+
"preset": "docutils",
172+
"input": "café naïve",
173+
"expected": "cafe-naive"
174+
},
175+
{
176+
"preset": "docutils",
177+
"input": "__init__ (API)",
178+
"expected": "init-api"
107179
}
108180
],
109181
"unique": [

‎tests/test_anchors.py‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,22 @@ def test_print_anchors_gitlab():
3434
assert re.findall(r'id="([^"]*)"', out) == ["anchor-20"]
3535

3636

37+
def test_print_anchors_docutils():
38+
"""``--slug-func docutils`` applies the make_id-style preset (with dedup)."""
39+
out = _run_cli(
40+
"# Hello World\n\n# Straße & Œuvre\n\n# Hello World",
41+
"-l",
42+
"1",
43+
"--slug-func",
44+
"docutils",
45+
)
46+
assert re.findall(r'id="([^"]*)"', out) == [
47+
"hello-world",
48+
"strasze-oeuvre",
49+
"hello-world-1",
50+
]
51+
52+
3753
# a corpus exercising duplicates, dotted/digit titles, non-latin scripts,
3854
# inline code, emphasis and internal whitespace
3955
CROSS_CHECK_CORPUS = """\
@@ -84,3 +100,54 @@ def test_anchor_slugs_match_renderer():
84100
]
85101

86102
assert cli_ids == doc_slugs
103+
104+
105+
# docutils cross-check corpus: duplicates, dotted/digit titles, digraphs,
106+
# inline code and internal whitespace, all with *non-empty* docutils slugs.
107+
# Empty-slug titles (e.g. `# 2.0`, `# 中文标题`) are deliberately excluded:
108+
# the renderer drops them from deduplication (empty slug -> no anchor), a
109+
# renderer-only rule that the CLI's anchors plugin does not implement.
110+
DOCUTILS_CROSS_CHECK_CORPUS = """\
111+
# Dup
112+
113+
# Dup
114+
115+
# Ubuntu 20.04
116+
117+
# lxc.env for environment variables
118+
119+
# 3rd party
120+
121+
# Straße & Œuvre
122+
123+
# Some `code` here
124+
125+
# Some *emphasis* text
126+
127+
# x y
128+
"""
129+
130+
131+
def test_anchor_slugs_match_renderer_docutils():
132+
"""The CLI and the docutils renderer agree under the ``docutils`` preset."""
133+
cli_out = _run_cli(
134+
DOCUTILS_CROSS_CHECK_CORPUS, "-l", "6", "--slug-func", "docutils"
135+
)
136+
cli_ids = re.findall(r'id="([^"]*)"', cli_out)
137+
138+
doctree = publish_doctree(
139+
DOCUTILS_CROSS_CHECK_CORPUS,
140+
parser=Parser(),
141+
settings_overrides={
142+
"myst_heading_anchors": 6,
143+
"doctitle_xform": False,
144+
"myst_heading_slug_func": "docutils",
145+
},
146+
)
147+
doc_slugs = [
148+
section["slug"]
149+
for section in doctree.findall(nodes.section)
150+
if "slug" in section
151+
]
152+
153+
assert cli_ids == doc_slugs

‎tests/test_docutils.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -597,6 +597,48 @@ def test_text_node_line_stamping():
597597
assert lines["note body"] is None
598598

599599

600+
def test_empty_slug_means_no_anchor():
601+
"""A heading whose title slugifies to nothing gets no anchor.
602+
603+
In particular the empty slug takes no part in deduplication, so a later
604+
empty-slugging heading must not receive a nonsense ``-1`` anchor id.
605+
"""
606+
doctree = publish_doctree(
607+
source="# !!!\n\n# ???\n",
608+
parser=Parser(),
609+
settings_overrides={
610+
"warning_stream": io.StringIO(),
611+
"doctitle_xform": False,
612+
"myst_heading_anchors": 2,
613+
},
614+
)
615+
for section in doctree.findall(nodes.section):
616+
assert "slug" not in section.attributes, section.attributes
617+
assert section["ids"] and all(not i.startswith("-") for i in section["ids"]), (
618+
section["ids"]
619+
)
620+
621+
622+
def test_docutils_slug_preset():
623+
"""The ``docutils`` preset produces make_id-style anchors.
624+
625+
Non-Latin titles slugify to nothing under this preset, so those headings
626+
get no anchor.
627+
"""
628+
doctree = publish_doctree(
629+
source="# Привет\n\n# 2.0\n\n# Hello World\n\n# Straße & Œuvre\n",
630+
parser=Parser(),
631+
settings_overrides={
632+
"warning_stream": io.StringIO(),
633+
"doctitle_xform": False,
634+
"myst_heading_anchors": 2,
635+
"myst_heading_slug_func": "docutils",
636+
},
637+
)
638+
slugs = [section.get("slug") for section in doctree.findall(nodes.section)]
639+
assert slugs == [None, None, "hello-world", "strasze-oeuvre"]
640+
641+
600642
def test_definition_list_orphan_definition():
601643
"""A definition with no preceding term errors, but keeps its content.
602644

0 commit comments

Comments
 (0)