Skip to content

Commit 82bce53

Browse files
tedsluisclaude
andcommitted
docs: sitemap.xml lists only the site root (option B)
Researched and confirmed: 68 of the site's 69 pages return a genuine HTTP 404 status (GitHub Pages has no server-side rewrite, so routerMode: 'history' relies on the 404.html fallback trick — correct content, wrong status code). Google's own documentation confirms a 4xx status is a hard stop for indexing regardless of rendered content (developers.google.com/search/docs/crawling-indexing/http-network-errors). A sitemap listing all 69 would just be 68 confirmed-dead entries in Search Console's coverage report. generate_sitemap.py now emits a single <url> entry for the site root only, which does return a real 200. Documented the reasoning in the script's own docstring and in a new MAINTAINING_DOCS_SITE.md section, including the path back (build-time pre-rendering to static files per route) if this gets revisited later. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bjo12itgeoZW8rPFPXUZvz
1 parent 4e2a86d commit 82bce53

3 files changed

Lines changed: 51 additions & 308 deletions

File tree

MAINTAINING_DOCS_SITE.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,27 @@ and kept correct mechanically, not by hand:
5555
If you add a brand-new page, run `./scripts/ensure_footers.py` once (same as the sidebar-generation
5656
step above) rather than typing the footer by hand.
5757

58+
## sitemap.xml lists the site root only, on purpose
59+
60+
`scripts/generate_sitemap.py` (run automatically by
61+
`.github/workflows/update-sitemap.yml` on every push touching a `.md` file) generates
62+
`sitemap.xml` with exactly one `<url>` entry — the site root — not one per page.
63+
64+
This isn't an oversight. `index.html` uses `routerMode: 'history'`, and GitHub Pages has no
65+
server-side rewrite support, so every URL except the bare root is served via the `404.html`
66+
fallback trick — with a genuine HTTP 404 status, even though the page renders correctly once
67+
Docsify's JS boots. Google's own documentation is explicit that a 4xx status is a hard stop for
68+
indexing regardless of rendered content (see
69+
[HTTP status codes and network/DNS errors](https://developers.google.com/search/docs/crawling-indexing/http-network-errors)).
70+
Listing all pages would just be dozens of confirmed-dead entries in Search Console's coverage
71+
report — decided 2026-08-25 to keep the sitemap to the one URL that's actually indexable as
72+
things stand, rather than that.
73+
74+
If a future change gives individual pages a real HTTP 200 (e.g. a build step that pre-renders
75+
each route to a static file, discussed but not implemented as of this decision — see
76+
`scripts/generate_sitemap.py`'s own docstring), `generate_sitemap.py`'s scope should expand back
77+
to all pages.
78+
5879
## Checking it worked
5980

6081
`https://tedsluis.github.io/opencontrolpixelbudspro2/` — GitHub Pages builds typically go live

scripts/generate_sitemap.py

Lines changed: 30 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,25 @@
11
#!/usr/bin/env python3
2-
"""Generates sitemap.xml for the Docsify site (history-mode, real path-based URLs).
3-
4-
Reuses the exact file set `ensure_footers.py`/`lint_docs.py` already treat as
5-
"doc pages" (`footer_files()` — every tracked `.md` file except `_sidebar.md`,
6-
`CLAUDE.md`, and `AUDIT_REPORT_*.md`, the last excluded by
7-
`all_markdown_files()`) so this can't silently drift from what the footer
8-
system considers a real page.
9-
10-
Each URL uses `docsify_route()` (same helper the footer URLs use) so the
11-
sitemap always matches whatever path scheme `index.html`'s routerMode
12-
actually serves — history mode, no `#`. `README.md` maps to the site root
13-
(`BASE_URL/`), matching `index.html`'s `homepage: 'README.md'`.
14-
15-
`<lastmod>` is each file's last commit date (`git log -1 --format=%cI`) — a
16-
full, unshallowed checkout is required for this to be accurate (see
17-
`.github/workflows/update-sitemap.yml`'s `fetch-depth: 0`); a file with no
18-
resolvable git history (e.g. never committed yet) is emitted without one
19-
rather than guessing.
2+
"""Generates sitemap.xml for the Docsify site — the site root only.
3+
4+
Root-only by deliberate choice (2026-08-25), not an oversight: index.html
5+
serves routerMode: 'history' via GitHub Pages' 404.html fallback trick (no
6+
server-side rewrite support there), which means every URL except the bare
7+
site root returns a genuine HTTP 404 status — confirmed live by curl against
8+
all 69 pages. Per Google's own documentation, a 4xx status is a hard stop:
9+
"any content Google receives from URLs that return a 4xx status code is
10+
ignored" (developers.google.com/search/docs/crawling-indexing/http-network-errors).
11+
A sitemap listing all 69 pages would just be 68 confirmed dead entries in
12+
Search Console's coverage report — this lists only the one URL that's
13+
actually indexable as things stand.
14+
15+
If a future change makes individual pages return real 200s (e.g. build-time
16+
pre-rendering to static files per route), this script's scope should expand
17+
back to all pages — see MAINTAINING_DOCS_SITE.md and git history around
18+
2026-08-25 for that discussion and the options considered.
19+
20+
`<lastmod>` is README.md's own last commit date (`git log -1 --format=%cI`) —
21+
a full, unshallowed checkout is required for this to be accurate (see
22+
`.github/workflows/update-sitemap.yml`'s `fetch-depth: 0`).
2023
2124
Usage: ./scripts/generate_sitemap.py
2225
"""
@@ -29,7 +32,7 @@
2932
from xml.sax.saxutils import escape
3033

3134
sys.path.insert(0, str(Path(__file__).resolve().parent))
32-
from lint_docs import REPO_ROOT, docsify_route, footer_files # noqa: E402
35+
from lint_docs import REPO_ROOT # noqa: E402
3336

3437
BASE_URL = "https://tedsluis.github.io/opencontrolpixelbudspro2"
3538

@@ -46,34 +49,25 @@ def last_commit_date(path: Path) -> str | None:
4649
return date or None
4750

4851

49-
def page_url(path: Path) -> str:
50-
rel = path.relative_to(REPO_ROOT).as_posix()
51-
if rel == "README.md":
52-
return f"{BASE_URL}/"
53-
return f"{BASE_URL}/{docsify_route(rel)}"
54-
55-
5652
def main() -> int:
57-
files = sorted(footer_files(), key=lambda p: p.relative_to(REPO_ROOT).as_posix())
53+
readme = REPO_ROOT / "README.md"
54+
lastmod = last_commit_date(readme)
5855

5956
lines = [
6057
'<?xml version="1.0" encoding="UTF-8"?>',
6158
'<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
59+
" <url>",
60+
f" <loc>{escape(BASE_URL)}/</loc>",
6261
]
63-
for path in files:
64-
url = page_url(path)
65-
lastmod = last_commit_date(path)
66-
lines.append(" <url>")
67-
lines.append(f" <loc>{escape(url)}</loc>")
68-
if lastmod:
69-
lines.append(f" <lastmod>{lastmod}</lastmod>")
70-
lines.append(" </url>")
62+
if lastmod:
63+
lines.append(f" <lastmod>{lastmod}</lastmod>")
64+
lines.append(" </url>")
7165
lines.append("</urlset>")
7266
lines.append("")
7367

7468
out_path = REPO_ROOT / "sitemap.xml"
7569
out_path.write_text("\n".join(lines), encoding="utf-8")
76-
print(f"Wrote {out_path.relative_to(REPO_ROOT)} with {len(files)} URLs")
70+
print(f"Wrote {out_path.relative_to(REPO_ROOT)} with 1 URL (root only — see this script's docstring)")
7771
return 0
7872

7973

0 commit comments

Comments
 (0)