Improve docs SEO/AEO: fix broken canonical URLs, add meta descriptions and llms.txt

[twiecki]

What & why

Google Search Console (export 2026-08-31) shows the docs site ranks well for high-intent queries but has weak indexing and poor snippet CTR:

• 549 pages "Crawled – currently not indexed" vs only ~173 indexed. Root cause found: every page emits a malformed canonical URL with a double slash (https://causalpy.readthedocs.io//en/stable/...). The theme's layout.html joins {{ baseurl }}/{{ language }}/... and our html_context["baseurl"] ended with a trailing slash. (pymc-marketing.io has the same bug — flagged separately.)
• Top method pages rank at position 6–9 for e.g. "staggered did" (695 impressions), "inverse propensity weighting" (500), "interrupted time series", "synthetic control python" — but CTR is 0.4–1.4% because pages ship no meta description (Google synthesizes snippets) and no OpenGraph tags.

Changes

Canonical fix (highest impact)

• conf.py: remove the trailing slash from html_context["baseurl"] → canonicals are now valid absolute URLs pointing at /en/stable/, consolidating latest/stable duplication. Longer term this should also be hardened upstream in labs-sphinx-theme.

Meta descriptions + social cards

• Add sphinxext-opengraph: emits og:* tags sitewide and auto-generates <meta name="description"> from the page's opening text for every page that lacks an explicit one.
• Hand-written descriptions via MyST front matter for the Markdown pages with the most impressions: homepage, gallery index (new description field in gallery.yaml, supported by generate_gallery.py), estimands, design notation, reporting statistics, written resources. (Note: MyST-NB does not parse front matter inside .ipynb first cells — verified locally — so notebooks rely on the auto-description from their opening paragraph; the synthetic control notebook gained a proper intro paragraph for this reason.)

Title/H1 tuning for query match (top underperformers only, gallery cards kept in sync)

• "Staggered Difference-in-Differences" → "Staggered Difference-in-Differences (DiD) in Python"
• "Inverse Propensity Score Weighting with pymc" → "Inverse Propensity Score Weighting (IPW) in Python"
• "Bayesian Interrupted Time Series" → "Bayesian Interrupted Time Series (ITS) Analysis in Python"
• "Synthetic control with pymc models" → "Bayesian Synthetic Control in Python"
• README/homepage H1 → "Causal Inference for Quasi-Experiments in Python" ("causal inference python" currently ranks ~position 50)

AEO (answer-engine optimization)

• Add llms.txt (served at /en/<version>/llms.txt): a structured summary of the library with canonical links, for LLM crawlers (ChatGPT, Claude, Perplexity).
• Add Schema.org SoftwareSourceCode JSON-LD on the landing page via a small layout.html template override.

Verified locally

• Full Sphinx build passes; canonicals render as https://causalpy.readthedocs.io/en/stable/... (single slash).
• Meta descriptions present in built HTML for all touched pages; no front matter leaks into rendered pages.
• llms.txt and robots.txt copied to the build root; JSON-LD present on the homepage only.
• All pre-commit hooks pass (including notebook validation and gallery sync).

Follow-ups (not in this PR)

• RTD dashboard: add an exact redirect /llms.txt → /en/stable/llms.txt so the file also exists at the conventional root path.
• The 32 hard 404s in GSC need the URL list from the GSC UI (the coverage export only contains counts) — worth a pass to add rediraffe entries.
• Fix the canonical construction upstream in labs-sphinx-theme (normalize baseurl, and prefer directory-style canonicals for index pages); pymc-marketing needs the same conf.py fix until then.

🤖 Generated with Claude Code

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[read-the-docs-community]

Documentation build overview

📚 causalpy | 🛠️ Build #34303700 | 📁 Comparing 98d34a2 against latest (994f665)

  🔍 Preview build  

9 files changed · ± 9 modified

± Modified

• 404.html
• index.html
• notebooks/index.html
• notebooks/interrupted-time-series-post-intervention-analysis.html
• notebooks/interrupted-time-series-pymc.html
• notebooks/inverse-propensity-pymc.html
• notebooks/sensitivity_checks.html
• notebooks/staggered-difference-in-differences-pymc.html
• notebooks/synthetic-control-pymc.html

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.03%. Comparing base (994f665) to head (98d34a2).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1181   +/-   ##
=======================================
  Coverage   96.03%   96.03%           
=======================================
  Files         105      105           
  Lines       16363    16370    +7     
  Branches      916      916           
=======================================
+ Hits        15714    15721    +7     
  Misses        485      485           
  Partials      164      164           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.