Skip to content

Commit 254c090

Browse files
fix(docs): SEO improvements (#1858)
* fix(docs): make the generated reference site indexable Four of the errors in the September Ahrefs Site Audit of langfuse.com land on python.reference.langfuse.com, and all four come from how pdoc's output is generated rather than from the SDK. pdoc writes an `index.html` that is only a `<meta http-equiv="refresh">` stub, because a single root module makes `langfuse.html` the entry point. That stub has no title, no links and no canonical URL, so `/` and `/index.html` read as duplicate, soft-redirecting pages. Worse, the host serves that same stub with a `200` for *any* unmatched path when the output contains no `404.html`, so every stale or mistyped URL becomes another indexable copy of it -- which is how two nonexistent `…/concepts/models.md` paths ended up in the crawl. Separately, pdoc inlines the full source of every symbol, which put `langfuse.html` at 2.7 MB, over the crawler's 2 MB limit and slow for readers. - `pdoc-templates/index.html.jinja2` sets `root_module_name` to false, the escape hatch pdoc's own template documents, and fills the main column, so `/` is a real landing page: title, description, canonical, and links to `langfuse`, `langfuse.experiment` and `langfuse.api`. - `pdoc-templates/module.html.jinja2` adds a self-referencing canonical to every module page, without the `.html` suffix -- the site is served with clean URLs, so `/langfuse.html` 308-redirects to `/langfuse` and a canonical pointing there would point at a redirect. - `pdoc-templates/404.html` is copied into the output so unmatched paths get a real 404 instead of a 200. - `--no-show-source` takes `langfuse.html` from 2.7 MB to 484 KB. `--edit-url` replaces the inline source with a link to the module on GitHub. `scripts/build_reference_docs.sh` wraps all of this, so the published site cannot be built correctly by accident; CONTRIBUTING.md and AGENTS.md now point at the script instead of the raw pdoc command. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(docs): harden the reference build against a base URL without a trailing slash Review follow-ups on the reference-site build: - `PDOC_CANONICAL_BASE_URL` was concatenated straight onto the module path, so an origin given without a trailing slash (`https://staging.example.com`, a perfectly valid form) produced `https://staging.example.comlangfuse` on every module page -- the same class of broken canonical this change exists to fix. The base URL is now normalized to exactly one trailing slash. - `OUT_DIR` was captured before the script cd'd to the repo root, so a relative output path resolved against the repo instead of the caller's directory. Relative paths are now made absolute first. - Recorded why the landing-page links keep their `.html` suffix, so the next reader does not mistake it for an oversight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Update CONTRIBUTING.md --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent 7976f2a commit 254c090

6 files changed

Lines changed: 192 additions & 6 deletions

File tree

‎AGENTS.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,9 +154,11 @@ If you change CI bootstrap:
154154

155155
```bash
156156
uv build --no-sources
157-
uv run --group docs pdoc -o docs/ --docformat google --logo "https://langfuse.com/langfuse_logo.svg" langfuse
157+
bash scripts/build_reference_docs.sh
158158
```
159159

160+
Build the reference through that script, not by calling pdoc directly -- it applies the `pdoc-templates/` overrides and the 404 page that the hosted site needs. See "SDK Reference" in `CONTRIBUTING.md`.
161+
160162
Releases are handled by GitHub Actions. Do not build an ad hoc local release flow into repository instructions.
161163

162164
## External Docs

‎CONTRIBUTING.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -125,18 +125,18 @@ The workflow will automatically:
125125

126126
Note: The generated SDK reference is currently work in progress.
127127

128-
The SDK reference is generated via pdoc. The docs dependency group is installed on demand when you run the documentation commands.
128+
The SDK reference is generated via pdoc and published at [python.reference.langfuse.com](https://python.reference.langfuse.com). The docs dependency group is installed on demand when you run the documentation commands.
129129

130-
To update the reference, run the following command:
130+
To build the reference into `docs/`, run:
131131

132132
```sh
133-
uv run --group docs pdoc -o docs/ --docformat google --logo "https://langfuse.com/langfuse_logo.svg" langfuse
133+
bash scripts/build_reference_docs.sh
134134
```
135135

136-
To run the reference locally, you can use the following command:
136+
To browse the reference locally with live reload, run pdoc's dev server:
137137

138138
```sh
139-
uv run --group docs pdoc --docformat google --logo "https://langfuse.com/langfuse_logo.svg" langfuse
139+
uv run --group docs pdoc --docformat google --logo "https://langfuse.com/langfuse_logo.svg" --template-directory pdoc-templates langfuse
140140
```
141141

142142
## Credits

‎pdoc-templates/404.html‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
<!doctype html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="utf-8" />
5+
<meta name="viewport" content="width=device-width, initial-scale=1" />
6+
<meta name="robots" content="noindex" />
7+
<title>Page not found &ndash; Langfuse Python SDK API reference</title>
8+
<style>
9+
body {
10+
margin: 0;
11+
min-height: 100vh;
12+
display: flex;
13+
align-items: center;
14+
justify-content: center;
15+
font-family:
16+
system-ui,
17+
-apple-system,
18+
"Segoe UI",
19+
Roboto,
20+
"Helvetica Neue",
21+
Arial,
22+
sans-serif;
23+
color: #212529;
24+
background: #fff;
25+
}
26+
main {
27+
max-width: 32rem;
28+
padding: 2rem;
29+
text-align: center;
30+
}
31+
img {
32+
max-width: 12rem;
33+
margin-bottom: 2rem;
34+
}
35+
h1 {
36+
font-size: 1.5rem;
37+
margin: 0 0 0.75rem;
38+
}
39+
p {
40+
margin: 0 0 1.5rem;
41+
line-height: 1.6;
42+
color: #495057;
43+
}
44+
a {
45+
color: #0d6efd;
46+
}
47+
</style>
48+
</head>
49+
<body>
50+
<main>
51+
<img src="https://langfuse.com/langfuse_logo.svg" alt="Langfuse" />
52+
<h1>Page not found</h1>
53+
<p>
54+
This page is not part of the Langfuse Python SDK API reference. The
55+
symbol may have been renamed or removed in a later release.
56+
</p>
57+
<p>
58+
<a href="/">Browse the API reference</a> &middot;
59+
<a href="https://langfuse.com/docs">Langfuse documentation</a>
60+
</p>
61+
</main>
62+
</body>
63+
</html>

‎pdoc-templates/index.html.jinja2‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
{#
2+
Root page of https://python.reference.langfuse.com.
3+
4+
pdoc's default index is a bare `<meta http-equiv="refresh">` stub with no
5+
title, no links and no canonical URL, because a single root module makes
6+
`langfuse.html` the entry point. Search engines read that stub as a
7+
soft-redirecting duplicate of `/index.html`. Setting `root_module_name` to
8+
false -- the escape hatch the default template documents -- renders the real
9+
module list instead, and the `content` block below gives that page something
10+
to say, since pdoc's own version leaves the main column empty.
11+
#}
12+
{% set canonical_base_url = env.get("PDOC_CANONICAL_BASE_URL", "https://python.reference.langfuse.com/").rstrip("/") ~ "/" %}
13+
{% set root_module_name = false %}
14+
{% extends "default/index.html.jinja2" %}
15+
16+
{% block title %}Langfuse Python SDK API reference{% endblock %}
17+
18+
{% block head %}
19+
<link rel="canonical" href="{{ canonical_base_url }}"/>
20+
<meta name="description"
21+
content="Generated API reference for the Langfuse Python SDK: clients, tracing, prompt management, datasets, evaluation and the underlying API models."/>
22+
{% endblock %}
23+
24+
{% block content %}
25+
<header class="pdoc">
26+
{{ self.logo() }}
27+
{% if search %}
28+
<input type="search" placeholder="Search API Documentation..." aria-label="search box">
29+
{% endif %}
30+
</header>
31+
<main class="pdoc">
32+
<section>
33+
<h1>Langfuse Python SDK API reference</h1>
34+
<p>
35+
Generated reference for the <code>langfuse</code> package. It documents
36+
every public symbol; the hand-written guides on
37+
<a href="https://langfuse.com/docs">langfuse.com/docs</a> are the better
38+
starting point if you are setting Langfuse up for the first time.
39+
</p>
40+
<h2>Start here</h2>
41+
{#
42+
These keep the `.html` suffix even though the hosted site serves
43+
clean URLs. pdoc generates its own sidebar links that way, and
44+
extensionless paths 404 both under pdoc's dev server and when the
45+
built output is served locally, so the one 308 hop in production is
46+
the cheaper trade. Canonical URLs are a different case -- those must
47+
not point at a redirect, which is why they drop the suffix.
48+
#}
49+
<ul>
50+
<li><a href="langfuse.html"><code>langfuse</code></a> &mdash; the
51+
<code>Langfuse</code> client, tracing decorators and context helpers.
52+
Most code only needs this module.</li>
53+
<li><a href="langfuse/experiment.html"><code>langfuse.experiment</code></a>
54+
&mdash; running experiments over datasets and scoring the results.</li>
55+
<li><a href="langfuse/api.html"><code>langfuse.api</code></a> &mdash; the
56+
generated low-level API client and its request and response models.</li>
57+
</ul>
58+
<h2>Elsewhere</h2>
59+
<ul>
60+
<li><a href="https://langfuse.com/docs/observability/sdk/python/overview">Python SDK documentation</a></li>
61+
<li><a href="https://github.com/langfuse/langfuse-python">langfuse-python on GitHub</a></li>
62+
<li><a href="https://pypi.org/project/langfuse/">langfuse on PyPI</a></li>
63+
</ul>
64+
<p>Every module is listed in the sidebar, and the search box covers all of them.</p>
65+
</section>
66+
</main>
67+
{% if search %}
68+
{% include "search.html.jinja2" %}
69+
{% endif %}
70+
{% endblock %}

‎pdoc-templates/module.html.jinja2‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{#
2+
Adds a self-referencing canonical URL to every module page.
3+
4+
The `.html` suffix is deliberately dropped: Cloudflare Pages serves this site
5+
with clean URLs and 308-redirects `/langfuse.html` to `/langfuse`, so a
6+
canonical pointing at the `.html` path would point at a redirect.
7+
#}
8+
{% set canonical_base_url = env.get("PDOC_CANONICAL_BASE_URL", "https://python.reference.langfuse.com/").rstrip("/") ~ "/" %}
9+
{% extends "default/module.html.jinja2" %}
10+
11+
{% block head %}
12+
{{ super() }}
13+
<link rel="canonical"
14+
href="{{ canonical_base_url }}{{ module.modulename.replace('.', '/') }}"/>
15+
{% endblock %}

‎scripts/build_reference_docs.sh‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/usr/bin/env bash
2+
# Builds the API reference published at https://python.reference.langfuse.com.
3+
#
4+
# Use this instead of calling pdoc directly: it applies the template overrides
5+
# in pdoc-templates/ and ships the 404 page, both of which the hosted site
6+
# needs. Set PDOC_CANONICAL_BASE_URL to build for a different origin.
7+
set -euo pipefail
8+
9+
OUT_DIR="${1:-docs}"
10+
11+
# Resolve a relative output path against the caller's working directory before
12+
# cd'ing to the repo root, so `build_reference_docs.sh out` from elsewhere does
13+
# not silently write to <repo>/out.
14+
case "$OUT_DIR" in
15+
/*) ;;
16+
*) OUT_DIR="$PWD/$OUT_DIR" ;;
17+
esac
18+
19+
cd "$(dirname "$0")/.."
20+
21+
uv run --group docs pdoc \
22+
-o "$OUT_DIR" \
23+
--docformat google \
24+
--logo "https://langfuse.com/langfuse_logo.svg" \
25+
--logo-link "https://langfuse.com" \
26+
--template-directory pdoc-templates \
27+
--edit-url "langfuse=https://github.com/langfuse/langfuse-python/blob/main/langfuse/" \
28+
--no-show-source \
29+
langfuse
30+
31+
# Cloudflare Pages serves the closest index.html with a 200 for any unmatched
32+
# path unless the output contains a 404.html, which turns every stale or
33+
# mistyped URL into an indexable duplicate of the landing page.
34+
cp pdoc-templates/404.html "$OUT_DIR/404.html"
35+
36+
echo "Reference docs written to $OUT_DIR/"

0 commit comments

Comments
 (0)