Skip to content

Commit db405db

Browse files
committed
Add crawlable static routes and SEO sitemap
1 parent d1b5bd8 commit db405db

23 files changed

Lines changed: 658 additions & 60 deletions

‎.agents/skills/pre-production-checklist/SKILL.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,21 @@ Run this gate whenever public routes, the canonical domain, page purpose, contac
3535

3636
This gate passes only when the file accurately describes the current public site, all listed destinations are intentional, and the deployed copy is reachable without authentication.
3737

38-
## Gate 3 — Officer profile publication
38+
## Gate 3 — Search discovery and route metadata
39+
40+
Run this gate whenever public routes, metadata, canonical-domain behavior, or the Pages build changes.
41+
42+
1. Treat `src/data/siteIdentity.json` and `src/data/sitePages.json` as the shared registries for public identity, canonical origin, indexable routes, and route-specific titles and descriptions. Compare them with the public routes in `src/App.tsx` and the main-page links in `public/llms.txt`.
43+
2. Run `npm test` and `npm run build`. Confirm every registered route produces `dist/<route>/index.html` with rendered page content rather than an empty React root.
44+
3. Serve `dist` with a plain static server and request every canonical route directly. Each registered route must return `200`; unknown and `/concepts` routes must remain non-indexable.
45+
4. Inspect each generated page's initial HTML. Confirm it has one unique title, description, self-referencing canonical URL, Open Graph URL, `index, follow` directive, and valid Organization/WebPage JSON-LD.
46+
5. Confirm `dist/sitemap.xml` lists every registered production canonical exactly once and no staging, concept-review, or unknown URL. Confirm `dist/robots.txt` allows crawling and references the production sitemap.
47+
6. Run a staging build with `VITE_NOINDEX=true` and the staging `VITE_SITE_ORIGIN`. Confirm every generated page says `noindex, nofollow`, `dist/robots.txt` allows crawlers to fetch and observe that directive without advertising a sitemap, and no sitemap is published. On GitHub project Pages, remember that a robots file below the host root is not a host-wide crawl policy.
48+
7. After deployment, request the production routes, `/robots.txt`, and `/sitemap.xml` directly. In Google Search Console, submit the sitemap and use URL Inspection on the homepage and at least one inner page.
49+
50+
This gate passes only when production routes are directly crawlable with consistent canonical metadata, preview routes cannot become duplicate search results, and the deployed sitemap matches the public route registry.
51+
52+
## Gate 4 — Officer profile publication
3953

4054
Run this gate whenever the officer roster, a form-response snapshot, profile links, bios, or officer-data build logic changes.
4155

‎.github/workflows/staging-pages.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ jobs:
4141
run: npm run build:preview
4242
env:
4343
VITE_BASE_PATH: /compbioatberkeley.github.io/
44+
VITE_SITE_ORIGIN: https://hilo-hilo.github.io/compbioatberkeley.github.io
45+
VITE_NOINDEX: "true"
4446

4547
- name: Configure Pages
4648
uses: actions/configure-pages@v5

‎README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,14 @@ npm run validate:officers
4343

4444
The validator reports unmatched responses, invalid public links, missing assets, and officers still waiting for a current profile response. See `scripts/README.md` for the update workflow and `.agents/skills/pre-production-checklist/SKILL.md` before a production promotion.
4545

46+
## Search and static routes
47+
48+
- `src/data/siteIdentity.json` and `src/data/sitePages.json` are the shared registries for the public identity, indexable routes, page titles, and descriptions.
49+
- `npm run build` creates rendered HTML at each route directory, then generates and validates `dist/sitemap.xml` and `dist/robots.txt`.
50+
- `src/components/RouteMetadata.tsx` keeps metadata synchronized during client-side navigation.
51+
- Production builds are indexable by default. Staging sets `VITE_NOINDEX=true` and a staging `VITE_SITE_ORIGIN`, which marks every page `noindex, nofollow` and suppresses the sitemap. Its robots file allows fetching so crawlers can observe the page-level directive.
52+
- `public/404.html` is always `noindex`; internal `/concepts` review routes are intentionally absent from the sitemap and static route output.
53+
4654
## Deployment
4755

4856
- Pushes to `dev` deploy the staging GitHub Pages site from the `Hilo-Hilo` staging repository.

‎index.html‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
<meta charset="UTF-8" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
66
<title>Computational Biology at Berkeley</title>
7-
<meta name="description" content="A student-driven community exploring the intersection of computer science, biology, and healthcare at UC Berkeley." />
7+
<meta name="description" content="A UC Berkeley student organization connecting biology, computer science, and healthcare through education, research, projects, and community." />
88
<meta name="author" content="Computational Biology at Berkeley" />
99
<meta name="theme-color" content="#FFFFFF" />
1010
<link rel="canonical" href="https://compbioatberkeley.github.io/" />
@@ -78,14 +78,14 @@
7878
</script>
7979

8080
<meta property="og:title" content="Computational Biology at Berkeley" />
81-
<meta property="og:description" content="A student-driven community exploring the intersection of computer science, biology, and healthcare at UC Berkeley." />
81+
<meta property="og:description" content="A UC Berkeley student organization connecting biology, computer science, and healthcare through education, research, projects, and community." />
8282
<meta property="og:type" content="website" />
8383
<meta property="og:url" content="https://compbioatberkeley.github.io/" />
8484
<meta property="og:image" content="https://compbioatberkeley.github.io/comp-bio-logo-berkeley.svg" />
8585
<meta property="og:image:type" content="image/svg+xml" />
8686
<meta name="twitter:card" content="summary" />
8787
<meta name="twitter:title" content="Computational Biology at Berkeley" />
88-
<meta name="twitter:description" content="A student-driven community exploring computer science, biology, and healthcare at UC Berkeley." />
88+
<meta name="twitter:description" content="A UC Berkeley student organization connecting biology, computer science, and healthcare through education, research, projects, and community." />
8989
<meta name="twitter:image" content="https://compbioatberkeley.github.io/comp-bio-logo-berkeley.svg" />
9090
</head>
9191

‎package.json‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,12 @@
1212
"officers:notion:dry-run": "node scripts/sync-officer-profiles-notion.js --term fa26 --dry-run",
1313
"officers:notion:check": "node scripts/sync-officer-profiles-notion.js --term fa26 --check",
1414
"officers:notion:write": "node scripts/sync-officer-profiles-notion.js --term fa26 --write",
15-
"build": "npm run validate:officers && vite build",
16-
"build:dev": "npm run validate:officers && vite build --mode development",
17-
"build:preview": "npm run validate:officers && vite build",
15+
"build:client": "vite build",
16+
"build:server": "vite build --ssr src/entry-server.tsx --outDir .seo-ssr --emptyOutDir",
17+
"seo:check": "node scripts/validate-seo-build.mjs",
18+
"build": "npm run validate:officers && npm run build:client && npm run build:server && node scripts/prerender-site.mjs && npm run seo:check",
19+
"build:dev": "npm run validate:officers && vite build --mode development && vite build --mode development --ssr src/entry-server.tsx --outDir .seo-ssr --emptyOutDir && node scripts/prerender-site.mjs && npm run seo:check",
20+
"build:preview": "npm run build",
1821
"lint": "eslint .",
1922
"preview": "vite preview"
2023
},

‎public/404.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
<head>
44
<meta charset="utf-8" />
55
<title>Computational Biology at Berkeley</title>
6+
<meta name="robots" content="noindex, nofollow" />
67
<script>
78
(function () {
89
var projectName = "compbioatberkeley.github.io";

‎public/llms.txt‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,11 @@ The organization welcomes students at all experience levels. Use the pages below
77
## Main pages
88

99
- [Home](https://compbioatberkeley.github.io/): Overview of the organization, its objectives, and current calls to action.
10-
- [About](https://compbioatberkeley.github.io/about): Mission and work across education, research, professional networking, and community impact.
11-
- [Calendar](https://compbioatberkeley.github.io/calendar): Current workshops, meetings, and special events in Pacific time.
12-
- [Collaborations](https://compbioatberkeley.github.io/collaborations): Partnership opportunities for students, faculty, alumni, research groups, and industry.
13-
- [Officers](https://compbioatberkeley.github.io/officers): Fall 2026 and previous student leadership rosters, including submitted public bios and profile links.
14-
- [Join](https://compbioatberkeley.github.io/signup): Interest form, newsletter signup, and community channels.
10+
- [About](https://compbioatberkeley.github.io/about/): Mission and work across education, research, professional networking, and community impact.
11+
- [Calendar](https://compbioatberkeley.github.io/calendar/): Current workshops, meetings, and special events in Pacific time.
12+
- [Collaborations](https://compbioatberkeley.github.io/collaborations/): Partnership opportunities for students, faculty, alumni, research groups, and industry.
13+
- [Officers](https://compbioatberkeley.github.io/officers/): Fall 2026 and previous student leadership rosters, including submitted public bios and profile links.
14+
- [Join](https://compbioatberkeley.github.io/signup/): Interest form, newsletter signup, and community channels.
1515

1616
## Contact and community
1717

‎public/robots.txt‎

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,4 @@
1-
User-agent: Googlebot
2-
Allow: /
3-
4-
User-agent: Bingbot
5-
Allow: /
6-
7-
User-agent: Twitterbot
8-
Allow: /
9-
10-
User-agent: facebookexternalhit
11-
Allow: /
12-
131
User-agent: *
142
Allow: /
3+
4+
Sitemap: https://compbioatberkeley.github.io/sitemap.xml

‎scripts/lib/site-seo.js‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
const escapeXml = (value) =>
2+
String(value)
3+
.replaceAll("&", "&amp;")
4+
.replaceAll("<", "&lt;")
5+
.replaceAll(">", "&gt;")
6+
.replaceAll('"', "&quot;")
7+
.replaceAll("'", "&apos;");
8+
9+
const canonicalUrl = (siteOrigin, pathname) =>
10+
new URL(pathname.replace(/^\//, ""), `${siteOrigin.replace(/\/$/, "")}/`).href;
11+
12+
const escapeHtml = (value) =>
13+
String(value)
14+
.replaceAll("&", "&amp;")
15+
.replaceAll("<", "&lt;")
16+
.replaceAll(">", "&gt;")
17+
.replaceAll('"', "&quot;");
18+
19+
const replaceRequired = (html, pattern, replacement, label) => {
20+
if (!pattern.test(html)) {
21+
throw new Error(`Unable to find ${label} in the Vite HTML template.`);
22+
}
23+
return html.replace(pattern, replacement);
24+
};
25+
26+
const jsonForHtml = (value) => JSON.stringify(value).replaceAll("<", "\\u003c");
27+
28+
export const buildSitemapXml = ({ pages, siteOrigin, lastModified }) => {
29+
const urls = pages
30+
.map(
31+
(page) => ` <url>
32+
<loc>${escapeXml(canonicalUrl(siteOrigin, page.path))}</loc>
33+
<lastmod>${escapeXml(lastModified)}</lastmod>
34+
</url>`,
35+
)
36+
.join("\n");
37+
38+
return `<?xml version="1.0" encoding="UTF-8"?>
39+
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
40+
${urls}
41+
</urlset>
42+
`;
43+
};
44+
45+
export const buildRobotsTxt = ({ noindex, siteOrigin }) => {
46+
// Let crawlers fetch staging pages so they can observe the page-level noindex.
47+
// A robots.txt file under a GitHub project-pages subpath is not host-wide policy.
48+
if (noindex) return "User-agent: *\nAllow: /\n";
49+
const sitemapUrl = canonicalUrl(siteOrigin, "/sitemap.xml");
50+
return `User-agent: *\nAllow: /\n\nSitemap: ${sitemapUrl}\n`;
51+
};
52+
53+
export const buildPageHtml = ({
54+
template,
55+
appHtml,
56+
page,
57+
siteIdentity,
58+
siteOrigin,
59+
noindex,
60+
}) => {
61+
const url = canonicalUrl(siteOrigin, page.path);
62+
const title = escapeHtml(page.title);
63+
const description = escapeHtml(page.description);
64+
const structuredData = [
65+
{
66+
"@context": "https://schema.org",
67+
"@type": "EducationalOrganization",
68+
name: siteIdentity.name,
69+
alternateName: siteIdentity.alternateName,
70+
url: canonicalUrl(siteOrigin, "/"),
71+
logo: canonicalUrl(siteOrigin, siteIdentity.logoPath),
72+
sameAs: siteIdentity.sameAs,
73+
},
74+
{
75+
"@context": "https://schema.org",
76+
"@type": "WebPage",
77+
name: page.title,
78+
description: page.description,
79+
url,
80+
isPartOf: {
81+
"@type": "WebSite",
82+
name: siteIdentity.name,
83+
url: canonicalUrl(siteOrigin, "/"),
84+
},
85+
},
86+
];
87+
88+
let html = replaceRequired(
89+
template,
90+
/<title>[^<]*<\/title>/i,
91+
`<title>${title}</title>`,
92+
"title",
93+
);
94+
html = replaceRequired(
95+
html,
96+
/<meta\s+name="description"\s+content="[^"]*"\s*\/?\s*>/i,
97+
`<meta name="description" content="${description}" />`,
98+
"meta description",
99+
);
100+
html = replaceRequired(
101+
html,
102+
/<link\s+rel="canonical"\s+href="[^"]*"\s*\/?\s*>/i,
103+
`<link rel="canonical" href="${escapeHtml(url)}" />`,
104+
"canonical link",
105+
);
106+
107+
const replacements = [
108+
["og:title", title],
109+
["og:description", description],
110+
["og:url", escapeHtml(url)],
111+
];
112+
for (const [property, content] of replacements) {
113+
html = replaceRequired(
114+
html,
115+
new RegExp(`<meta\\s+property="${property}"\\s+content="[^"]*"\\s*\\/?\\s*>`, "i"),
116+
`<meta property="${property}" content="${content}" />`,
117+
property,
118+
);
119+
}
120+
121+
const twitterReplacements = [
122+
["twitter:title", title],
123+
["twitter:description", description],
124+
];
125+
for (const [name, content] of twitterReplacements) {
126+
html = replaceRequired(
127+
html,
128+
new RegExp(`<meta\\s+name="${name}"\\s+content="[^"]*"\\s*\\/?\\s*>`, "i"),
129+
`<meta name="${name}" content="${content}" />`,
130+
name,
131+
);
132+
}
133+
134+
html = html.replace(/\s*<meta\s+name="robots"[^>]*>/i, "");
135+
const robotsMeta = noindex
136+
? '<meta name="robots" content="noindex, nofollow" />'
137+
: '<meta name="robots" content="index, follow" />';
138+
html = html.replace(
139+
"</head>",
140+
` ${robotsMeta}\n <script id="site-structured-data" type="application/ld+json">${jsonForHtml(structuredData)}</script>\n </head>`,
141+
);
142+
html = replaceRequired(
143+
html,
144+
/<div\s+id="root"\s*><\/div>/i,
145+
`<div id="root">${appHtml}</div>`,
146+
"React root",
147+
);
148+
149+
return html;
150+
};

‎scripts/lib/site-seo.test.js‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
import assert from "node:assert/strict";
2+
import fs from "node:fs";
3+
import test from "node:test";
4+
5+
import { buildPageHtml, buildRobotsTxt, buildSitemapXml } from "./site-seo.js";
6+
7+
const pages = [
8+
{ path: "/", title: "Home", description: "Home page" },
9+
{ path: "/about/", title: "About", description: "About page" },
10+
];
11+
const siteIdentity = {
12+
name: "Example Organization",
13+
alternateName: "Example Org",
14+
logoPath: "/logo.svg",
15+
sameAs: ["https://social.example.com/example"],
16+
};
17+
18+
const htmlTemplate = `<!doctype html><html><head>
19+
<title>Fallback</title>
20+
<meta name="description" content="Fallback description" />
21+
<link rel="canonical" href="https://example.com/" />
22+
<meta property="og:title" content="Fallback" />
23+
<meta property="og:description" content="Fallback description" />
24+
<meta property="og:url" content="https://example.com/" />
25+
<meta name="twitter:title" content="Fallback" />
26+
<meta name="twitter:description" content="Fallback description" />
27+
</head><body><div id="root"></div></body></html>`;
28+
29+
test("the shared public-page registry covers every indexable site route", () => {
30+
const registry = JSON.parse(
31+
fs.readFileSync(new URL("../../src/data/sitePages.json", import.meta.url), "utf8"),
32+
);
33+
34+
assert.deepEqual(
35+
registry.map((page) => page.path),
36+
["/", "/about/", "/calendar/", "/collaborations/", "/officers/", "/signup/"],
37+
);
38+
assert.ok(registry.every((page) => page.title && page.description));
39+
assert.ok(registry.every((page) => !page.path.startsWith("/concepts")));
40+
});
41+
42+
test("builds a sitemap containing each canonical public URL exactly once", () => {
43+
const sitemap = buildSitemapXml({
44+
pages,
45+
siteOrigin: "https://example.com/",
46+
lastModified: "2026-07-31",
47+
});
48+
49+
assert.match(sitemap, /<loc>https:\/\/example\.com\/<\/loc>/);
50+
assert.match(sitemap, /<loc>https:\/\/example\.com\/about\/<\/loc>/);
51+
assert.equal((sitemap.match(/<url>/g) ?? []).length, pages.length);
52+
assert.equal((sitemap.match(/https:\/\/example\.com\/about\//g) ?? []).length, 1);
53+
assert.match(sitemap, /<lastmod>2026-07-31<\/lastmod>/);
54+
});
55+
56+
test("embeds rendered content and unique route metadata in the initial HTML", () => {
57+
const html = buildPageHtml({
58+
template: htmlTemplate,
59+
appHtml: "<main><h1>About us</h1></main>",
60+
page: pages[1],
61+
siteIdentity,
62+
siteOrigin: "https://example.com/",
63+
noindex: false,
64+
});
65+
66+
assert.match(html, /<title>About<\/title>/);
67+
assert.match(html, /name="description" content="About page"/);
68+
assert.match(html, /rel="canonical" href="https:\/\/example\.com\/about\/"/);
69+
assert.match(html, /property="og:url" content="https:\/\/example\.com\/about\/"/);
70+
assert.match(html, /<main><h1>About us<\/h1><\/main>/);
71+
assert.match(html, /"@type":"WebPage"/);
72+
assert.match(html, /"@type":"EducationalOrganization"/);
73+
assert.doesNotMatch(html, /name="robots" content="noindex/);
74+
});
75+
76+
test("marks staging output as noindex and lets crawlers observe that directive", () => {
77+
const html = buildPageHtml({
78+
template: htmlTemplate,
79+
appHtml: "<main>Preview</main>",
80+
page: pages[0],
81+
siteIdentity,
82+
siteOrigin: "https://preview.example.com/",
83+
noindex: true,
84+
});
85+
86+
assert.match(html, /name="robots" content="noindex, nofollow"/);
87+
assert.equal(buildRobotsTxt({ noindex: true }), "User-agent: *\nAllow: /\n");
88+
assert.equal(
89+
buildRobotsTxt({ noindex: false, siteOrigin: "https://example.com/" }),
90+
"User-agent: *\nAllow: /\n\nSitemap: https://example.com/sitemap.xml\n",
91+
);
92+
});

0 commit comments

Comments
 (0)