Skip to content

Commit ec6c7c5

Browse files
Add Automation/: the release pipeline that builds ducksteps
Publishes the scripts alongside the releases they produce, so anyone downloading a build can read how it was made: which compiler flags, which PGO corpus, how the CVE list was resolved, and what was checked before it shipped. Synced automatically on every publish by publish.sync_automation, so this directory always matches the pipeline that built the newest release rather than drifting from it. .env and state.json are excluded by name, not merely by pattern. .env holds the ntfy topics, which in practice are the password for the approval buttons, and state.json holds local build state and draft URLs. The sync refuses both regardless of what the file globs match, since this directory is public.
1 parent fe21907 commit ec6c7c5

13 files changed

Lines changed: 3696 additions & 0 deletions

Automation/.gitignore

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Secrets and machine-local state
2+
.env
3+
*.key
4+
state.json
5+
*.lock
6+
7+
# Python
8+
__pycache__/
9+
*.pyc
10+
11+
# Run artifacts. Regenerated by every build and worth nothing in history:
12+
# logs are per-run, smoke screenshots are ~150KB each, and releases/ holds the actual
13+
# shipped installers and archives (roughly 340MB per release).
14+
logs/*.log
15+
logs/*.png
16+
releases/

Automation/README.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# ducksteps release automation
2+
3+
The scripts that build and publish ducksteps. Kept here so the pipeline that produced a
4+
release is readable alongside the release itself. Synced automatically on every publish,
5+
so what is in this directory is what built the newest release.
6+
7+
| | |
8+
|---|---|
9+
| `watcher.py` | Polls Mozilla's product-details for a new ESR tag and decides whether there is anything to build. Runs twice daily and never touches the source tree. |
10+
| `orchestrator.py` | The build pipeline. 23 phases, resumable from the last completed one. |
11+
| `advisory.py` | Resolves the MFSA and CVE list from Mozilla's published advisory data. Never guesses: if the sources disagree or cannot be reached, it halts. |
12+
| `render.py` | Builds the release notes and changelog entry. |
13+
| `publish.py` | Syncs this repo, creates the GitHub release, and applies the house formatting rules. |
14+
| `vt.py` | VirusTotal submission, verdict classification, and the repackaging ladder used when something is flagged. |
15+
| `notify.py` | Push notifications and the approval gates. |
16+
| `common.py` | Config, state, locking, and shared text rules. |
17+
| `setup_scheduler.ps1` | Registers the two Windows Task Scheduler tasks. |
18+
| `SCHEDULER.md` | Read this before the first build. Explains why the session must stay unlocked. |
19+
20+
## How a release happens
21+
22+
1. `watcher.py` sees a new ESR tag upstream and sends a notification.
23+
2. Approving it starts `orchestrator.py`, which rebases the patch stack, builds both
24+
variants with PGO and full LTO, smoke-tests each one, packages them, and submits all
25+
four artifacts to VirusTotal.
26+
3. It resolves the CVE list, drafts the release notes, and creates a draft release.
27+
4. Approving that publishes the release, updates this repo, and opens a discussion.
28+
29+
Two approvals, both answerable from a phone. Everything between them is unattended.
30+
31+
## What is not here
32+
33+
`.env` and `state.json` are deliberately excluded. `.env` holds the notification topics,
34+
which are in practice the password for the approval buttons, and `state.json` holds local
35+
build state. Anyone running these scripts needs to supply their own `.env` and adjust the
36+
paths at the top of `config.toml`.

Automation/SCHEDULER.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Task Scheduler setup
2+
3+
## Read this before the first real build
4+
5+
**The console session must stay unlocked for the entire build.** Sleep and hibernate
6+
must be disabled for the duration; screen-off (monitor sleep) is fine, that doesn't
7+
affect this.
8+
9+
**Why:** Session 0 and locked sessions drop hardware compositing. PGO training drives a
10+
real Firefox window through 88 sites for ~134 minutes to record which code paths
11+
actually run under WebRender and the GPU compositor. Without real compositing, that
12+
window either doesn't render normally or renders through a software fallback path, so
13+
the profile gets recorded, the build finishes, everything *looks* fine, and the
14+
resulting binary is measurably worse in exactly the paths PGO was supposed to optimize.
15+
There's no error, no warning, nothing in the log: the only symptom is a binary that's
16+
quietly worse than it should be. This is exactly why Invariant 2 (never Session 0) and
17+
the orchestrator task's "run only when logged on" setting exist.
18+
19+
In practice: don't lock your session, don't let Windows sleep, and don't RDP into this
20+
machine to run a build (RDP sessions used to force the same GPU-less compositing path
21+
as Session 0; if you ever need remote access during a build, confirm your Windows
22+
build's RDP client actually gives you hardware acceleration before trusting the result).
23+
24+
---
25+
26+
## What gets registered
27+
28+
Running `setup_scheduler.ps1` creates two Task Scheduler tasks:
29+
30+
| Task | Trigger | Runs whether logged on or not? |
31+
|---|---|---|
32+
| `ducksteps watcher` | Daily, 9:00 AM and 5:00 PM | Yes |
33+
| `ducksteps orchestrator` | None (manual start only) | No, requires an interactive logon |
34+
35+
**`ducksteps watcher`** polls Mozilla, decides if there's something new to build, and
36+
sends the Gate 1 notification if so. It runs in a few seconds and never touches the
37+
source tree beyond a read-only `git ls-remote`. Safe to run unattended, which is why it
38+
can run whether you're logged on or not.
39+
40+
**`ducksteps orchestrator`** runs the actual 10-hour build pipeline. It has **no
41+
automatic trigger on purpose** - start it yourself once you've approved a release from
42+
your phone:
43+
44+
- Task Scheduler UI: right-click "ducksteps orchestrator" -> Run
45+
- Or from an elevated or regular prompt: `schtasks /run /tn "ducksteps orchestrator"`
46+
47+
If a build gets interrupted (reboot, crash, you closed it), don't re-run the task as-is:
48+
open `start-shell.bat` yourself and run `python orchestrator.py --resume` directly. The
49+
registered task always does a fresh start, which correctly refuses to clobber an
50+
in-progress build and will just tell you to use `--resume` instead - resuming from a
51+
random restart isn't a case worth a second scheduled task for.
52+
53+
## Running the setup script
54+
55+
```powershell
56+
D:\ducksteps\automation\setup_scheduler.ps1
57+
```
58+
59+
**Run this from an elevated (Run as Administrator) PowerShell prompt.** The watcher
60+
task's logon type (S4U: runs whether logged on or not, without storing your Windows
61+
password anywhere) requires elevation to register. Without it, registration fails
62+
partway through with "Access is denied" - confirmed directly, not assumed, while
63+
building this.
64+
65+
Safe to re-run any time (e.g. after moving the automation directory, or changing your
66+
Python install path at the top of the script): it re-registers both tasks in place
67+
rather than erroring on "task already exists."
68+
69+
## Checking on things
70+
71+
```powershell
72+
Get-ScheduledTask -TaskName "ducksteps*"
73+
Get-ScheduledTaskInfo -TaskName "ducksteps watcher"
74+
Get-ScheduledTaskInfo -TaskName "ducksteps orchestrator"
75+
```
76+
77+
`logs/` in the automation directory has the real record of what each run actually did;
78+
Task Scheduler's own "Last Run Result" only tells you the process exit code.
79+
80+
## Removing everything
81+
82+
```powershell
83+
Unregister-ScheduledTask -TaskName "ducksteps watcher" -Confirm:$false
84+
Unregister-ScheduledTask -TaskName "ducksteps orchestrator" -Confirm:$false
85+
```

Automation/advisory.py

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
from __future__ import annotations
2+
3+
import re
4+
from dataclasses import dataclass, field
5+
from datetime import datetime, timezone
6+
7+
import requests
8+
import yaml
9+
10+
import common
11+
12+
GITHUB_API_BASE = "https://api.github.com/repos/mozilla/foundation-security-advisories"
13+
ADVISORIES_INDEX_URL = "https://www.mozilla.org/en-US/security/advisories/"
14+
15+
16+
class AdvisoryError(Exception):
17+
"""Neither the structured YAML source nor the mozilla.org fallback could give a
18+
definitive answer. Never guessed at - the caller must halt, per Phase 6's whole reason
19+
for existing: an LLM must never be left to invent a CVE list."""
20+
21+
22+
@dataclass
23+
class Advisory:
24+
mfsa_number: str | None
25+
mfsa_url: str | None
26+
announced_date: str | None
27+
cves: list = field(default_factory=list) # [{id, title, impact, url}]
28+
29+
@classmethod
30+
def empty(cls) -> "Advisory":
31+
return cls(mfsa_number=None, mfsa_url=None, announced_date=None, cves=[])
32+
33+
34+
def _fixed_in_candidates(version) -> list:
35+
"""The strings Mozilla's own `fixed_in` uses for this release, most specific first.
36+
37+
Two-component versions ("153.0") are the first release of a new ESR line, and upstream
38+
labels those differently: the ESR ships the same code as the matching rapid release on
39+
that day, so the advisory's fixed_in tends to say "Firefox 153" rather than
40+
"Firefox ESR 153.0". Later releases on the line ("153.1.0") go back to the ordinary
41+
"Firefox ESR x.y.z" form. Only the two-component case gets the plain-Firefox
42+
candidates - offering them for a point release would let an unrelated rapid-release
43+
advisory match.
44+
"""
45+
parts = version.split(".")
46+
candidates = [f"Firefox ESR {version}"]
47+
if len(parts) == 3:
48+
candidates.append(f"Firefox ESR {parts[0]}.{parts[1]}")
49+
else:
50+
candidates.append(f"Firefox {version}")
51+
candidates.append(f"Firefox {parts[0]}")
52+
return candidates
53+
54+
55+
def _matches_version(fixed_in_list, version) -> bool:
56+
candidates = set(_fixed_in_candidates(version))
57+
return any((entry or "").strip() in candidates for entry in (fixed_in_list or []))
58+
59+
60+
def _list_advisory_files(year, timeout=30):
61+
"""Returns [(filename, download_url), ...] for that year's mfsa*.yml files, or []
62+
if the year directory doesn't exist yet (e.g. checking next year too early)."""
63+
url = f"{GITHUB_API_BASE}/contents/announce/{year}"
64+
r = requests.get(url, timeout=timeout)
65+
if r.status_code == 404:
66+
return []
67+
r.raise_for_status()
68+
return [
69+
(entry["name"], entry["download_url"])
70+
for entry in r.json()
71+
if entry["name"].startswith("mfsa") and entry["name"].endswith(".yml")
72+
]
73+
74+
75+
def _fetch_yaml(download_url, timeout=30):
76+
r = requests.get(download_url, timeout=timeout)
77+
r.raise_for_status()
78+
return yaml.safe_load(r.text)
79+
80+
81+
def _non_windows_marker(details, markers):
82+
"""The marker that scopes this advisory to a platform we don't build for, or None.
83+
84+
Matches against title and description together, because upstream puts the scoping in
85+
whichever of the two it feels like on the day ("... in Firefox for Android" in a title,
86+
"Note: this bug only affects Android" in a description). Returns the matched marker
87+
rather than a bool so the caller can log WHY something was dropped.
88+
"""
89+
if not markers:
90+
return None
91+
haystack = " ".join([
92+
(details or {}).get("title", "") or "",
93+
(details or {}).get("description", "") or "",
94+
]).lower()
95+
for marker in markers:
96+
if marker.lower() in haystack:
97+
return marker
98+
return None
99+
100+
101+
def _advisory_from_yaml(filename, data, non_windows_markers=(), logger=None) -> Advisory:
102+
mfsa_number = filename.removeprefix("mfsa").removesuffix(".yml") # "mfsa2026-13.yml" -> "2026-13"
103+
cves = []
104+
dropped = []
105+
for cve_id, details in (data.get("advisories") or {}).items():
106+
marker = _non_windows_marker(details, non_windows_markers)
107+
if marker:
108+
dropped.append((cve_id, marker))
109+
continue
110+
cves.append({
111+
"id": cve_id,
112+
"title": common.strip_em_dashes((details or {}).get("title", "")),
113+
"impact": (details or {}).get("impact", "").lower(),
114+
"url": f"https://www.cve.org/CVERecord?id={cve_id}",
115+
})
116+
if dropped and logger:
117+
# Logged individually and by name: a reader of this log has to be able to
118+
# reconstruct exactly which upstream CVEs were left out of a shipped release.
119+
logger.info("advisory %s: keeping %d of %d CVEs, dropped %d as non-Windows",
120+
mfsa_number, len(cves), len(cves) + len(dropped), len(dropped))
121+
for cve_id, marker in dropped:
122+
logger.info(" dropped %s (matched %r)", cve_id, marker)
123+
return Advisory(
124+
mfsa_number=mfsa_number,
125+
mfsa_url=f"https://www.mozilla.org/en-US/security/advisories/mfsa{mfsa_number}/",
126+
announced_date=data.get("announced"),
127+
cves=cves,
128+
)
129+
130+
131+
def _candidate_years():
132+
now = datetime.now(timezone.utc)
133+
return [now.year, now.year - 1]
134+
135+
136+
def _try_yaml_search(version, non_windows_markers=(), logger=None):
137+
"""Returns (Advisory_or_None, error_or_None). None/None means the source was reachable
138+
and searched completely but genuinely has nothing for this version."""
139+
try:
140+
for year in _candidate_years():
141+
for name, download_url in _list_advisory_files(year):
142+
data = _fetch_yaml(download_url)
143+
if data and _matches_version(data.get("fixed_in", []), version):
144+
if logger:
145+
logger.info("found %s for %s via structured YAML", name, version)
146+
return _advisory_from_yaml(name, data, non_windows_markers, logger), None
147+
return None, None
148+
except (requests.RequestException, yaml.YAMLError) as exc:
149+
if logger:
150+
logger.warning("YAML advisory search failed: %s", exc)
151+
return None, exc
152+
153+
154+
def _try_scrape_search(version, logger=None):
155+
"""Fallback: confirms an MFSA exists and its number/URL, nothing more - the index page
156+
doesn't carry CVE-level detail, so this can never be used to build a full CVE list."""
157+
try:
158+
r = requests.get(ADVISORIES_INDEX_URL, timeout=30)
159+
r.raise_for_status()
160+
except requests.RequestException as exc:
161+
if logger:
162+
logger.warning("advisories index scrape failed: %s", exc)
163+
return None, exc
164+
165+
candidates = _fixed_in_candidates(version)
166+
for line in r.text.splitlines():
167+
if any(c in line for c in candidates):
168+
match = re.search(r"mfsa(\d{4}-\d+)", line, re.IGNORECASE)
169+
if match:
170+
mfsa_number = match.group(1)
171+
if logger:
172+
logger.info("found MFSA %s for %s via mozilla.org fallback scrape", mfsa_number, version)
173+
return {
174+
"mfsa_number": mfsa_number,
175+
"mfsa_url": f"https://www.mozilla.org/en-US/security/advisories/mfsa{mfsa_number}/",
176+
}, None
177+
return None, None
178+
179+
180+
def resolve_advisory(version, non_windows_markers=(), logger=None) -> Advisory:
181+
"""Resolves the MFSA/CVE data for `version` (e.g. "140.14.0"). Returns Advisory.empty()
182+
for the legitimate "no security content this release" case - only raises AdvisoryError
183+
when neither source could give a definitive answer, or when they disagree.
184+
185+
non_windows_markers drops CVEs upstream scopes to platforms ducksteps doesn't build
186+
for (see [advisory] in config.toml). It only ever filters the CVE list; it never
187+
affects whether an MFSA is considered found, so an advisory whose every CVE is
188+
Android-only still resolves and still reports its MFSA number rather than
189+
masquerading as "no security content".
190+
"""
191+
yaml_advisory, yaml_error = _try_yaml_search(version, non_windows_markers, logger=logger)
192+
if yaml_advisory is not None:
193+
return yaml_advisory
194+
195+
scrape_hit, scrape_error = _try_scrape_search(version, logger=logger)
196+
197+
if yaml_error is None and scrape_error is None:
198+
if scrape_hit is None:
199+
return Advisory.empty()
200+
raise AdvisoryError(
201+
f"mozilla.org shows {scrape_hit['mfsa_number']} for {version} but the structured "
202+
f"YAML repo doesn't - sources disagree, halting rather than guessing CVE details."
203+
)
204+
205+
if scrape_hit is not None:
206+
raise AdvisoryError(
207+
f"MFSA {scrape_hit['mfsa_number']} confirmed for {version} via the mozilla.org "
208+
f"fallback, but the structured YAML source errored ({yaml_error}) - cannot build "
209+
f"a reliable CVE list from a scrape alone."
210+
)
211+
212+
if yaml_error is not None and scrape_error is not None:
213+
raise AdvisoryError(
214+
f"neither advisory source could be reached for {version}: YAML={yaml_error}, scrape={scrape_error}"
215+
)
216+
217+
if logger:
218+
logger.warning(
219+
"one advisory source errored (yaml=%s, scrape=%s) but the other found nothing for "
220+
"%s and was fully reachable; proceeding as no security content",
221+
yaml_error, scrape_error, version,
222+
)
223+
return Advisory.empty()

0 commit comments

Comments
 (0)