Skip to content

Commit f5e0c06

Browse files
committed
ci: Tighten comments around the model regeneration workflow
1 parent 6df1be6 commit f5e0c06

4 files changed

Lines changed: 62 additions & 87 deletions

File tree

‎.github/workflows/on_schedule_regenerate_models.yaml‎

Lines changed: 31 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,17 @@
1-
# This workflow keeps the generated API models in sync with the published OpenAPI specification. Every night it
2-
# regenerates the models from the published specification and opens a pull request when the result differs from
3-
# master.
1+
# Keeps the generated API models in sync with the published OpenAPI specification: every night it regenerates them
2+
# and opens a pull request when the result differs from master.
43
#
54
# Two invariants make it safe to run unattended:
6-
# 1. Generation always happens on master, so the output is a function of the current spec and the current codegen
7-
# tooling - never of whatever an older auto-update branch happens to carry.
8-
# 2. The auto-update branch is rebuilt from master instead of appended to, so the pull request diff is always
9-
# "current spec vs current master" and can never resurrect a stale generated file.
5+
# 1. Generation always happens on master, so the output follows the current spec and the current codegen tooling,
6+
# never whatever an older auto-update branch carries.
7+
# 2. The auto-update branch is rebuilt from master instead of appended to, so the diff is always "current spec vs
8+
# current master" and can't resurrect a stale generated file.
109

1110
name: Regenerate models
1211

1312
on:
14-
# Runs when manually triggered from the GitHub UI.
1513
workflow_dispatch:
1614

17-
# Runs every day at 02:00 UTC.
1815
schedule:
1916
- cron: "0 2 * * *"
2017

@@ -29,8 +26,8 @@ permissions:
2926
env:
3027
PYTHON_VERSION: 3.14
3128
BRANCH_NAME: ci/regenerate-models
32-
# A valid Conventional Commits title, so the pull request is mergeable without anyone having to rename it.
33-
# Reviewers retitle it to `fix:`/`feat:` when the diff is user-facing - see the pull request body.
29+
# Valid Conventional Commits, so the pull request is mergeable as-is; reviewers retitle it to `fix:`/`feat:` when
30+
# the diff is user-facing.
3431
PR_TITLE: "chore: Regenerate models from the published OpenAPI spec"
3532
ASSIGNEE: vdusek
3633
LABEL: t-tooling
@@ -56,22 +53,18 @@ jobs:
5653
- name: Install dependencies
5754
run: uv run poe install-dev
5855

59-
# Read before regenerating overwrites it, so the pull request can say whether the specification moved at all.
56+
# Read before regeneration overwrites it, so the pull request can say whether the specification moved.
6057
- name: Read the recorded specification version
6158
id: previous-spec
6259
run: echo "version=$(uv run python -m scripts.openapi_spec recorded-version)" >> "$GITHUB_OUTPUT"
6360

64-
# Downloads the published specification, generates from it, and records its version in `pyproject.toml`.
61+
# Downloads the specification, generates from it, and records its version in `pyproject.toml`.
6562
- name: Regenerate models
6663
run: uv run poe generate-models
6764

68-
# Gate on the generated models rather than on the specification version: that version is a build timestamp
69-
# that moves on rebuilds regardless of client impact, and doesn't reliably move at all when the
70-
# specification content does (see the pull request body step below). Because nothing is committed unless the
71-
# models moved, the recorded version follows the models rather than the published specification.
72-
#
73-
# Compared against HEAD rather than the index, so the result doesn't depend on nothing having been staged
74-
# earlier in the job.
65+
# Gate on the generated models, not on the specification version: that stamp moves on rebuilds regardless of
66+
# client impact, and doesn't reliably move when the content does (see the pull request body step below).
67+
# Compared against HEAD rather than the index, so nothing staged earlier in the job can hide a change.
7568
- name: Check for model changes
7669
id: changes
7770
run: |
@@ -83,18 +76,17 @@ jobs:
8376
echo "has-changes=true" >> "$GITHUB_OUTPUT"
8477
fi
8578
86-
# A previous run may have already opened a pull request with exactly these models. Leave it alone rather than
87-
# replacing it with an identical one, so an open pull request doesn't churn while the specification is stable.
88-
# Only the generated files are compared: the branch is also behind on everything else master merged since,
89-
# which says nothing about whether the models up for review are still the right ones.
79+
# A previous run may already have these exact models up for review; leave it alone instead of churning an open
80+
# pull request. Only the generated files are compared - the branch being behind master says nothing about
81+
# whether the models on it are still the right ones.
9082
- name: Check whether the models are already up for review
9183
id: review
9284
if: steps.changes.outputs.has-changes == 'true'
9385
env:
9486
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
9587
run: |
96-
# `--exit-code` reports 2 for "no such branch", so anything else is a real failure and must not be read
97-
# as an absent branch - that would retire a current pull request over a transient network error.
88+
# `--exit-code` reports 2 for "no such branch"; any other failure must not read as an absent branch, or a
89+
# transient network error would retire a current pull request.
9890
git ls-remote --exit-code --heads origin "$BRANCH_NAME" >/dev/null 2>&1 && status=0 || status=$?
9991
if [[ "$status" == "2" ]]; then
10092
echo "No auto-update branch exists yet."
@@ -105,13 +97,10 @@ jobs:
10597
exit 1
10698
fi
10799
108-
# The branch only counts as "already up for review" while a pull request is actually open on it. A
109-
# leftover branch - an earlier run that failed before opening one, or a review that closed the pull
110-
# request without deleting the branch - must be rebuilt, otherwise its contents would look proposed
111-
# and silently suppress every future regeneration.
112-
#
113-
# An empty result only counts once the query itself succeeded: `gh` exits non-zero on an API error, and
114-
# treating that as "no open pull request" would replace a pull request someone may be reviewing.
100+
# The branch only counts as proposed while a pull request is open on it: a leftover branch - a run that
101+
# failed before opening one, or a review that closed it without deleting the branch - must be rebuilt, or
102+
# its contents would look proposed and silently suppress every future regeneration. An empty result counts
103+
# only once the query itself succeeded, otherwise an API error would replace a live pull request.
115104
if ! PR_NUMBER=$(gh pr list --head "$BRANCH_NAME" --base master --state open --json number --jq '.[0].number // empty'); then
116105
echo "Failed to list open pull requests for $BRANCH_NAME." >&2
117106
exit 1
@@ -150,8 +139,8 @@ jobs:
150139
echo "Deleted leftover branch $BRANCH_NAME (no open pull request)."
151140
fi
152141
153-
# Creates the branch at the current master commit and lands the regenerated files on it as a single signed
154-
# ("Verified") commit via GitHub's createCommitOnBranch GraphQL mutation.
142+
# Creates the branch at the current master commit and lands the regenerated files as one signed ("Verified")
143+
# commit, via GitHub's createCommitOnBranch mutation.
155144
- name: Commit the regenerated models
156145
id: commit
157146
if: steps.changes.outputs.has-changes == 'true' && steps.review.outputs.is-new == 'true'
@@ -176,18 +165,17 @@ jobs:
176165
run: |
177166
SPEC_VERSION=$(uv run python -m scripts.openapi_spec recorded-version)
178167
179-
# A moved version stamp proves the specification changed. An unchanged one proves nothing: apify-docs
180-
# bumps `components/version.yaml` in a follow-up commit marked `[skip ci]`, so a deploy publishes new
181-
# specification content still carrying the previous stamp. Say which case this is rather than let the
182-
# reviewer assume the stamp is a content identity.
168+
# A moved stamp proves the specification changed; an unchanged one proves nothing, because apify-docs bumps
169+
# `components/version.yaml` in a follow-up `[skip ci]` commit and a deploy can publish new content under
170+
# the old stamp. Say which case this is instead of letting the reviewer read it as a content identity.
183171
if [[ "$SPEC_VERSION" == "$PREVIOUS_SPEC_VERSION" ]]; then
184-
SPEC_LINE="- Specification version: \`${SPEC_VERSION}\` - **unchanged**. The published stamp lags its content by a deploy, so this diff is either an unstamped specification change or a codegen/formatter change - the diff itself is the only way to tell."
172+
SPEC_LINE="- Specification version: \`${SPEC_VERSION}\` - **unchanged**, which doesn't rule out a specification change: the published stamp can lag its content by a deploy. Only the diff tells."
185173
else
186174
SPEC_LINE="- Specification version: \`${PREVIOUS_SPEC_VERSION}\` -> \`${SPEC_VERSION}\`."
187175
fi
188176
189177
BODY=$(printf '%s\n' \
190-
"- Regenerates the Pydantic models, TypedDicts, and literal aliases from the [published OpenAPI specification](https://docs.apify.com/api/openapi.json), and records the version they were generated from in \`pyproject.toml\`." \
178+
"- Regenerates the Pydantic models, TypedDicts, and literal aliases from the [published OpenAPI specification](https://docs.apify.com/api/openapi.json), and records its version in \`pyproject.toml\`." \
191179
"${SPEC_LINE}" \
192180
"" \
193181
"> [!IMPORTANT]" \
@@ -203,9 +191,8 @@ jobs:
203191
--assignee "$ASSIGNEE" \
204192
--label "$LABEL"
205193
206-
# Alert the team when a scheduled run fails. Without this a broken sync stops model regeneration silently:
207-
# for a scheduled workflow GitHub only notifies whoever last touched the cron, and this workflow is not part
208-
# of the checks that releases wait on. Skipped on manual dispatch so ad-hoc triggers don't spam the channel.
194+
# Without this a broken sync stops regeneration silently: GitHub only notifies whoever last touched the cron, and
195+
# no release waits on this workflow. Skipped on manual dispatch so ad-hoc triggers don't spam the channel.
209196
notify_on_failure:
210197
name: Notify Slack on failure
211198
needs: regenerate-models

‎.rules.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -82,11 +82,11 @@ Each input-side TypedDict ships in two casings: snake_case (`RequestDict`) and c
8282
- To regenerate locally:
8383
- From the live published spec: `uv run poe generate-models`
8484
- From a local spec file: `uv run poe generate-models-from-file path/to/openapi.json`
85-
- `scripts/openapi_spec.py` downloads the spec once into git-ignored `tmp/openapi.json` (both codegen passes read that copy, so a spec redeployed mid-run can't leave the generated files describing two different inputs), and afterwards records the spec's `info.version` in `pyproject.toml` under `[tool.apify.openapi-spec]`
86-
- That version is a **coarse marker, not a content identity**: the spec is served latest-only, so it can't fetch that exact document back, and apify-docs bumps `components/version.yaml` in a follow-up `[skip ci]` commit — so a deploy publishes new spec content still carrying the previous stamp. A moved value proves the spec changed; an unchanged value proves nothing
87-
- The recording is unconditional; the nightly workflow is what keeps it honest, by committing nothing unless the models changed. So the recorded version lands together with the generated files and stays behind the published spec while spec changes leave the models unchanged. A local `generate-models` that only moves that line is the same case — drop it instead of committing it alone. `generate-models-from-file` deliberately leaves the line alone, so models generated from a candidate spec must not be committed
88-
- In CI, the `Regenerate models` workflow (`on_schedule_regenerate_models.yaml`) runs nightly at 02:00 UTC: it regenerates from the published spec and opens a PR when the models change. It always generates on master and rebuilds its `ci/regenerate-models` branch from master instead of appending, so the PR diff is always "current spec vs current master". The PR opens with a mergeable `chore:` title — retitle it to `fix:`/`feat:` when the diff is user-facing, so it lands in the changelog and triggers a release
89-
- The gate is the **generated models**, not the spec version: the version is a build timestamp that moves on spec rebuilds regardless of client impact, and (per above) doesn't reliably move at all — so it can't decide whether to open a PR
85+
- `scripts/openapi_spec.py` downloads the spec once into git-ignored `tmp/openapi.json` — both codegen passes read that copy, so a spec redeployed mid-run can't leave the generated files describing two different inputs — then records its `info.version` in `pyproject.toml` under `[tool.apify.openapi-spec]`
86+
- That version is a **coarse marker, not a content identity**: the spec is served latest-only, and apify-docs bumps `components/version.yaml` in a follow-up `[skip ci]` commit, so a deploy can publish new content under the old stamp. A moved value proves the spec changed; an unchanged value proves nothing
87+
- Recording is unconditional; the nightly workflow keeps it honest by committing nothing unless the models changed. A local `generate-models` that only moves that line should be dropped, not committed alone. `generate-models-from-file` leaves the line alone, so models from a candidate spec must not be committed
88+
- In CI, the `Regenerate models` workflow (`on_schedule_regenerate_models.yaml`) runs nightly at 02:00 UTC and opens a PR when the models change. It always generates on master and rebuilds its `ci/regenerate-models` branch from master instead of appending, so the PR diff is always "current spec vs current master". Retitle the `chore:` PR to `fix:`/`feat:` when the diff is user-facing, so it lands in the changelog and triggers a release
89+
- The gate is the **generated models**, not the spec version — which moves on rebuilds regardless of client impact, and doesn't reliably move when the content does
9090
- Manual regeneration is also possible from the GitHub Actions UI (`Regenerate models` workflow)
9191

9292
## Code Conventions

‎pyproject.toml‎

Lines changed: 10 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -247,17 +247,13 @@ disable_timestamp = true
247247
keep_model_order = true
248248

249249
# Which published OpenAPI specification the committed `_models.py`, `_typeddicts.py`, and `_literals.py` were
250-
# generated from. This is a coarse marker, not a content identity: the specification is served latest-only, so the
251-
# version cannot fetch that exact document back, and `apify/apify-docs` bumps
252-
# `apify-api/openapi/components/version.yaml` in a follow-up commit marked `[skip ci]` - a deploy therefore
253-
# publishes new specification content that still carries the previous stamp. A moved value proves the
254-
# specification changed; an unchanged one proves nothing.
250+
# generated from. A coarse marker, not a content identity: the specification is served latest-only, so the version
251+
# can't fetch that document back, and apify-docs bumps `components/version.yaml` in a follow-up `[skip ci]` commit,
252+
# so a deploy can publish new content under the old stamp.
255253
#
256-
# `poe generate-models` records whatever it downloaded, unconditionally. What keeps the recorded version honest is
257-
# the nightly workflow committing nothing unless the models changed: the value therefore lands together with the
258-
# generated files, and stays behind the published specification while specification changes leave the models
259-
# unchanged. A local regeneration that only moves this line is that same case - drop the change rather than
260-
# committing it on its own.
254+
# `poe generate-models` records whatever it downloaded, unconditionally; what keeps the value honest is the nightly
255+
# workflow committing nothing unless the models changed. A local regeneration that only moves this line is that
256+
# same case - drop it rather than committing it alone.
261257
[tool.apify.openapi-spec]
262258
version = "v2-2026-07-28T083939Z"
263259

@@ -308,9 +304,8 @@ cwd = "website"
308304
shell = "./build_api_reference.sh && pnpm install && uv run pnpm start"
309305
cwd = "website"
310306

311-
# Both codegen passes read one downloaded copy of the specification, so a specification redeployed mid-run cannot
312-
# leave the generated files describing two different inputs. The version is recorded last, once generation
313-
# succeeded, so it always names the specification the generated files follow from.
307+
# Both passes read one downloaded copy, so a specification redeployed mid-run can't leave the generated files
308+
# describing two different inputs. The version is recorded last, once generation succeeded.
314309
#
315310
# The `--alias-generator to_camel` flag lives on the `_models.py` command (not in `[tool.datamodel-codegen]`)
316311
# because datamodel-codegen only allows it for `pydantic_v2.BaseModel` output and would reject the TypedDict run.
@@ -327,9 +322,8 @@ uv run python -m scripts.openapi_spec fetch \
327322
&& uv run python -m scripts.openapi_spec record-version
328323
"""
329324

330-
# A candidate specification that isn't published yet, so the recorded version is deliberately left alone - it
331-
# names the published specification. Committing models generated this way would make it name a document they did
332-
# not come from, hence the warning.
325+
# For a candidate specification that isn't published yet. The recorded version deliberately stays put - it names
326+
# the published specification - so models generated this way must not be committed, hence the warning.
333327
[tool.poe.tasks.generate-models-from-file]
334328
shell = """
335329
uv run datamodel-codegen --input $input_file \

0 commit comments

Comments
 (0)