Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions .github/actions/webperf-snippets/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,20 @@ inputs:
description: Workflow to run (core-web-vitals, loading, audit, interaction)
default: core-web-vitals
budget-lcp:
description: LCP budget in ms. The run fails if it is exceeded. Empty disables the check.
default: "2500"
description: LCP budget in ms. The run fails if it is exceeded or LCP was not measured. "auto" is 2500 with the core-web-vitals workflow and no budget with the others, which do not measure LCP. Empty disables the check.
default: auto
budget-fcp:
description: FCP budget in ms, for the loading workflow. The run fails if it is exceeded or FCP was not measured. Empty disables the check.
default: ""
budget-ttfb:
description: TTFB budget in ms, for the loading and audit workflows. The run fails if it is exceeded or TTFB was not measured. Empty disables the check.
default: ""
budget-cls:
description: CLS budget. The run fails if it is exceeded. Empty disables the check.
default: "0.1"
description: CLS budget. The run fails if it is exceeded or CLS was not measured. "auto" is 0.1 with the core-web-vitals workflow and no budget with the others, which do not measure CLS. Empty disables the check.
default: auto
budget-inp:
description: INP budget in ms, for the interaction workflow with clicks or key presses. The run fails if it is exceeded or INP was not measured. Empty disables the check.
default: ""
fail-on-budget:
description: Fail the step when a budget is exceeded or a snippet errors. Set to "false" to read the exit-code output and decide yourself, for example to post the markdown first.
default: "true"
Expand Down Expand Up @@ -53,6 +62,9 @@ runs:
WEBPERF_URL: ${{ inputs.url }}
WEBPERF_WORKFLOW: ${{ inputs.workflow }}
WEBPERF_BUDGET_LCP: ${{ inputs.budget-lcp }}
WEBPERF_BUDGET_FCP: ${{ inputs.budget-fcp }}
WEBPERF_BUDGET_TTFB: ${{ inputs.budget-ttfb }}
WEBPERF_BUDGET_CLS: ${{ inputs.budget-cls }}
WEBPERF_BUDGET_INP: ${{ inputs.budget-inp }}
WEBPERF_FAIL_ON_BUDGET: ${{ inputs.fail-on-budget }}
run: bash "${{ github.action_path }}/run.sh"
22 changes: 19 additions & 3 deletions .github/actions/webperf-snippets/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,25 @@
set -uo pipefail

read -r -a cmd <<< "${WEBPERF_CMD:?WEBPERF_CMD is required}"
args=("${WEBPERF_URL:?WEBPERF_URL is required}" --workflow "${WEBPERF_WORKFLOW:-core-web-vitals}" --markdown)
[ -n "${WEBPERF_BUDGET_LCP:-}" ] && args+=(--budget-lcp "$WEBPERF_BUDGET_LCP")
[ -n "${WEBPERF_BUDGET_CLS:-}" ] && args+=(--budget-cls "$WEBPERF_BUDGET_CLS")
workflow="${WEBPERF_WORKFLOW:-core-web-vitals}"
args=("${WEBPERF_URL:?WEBPERF_URL is required}" --workflow "$workflow" --markdown)

# Adds a budget flag when its input has a value. "auto", the default of LCP and CLS, means the default
# value with the core-web-vitals workflow, the one that measures them, and no budget with the others:
# the CLI rejects a budget of a metric the workflow does not measure.
budget() {
local option=$1 value=$2 default=${3:-}
if [ "$value" = "auto" ]; then
[ "$workflow" = "core-web-vitals" ] && value=$default || value=""
fi
[ -n "$value" ] && args+=("$option" "$value")
return 0
}
budget --budget-lcp "${WEBPERF_BUDGET_LCP:-}" 2500
budget --budget-fcp "${WEBPERF_BUDGET_FCP:-}"
budget --budget-ttfb "${WEBPERF_BUDGET_TTFB:-}"
budget --budget-cls "${WEBPERF_BUDGET_CLS:-}" 0.1
budget --budget-inp "${WEBPERF_BUDGET_INP:-}"

markdown=$("${cmd[@]}" "${args[@]}")
code=$?
Expand Down
5 changes: 5 additions & 0 deletions cli/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,12 @@ Format based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The CL

## [Unreleased]

### Added
- `--budget-fcp`, `--budget-ttfb` and `--budget-inp`, next to `--budget-lcp` and `--budget-cls`: a budget for each metric the CLI measures. The interaction workflow measures INP, in a step `INP` that needs a click or a key press like `interactions` and `input-latency`. The GitHub Action takes `budget-fcp`, `budget-ttfb` and `budget-inp`.

### Changed
- A budget never passes unchecked. One whose metric the run does not measure, such as `--budget-lcp` with `--workflow audit`, exits with `2` before the page loads, naming the workflow or snippet that measures it; one whose metric was not measured, such as INP without interactions or an LCP that failed, exits with `1`. A value that is not a number of 0 or more exits with `2`; it passed before. Each failed budget goes to stderr also with `--json`.
- The `budget-lcp` and `budget-cls` inputs of the GitHub Action default to `auto`: 2500 and 0.1 with the `core-web-vitals` workflow, and no budget with the others, which do not measure LCP or CLS. An empty value still disables the check.
- A tracking snippet with nothing recorded, such as INP without interactions, is reported as `not measured` with its message, and no longer counts as a failed snippet: the command exits with `0` for it. After a click or a key press the CLI reads it again while it is still tracking, as it did while it returned an error.

---
Expand Down
16 changes: 12 additions & 4 deletions cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,8 @@ CI gating:
npx webperf-snippets https://example.com --budget-lcp 2500 --budget-cls 0.1
```

Each budget compares the value measured, or the median with `--runs`, and fails when the value is above it. `--budget-lcp` and `--budget-cls` need the `core-web-vitals` workflow, `--budget-fcp` the `loading` workflow, `--budget-ttfb` the `loading` or `audit` workflow, and `--budget-inp` the `interaction` workflow with a click or a key press; `--snippet` with the snippet of the metric works too. A budget of a metric that the run does not measure exits with code `2` before the page loads, and one whose metric was not measured, such as INP without interactions, exits with code `1`: a budget never passes unchecked. Each failed budget is written to stderr, also with `--json`.

### Options

| Option | Description |
Expand All @@ -325,8 +327,11 @@ npx webperf-snippets https://example.com --budget-lcp 2500 --budget-cls 0.1
| `--crux-key <key>` | Add CrUX field data (p75) next to the measured values. Prefer `CRUX_API_KEY`. |
| `--report-to <url>` | POST the results to this `https` URL after the run. No external calls without it. |
| `--api-key <key>` | Sent as a Bearer token with `--report-to`. Prefer `PERF_REVIEWS_API_KEY`. |
| `--budget-lcp <ms>` | Exit `1` if LCP exceeds this value. |
| `--budget-cls <score>` | Exit `1` if CLS exceeds this value. |
| `--budget-lcp <ms>` | Exit `1` if LCP exceeds this value or was not measured. |
| `--budget-fcp <ms>` | Exit `1` if FCP exceeds this value or was not measured. |
| `--budget-ttfb <ms>` | Exit `1` if TTFB exceeds this value or was not measured. |
| `--budget-cls <score>` | Exit `1` if CLS exceeds this value or was not measured. |
| `--budget-inp <ms>` | Exit `1` if INP exceeds this value or was not measured. |
| `--verbose` | Show all items, including passing checks. |
| `--headed` | Show the browser window (debug). |
| `-h, --help` | Show help. |
Expand Down Expand Up @@ -453,8 +458,11 @@ jobs:
| ---------------- | ------------------- | -------------------------------------------------------------------------------------------- |
| `url` | required | URL to measure. |
| `workflow` | `core-web-vitals` | Workflow to run: `core-web-vitals`, `loading`, `audit` or `interaction`. |
| `budget-lcp` | `2500` | LCP budget in ms. An empty value disables the check. |
| `budget-cls` | `0.1` | CLS budget. An empty value disables the check. |
| `budget-lcp` | `auto` | LCP budget in ms. `auto` is `2500` with `core-web-vitals` and no budget with the other workflows. An empty value disables the check. |
| `budget-fcp` | | FCP budget in ms, for `loading`. |
| `budget-ttfb` | | TTFB budget in ms, for `loading` or `audit`. |
| `budget-cls` | `auto` | CLS budget. `auto` is `0.1` with `core-web-vitals` and no budget with the other workflows. An empty value disables the check. |
| `budget-inp` | | INP budget in ms, for `interaction` with a click or a key press. |
| `fail-on-budget` | `true` | Fail the step when a budget is exceeded or a snippet errors. Use `"false"` to read `exit-code` and decide yourself. |
| `cli-version` | `latest` | Version of the `webperf-snippets` npm package to run. |

Expand Down
70 changes: 33 additions & 37 deletions cli/src/bin.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import { parseOwnDomains } from "./own-domains.js";
import { MEDIAN_METRICS, summarizeRuns } from "./runs.js";
import { loadMatrix } from "./matrix.js";
import { blockingErrors } from "./blocking.js";
import { MEASURED_BY, checkBudgets, parseBudgets, unmeasuredMetrics, violationsOf } from "./budgets.js";
import { measurementConditions } from "./conditions.js";
import { loadBaseline, parseThreshold, thresholdsWith, compareRun, compareMatrix } from "./baseline.js";
import { describeRegressions, describeMatrixRegressions, matrixComparisonView } from "./reporters/baseline-view.js";
Expand Down Expand Up @@ -49,8 +50,11 @@ Options:
--wait <ms> Post-load wait before evaluating (default: 3000)
--cpu <factor> Slow the CPU down by this factor, such as 4 (no throttling by default)
--network <preset> Throttle the network as DevTools does: fast-4g, slow-4g, 3g
--budget-lcp <ms> Exit 1 if LCP exceeds this value
--budget-cls <score> Exit 1 if CLS exceeds this value
--budget-lcp <ms> Exit 1 if LCP exceeds this value, or was not measured
--budget-fcp <ms> Exit 1 if FCP exceeds this value, or was not measured
--budget-ttfb <ms> Exit 1 if TTFB exceeds this value, or was not measured
--budget-cls <score> Exit 1 if CLS exceeds this value, or was not measured
--budget-inp <ms> Exit 1 if INP exceeds this value, or was not measured (interaction workflow)
--interact-script <path> JSON file with interactions to run after the load; the
Interaction snippets are installed first and read last
Actions: scroll, click, hover, type, wait
Expand Down Expand Up @@ -124,30 +128,13 @@ function parseRunCount(raw) {
return /^[1-9]$/.test(raw.trim()) ? Number(raw) : null;
}

function checkBudgets({ results, runs }, values) {
const violations = [];
const lcpBudget = values["budget-lcp"] ? Number(values["budget-lcp"]) : null;
const clsBudget = values["budget-cls"] ? Number(values["budget-cls"]) : null;

// With several runs the budgets compare the median of each metric, not one sample
if (runs) {
const lcp = runs.summary.LCP?.median;
const cls = runs.summary.CLS?.median;
if (lcp != null && lcpBudget != null && lcp > lcpBudget) violations.push(`LCP ${lcp}ms exceeds budget ${lcpBudget}ms`);
if (cls != null && clsBudget != null && cls > clsBudget) violations.push(`CLS ${cls} exceeds budget ${clsBudget}`);
return violations;
}

for (const r of results) {
if (r.status !== "ok") continue;
if (r.metric === "LCP" && lcpBudget != null && r.value > lcpBudget) {
violations.push(`LCP ${r.value}ms exceeds budget ${lcpBudget}ms`);
}
if (r.metric === "CLS" && clsBudget != null && r.value > clsBudget) {
violations.push(`CLS ${r.value} exceeds budget ${clsBudget}`);
}
}
return violations;
// What to run instead, when a budget names a metric the run does not measure
function budgetUsage(metric, run) {
const workflows = Object.values(WORKFLOWS)
.filter((w) => w.steps.some((step) => MEASURED_BY[metric].includes(step.path)))
.map((w) => w.id);
const instead = [...workflows.map((id) => `--workflow ${id}`), `--snippet ${MEASURED_BY[metric][0].split("/").pop()}`].join(" or ");
return `--budget-${metric.toLowerCase()} needs a run that measures ${metric}, and ${run} does not. Use ${instead}`;
}

async function main() {
Expand All @@ -168,7 +155,10 @@ async function main() {
markdown: { type: "boolean" },
wait: { type: "string" },
"budget-lcp": { type: "string" },
"budget-fcp": { type: "string" },
"budget-ttfb": { type: "string" },
"budget-cls": { type: "string" },
"budget-inp": { type: "string" },
viewport: { type: "string" },
cpu: { type: "string" },
network: { type: "string" },
Expand Down Expand Up @@ -275,10 +265,20 @@ async function main() {
if (values["fail-on-regression"]) fail("--fail-on-regression needs --baseline");
}

let budgets;
try {
budgets = parseBudgets(values);
} catch (err) {
fail(err.message);
}

// What each page load runs, a snippet or a workflow, under the conditions of the load
const items = values.snippet ? buildSnippetItem(values) : null;
const workflow = values.snippet ? null : WORKFLOWS[values.workflow ?? "core-web-vitals"];
if (!values.snippet && !workflow) fail(`Unknown workflow: ${values.workflow}`);
// A budget of a metric that no snippet of the run measures could never pass: a usage error
const [unmeasured] = unmeasuredMetrics(budgets, items ? [items[0].path] : workflow.steps.map((step) => step.path));
if (unmeasured) fail(budgetUsage(unmeasured, items ? `--snippet ${values.snippet}` : `the ${workflow.id} workflow`));
const measureWith = (conditions) => {
const options = { waitMs, headless: !values.headed, interactScript, ...conditions };
return items ? runSnippets({ ...options, items }) : runMeasurement({ ...options, workflow, rules: RULES });
Expand All @@ -294,7 +294,7 @@ async function main() {
} catch (err) {
fail(err.message);
}
await runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds });
await runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds, budgets });
return;
}

Expand Down Expand Up @@ -349,12 +349,10 @@ async function main() {
}
}

// Exit codes.
const violations = checkBudgets(payload, values);
// Exit codes. stderr names the budgets that failed, also with --json, whose report is on stdout
const violations = violationsOf(checkBudgets(payload, budgets));
if (violations.length > 0) {
if (!values.json) {
for (const v of violations) process.stderr.write(`Budget violation: ${v}\n`);
}
for (const v of violations) process.stderr.write(`Budget violation: ${v}\n`);
process.exit(1);
}
// Without --fail-on-regression the comparison only informs, and the exit code is the one of today
Expand Down Expand Up @@ -385,7 +383,7 @@ async function measureRuns(measure, count, medianBy, progress) {

// Every combination of the matrix, in order, each one `matrix.runs` times. Each combination reports
// its median run, and the exit code follows the same rules as one page, for every combination.
async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds }) {
async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, reportUrl, conditionsFor, baseline, thresholds, budgets }) {
for (const warning of matrix.warnings) process.stderr.write(`Warning: ${warning}\n`);
const viewportOf = new Map(matrix.combinations.map((c) => [c.id, c.viewport]));

Expand Down Expand Up @@ -464,10 +462,8 @@ async function runMatrix(matrixPath, matrix, measureWith, { values, waitMs, repo
}
}

const violations = combinations.filter((c) => !c.error).flatMap((c) => checkBudgets(c, values).map((v) => `${v} (${c.id})`));
if (!values.json) {
for (const v of violations) process.stderr.write(`Budget violation: ${v}\n`);
}
const violations = combinations.filter((c) => !c.error).flatMap((c) => violationsOf(checkBudgets(c, budgets)).map((v) => `${v} (${c.id})`));
for (const v of violations) process.stderr.write(`Budget violation: ${v}\n`);
const failed = combinations.some((c) => c.error || blockingErrors(c.results).length > 0);
// Without --fail-on-regression the comparison only informs
const regressions = values["fail-on-regression"] ? describeMatrixRegressions(combinations) : [];
Expand Down
Loading
Loading