feat: KEEP-1042 age out workflow execution data on a schedule

[OleksandrUA]

What and why

Nothing in the database or the app deleted a workflow execution row on an age basis.
workflow_execution_logs reached 62 GB / 23.1M rows on prod and caused two failures in two
days: the RDS volume alarm on 2026-09-01, and 87 minutes of 99.7% database CPU on
2026-09-02, when the analytics queries de-TOASTed jsonb out of it row by row, starved the
dispatchers, and stopped workflows from starting.

The product has sold retention since lib/billing/plans.ts was written - logRetentionDays
is 7 free, 30 pro, 90 business, 365 enterprise, and the trial upsell modal advertises
"30-day log retention" - but that value had no server-side reader anywhere. This PR makes it
real.

What it does

An hourly CronJob calls GET /api/internal/retention, authorized by the same internal-service
HMAC every other scheduled route uses. Five passes, child rows before parent rows, because
every foreign key into workflow_executions is ON DELETE NO ACTION and nothing cascades:

pass	what it removes
logs_floor	step logs past the longest window in use, every org, no join
logs_plan_window	step logs past the window their org's plan sells
output_raw	output_raw nulled once a run can no longer resume
logs_soft_deleted	step logs a user purged from the UI, after a grace period
executions_flat_window	run rows past a flat 400-day window, behind its own switch

output_raw is where the space comes from. It is the unredacted twin of output and costs
about the same on disk, so nulling it halves the payload of every aged row without deleting
anything a user reads. The redacted output the UI shows stays for the full plan window.

How each pass finds its lower bound

None of them uses a fixed lookback. An earlier revision did, and it meant a run only ever saw
rows that had crossed their boundary in the last few days: on prod that left 1.96M step-log
rows and 19M output_raw payloads that nothing would ever reach. The two passes that need a
lower bound take it from real progress instead - a per-organization watermark for the
plan-window pass, a self-pruning partial index for the output_raw pass - so the backlog
drains on its own and a drained table costs nothing to re-check.

The watermark stops at the oldest run the pass had to skip, not at the cutoff. The drain
query excludes runs that can still resume, so an empty page does not mean an empty range, and
advancing past those rows put them below an inclusive lower bound for good. The backstop pass
then records its own cutoff for every organization, which bounds a permanently stuck run and
gives the organizations it serves alone a watermark they would otherwise never have.

The one place the plan window does not apply

logRetentionDays is a log promise, so it drives the step logs only. workflow_executions
gets one flat, long window instead, behind EXECUTION_RETENTION_EXECUTIONS_ENABLED, which
ships off everywhere.

Every billing count reads that table by started_at, filters billable = true, joins
workflows for the org, and never filters deleted_at. countMonthlyExecutions counts from
the start of the UTC month, and the invoices page recounts every past period on every load
with no floor. No durable record survives a period without overage, free plans never get one,
and the provider holds no recoverable figure. So deleting a run row rewrites what a customer
was billed. That pass stays off until a per-period usage record exists (KEEP-1338). The first
prod row crosses 400 days on 2027-02-10.

Readers that would have gone wrong

Three pre-existing bugs that retention turns into wrong numbers, plus one the cutoff itself
introduced:

• The analytics gas filter refiled a wallet-paid run as free once its steps were gone. It
  reads workflow_executions.gas_used_wei and gas_credit_usage now, both of which the job
  never touches.
• The network filter and facet counts dropped a run entirely once its logs went. They read
  transaction_hashes on the run row as well, which is written at finalize and never purged.
• The monthly digest counted sponsored transactions off output_raw and would have reported
  about a quarter of them. It counts off output, which carries the same value - verified as
  0 rows of 1,717,501 on staging where output_raw is set and output is not.
• The runs table said "these have been removed" for any run past the plan window, whatever
  the job was doing. The cutoff comes from the purge watermark now, is null while the job is
  disabled or dry, and only ever explains a cell that is already empty.

The analytics range also gets a floor and both ends get validated: customStart was
unbounded and customEnd returned "Invalid time value" for anything unparseable.

Configuration

Eleven env vars, all documented in .env.example. The job is off until
EXECUTION_RETENTION_ENABLED is true, and EXECUTION_RETENTION_DRY_RUN reports what each
pass would touch while deleting nothing. An empty, unparseable or non-positive value falls
back to the built-in default rather than to 0, because a 0 window would mean "delete
everything". EXECUTION_RETENTION_DAYS has a hard 400-day floor and can only lengthen.

Staging ships enabled-but-dry. Prod ships off.

Metrics

keeperhub_execution_retention_rows_purged_total{pass},
keeperhub_execution_retention_runs_total{result},
keeperhub_execution_retention_last_success_timestamp_seconds and the per-window
keeperhub_execution_retention_window_organizations{retention_days} on the API registry;
keeperhub_execution_log_oldest_age_seconds and keeperhub_execution_table_bytes{table}
from the db-metrics collector.

The last-success gauge is the health signal, not the oldest-row age: enterprise organizations
keep a year of logs and therefore pin min(started_at), so that gauge would not move at all
if only the short-window passes broke. Alert rules are in techops-services/infrastructure.

Migrations 0153 and 0154 - 0153 needs db-prep

Three index lookups the job depends on, behind -- @requires-db-prep, plus the watermark
table:

• idx_workflow_executions_started_at - no existing index leads with an unfiltered
  started_at, so the flat pass would sequentially scan on every run.
• idx_exec_logs_deleted_at, partial on deleted_at IS NOT NULL. Holds only the rows waiting
  out their grace period and empties itself as they go.
• idx_exec_logs_output_raw_pending, partial on output_raw IS NOT NULL. This is what makes
  the unbounded output_raw scan viable; it self-prunes as the backlog drains.

All three are built CONCURRENTLY on staging and verified valid; the migration body is
IF NOT EXISTS so the deploy no-ops. The exact statements are in the migration header.

Testing

• tests/db/execution-retention-purge.db.test.ts runs the real purge against a real Postgres
  in a new ungated test-db CI job. Six organizations across every window, the config and the
  clock injected, assertions against rows rather than against the report. It covers the cases
  that only a two-run test can see: a run that is non-terminal when the pass sweeps past it
  and finalizes afterwards, a dry run over more rows than one batch, a batch smaller than the
  candidate set, a budget cut mid-organization, and the NOT IN NULL trap that would turn the
  run-row pass into a silent no-op. About two seconds.
• tests/unit/execution-retention-windows.test.ts - config parsing, per-org window
  resolution, and the schedule grouping.
• tests/unit/execution-retention-purge.test.ts - pass order, dry run writing nothing, the
  runtime budget, and the watermark clamp.
• tests/unit/analytics-time-range-bounds.test.ts - both ends of a custom range.
• tests/integration/retention-route.test.ts - route contract and the metric counters.
• Verified end to end in PR environment pr-2344, which now runs the CronJob for real: the
  job has run unattended every 10 minutes for over 16 hours with no failures.
• Full local pipeline: pnpm check, pnpm type-check, pnpm build, pnpm test:unit,
  pnpm test:integration, pnpm test:db, and pnpm db:migrate plus the drizzle drift check
  on a fresh database.

[joelorzet]

Questions on what this changes for users.

Two tables, because everything below comes from them:

• workflow_executions is one row per run. This PR keeps it 400 days.
• workflow_execution_logs is one row per step inside a run. This PR erases it at the plan window: 7 days free, 30 pro, 90 business, 365 enterprise.

1. A free org will see two different numbers on the same screen.

The "Gas spent" number at the top of analytics reads workflow_executions. The "Gas by network" chart under it reads workflow_execution_logs.

The page still offers a 30d button and a custom range with no limit. A free org clicks 30d: the number on top covers 30 days, the chart under it covers 7, and nothing says why they disagree.

There are no UI changes in this PR, 15 files and none in components/. Is a UI change coming with it, or do we ship the cron and leave the page like that?

2. Runs will open to an empty step list.

A free org opens a run from 20 days ago. It is still in the list with its status and duration, because that row lives 400 days. Click into it and there are no steps, because they were erased at 7 days. It looks the same as a run that never had any.

3. The gas filter will put runs in the wrong bucket, not hide them.

The filter decides "free" by asking whether the run has a step that burned gas and whether it has a gas credit row. If neither, it is free.

Erase the steps and an old run that really did pay gas from the wallet answers no to both. It moves out of wallet and shows up under free. Not missing, wrong.

4. The monthly digest will report about a quarter of the sponsored transactions.

The digest counts sponsored transactions with output_raw->>'sponsored' = 'true' on the step logs. stripExpiredOutputRaw sets output_raw to NULL after 7 days, for every org on every plan, and NULL does not match.

The monthly digest covers the previous full calendar month. Only its last 7 days will still have the marker. The email will still label the number as the month's total.

5. output_raw ignores the plan the customer pays for.

Every other part of this job uses the org's plan window. This one is a flat 7 days for everybody. Business pays for 90 days and loses the field at 7. Enterprise pays for 365 and loses it at 7. It is returned by GET /api/workflows/executions/{id}/logs and by get_execution_logs over MCP, so a customer can watch it disappear.

Why is this one flat?

6. We need to check every current enterprise org before this runs.

We fill in plan and plan_overrides by hand when an enterprise org is created. Until now a wrong or missing value cost nothing, because no code read it. This PR makes it decide what gets erased.

If the subscription row is missing, resolveRetentionDays falls back to defaultLogRetentionDays, which is 7, and that org's logs go at 7 days like a free org.

So before dry-run comes off we need to go through every enterprise org and confirm its row carries the right retention value, and then confirm the cron resolves that org to the window we expect and not to 7.

The job only reports a row count today, so it cannot be used for that check. Can it report the window and the org count per group, so the manual rows can be verified against what it actually resolved?

7. I do not think this erases much on prod.

The two per-org steps only select a 7 day slice, from retentionDays + lookbackDays ago up to retentionDays ago. For a free org that is rows aged 7 to 14 days. Anything aged 15 to 400 days is not selected by anything.

The two steps that scan from the oldest row cut at 400 days, and the migration comment in this PR says the oldest row on prod is from 2026-01-06, about 244 days ago. So they match nothing.

If the goal is the 62 GB, this does not reach it. Is that intended, and is a catch-up run planned separately?

8. In about five months, invoice history starts changing.

Once prod rows cross 400 days, run rows start being erased. The invoices page does not read a stored figure, it recounts workflow_executions on every load, so a past invoice will show a smaller number than we billed.

overage_billing_records stores the real figures, but overage.ts only writes a row when there was overage, so every month an org stayed inside its plan has no record. For those months the rows are the only proof. Once they go we cannot defend a charge a customer disputes, and we cannot catch one we got wrong in our own favor.

That means new storage, not a tweak here. We need a table that records, per org per billing period, the plan and tier at the time, the execution limit, the billable workflow executions and the direct executions, and the charge. overage_billing_records already has most of those columns but only gets a row when there was overage, so it is a starting shape and not the answer. Then the invoices page reads that record for closed periods and only recounts the open one.

This needs to be a tracked follow-up that lands before dry-run comes off, not after. Once the first run rows are erased the numbers are gone and we cannot rebuild the record from anything.

9. EXECUTION_RETENTION_DAYS has no clamp.

Everything in point 8 is 400 days away only while that variable stays at 400. It is read with no floor. minLogRetentionDays gets a Math.max in getRetentionConfig and this one does not, and it sits in the same values.yaml block as the log windows. A low value there moves live quotas and every past invoice in one hourly run. Can it get a clamp?

And who turns dry-run off, and when?

Posting this as discussion rather than a blocking review.

[OleksandrUA]

Joel, thank you for this. Point 7 is right and it is the one that matters, so I start there.

7. Correct, and worse than you wrote. My numbers were wrong

You found the actual defect. Both per-organization passes bounded the slice on both sides, [now - (window + lookback), now - window), so a run only ever saw rows which crossed their boundary in last week. Everything older than that, and younger than the floor, was selected by nothing. And floor pass matched nothing at all, because it ran at fixed 400 days while oldest prod row is from 2026-01-06.

I MEasured it on prod rather than arguing. 0.5% sample, scaled:

bucket	rows	payload
inside the plan window, correctly kept	22,680,400	45 GB
pass-2 slice, deleted today	178,600	180 MB
stranded, selected by nothing	1,963,000	1,179 MB

And the same bound applied to output_raw, which was supposed to be whole point of the design:

bucket	rows	output_raw
inside 7 days, kept	2,087,800	1394 MB
pass-3 slice, stripped today	1,673,800	1378 MB
stranded, keeps it forever	19,075,200	14 GB

So this PR freed about 1.5 GB of 62 GB, not the ~16 GB I claimed in description and in Linear. That claim was simply wrong and I corrected it in the ticket. My own live check did not catch it because I ran it with a 999-day lookback, which is exactly the crutch that hid the bug.

The lookback is gone now. The two passes that need a lower bound take it from real progress :

• the plan-window pass keeps a per-organization watermark (execution_retention_progress) and advances it only when that organization's range fully drains, so an interrupted run resumes instead of skipping;
• the output_raw pass has no lower bound at all, and leans on partial index on (started_at) WHERE output_raw IS NOT NULL, which holds the backlog while it drains and then shrinks to only the rows inside window.

I also moved the floor pass to run at the longest window actually in use, not at a constant. That way organizations holding most of the table are served by plain index range with no join, and the per-organization pass only ever walks the short windows- which on prod is 2.23M executions rather than millions.

No separate catch-up job. The hourly run drains it on its own. I re-ran the live check on a real Postgres, this time with rows seeded at 2, 9, 20, 45, 120, 300 and 500 days and no lookback anywhere: all 20 assertions pass, including that second run does exactly nothing.

9. Correct. Clamped

EXECUTION_RETENTION_DAYS now has a hard 400-day floor in code, so it can lengthen the window and never shorten it. Every other window is clamped up to EXECUTION_RETENTION_MIN_DAYS, which only defaultLogRetentionDays had before.

On "who turns dry-run off, and when" :I do, and only after the dry run reports what point 6 asks for. It is a values.yaml change, so it is a PR, not a console click.

8. Correct on every assertion. I gated the pass off

I checked all of it. The invoices page recounts on every load with no stored figure, overage.ts writes a record only when overageCount > 0, and it returns even earlier when the plan has overage disabled- so free-plan orgs never get a row at all. I looked for any other per-period snapshot: execution_quota_notifications stores usage at a threshold crossing on calendar month, not billing period; gas_credit_allocations and gas_sponsorship_monthly store gas. Stripe holds nothing usable either, no metering API is used and the overage line item carries overageCount, not totalExecutions.

ONe thing which makes it a bit worse than you wrote :the job never touches direct_executions, so a >400-day invoice would show only the direct half, near zero for most orgs.

Rather than race the follow-up, I put run-row deletion behind its own switch, EXECUTION_RETENTION_EXECUTIONS_ENABLED, default false, off in both environments. One boolean and the whole invoice risk leaves this PR. I will file the usage-record ticket - per org, per billing period, with plan, tier, limit, billable workflow executions, direct executions and charge - as hard blocker on ever turning it on. We have until 2027-02-10 before the first row crosses 400 days, so there is no rush, only a dependency.

6. Correct, and the exposure is larger than enterprise

I checked prod :975 of 1409 organizations have no organization_subscriptions row, 69%, and they own 3,350 workflows. Every one of them resolves to defaultLogRetentionDays, which is 7.

In fairness that fallback is not arbitrary- getOrgPlan also reads a missing row as free plan, so retention matches entitlement path exactly. But your point stands: nobody ever had to look at that number before.

All four enterprise orgs (ajna, KeeperHub-Demo, keeperhub-observability, techops-services) carry plan = 'enterprise' with no plan_overrides, so they resolve to 365 correctly today.

The job now reports { retentionDays, organizationCount, rows } per window in the route response, and emits keeperhub_execution_retention_window_organizations alongside it. So dry run is now the pre-flight check you asked for, and I would not turn dry-run off before reading it.

3. Correct, and KEEP-1334 did not fix it

I checked, because I hoped it had. KEEP-1334 moved the filter from COALESCE(step-log column, step-log JSONB) to the step-log column alone- both arms are workflow_execution_logs, so nothing changed for this.

FIxed properly now. The buckets read workflow_executions.gas_used_wei and gas_credit_usage, neither of which retention purges. I did not want to change analytics semantics blindly, so I compared old rules against new on 30 days of prod first :identical buckets for all 1,938,753 runs, zero disagreement. So it is retention-proof and behaviour-preserving at same time.

4. Correct, and the arithmetic is yours

Monthly cadence exists, window is previous full calendar month, and with the strip running hourly about 6.4 of 28-31 days keep their marker. So roughly a quarter, as you said, printed under a Monthly summary heading with no caveat.

It needed one word. sponsored is not a redacted key, so output->>'sponsored' carries same value as output_raw on every row, and pass 3 never touches output. The digest counts off output now.

Two things I noticed there which are not mine to fix in this PR, but somebody should know :the digest is requiredPlan: "pro", so a 30-day window still clips first day or two of a reported month; and the web3 steps stamp sponsored: true on failure branch too, so the digest already counts sponsored failures as sponsored transactions. Analytics guards against that, digest does not.

1. Mechanism correct, the example is dead code

The two queries really do read different tables, you are right about that. But GasBreakdownChart is defined and never rendered- analytics-page.tsx renders header, KPI cards, time series, filters and runs table only. So that particular side-by-side cannot happen today.

The problem is real on the screen which IS rendered, though, and I would say this makes your point stronger rather than weaker :the runs table's per-run Gas and Network cells come from the step-log subquery, and so do the network and gas filter dropdown counts. A purged run keeps its KPI contribution and loses its row-level cell.

2. Correct

Fixed together with 1. The runs response now carries the organization's step-log cutoff, so a run older than it renders an em dash with reason on hover instead of blank cell, and the expanded view says the steps are past the retention window instead of No step logs available.

I also gave the analytics range a floor while I was in there. customStart came straight off query string with no lower bound, so ?range=custom&customStart=2020-01-01 scanned whole table and bypassed the cache- I would say it is cheapest way to reproduce 2026-09-02. Same change fixes range=custom with no customStart, which produced an Invalid Date that every downstream comparison answered false to.

5. Fair question. I would like to hold the flat window, here is why

output_raw is nulled at flat 7 days for everybody, yes. But the customer does not lose their log, they lose the unmasked duplicate of it.

Both columns are written in the same UPDATE from the same payload, and output is the identical structure with only secret-keyed values masked. sponsored, transactionHash, chainId, gasUsed are byte-identical between the two. I checked on prod: 0 rows in a 0.5% sample have output_raw populated while output is null.

So what goes at 7 days is an unredacted copy of secrets which I would say we should perhaps not keep for a year anyway. And if I align it to the plan window instead, enterprise holds 83% of the payload at 365 days, so it gives back ~14 GB and we are back to freeing almost nothing.

Where I think you are plainly right is that a stripped row is byte-identical to one which never had a payload. If you want, I would suggest to add a marker so the API can say "stripped by retention" rather than returning bare null. Happy to do it in this PR if you think it is needed.

---

Also, unrelated to your review :KEEP-1328 took migration 0151 on staging while this was open, so mine is renumbered to 0152, and 0153 adds the progress table. All three indexes are built CONCURRENTLY on staging and verified valid, so db-prepped-staging stands.

Full pipeline is green again- pnpm check, both type-checks, build, check:api-docs, 22,247 unit tests, 1,046 integration tests, and db:migrate plus the drizzle drift check on a fresh database.

[github-actions]

PR Environment Deployed

Your PR environment has been deployed!

Environment Details:

• URL: https://app-pr-2344.keeperhub.com
• Namespace: pr-2344
• Cluster: techops-staging
• Image Tag: f1b5ad6
• Commit: f1b5ad62586850754b80da327858fc6e5bbca092

Components:

• Keeperhub Application
• PostgreSQL Database (isolated instance)
• LocalStack (SQS emulation)
• Redis (isolated instance)
• Schedule Dispatcher (staging image)
• Block Dispatcher (staging image)
• Event Tracker (staging image)

The environment will be automatically cleaned up when this PR is closed or merged.

[github-actions]

PR Environment Deployed

Your PR environment has been deployed!

Environment Details:

• URL: https://app-pr-2344.keeperhub.com
• Namespace: pr-2344
• Cluster: techops-staging
• Image Tag: 96285e3
• Commit: 96285e36c1e9840181ea41c650d09cc6c430df2b

Components:

• Keeperhub Application
• PostgreSQL Database (isolated instance)
• LocalStack (SQS emulation)
• Redis (isolated instance)
• Schedule Dispatcher (staging image)
• Block Dispatcher (staging image)
• Event Tracker (staging image)

The environment will be automatically cleaned up when this PR is closed or merged.

[joelorzet]

👋 Hi @OleksandrUA, reviewed this again and most of what I raised is addressed. The watermark replacing the lookback, the hard clamp on EXECUTION_RETENTION_DAYS, the run-row pass behind its own switch, the dry run counting properly, and the digest reading output all check out. I verified the digest one: sponsored is not in SENSITIVE_KEYS and matches none of the patterns in lib/utils/redact.ts, so the two columns do carry the same value. I also went looking for a hole in the floor pass running without the resumable guard at a 7 day floor and did not find one, the reaper and the reconciler both close well inside it.

Two things left.

1. Plan downgrade deletes data within the hour, and it is not recoverable.

The deletion is a plain DELETE with no archive, and the watermark only moves forward, so an upgrade never restores anything. That is fine if it is the intent. The downgrade direction is the one that needs a decision.

handle-billing-event.ts sets plan: "free" on subscription.deleted once the period has ended. An unconverted Pro trial and a cancellation both land there. resolveRetentionDays then returns 7 instead of 30 on the next hourly run, the watermark is sitting at 30 days, and the pass deletes the 23 days in between. No grace, no notice, no way back. The cancellation grace window itself is fine, since the plan only drops at period end.

So is the intent a straight wipe at the plan window, or should there be an internal retention policy behind it? A grace period on downgrade, an archive, or a soft delete before the hard delete would each change what this pass does. If it is a straight wipe, the plan copy should say so, because "30-day log retention" reads as a ceiling today rather than as "these are destroyed the hour your plan ends."

2. The two run row columns everything now leans on have unmeasured gaps.

Moving the gas buckets and the network dimension off the step logs is the right fix, and it makes workflow_executions.gas_used_wei and transaction_hashes the source of truth for what the runs table shows.

gas_used_wei is filled at finalize and backfilled for history by scripts/backfill-workflow-gas.ts, which derives from step log JSONB. Your equivalence check covered 30 days, all of it after finalize started writing the column, so it does not exercise the backfilled range. A gas bearing run in that range with a NULL there shows a blank Gas cell and lands in the free bucket. transaction_hashes has no backfill script at all, so runs older than 0071 are carried by the step log arm alone and drop out of network filtering once purged.

Both become permanent the moment the step logs go, because the backfill derives from the rows being deleted. One count before dry run comes off would settle it:

SELECT count(*) FROM workflow_executions
WHERE gas_used_wei IS NULL
  AND started_at < now() - interval '30 days'
  AND id IN (SELECT execution_id FROM workflow_execution_logs WHERE gas_used_wei > 0);

[github-actions]

PR Environment Deployed

Your PR environment has been deployed!

Environment Details:

• URL: https://app-pr-2344.keeperhub.com
• Namespace: pr-2344
• Cluster: techops-staging
• Image Tag: 6bd83fc
• Commit: 6bd83fce59cd6c9e96e53a431ea807578e84a288

Components:

• Keeperhub Application
• PostgreSQL Database (isolated instance)
• LocalStack (SQS emulation)
• Redis (isolated instance)
• Schedule Dispatcher (staging image)
• Block Dispatcher (staging image)
• Event Tracker (staging image)

The environment will be automatically cleaned up when this PR is closed or merged.