Repository navigation
Commit 678c1a4
authored
fix(account-deletion): durably purge a deleted account's storage without touching live objects (#9146)
* fix(account-deletion): durably purge a deleted workspace's storage by prefix
Account deletion found the objects to erase by reading their rows before the
teardown transaction and deleting them in memory after commit. Anything that
read could not see stayed in storage forever: rows written between the read
and the workspace lock, rowless objects (table snapshots, compiled documents,
table-import leftovers, execution objects whose metadata insert failed),
everything past the 100k in-memory cap, everything after a crash following
commit, and the account's private files in workspaces that survive it, whose
rows cascade away with the user.
Inside the deletion transaction, after the workspace rows are locked, account
deletion now queues delayed outbox work that commits or rolls back with it:
- one prefix sweep per deleted workspace across workspace/, table-import/,
execution/, table-snapshots/, copilot-doc-compiled/ and workspace-logos/
in the bucket serving each, with a trailing-slash prefix built from a
validated id;
- exact-key cleanup for what cascades outside those prefixes: knowledge-base
documents, Copilot uploads and other non-prefixed files of deleted
workspaces, and the account's private files elsewhere (attachments of its
own chats, private contexts, files with no workspace). Handed-off, Project,
organization files and attachments of someone else's chat are not queued.
Both wait out every upload URL signed before the deletion, refuse as a no-op
when the workspace or account row exists again (restore, PITR), and keep any
key a surviving workspace_files, workspace_file, workspace_file_version,
document or execution_large_values row still references. The sweep reuses the
Project prefix cleanup's provider listing and deletion, generalized by storage
context and given an in-run provider cursor; Project cleanup still lists one
page per run and deletes exactly as before. The existing in-memory purge is
unchanged and still runs.
Large execution values now pass cleanupOnMetadataFailure, so a failed
metadata insert deletes the freshly keyed object instead of orphaning it, as
execution files already do.
* fix(account-deletion): never purge objects that live data still names
Review of the durable storage sweep found ways it, or the in-memory purge it
backs up, could delete an object another tenant still uses.
Reference guard. Every account-deletion purge now runs one check before
deleting a key: the durable prefix sweep (per listed key), the durable
exact-key cleanup, and the post-commit in-memory purge. A key is kept when a
row still names it (workspace_files.key, workspace_file.key,
workspace_file_version.key, document.storage_key, execution_large_values.key)
or a stored asset URL names it: user.image, workspace.logo_url,
organization.logo, organization.whitelabel_settings, organization.metadata,
custom_block.icon_url, chat.customizations (deployed-chat logo and avatar) and
oauth_client.icon. URLs are mapped to keys with extractStorageKey, so serve
paths with or without a provider segment and direct bucket URLs both match;
over-matching only keeps an object. References embedded in free-form JSON or
text (workflow sub-block values, chat messages, Markdown links) are not
consulted; for the deleted account's own chat attachments that is the intended
erasure. This keeps whitelabel logos that were uploaded under an admin's
personal workspace prefix, and legacy kb/ objects shared across workspaces,
which the in-memory purge used to delete unchecked.
Exact-key cleanup also keeps any key under a workspace-keyed root whose
workspace still exists, since its members can read it by key through the
metadata fallback. Workspace-less rows are queued only for the Copilot
context: the workspace binding check was added NOT VALID, so legacy
shared-context rows without a workspace can name a surviving workspace's
object. Profile pictures are taken only from the account's own user.image,
never from rows, because whitelabel logos were once stored as
profile-pictures rows under an admin's account; organization logos are never
queued either.
Prefix deletion now counts only objects actually removed. A tolerated ENOENT
(locally, any file whose on-disk name no sanitized key resolves to) used to
count as progress, so a page of such files re-listed forever without spending
an attempt. The workspace sweep and the Project prefix cleanup (which had the
same flaw) now defer and spend an attempt when a full page removes nothing.
A local metadata sidecar is kept whenever its object is. Versions of the
account's private files are queued in keyset pages that stop at the cap
instead of one unbounded query inside the deletion transaction.
* fix(account-deletion): bound and harden the storage reference guard
A stored URL with malformed percent-encoding (for example
`/api/files/serve/%E0%A4%A`, which any workspace admin can store as a logo URL)
made extractStorageKey throw. The guard then rejected on every call, so every
storage purge on the platform failed closed and deleted nothing. The guard now
parses each value under its own try and falls back to the raw, undecoded path.
extractStorageKey itself is unchanged for its other callers.
The URL-column check read six unindexed tables on almost every guard call.
Only keys under the prefixes the file-serve route answers without a row or a
session (profile-pictures/, og-images/, workspace-logos/,
organization-logos/, per isPublicByKeyPrefix in the serve route) can be live
through a stored URL alone. The guard now reads URL columns only for such
keys, and only rows containing the key's longest URL-safe run, which
percent-encoding never splits. Each run is read at most once per guard, 50 per
query. A key with no run long enough falls back to the unfiltered read. Lookup
failures still throw, so nothing is deleted.
The workspace sweep now defers with a distinct reason when a full local page
removes nothing, so the test for it can tell a stall from a deadline.
* refactor(account-deletion): split the storage reference guard and share storage primitives
- keysNamedByUrl also yields a leniently decoded candidate when a stored URL's
percent-encoding is malformed, so a legacy key holding a literal '%' is kept
- correct the guard's TSDoc: rowless keys stay reachable through key paths under a
live workspace and object metadata, not only through a row
- move the reference guard to lib/users/storage-reference-guard.ts
- move prefix listing/deletion, the upload-TTL cleanup delay, and the outbox
deadline margin from lib/projects/files/prefix-cleanup.ts to
lib/uploads/core/prefix-sweep.ts; drop the projectUploadCleanupAvailableAt alias
- share WORKSPACE_SHARED_FILE_CONTEXTS between member handoff and account cleanup
- isPublicStorageKey replaces the public-prefix lists in the serve route and guard
* fix(account-deletion): read durable storage keys before the deletion locks
- collectAccountStorageObjects runs the exact-key keyset reads before the
transaction, beside collectAccountStorageKeys; inside it,
enqueueAccountStorageCleanupInTx now only inserts the prefix-sweep events and
the pre-collected exact-key events, so teammates' writes in shared workspaces
never wait on those reads
- Project prefix cleanup completes on a short listing page again, as before
- the file-serve route keeps its original public-prefix check
* fix(account-deletion): close the storage cleanup gaps review found
- integration fixture rows get unique names: the active-name unique index on
workspace_files rejected the second 'fixture.txt' row, failing seed()
- the reference guard reads stored asset URLs unfiltered and decodes them, so
a URL that percent-encodes the key's characters still keeps its object; the
world-readable-key gate still decides whether any URL column is read
- enqueueAccountStorageCleanupInTx re-reads the account's rows written since
the pre-transaction collection (via the user_id and uploaded_by indexes)
under the account lock, so a late Copilot or knowledge-base upload is queued
- local listings walk keys in sorted order and page after the last key, so a
workspace sweep advances past a full page of kept objects
* docs(account-deletion): describe the in-transaction delta read accurately
* fix(account-deletion): narrow stored-URL reads and re-check workspaces per page
- fix TS2769: the collection scope passed a readonly id list to notInArray
- the reference guard reads stored asset URL rows only where they mention a
public key's owning id run, paged, and keeps a public key with no such run
instead of reading every URL column
- the account deletion request no longer purges world-readable keys; the
account's own avatar is queued for the delayed durable cleanup instead
- the workspace prefix sweep re-checks that the workspace is still gone before
every page it deletes from
* fix(account-deletion): queue an avatar only when the account uploaded it
user.image accepts any serve URL, so it names a key without proving whose it
is. The avatar is queued for durable cleanup only when the account's completed
profile_picture upload session, or an older profile-picture workspace_files row
of the account, names the same key; otherwise it is left in place.
* fix(account-deletion): apply every purge guard on every call
- the reference guard keeps keys under a live workspace's keyed roots itself,
so the request purge gets the check the durable cleanup already had
- stored asset URLs are re-read for every guard call rather than cached for
the sweep, so a reference written mid-sweep still keeps its object
- an avatar key that is not a safe profile-pictures key is never queued
* fix(account-deletion): queue the account's avatar and let the guard keep shared ones
An ownership lookup left avatars uploaded more than a week before deletion in
storage, since their upload sessions are gone by then. The key user.image
names is queued again, and the run-time reference guard keeps it whenever
another account's user.image or a stored asset URL still names it.1 parent 99ed5fb commit 678c1a4
20 files changed
Lines changed: 2474 additions & 166 deletions
File tree
- apps/sim/lib
- core/outbox
- execution/payloads
- projects/files
- application
- uploads
- core
- utils
- users
- workspaces
- packages/testing/src/mocks
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
48 | 52 | | |
49 | 53 | | |
50 | 54 | | |
| |||
239 | 243 | | |
240 | 244 | | |
241 | 245 | | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
242 | 253 | | |
243 | 254 | | |
244 | 255 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
| 93 | + | |
93 | 94 | | |
94 | 95 | | |
95 | 96 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
| 32 | + | |
36 | 33 | | |
37 | 34 | | |
38 | 35 | | |
| |||
43 | 40 | | |
44 | 41 | | |
45 | 42 | | |
| 43 | + | |
46 | 44 | | |
47 | 45 | | |
48 | 46 | | |
| |||
321 | 319 | | |
322 | 320 | | |
323 | 321 | | |
324 | | - | |
| 322 | + | |
325 | 323 | | |
326 | 324 | | |
327 | 325 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
0 commit comments