Port upstream asynchronous, resumable stream deletion (DELETE returns 202) #151

Summary

Upstream made stream deletion asynchronous and resumable in parseablehq/parseable#1770 ("Make stream deletion asynchronous, resumable across restarts", merged 2026-11-11). This fork still deletes synchronously and returns 200 OK. Port the upstream change, adapted to the metastore boundary, so that:

• DELETE /logstream/{stream} writes a durable deletion marker, flags the stream as deleting, and returns 202 Accepted before the object-store sweep starts.
• The sweep runs in a background task, deduplicated per stream.
• A deletion interrupted by a crash or restart resumes on startup.
• Create, update, ingest and query against a stream that is being deleted return 409 Conflict.

This closes #143 and covers most of #142.

Where the fork stands

• The groundwork from parseablehq/parseable#1768 is already on main: the .tombstones directory, is_tombstoned, Stream::mark_deleting() / is_deleting(), and is_deleting() checks in the ingest, query and stream handlers.
• Nothing in production code calls mark_deleting() or writes a tombstone, so those checks never fire (see Stream deletion is not fenced against concurrent ingest, upload and running queries; the deleting guards are never armed #142).
• Both delete handlers (src/handlers/http/logstream.rs, src/handlers/http/modal/query/querier_logstream.rs) return 200 OK after the full sweep.

Why this is not a straight cherry-pick

1. Overlap with the metastore work. The upstream change is about 1,000 lines in the stream handlers, src/storage/object_storage.rs, src/storage/localfs.rs, src/parseable/streams.rs and src/parseable/mod.rs. refactor: route metadata lifecycle operations through the Metastore trait #146 routes stream deletion through the Metastore trait (is_stream_deleting, delete_stream), and feat: opt-in PostgreSQL metastore with import, export and verify tooling #148 keeps its own deletion marker in PostgreSQL. Upstream stores the tombstone in the bucket. The port has to put deletion state behind the trait so both backends work, and has to land after refactor: route metadata lifecycle operations through the Metastore trait #146 and feat: opt-in PostgreSQL metastore with import, export and verify tooling #148.
2. Open review findings upstream. The automated review on the upstream pull request listed problems that must be checked against the merged code, and fixed here if they still apply:
   • a queued or repeated deletion job can run after the original finished and delete a stream recreated under the same name;
   • a repeated DELETE re-runs cleanup while the first job is still going;
   • an ingestor that misses the notification and whose sync tick falls after the marker is cleared keeps accepting events for the deleted stream;
   • that ingestor's staged files survive and are uploaded into a recreated stream of the same name;
   • if clearing the marker fails on a query node, later DELETE calls return 202 without starting a job, so the name stays blocked until restart;
   • ingestor statistics for the deleted stream are not cleared;
   • the dedup tests do not exercise the real claim and guard.
3. API contract change. Anything that expects exactly 200 from stream deletion needs updating: the frontend, the tool executor, and the pb CLI version used in tests.

Acceptance criteria

• DELETE /logstream/{stream} returns 202 once the deletion marker is durable; standalone, query and ingestor modes.
• Deletion resumes after a restart in object-store mode and in PostgreSQL mode.
• 409 on create, update, ingest (including OTel) and query while deletion is pending.
• A stream recreated under the same name is never touched by an old deletion job (test).
• Each upstream review finding above is either shown not to apply or fixed, with a test.
• Frontend, tool executor and docs updated for 202.
• CI goes back to the upstream Quest image: remove the pinned Quest source commit from .github/workflows/integration-test.yaml.

Context

CI runs the upstream Quest integration suite from the floating quay.io/parseablehq/quest:main image. Since parseablehq/quest#133 that suite expects 202 on stream deletion, so TestSmokeRbacBasic and TestSmokeRoles fail on main and on every pull request. As a stopgap, CI builds Quest from the last source commit before that change. This issue tracks removing the stopgap.