Skip to content

Add external body storage for audit - #6029

Draft
rbev wants to merge 2 commits into
masterfrom
audit-external-bodies
Draft

rbev wants to merge 2 commits into
masterfrom
audit-external-bodies

Conversation

@rbev

@rbev rbev commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Problem

Audit's SQL Server and PostgreSQL persisters discard oversized and binary message bodies. Oversized text retains a searchable prefix, but the body API cannot return the complete body. Error's SQL persisters already support external storage for these bodies.

Change

Reuse Error's FileSystem, Azure Blob, and S3 providers for Audit by extracting them into ServiceControl.Persistence.BodyStorage.

  • Keep small valid text bodies inline. Store oversized, binary, content-encoded, non-UTF-8, and NUL-containing bodies externally.
  • Retain a UTF-8-safe inline search prefix for oversized text and serve the complete body through the body API.
  • Await external writes before inserting message rows, including when disposing an abandoned ingestion batch.
  • Delete external bodies before removing expired rows or partitions. A storage failure leaves the database records available for the next sweep.
  • Provision storage during setup and register the existing disk-space custom check for FileSystem storage.
  • Update persistence manifests, test fixtures, development configuration, and the testing tool.

RavenDB behavior is unchanged. Existing TooLarge and NotText rows retain their previous API responses; discarded bodies cannot be recovered. The new external-body state uses the existing integer column, so no schema migration is needed.

Required configuration

External storage is required for Audit SQL Server and PostgreSQL instances. Existing instances must add ServiceControl.Audit/MessageBody/StorageType and the selected provider's settings before restarting after an upgrade.

<add key="ServiceControl.Audit/MessageBody/StorageType" value="FileSystem" />
<add key="ServiceControl.Audit/MessageBody/FileSystem/StoragePath" value="D:\ServiceControl.Audit\bodies" />

MaxBodySizeToStore becomes the inline threshold, not a limit on the complete stored body. Azure defaults to the audit-bodies container; S3 defaults to the audit-bodies/ key prefix.

Each instance must use a separate directory, container, or key prefix. Sharing a location can let one instance's retention process delete another instance's bodies.

Design constraints

Sharing the providers avoids duplicating their storage formats, compression, authentication, and stream handling. Error retains its existing configuration and behavior.

Audit retention deletes by hourly prefixes on SQL Server and daily prefixes on PostgreSQL. This preserves PostgreSQL's partition-drop path without scanning message rows. SQL Server only visits hours containing rows, so bodies orphaned by failed inserts into otherwise empty hours can require manual cleanup.

The current design and configuration are documented in Message body storage.

Verification

  • Passed 135 infrastructure tests, 36 focused Audit tests for each SQL provider, and 36 Error body-storage regression tests.
  • Built the Audit persistence and acceptance-test projects for both SQL providers.
  • Built Error SQL Server and Audit PostgreSQL persisters in Release configuration.
  • Added database-backed coverage for complete body retrieval, legacy states, HTTP downloads, retention failures, and partially expired PostgreSQL days.
  • Completed independent source review and resolved its findings. git diff --check passes.

Database-backed Audit, Azure/S3, and Windows packaging tests were not run because the required services or environment were unavailable. They remain release validation requirements.

The full Audit unit suite has an existing PlatformSampleSettings snapshot failure involving hostname/BOM differences. The testing-tool AppHost build is blocked by the existing MessagePack 2.5.302 vulnerability warning, GHSA-qhrr-8q5h-9q3h. Neither issue was changed or suppressed.

@rbev rbev self-assigned this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant