Automatic Message Payload Preservation in DLQ Stream

[Atanusaha143]

Problem Statement

When a message reaches its maximum delivery attempts in JetStream, only an advisory message with metadata is sent to the DLQ stream. The original message payload is lost because it gets removed from the source stream after hitting max_deliver. This makes it impossible to:

1. Inspect the actual message content that caused the failure
2. Manually reprocess failed messages after fixing underlying issues
3. Debug and troubleshoot processing failures effectively
4. Maintain audit trails of failed messages

Currently, there's no built-in JetStream configuration to automatically preserve message payloads in a DLQ when max deliveries is reached.

Current DLQ Stream Configuration

Stream: booking_dlq_stream
Created: 2025-09-04 11:53:48
Subjects: $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.booking_stream.booking_consumer
Replicas: 1
Storage: File

Options:
  Retention: WorkQueue
  Acknowledgments: true
  Discard Policy: Old
  Duplicate Window: 5m0s
  Direct Get: true
  Allows Msg Delete: true
  Allows Purge: true
  Allows Rollups: false

Limits:
  Maximum Messages: unlimited
  Maximum Per Subject: unlimited
  Maximum Bytes: unlimited
  Maximum Age: unlimited
  Maximum Message Size: unlimited
  Maximum Consumers: unlimited

State:
  Messages: 4
  Bytes: 1.5 KiB
  First Sequence: 1 @ 2025-11-14 13:03:51
  Last Sequence: 4 @ 2025-11-14 17:44:21
  Active Consumers: 0
  Number of Subjects: 1

Current DLQ Message Content

When retrieving a message from the DLQ stream by sequence number:

$ nats stream get booking_dlq_stream 1 --json

Response:

{
  "subject": "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.booking_stream.booking_consumer",
  "seq": 1,
  "data": "eyJ0eXBlIjoiaW8ubmF0cy5qZXRzdHJlYW0uYWR2aXNvcnkudjEubWF4X2RlbGl2ZXIiLCJpZCI6IlVDV1J1bHM0aWFENXhDT0NoV2c4anUiLCJ0aW1lc3RhbXAiOiIyMDI1LTExLTE0VDA3OjAzOjUxLjg2MTExMzY3OFoiLCJzdHJlYW0iOiJkZXZfc2JfcmVmdW5kX2Jvb2tpbmdfc3RyZWFtIiwiY29uc3VtZXIiOiJkZXZfc2JfcmVmdW5kX2Jvb2tpbmdfY29uc3VtZXIiLCJzdHJlYW1fc2VxIjoxMCwiZGVsaXZlcmllcyI6NX0=",
  "time": "2025-11-14T07:03:51.862562668Z"
}

Decoded advisory data:

{
  "type": "io.nats.jetstream.advisory.v1.max_deliver",
  "id": "UCWRuls4iaD5xCOChWg8ju",
  "timestamp": "2025-11-14T07:03:51.861113678Z",
  "stream": "booking_stream",
  "consumer": "booking_consumer",
  "stream_seq": 10,
  "deliveries": 5
}

Issue: The advisory only contains metadata (sequence number, delivery count, timestamp) but no original message payload.

Attempting to Retrieve Original Message

According to the documentation:

"If you also need to have access to the message in question then you can use the message's sequence number included in the advisory to retrieve that specific message by sequence number from the stream."

However, attempting to retrieve the original message fails:

$ nats stream get booking_stream 10
Error: message not found (stream_seq: 10)

Why? The original message at sequence 10 was removed from the source stream after hitting max deliveries. The documentation assumes the stream retains messages after max deliveries, which is not the default behavior.

How to Get Original Message (Current Workarounds)

Currently, the only way to preserve message payloads in a DLQ is through application-level implementation:

Proposed Feature Request

Add a consumer configuration option to automatically republish messages (including payload) to a DLQ stream when max deliveries is reached:

Option 1: Consumer Configuration

nats consumer add my_stream my_consumer \
  --max-deliver=5 \
  --dlq-stream=my_dlq_stream \
  --dlq-subject=my.dlq \
  --dlq-include-payload=true

Option 2: Consumer Config YAML

max_deliver: 5
dlq:
  stream: my_dlq_stream
  subject: my.dlq
  include_payload: true
  include_headers: true
  include_metadata: true

Expected Behavior

1. Message fails and reaches max deliveries (e.g., 5)
2. Before removal, NATS automatically publishes to DLQ stream with:
   • Original message payload
   • Original headers
   • Metadata (delivery count, error info, original subject, sequence)
3. Message is then removed from source stream or terminated
4. DLQ message can be inspected, reprocessed, or archived

Benefits

1. Zero code changes required - pure configuration
2. Guaranteed payload preservation - handled by server
3. Consistent DLQ behavior across all consumers
4. Better debugging - full context of failures
5. Replay capability - can reprocess after fixes
6. Audit compliance - complete trail of failed messages

Related Discussions

• Similar request in other message systems (AWS SQS, RabbitMQ have built-in DLQ)
• NATS docs mention using sequence number to retrieve messages, but this fails when messages are removed from the original stream

Environment

• NATS Server Version: 2.12.1
• JetStream Enabled: Yes

[ripienaar]

We have considered this before but has quite a lot of user feeedback that this is a bad idea due to the payloads potentially being sensitive and DLQ advisories going to different locations outside of nats

it might be something that would make sense as a pre stream opt in though

[Atanusaha143]

this is a bad idea due to the payloads potentially being sensitive

I agree with you. That's why I proposed it through the consumer where sending the payload could be considered as a choice weather to send it or not. Then it would be a toggle away to enable or disable it.

it might be something that would make sense as a pre stream opt in though

Can you describe a bit regarding this?

[ripienaar]

Yes sorry I meant a per consumer option not per stream.

[Atanusaha143]

What's the plan then?

[ripienaar]

I don’t think there is a plan per se. We have many feature requests and do prioritisation twice a year or so based on many factors.

A fast way to make progress would be to contribute something via a pull request

[jgriegershs]

@Atanusaha143 , @ripienaar
I am a bit confused here. As the documentation states, messages are only removed from a stream (with retention=work) when they are ack'ed or reach their end of live (TTL). When a message hits max-deliver, it stays in the worker queue and an advisory is emitted to $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>, enabling fetching the dead-lettered message with nats s get my-work-queue msg-seq-from-advisory including the payload.

As far as I can tell, this works with v2.12.4 and R1 streams. With R3, messages might get lost (I am about to create an issues for that).

[ravithejatech714-create]

This is NATS rather than Kafka, but the operational problem is very familiar: a DLQ is only useful if operators can inspect the failed payload, retain provenance, decide whether it is safe to retry, and replay it with an audit trail.

ReplayGuard is tackling that control-layer workflow for Kafka. The payload/headers/metadata preservation you describe is especially close to how we think about recoverable provenance.

https://replayguard.in/