diff --git a/CHANGELOG.md b/CHANGELOG.md index 77ccc8e4..6a6c0363 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,41 @@ archived by series under [docs/changelog/](docs/changelog/); see the ### Added +- **The inbound pending-decryption queue is persisted.** A frame that arrives + before its sender's MLS session is ready is written to protocol-state storage + (`pending_decrypt_entries`, sealed like the outbound pending queue) when it + is admitted, deleted when it is drained, evicted or discarded (blocking the + sender discards it), and restored on the next launch. A restart during a slow handshake used to lose it, and a + relay that pushes ciphertext without store-and-forward holds no second copy + to ask for. A record older than seven days on disk is dropped on restore and + reported as `PENDING_QUEUE_DROPPED` with reason `expired_persisted`. +- **The deduplicator's seen set is persisted.** Up to 2000 inbound ids, newest + first, sealed like the pending queues, written in batches (32 changes or 5 + seconds on the process tick, + flushed on stop) and restored beside the Lamport clock, so the relay-socket + copy of a message the app already consumed from a push injection is + recognised as a duplicate across a restart instead of reaching a spent + ratchet generation and surfacing as a decryption failure. A runtime + `updateDedupConfig` carries the set across instead of clearing it. A + sender's own outgoing ids are tracked for the process only and never + written. +- **`PENDING_QUEUE_DROPPED` is emitted for text messages**, not only for media + chunks, whenever the pending-decryption queue evicts a frame (overflow, TTL, + or the persisted age bound). Text evictions were metrics-only, which left an + app unable to tell "the sender went quiet" from "the SDK evicted their + message". The event is advisory: the frame was never ACKed, so a sender + still retrying resends it. +- **Inbound message events are held while nothing can take them.** + `message_received`, `file_received` and `message_decryption_failed` each + report one message the core has already ACKed and will never restate, so a + drop between the core and the app's handler was a lost message. Each React + Native bridge now holds up to 256 of them, keyed by message id, while + JavaScript has no listener or no live React instance (a backgrounded app, a + push injection before the first subscribe), and replays them on subscribe + and on foreground; the TypeScript layer does the same across the gap to the + app's first `on()`. Held events are scoped to the session that produced + them, so a message held for a torn-down account never surfaces in the next + one. - **`enableTelemetry` / `enable_telemetry` on every binding.** Takes the API key and app id the developer portal issued; the binding fills in the platform and owns the application lifecycle, so there is no per-event or per-lifecycle @@ -107,6 +142,33 @@ the client's answers were wrong on a device rather than merely different: ### Changed +- **A DM the relay only pushed is parked, not left awaiting an ACK.** The relay + answers `MessageSent { pushed: true }` when the recipient has no live socket + and the ciphertext went out in a device push; it keeps no copy. The React + Native bridges and the Python relay client now report that id to the core as + `relay_pushed`, which drops + the pending ACK, keeps the outbox entry, schedules the reachability probe + and presence-watches the recipient, as a `DeliveryError` does, with two + differences because the push may have delivered the frame: connection + requests and Welcomes stay on their ordinary path, and no + `MessageUndeliverable` is emitted, on the push itself or on any later probe + verdict for that frame. The outbox entry remembers it was pushed (persisted + with the entry), because the relay answers a probe of a pushed message with + `DeliveryError` rather than a second notification and the core would + otherwise read that as an ordinary unreachable verdict. +- **`pendingQueue.pendingTtlMs` defaults to 24 hours** (was 30 minutes) on + every binding, so a frame parked for a session still being set up outlives + a handshake on a phone opened once a day. The in-memory TTL restarts with + the process; the persisted copy is bounded separately at seven days. Memory + stays bounded by the per-peer and global caps. +- **`reliability.dedup` defaults to 2000 ids and 24 hours** (were 1000 and + 1 hour), sized for the persisted seen set. The Nostr replay overlap + (1 h 5 min) now fits inside the retention window, so a reconnect the next + day deduplicates it instead of re-processing it; `docs/nostr.md`, the + transport's subscription doc and the drift guard now state the absorption. + The React Native `updateDedupConfig` fallback for an omitted + `maxTrackedMessages` is 2000 as well (was 10000), so a `dedup` section that + sets only the retention no longer tracks five times the documented default. - **`TelemetryConfig` is reshaped** on every binding: `apiKey` and `appId` are required, `appVersion`, `debug`, `flushIntervalMs`, `maxBatchBytes`, `maxBufferedRecords` and `includeDeviceId` are new, `enablePollQueue` is @@ -232,6 +294,18 @@ the client's answers were wrong on a device rather than merely different: stays per-event and unaggregated by design; the rollup and session summary exist only inside the pipe, and a test pins that neither is ever an event. +### Fixed + +- **A peer-requested session reset deletes the persisted pending-decryption + records** of the frames it discards. Left on disk, the next launch restored + frames sealed to the deleted session and drained them into its replacement + as spurious decrypt failures. +- **Releasing a dropped group entry's replay protection reaches the persisted + seen set.** The release unmarked the id in memory only, so a record written + while the entry was buffered kept the id, the next launch restored it, and + the sender's redelivery was swallowed and re-ACKed as delivered for the whole + retention window. + ## [0.25.0] — 2026-09-08 > **A device on Reticulum could never receive anything. It can now.** Both diff --git a/README.md b/README.md index 711d0414..a9696875 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ config = ProtocolConfig( require_encryption=False, max_pending_per_peer=64, max_pending_global=4096, - pending_ttl_ms=1_800_000, # 30 min — matches the SDK default + pending_ttl_ms=86_400_000, # 24 h — matches the SDK default overflow_policy=OverflowPolicy.DROP_OLDEST, ) protocol = ProtocolManager( diff --git a/bindings/python/offline_protocol_sdk/internet_manager.py b/bindings/python/offline_protocol_sdk/internet_manager.py index c1f7c7c5..3b938d74 100644 --- a/bindings/python/offline_protocol_sdk/internet_manager.py +++ b/bindings/python/offline_protocol_sdk/internet_manager.py @@ -831,6 +831,13 @@ def _process_received(self, data: bytes) -> None: message_id if message_id else None, self._now_ms(), ) + # ``pushed: true``: the relay had no live socket for the recipient + # and handed the ciphertext to a device push. It keeps no copy, so + # this is the fact a DeliveryError carries (the recipient is not on + # the relay) with the frame possibly delivered by the push. Older + # relays omit the field, which reads as not pushed. + if msg.get("pushed") is True and message_id: + self._park_pushed_message(recipient, message_id) elif msg_type == "DeliveryError": recipient = msg.get("recipient", "") @@ -882,6 +889,42 @@ def _process_received(self, data: bytes) -> None: else: self._emit_diagnostic("debug", f"Unhandled relay message type: {msg_type}") + def _park_pushed_message(self, recipient: str, message_id: str) -> None: + """Parks the one frame the relay answered ``MessageSent { pushed: true }`` for. + + A port of ``parkPushedMessage`` in the iOS and Android bridges, and the + narrower sibling of the DeliveryError path above. The relay named + exactly one frame, so only that id is reported and the recipient's + other in-flight frames are left alone. The reason is the exact + ``relay_pushed`` token, not a ``recipient_unreachable`` tail: that + prefix fast-fails connection requests and fails Welcomes, while + ``relay_pushed`` parks a plain DM, leaves both of those alone and tells + the app nothing, since the push may have delivered the frame. + + The id is the one the relay echoed, never the tracker's fallback guess: + push outcomes come back out of order, so the oldest frame in flight is + the least likely to be the pushed one. An echo that names none of our + frames parks nothing, because the core ignores an id with no outbox + entry. + """ + try: + self._protocol.internet_send_failed_with_reason( + message_id=message_id, reason="relay_pushed" + ) + except Exception as exc: + logger.debug("internet_send_failed_with_reason(relay_pushed) failed: %s", exc) + if recipient: + try: + self._protocol.internet_peer_presence( + peer_id=recipient, online=False, last_seen_ms=None + ) + except Exception as exc: + logger.debug("internet_peer_presence(offline) failed: %s", exc) + self._emit_diagnostic("info", "Relay pushed message to offline recipient; parked", { + "recipient": recipient, + "messageId": message_id, + }) + # -- incoming message handlers -------------------------------------------- def _handle_incoming_message(self, msg: dict[str, Any]) -> None: diff --git a/bindings/python/tests/test_internet_manager.py b/bindings/python/tests/test_internet_manager.py index fd4a1fe8..f1dc3431 100644 --- a/bindings/python/tests/test_internet_manager.py +++ b/bindings/python/tests/test_internet_manager.py @@ -1094,6 +1094,69 @@ def test_message_sent_prevents_false_fail_of_delivered( assert "stuck" in failed +class TestMessageSentPushed: + """``MessageSent { pushed: true }`` parks the one frame the relay pushed. + + Pins the port of the bridges' ``parkPushedMessage``: the exact + ``relay_pushed`` token the core parks a plain DM on (never the + ``recipient_unreachable`` prefix, which fast-fails connection requests and + fails Welcomes), only the echoed id, and an offline presence for the + recipient. The Rust side pins the same literal in + ``relay_pushed_is_its_own_token``. + """ + + @staticmethod + def _pushed(recipient: str, message_id: str | None) -> bytes: + msg = {"type": "MessageSent", "recipient": recipient, "pushed": True} + if message_id is not None: + msg["message_id"] = message_id + return json.dumps(msg).encode() + + def test_pushed_frame_is_reported_under_the_exact_token( + self, mock_protocol: MagicMock + ) -> None: + mgr = InternetManager(mock_protocol, "dev-1") + mgr._inflight.record_sent("offZ", "pushed-id", mgr._now_ms()) + mgr._process_received(self._pushed("offZ", "pushed-id")) + mock_protocol.internet_send_failed_with_reason.assert_called_once_with( + message_id="pushed-id", reason="relay_pushed" + ) + mock_protocol.internet_peer_presence.assert_called_once_with( + peer_id="offZ", online=False, last_seen_ms=None + ) + + def test_push_leaves_the_recipients_other_frames_in_flight( + self, mock_protocol: MagicMock + ) -> None: + mgr = InternetManager(mock_protocol, "dev-1") + now = mgr._now_ms() + mgr._inflight.record_sent("offZ", "pushed-id", now) + mgr._inflight.record_sent("offZ", "still-flying", now) + mgr._process_received(self._pushed("offZ", "pushed-id")) + reported = [ + c.kwargs.get("message_id") + for c in mock_protocol.internet_send_failed_with_reason.call_args_list + ] + assert reported == ["pushed-id"] + # Still tracked, so a later DeliveryError for the recipient can fail it. + assert mgr._inflight.drain_recipient("offZ", mgr._now_ms()) == ["still-flying"] + + @pytest.mark.parametrize("extra", [{}, {"pushed": False}]) + def test_a_frame_the_relay_did_not_push_is_not_parked( + self, mock_protocol: MagicMock, extra: dict[str, object] + ) -> None: + mgr = InternetManager(mock_protocol, "dev-1") + msg = {"type": "MessageSent", "recipient": "offZ", "message_id": "sent-id", **extra} + mgr._process_received(json.dumps(msg).encode()) + mock_protocol.internet_send_failed_with_reason.assert_not_called() + mock_protocol.internet_peer_presence.assert_not_called() + + def test_a_push_without_an_id_parks_nothing(self, mock_protocol: MagicMock) -> None: + mgr = InternetManager(mock_protocol, "dev-1") + mgr._process_received(self._pushed("offZ", None)) + mock_protocol.internet_send_failed_with_reason.assert_not_called() + + # --------------------------------------------------------------------------- # The activity-adaptive send loop. Every piece of the policy is pinned here: # the drain-count contract, the backoff ramp, the inbound-frame wake, and the diff --git a/bindings/react-native/MeshSdk.podspec b/bindings/react-native/MeshSdk.podspec index 0debb40a..63b64062 100644 --- a/bindings/react-native/MeshSdk.podspec +++ b/bindings/react-native/MeshSdk.podspec @@ -28,6 +28,7 @@ Pod::Spec.new do |s| "ios/BleManager.swift", "ios/InternetManager.swift", "ios/ForcedPresenceCheckQueue.swift", + "ios/InboundEventBuffer.swift", "ios/InboundFragmentBuffer.swift", "ios/OutboundFragmentQueue.swift", "ios/AddressDeclarationPolicy.swift", diff --git a/bindings/react-native/README.md b/bindings/react-native/README.md index 0ba3c1b9..8e8a3f42 100644 --- a/bindings/react-native/README.md +++ b/bindings/react-native/README.md @@ -569,7 +569,7 @@ interface EncryptionConfig { interface PendingQueueConfig { maxPendingPerPeer?: number; // default: 64 maxPendingGlobal?: number; // default: 4096 - pendingTtlMs?: number; // default: 1800000 (30 min) + pendingTtlMs?: number; // default: 86400000 (24 h) overflowPolicy?: 'drop_oldest' | 'drop_newest'; // default: drop_oldest } ``` @@ -605,8 +605,8 @@ interface ReliabilityConfig { pendingMessageMaxLifetimeMs?: number; // default: 604800000 }; dedup?: { - maxTrackedMessages?: number; // default: 1000, must be > 0 - retentionTimeSecs?: number; // default: 3600, must be > 0 + maxTrackedMessages?: number; // default: 2000, must be > 0 + retentionTimeSecs?: number; // default: 86400, must be > 0 }; } ``` diff --git a/bindings/react-native/android/src/main/java/com/offlineprotocol/InternetManager.kt b/bindings/react-native/android/src/main/java/com/offlineprotocol/InternetManager.kt index 66520298..4a125912 100644 --- a/bindings/react-native/android/src/main/java/com/offlineprotocol/InternetManager.kt +++ b/bindings/react-native/android/src/main/java/com/offlineprotocol/InternetManager.kt @@ -1348,6 +1348,29 @@ class InternetManager( ) } + // `pushed: true` is the relay saying it had no live socket for + // the recipient and handed the ciphertext to a push + // notification instead. The relay does not store-and-forward, + // so this is the same fact a DeliveryError carries — the + // recipient is not on the relay right now — with the message + // *possibly* arriving through the push. Report this one id to + // the core as `relay_pushed`, which parks a plain DM as the + // unreachable path would (no ACK budget burnt against an + // offline peer, a reachability probe scheduled, the recipient + // watched) so that if the push is lost the presence edge + // re-drives it. Absent on older relays, which reads as + // `false`. + val pushed = json.optBoolean("pushed", false) + // Park the id the relay echoed, never the tracker's fallback + // guess above. A relay new enough to send `pushed` echoes our + // own id, and the guess (the oldest frame in flight) is least + // reliable exactly here, because push outcomes come back out of + // order. An echo that names none of our frames parks nothing: + // the core ignores an id with no outbox entry. + if (pushed && messageId != null && messageId.isNotEmpty()) { + parkPushedMessage(recipient, messageId) + } + if (messageId != null && messageId.isNotEmpty()) { // The server has confirmed the message was sent with this message_id // We need to notify the protocol so it can update the message ID @@ -1355,7 +1378,8 @@ class InternetManager( emitDiagnostic("debug", "MessageSent from relay server", mapOf( "messageId" to messageId, "recipient" to recipient, - "timestamp" to timestamp + "timestamp" to timestamp, + "pushed" to pushed )) // Note: The protocol SDK will handle the message_sent event internally // The frontend will receive it via the normal event stream @@ -2219,6 +2243,46 @@ class InternetManager( )) } + /** + * Parks one message the relay reports as `MessageSent { pushed: true }`. + * + * The narrower sibling of [handleRecipientUnreachable]: the relay named + * exactly one frame, so only that id is reported to the core and the + * recipient's other in-flight frames are left alone — a later + * `MessageSent` for each of them says what became of it. The reason is + * the exact `relay_pushed` token, not a `recipient_unreachable` tail: that + * prefix fast-fails connection requests and fails Welcomes, while + * `relay_pushed` parks a plain DM and leaves both of those alone, since + * the push may have delivered them. The recipient is presence-watched + * and an offline presence is fed to the core exactly as the DeliveryError + * path does, so the presence-online edge is what re-drives the parked + * message if the push never reaches the device. + */ + private fun parkPushedMessage(recipient: String, messageId: String) { + // Sentinel entries track app-authored raw SendMessage frames; their + // outcomes belong to the app, not the core (see handleRecipientUnreachable). + if (messageId.isEmpty() || messageId.startsWith(RAW_SEND_SENTINEL_PREFIX)) return + try { + protocol.internetSendFailedWithReason(messageId, "relay_pushed") + } catch (e: Exception) { + Log.e(TAG, "Failed to park pushed message $messageId", e) + } + // Same self guard as handleRecipientUnreachable: never watch self and + // never feed "self is offline" into the core. + if (recipient.isNotEmpty() && !isSelfPeer(recipient)) { + presenceWatch.watch(recipient, monotonicNowMs()) + try { + protocol.internetPeerPresence(recipient, false, null) + } catch (e: Exception) { + Log.e(TAG, "Failed to ingest offline presence for $recipient", e) + } + } + emitDiagnostic("info", "Relay pushed message to offline recipient; parked", mapOf( + "recipient" to recipient, + "messageId" to messageId + )) + } + /** * True when [peerId] names this device in either namespace — the profile * (relay username by convention) or the derived address the core stamps diff --git a/bindings/react-native/android/src/main/java/com/offlineprotocol/OfflineProtocolModule.kt b/bindings/react-native/android/src/main/java/com/offlineprotocol/OfflineProtocolModule.kt index 5b3afc6c..0a6dcd29 100644 --- a/bindings/react-native/android/src/main/java/com/offlineprotocol/OfflineProtocolModule.kt +++ b/bindings/react-native/android/src/main/java/com/offlineprotocol/OfflineProtocolModule.kt @@ -113,6 +113,33 @@ class OfflineProtocolModule(reactContext: ReactApplicationContext) : }, ) + /** + * Redelivers *inbound message* events that could not be handed to JS — + * `message_received`, `file_received`, `message_decryption_failed` — on + * the next subscribe or foreground. See [BUFFERED_INBOUND_EVENT_TYPES] for + * why these three qualify and [holdInboundEventIfBuffered] for the gate. + * + * A second [StickyEventDispatcher] rather than more keys in [stickyEvents] + * because the two hold different things. A one-shot event collapses per + * type and the buffer is sized for two keys; an inbound event is one + * message the core has already ACKed and will never restate, so every one + * must survive on its own key (`type:message_id`) and the cap has to be a + * real capacity — 256, the oldest dropped past it — rather than a backstop. + */ + private val inboundEvents = StickyEventDispatcher( + buffer = StickyEventBuffer(maxEntries = INBOUND_EVENT_BUFFER_CAPACITY), + canEmit = { canEmitToJs() }, + emit = { eventJson -> sendEvent(EVENT_NAME, eventParams(eventJson)) }, + schedule = { runnable -> + try { + reactApplicationContext.runOnJSQueueThread(runnable) + } catch (e: Exception) { + android.util.Log.w(NAME, "Could not schedule inbound event flush", e) + false + } + }, + ) + /** * The started activities, which is what tells the process apart from an * activity, and the watcher that maintains them. Held by identity rather @@ -200,6 +227,26 @@ class OfflineProtocolModule(reactContext: ReactApplicationContext) : /** How long `destroy` waits for an in-flight process tick to finish. */ private const val PROCESS_SHUTDOWN_TIMEOUT_MS = 2_000L + + /** + * The inbound event tags [inboundEvents] holds when JS cannot take + * them. Each reports one message the core has already ACKed, + * dedup-marked and dropped its queued copy of — the sender will not + * resend and nothing will restate it — so a drop here is a lost + * message, and the drop is ordinary: the React instance is down while + * the app is backgrounded, or a push injection lands before JS has + * subscribed. Must match `BUFFERED_INBOUND_EVENT_TYPES` in + * `src/constants.ts` and `InboundEventBuffer.bufferedEventTypes` on + * iOS; pinned by `react_native_buffered_inbound_event_set_matches_native`. + */ + private val BUFFERED_INBOUND_EVENT_TYPES: Set = setOf( + "message_received", + "file_received", + "message_decryption_failed", + ) + + /** Inbound events held at most; the oldest is dropped past it. */ + private const val INBOUND_EVENT_BUFFER_CAPACITY = 256 } private object Constants { @@ -688,10 +735,13 @@ class OfflineProtocolModule(reactContext: ReactApplicationContext) : // Set up event callback proto.setEventCallback(object : EventCallback { override fun onEvent(eventJson: String) { - val params = Arguments.createMap().apply { - putString("eventJson", eventJson) + // Held on the emit's own answer rather than on a gate read + // taken before it, so an emit that fails with the gate open + // is held too, as on iOS. The type/id parse that decides + // whether to hold still runs only after a refusal. + if (!sendEvent(EVENT_NAME, eventParams(eventJson))) { + holdInboundEventIfBuffered(eventJson) } - sendEvent(EVENT_NAME, params) } }) @@ -935,11 +985,44 @@ class OfflineProtocolModule(reactContext: ReactApplicationContext) : stickyEvents.send(key, eventJson) } - /** Redelivers held one-shot events, if JS looks able to take them now. */ + /** + * Redelivers held one-shot and inbound events, if JS looks able to take + * them now. + */ private fun flushStickyEvents() { stickyEvents.flush() + inboundEvents.flush() } + /** + * Holds an inbound message event for redelivery when JS could not take + * it, keyed `type:message_id` so every message survives on its own. + * Returns whether the event was one of [BUFFERED_INBOUND_EVENT_TYPES] and + * was handed to [inboundEvents]; anything else stays dropped, as before. + * + * Only called after [sendEvent] refused the event, so the parse here is + * off the hot path. An event of a buffered type that carries no id at + * all is keyed by arrival order rather than dropped: losing a message to + * a missing field would be the failure this buffer exists to prevent. + */ + private fun holdInboundEventIfBuffered(eventJson: String): Boolean { + val json = try { + JSONObject(eventJson) + } catch (e: Exception) { + return false + } + val type = json.optString("type", "") + if (type !in BUFFERED_INBOUND_EVENT_TYPES) return false + val id = json.optString("message_id", "") + .ifEmpty { json.optString("file_id", "") } + .ifEmpty { "seq-${inboundEventSequence.incrementAndGet()}" } + inboundEvents.send("$type:$id", eventJson) + return true + } + + /** Fallback key source for a buffered inbound event that carries no id. */ + private val inboundEventSequence = AtomicInteger(0) + private fun emitDiagnostic(level: String, message: String, context: Map = emptyMap()) { try { val json = JSONObject() @@ -1067,6 +1150,10 @@ class OfflineProtocolModule(reactContext: ReactApplicationContext) : // that closes it. Collapsing the two is how an event emitted by the // session destroy() just ended ends up held for the next one. stickyEvents.beginSession() + // The inbound buffer follows the same session edges: nothing + // buffered can be produced before this start(), and a message held + // for the previous session belongs to whatever identity ran it. + inboundEvents.beginSession() emitDiagnostic("info", "Starting protocol") protocol?.start() emitDiagnostic("info", "Protocol core started") @@ -2261,6 +2348,7 @@ class OfflineProtocolModule(reactContext: ReactApplicationContext) : // exotic one — and would otherwise be held under the new generation // for whichever session subscribes next. stickyEvents.endSession() + inboundEvents.endSession() currentConfig = null promise.resolve(null) } catch (e: Exception) { @@ -3380,8 +3468,8 @@ class OfflineProtocolModule(reactContext: ReactApplicationContext) : try { val json = JSONObject(configJson) val dedupConfig = DedupConfig( - maxTrackedMessages = json.optLong("maxTrackedMessages", 10000).toULong(), - retentionTimeSecs = json.optLong("retentionTimeSecs", 3600).toULong() + maxTrackedMessages = json.optLong("maxTrackedMessages", 2000).toULong(), + retentionTimeSecs = json.optLong("retentionTimeSecs", 86400).toULong() ) protocol?.updateDedupConfig(dedupConfig) diff --git a/bindings/react-native/android/src/main/java/com/offlineprotocol/ProtocolConfigParser.kt b/bindings/react-native/android/src/main/java/com/offlineprotocol/ProtocolConfigParser.kt index 3d06edb1..66a8ebf5 100644 --- a/bindings/react-native/android/src/main/java/com/offlineprotocol/ProtocolConfigParser.kt +++ b/bindings/react-native/android/src/main/java/com/offlineprotocol/ProtocolConfigParser.kt @@ -109,7 +109,7 @@ internal object ProtocolConfigParser { val pendingTtlMs = pendingQueueJson?.optLongCompat( "pendingTtlMs", "pending_ttl_ms" - ) ?: json.optLongCompat("pendingTtlMs", "pending_ttl_ms") ?: 1_800_000L + ) ?: json.optLongCompat("pendingTtlMs", "pending_ttl_ms") ?: 86_400_000L val overflowPolicyRaw = pendingQueueJson?.optStringCompat( "overflowPolicy", "overflow_policy" diff --git a/bindings/react-native/android/src/test/java/com/offlineprotocol/ProtocolConfigParserTest.kt b/bindings/react-native/android/src/test/java/com/offlineprotocol/ProtocolConfigParserTest.kt index 35a0c03c..5ca62f41 100644 --- a/bindings/react-native/android/src/test/java/com/offlineprotocol/ProtocolConfigParserTest.kt +++ b/bindings/react-native/android/src/test/java/com/offlineprotocol/ProtocolConfigParserTest.kt @@ -146,11 +146,11 @@ class ProtocolConfigParserTest { @Test fun pendingTtlFallsBackToTheRustDefault() { - // Mirrors DEFAULT_PENDING_TTL_MS (30 min); the iOS reader asserts the + // Mirrors DEFAULT_PENDING_TTL_MS (24 h); the iOS reader asserts the // same, and `rn_bridge_pending_ttl_fallbacks_match_rust_default` pins // all three bridge literals to the Rust constant. val config = parse("""{"appId":"app","userId":"alice"}""") - assertEquals(1_800_000L, config.pendingTtlMs.toLong()) + assertEquals(86_400_000L, config.pendingTtlMs.toLong()) } @Test diff --git a/bindings/react-native/android/src/test/java/com/offlineprotocol/StickyEventBufferTest.kt b/bindings/react-native/android/src/test/java/com/offlineprotocol/StickyEventBufferTest.kt index 75130911..647a750a 100644 --- a/bindings/react-native/android/src/test/java/com/offlineprotocol/StickyEventBufferTest.kt +++ b/bindings/react-native/android/src/test/java/com/offlineprotocol/StickyEventBufferTest.kt @@ -34,6 +34,44 @@ class StickyEventBufferTest { assertEquals("""{"type":"mesh_stopped_by_user"}""", drained[0].eventJson) } + @Test + fun twoInboundMessagesWithDifferentIdsBothSurvive() { + // The inbound use: every message_received is its own message, keyed + // `type:message_id`, so two of them must not collapse into one the + // way two copies of a one-shot event do. Sized as the module sizes + // its inbound buffer. + val buffer = StickyEventBuffer(maxEntries = 256) + assertTrue(buffer.holdNow("message_received:m1", """{"type":"message_received","message_id":"m1"}""")) + assertTrue(buffer.holdNow("message_received:m2", """{"type":"message_received","message_id":"m2"}""")) + + assertEquals(2, buffer.size) + val drained = buffer.drain() + assertEquals(listOf("message_received:m1", "message_received:m2"), drained.map { it.key }) + assertEquals( + listOf( + """{"type":"message_received","message_id":"m1"}""", + """{"type":"message_received","message_id":"m2"}""", + ), + drained.map { it.eventJson }, + ) + } + + @Test + fun inboundBufferDropsTheOldestPastItsCapacity() { + // 256 is a real capacity for the inbound buffer, not a backstop: past + // it the oldest message goes, never the newest. + val buffer = StickyEventBuffer(maxEntries = 256) + for (i in 0 until 257) { + buffer.holdNow("message_received:m$i", "{}") + } + + assertEquals(256, buffer.size) + val keys = buffer.drain().map { it.key } + assertFalse(keys.contains("message_received:m0")) + assertEquals("message_received:m1", keys.first()) + assertEquals("message_received:m256", keys.last()) + } + @Test fun holdReportsThatItTookTheEvent() { // The caller flushes on a true return, so a false one here would be a diff --git a/bindings/react-native/ios/EncryptionConfigReader.swift b/bindings/react-native/ios/EncryptionConfigReader.swift index cd73514a..9910c9ae 100644 --- a/bindings/react-native/ios/EncryptionConfigReader.swift +++ b/bindings/react-native/ios/EncryptionConfigReader.swift @@ -70,7 +70,7 @@ enum EncryptionConfigReader { ?? 4096 let pendingTtlMs = number(pendingQueue, "pendingTtlMs", "pending_ttl_ms") ?? number(raw, "pendingTtlMs", "pending_ttl_ms") - ?? 1_800_000 + ?? 86_400_000 let overflowPolicyRaw = string(pendingQueue, "overflowPolicy", "overflow_policy") ?? string(raw, "overflowPolicy", "overflow_policy") ?? "drop_oldest" diff --git a/bindings/react-native/ios/InboundEventBuffer.swift b/bindings/react-native/ios/InboundEventBuffer.swift new file mode 100644 index 00000000..2533f091 --- /dev/null +++ b/bindings/react-native/ios/InboundEventBuffer.swift @@ -0,0 +1,161 @@ +import Foundation + +/// Holds inbound message events that JavaScript could not take, so a message +/// the core has already acknowledged survives a window where nothing was +/// listening. +/// +/// By the time `message_received`, `file_received` or the +/// `message_decryption_failed` that stands in for one is emitted, the core has +/// sent the delivery ACK, dedup-marked the id and dropped its queued copy: the +/// sender will not resend, and nothing in the SDK will restate the event. A +/// drop between the core and the app is therefore a lost message — and the +/// drop is ordinary. The React instance is down while the app is backgrounded, +/// or a push injection delivers the ciphertext before JS has subscribed. The +/// one-shot events on this bridge are re-derived from a latch on foreground +/// (`restateInternetSupersededIfNeeded`); an inbound message has no latch to +/// re-derive from, so it has to be held. +/// +/// Unlike the one-shot set these do not collapse per type — every message is +/// its own fact — so entries are keyed `type:message_id`, kept FIFO, and the +/// cap is a real capacity (256, oldest dropped past it) rather than a backstop. +/// Re-holding an existing key replaces it in place, which is what makes a +/// flush that fails and re-holds idempotent. +/// +/// **Every entry is stamped with the session generation it was emitted for.** +/// `bumpGeneration()` runs on `create()` and `destroy()`; `hold` and `restore` +/// refuse entries from any other generation, so a message held for the +/// account the app just tore down cannot surface in whatever it constructs +/// next, and a flush already carrying entries across a teardown puts nothing +/// back. Mirrors the Android `StickyEventBuffer` session stamp. +/// +/// Lock-protected because writers and readers are different threads: holds +/// arrive from the core's event thread, flushes from the main queue. Values are +/// the event JSON, not a React payload, so the class is Foundation-only and the +/// SwiftPM harness covers it. +final class InboundEventBuffer { + + /// A held event: its `type:message_id` key, the JSON to re-emit, and the + /// session generation it was emitted for. + struct Entry: Equatable { + let key: String + let eventJson: String + let generation: Int + } + + /// The event tags this buffer holds. Must match + /// `BUFFERED_INBOUND_EVENT_TYPES` in `src/constants.ts` and + /// `OfflineProtocolModule.BUFFERED_INBOUND_EVENT_TYPES` on Android; pinned + /// by `react_native_buffered_inbound_event_set_matches_native`. + static let bufferedEventTypes: Set = [ + "message_received", + "file_received", + "message_decryption_failed", + ] + + /// Most events held; the oldest is dropped past it. + static let defaultMaxEntries = 256 + + private let lock = NSLock() + private var held: [Entry] = [] + private var generation = 0 + private var sequence = 0 + private let maxEntries: Int + + init(maxEntries: Int = InboundEventBuffer.defaultMaxEntries) { + self.maxEntries = maxEntries + } + + /// The session events are currently being held for. Read before the emit + /// is attempted and passed back to `hold`, so an emit that fails against a + /// session torn down while it was in flight leaves nothing behind. + func currentGeneration() -> Int { + lock.lock(); defer { lock.unlock() } + return generation + } + + /// The `type:message_id` key for an event, or nil when the event is not of + /// a buffered type. An event of a buffered type that carries no id at all + /// is keyed by arrival order rather than dropped — losing a message to a + /// missing field would be the failure this buffer exists to prevent. + func key(forEventJson eventJson: String) -> String? { + guard let data = eventJson.data(using: .utf8), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let type = json["type"] as? String, + InboundEventBuffer.bufferedEventTypes.contains(type) else { + return nil + } + if let id = json["message_id"] as? String, !id.isEmpty { + return "\(type):\(id)" + } + if let id = json["file_id"] as? String, !id.isEmpty { + return "\(type):\(id)" + } + lock.lock(); defer { lock.unlock() } + sequence += 1 + return "\(type):seq-\(sequence)" + } + + /// Holds `eventJson` under `key`, replacing an entry already held under + /// it. Returns false — and holds nothing — when `generation` is no longer + /// current. Past the cap the oldest entry is evicted. + @discardableResult + func hold(key: String, eventJson: String, generation: Int) -> Bool { + lock.lock(); defer { lock.unlock() } + guard generation == self.generation else { return false } + if let index = held.firstIndex(where: { $0.key == key }) { + held[index] = Entry(key: key, eventJson: eventJson, generation: generation) + } else { + held.append(Entry(key: key, eventJson: eventJson, generation: generation)) + trimToCapLocked() + } + return true + } + + /// Removes and returns everything held, oldest first. The caller emits and + /// hands back whatever JS refused through `restore`. + func drain() -> [Entry] { + lock.lock(); defer { lock.unlock() } + let drained = held + held.removeAll() + return drained + } + + /// Puts back entries a flush could not deliver, at the head so arrival + /// order holds, skipping any from a superseded generation or whose key has + /// been held again in the meantime. + func restore(_ entries: [Entry]) { + guard !entries.isEmpty else { return } + lock.lock(); defer { lock.unlock() } + let restorable = entries.filter { entry in + entry.generation == generation && !held.contains(where: { $0.key == entry.key }) + } + guard !restorable.isEmpty else { return } + held = restorable + held + trimToCapLocked() + } + + /// Starts a new session: refuses everything in flight for the old one and + /// discards what it held. Called on `create()` and `destroy()`. + func bumpGeneration() { + lock.lock(); defer { lock.unlock() } + generation += 1 + held.removeAll() + } + + var isEmpty: Bool { + lock.lock(); defer { lock.unlock() } + return held.isEmpty + } + + var count: Int { + lock.lock(); defer { lock.unlock() } + return held.count + } + + /// Caller must hold `lock`. + private func trimToCapLocked() { + if held.count > maxEntries { + held.removeFirst(held.count - maxEntries) + } + } +} diff --git a/bindings/react-native/ios/InternetManager.swift b/bindings/react-native/ios/InternetManager.swift index dad800e7..4a22849b 100644 --- a/bindings/react-native/ios/InternetManager.swift +++ b/bindings/react-native/ios/InternetManager.swift @@ -1643,6 +1643,28 @@ public class InternetManager: NSObject, TransportManager { ) } + // `pushed: true` is the relay saying it had no live socket for the + // recipient and handed the ciphertext to a push notification + // instead. The relay does not store-and-forward, so this is the + // same fact a DeliveryError carries — the recipient is not on the + // relay right now — with the message *possibly* arriving through + // the push. Report this one id to the core as `relay_pushed`, + // which parks a plain DM as the unreachable path would (no ACK + // budget burnt against an offline peer, a reachability probe + // scheduled, the recipient watched) so that if the push is lost + // the presence edge re-drives it. Absent on older relays, which + // reads as `false`. + let pushed = json["pushed"] as? Bool ?? false + // Park the id the relay echoed, never the tracker's fallback guess + // above. A relay new enough to send `pushed` echoes our own id, and + // the guess (the oldest frame in flight) is least reliable exactly + // here, because push outcomes come back out of order. An echo that + // names none of our frames parks nothing: the core ignores an id + // with no outbox entry. + if pushed, let messageId = sentMessageId, !messageId.isEmpty { + parkPushedMessage(recipient: sentRecipient, messageId: messageId) + } + if let messageId = sentMessageId, !messageId.isEmpty { let timestamp = json["timestamp"] as? String ?? "" @@ -1652,7 +1674,8 @@ public class InternetManager: NSObject, TransportManager { emitDiagnostic("debug", "MessageSent from relay server", context: [ "messageId": messageId, "recipient": sentRecipient, - "timestamp": timestamp + "timestamp": timestamp, + "pushed": pushed ]) // Note: The protocol SDK will handle the message_sent event internally // The frontend will receive it via the normal event stream @@ -2721,6 +2744,39 @@ public class InternetManager: NSObject, TransportManager { ]) } + /// Parks one message the relay reports as `MessageSent { pushed: true }`. + /// + /// The narrower sibling of `handleRecipientUnreachable`: the relay named + /// exactly one frame, so only that id is reported to the core and the + /// recipient's other in-flight frames are left alone — a later + /// `MessageSent` for each of them says what became of it. The reason is + /// the exact `relay_pushed` token, not a `recipient_unreachable` tail: that + /// prefix fast-fails connection requests and fails Welcomes, while + /// `relay_pushed` parks a plain DM and leaves both of those alone, since + /// the push may have delivered them. The recipient is presence-watched + /// and an offline presence is fed to the core exactly as the DeliveryError + /// path does, so the presence-online edge is what re-drives the parked + /// message if the push never reaches the device. + private func parkPushedMessage(recipient: String, messageId: String) { + // Sentinel entries track app-authored raw SendMessage frames; their + // outcomes belong to the app, not the core (see handleRecipientUnreachable). + if messageId.isEmpty || messageId.hasPrefix(Self.rawSendSentinelPrefix) { return } + protocolInstance.internetSendFailedWithReason( + messageId: messageId, + reason: "relay_pushed" + ) + // Same self guard as handleRecipientUnreachable: never watch self and + // never feed "self is offline" into the core. + if !recipient.isEmpty && !isSelfPeer(recipient) { + presenceWatch.watch(recipient, nowMs: monotonicNowMs()) + protocolInstance.internetPeerPresence(peerId: recipient, online: false, lastSeenMs: nil) + } + emitDiagnostic("info", "Relay pushed message to offline recipient; parked", context: [ + "recipient": recipient, + "messageId": messageId + ]) + } + /// True when `peerId` names this device in either namespace — the profile /// (relay username by convention) or the derived address the core stamps /// on its own frames. Peer ids reaching the presence plane come from both diff --git a/bindings/react-native/ios/OfflineProtocolModule.swift b/bindings/react-native/ios/OfflineProtocolModule.swift index 13683c8a..9590fa57 100644 --- a/bindings/react-native/ios/OfflineProtocolModule.swift +++ b/bindings/react-native/ios/OfflineProtocolModule.swift @@ -207,11 +207,59 @@ class OfflineProtocolModule: RCTEventEmitter { name: UIApplication.willEnterForegroundNotification, object: nil ) + NotificationCenter.default.addObserver( + self, + selector: #selector(applicationDidBecomeActive), + name: UIApplication.didBecomeActiveNotification, + object: nil + ) } private func removeBackgroundObservers() { NotificationCenter.default.removeObserver(self, name: UIApplication.didEnterBackgroundNotification, object: nil) NotificationCenter.default.removeObserver(self, name: UIApplication.willEnterForegroundNotification, object: nil) + NotificationCenter.default.removeObserver(self, name: UIApplication.didBecomeActiveNotification, object: nil) + } + + /// The foreground flush trigger for held inbound events, besides a + /// subscribe. An app whose listeners never went away still needs one: a + /// message held because the React instance was briefly down would + /// otherwise wait for a resubscribe that is never coming. Mirrors the + /// Android `onHostResume` flush. + @objc private func applicationDidBecomeActive() { + flushInboundEvents() + } + + // MARK: - Inbound event hold + + /// Redelivers inbound message events that could not be handed to JS. See + /// `InboundEventBuffer` for what is held and why, and + /// `EventCallbackImpl.onEvent` for the gate. + fileprivate let inboundEvents = InboundEventBuffer() + + /// Holds an inbound event JS could not take, if it is of a buffered type. + /// Called only after `sendEventToJS` has refused it, so the parse is off + /// the hot path. Main-thread confined like the emit it follows. + fileprivate func holdInboundEventIfBuffered(_ eventJson: String, generation: Int) { + guard let key = inboundEvents.key(forEventJson: eventJson) else { return } + inboundEvents.hold(key: key, eventJson: eventJson, generation: generation) + } + + /// Redelivers held inbound events, oldest first, if JS looks able to take + /// them now; whatever it refuses goes back. Main-thread confined: every + /// caller is either on main already (`startObserving`, the notification) + /// or hops there first (`addListener`). + fileprivate func flushInboundEvents() { + guard canEmitToJs, !inboundEvents.isEmpty else { return } + let drained = inboundEvents.drain() + var undelivered: [InboundEventBuffer.Entry] = [] + for entry in drained { + guard entry.generation == inboundEvents.currentGeneration() else { continue } + if !sendEventToJS(Events.onEvent, body: ["eventJson": entry.eventJson]) { + undelivered.append(entry) + } + } + inboundEvents.restore(undelivered) } /// Gates the foreground relay reconnect on how long the app actually stayed @@ -398,6 +446,10 @@ class OfflineProtocolModule: RCTEventEmitter { override func startObserving() { hasListeners = true + // A subscription is the moment a held inbound event becomes + // deliverable. Deferred so it runs after RN has finished registering + // the listener that triggered it. + DispatchQueue.main.async { [weak self] in self?.flushInboundEvents() } } override func stopObserving() { @@ -406,6 +458,10 @@ class OfflineProtocolModule: RCTEventEmitter { @objc override func addListener(_ eventName: String) { super.addListener(eventName) + // The other subscribe-shaped trigger: `startObserving` fires only on + // the first listener, and a message held while an app briefly had + // none is collected by the next one it adds. + DispatchQueue.main.async { [weak self] in self?.flushInboundEvents() } } @objc override func removeListeners(_ count: Double) { @@ -794,6 +850,9 @@ class OfflineProtocolModule: RCTEventEmitter { print("[OfflineProtocolModule] Creating OfflineProtocol instance...") let proto = try OfflineProtocol(config: config) print("[OfflineProtocolModule] OfflineProtocol instance created successfully") + // A new session: nothing held for the previous one may surface in + // this one, and an emit still in flight for it is refused. + inboundEvents.bumpGeneration() currentConfig = config emitDiagnostic(level: "info", message: "Protocol core created", context: [ "appId": config.appId, @@ -2481,6 +2540,9 @@ class OfflineProtocolModule: RCTEventEmitter { protocolInstance = nil meshServicesInstance = nil currentConfig = nil + // The app tore the SDK down itself; a message held for this account + // must not reach whatever it constructs next. + inboundEvents.bumpGeneration() resolver(nil) } @@ -3615,8 +3677,8 @@ class OfflineProtocolModule: RCTEventEmitter { } let dedupConfig = DedupConfig( - maxTrackedMessages: (config["maxTrackedMessages"] as? NSNumber)?.uint64Value ?? 10000, - retentionTimeSecs: (config["retentionTimeSecs"] as? NSNumber)?.uint64Value ?? 3600 + maxTrackedMessages: (config["maxTrackedMessages"] as? NSNumber)?.uint64Value ?? 2000, + retentionTimeSecs: (config["retentionTimeSecs"] as? NSNumber)?.uint64Value ?? 86400 ) try proto.updateDedupConfig(config: dedupConfig) @@ -5351,13 +5413,22 @@ class EventCallbackImpl: EventCallback, @unchecked Sendable { func onEvent(eventJson: String) { guard let emitter = emitter else { return } + // Read before the emit is attempted, so a hold after a failed emit + // cannot land in a session that began in between. + let generation = emitter.inboundEvents.currentGeneration() let body: [String: Any] = ["eventJson": eventJson] + let deliver = { + if !emitter.sendEventToJS(OfflineProtocolModule.Events.onEvent, body: body) { + // JS could not take it. Inbound message events are held for + // the next subscribe or foreground; everything else is + // periodic or re-derived and is dropped as before. + emitter.holdInboundEventIfBuffered(eventJson, generation: generation) + } + } if Thread.isMainThread { - emitter.sendEventToJS(OfflineProtocolModule.Events.onEvent, body: body) + deliver() } else { - DispatchQueue.main.async { - emitter.sendEventToJS(OfflineProtocolModule.Events.onEvent, body: body) - } + DispatchQueue.main.async(execute: deliver) } } } diff --git a/bindings/react-native/ios/Package.swift b/bindings/react-native/ios/Package.swift index 6dceaa05..1c6a09f3 100644 --- a/bindings/react-native/ios/Package.swift +++ b/bindings/react-native/ios/Package.swift @@ -73,6 +73,7 @@ let package = Package( "PeerIdentityBinding.swift", "GatewayAttachPolicy.swift", "GatewayVerdictTracker.swift", + "InboundEventBuffer.swift", "PeripheralRestorationAgeOutPolicy.swift", "PresenceWatchPolicy.swift", "ProtocolStateStorage.swift", @@ -107,6 +108,7 @@ let package = Package( "ForcedPresenceCheckQueueTests.swift", "GatewayAttachPolicyTests.swift", "GatewayVerdictTrackerTests.swift", + "InboundEventBufferTests.swift", "ForegroundReconnectPolicyTests.swift", "InboundFragmentBufferTests.swift", "LegacyRelayMessageTests.swift", diff --git a/bindings/react-native/ios/tests/EncryptionConfigReaderTests.swift b/bindings/react-native/ios/tests/EncryptionConfigReaderTests.swift index 15fde81e..f0912697 100644 --- a/bindings/react-native/ios/tests/EncryptionConfigReaderTests.swift +++ b/bindings/react-native/ios/tests/EncryptionConfigReaderTests.swift @@ -27,9 +27,9 @@ final class EncryptionConfigReaderTests: XCTestCase { XCTAssertTrue(values.cryptoRecoveryEnabled) XCTAssertEqual(values.maxPendingPerPeer, 64) XCTAssertEqual(values.maxPendingGlobal, 4096) - // Mirrors DEFAULT_PENDING_TTL_MS (30 min). Pinned on the Rust side too + // Mirrors DEFAULT_PENDING_TTL_MS (24 h). Pinned on the Rust side too // by `rn_bridge_pending_ttl_fallbacks_match_rust_default`. - XCTAssertEqual(values.pendingTtlMs, 1_800_000) + XCTAssertEqual(values.pendingTtlMs, 86_400_000) XCTAssertEqual(values.overflowPolicyRaw, "drop_oldest") } diff --git a/bindings/react-native/ios/tests/InboundEventBufferTests.swift b/bindings/react-native/ios/tests/InboundEventBufferTests.swift new file mode 100644 index 00000000..1ba176e7 --- /dev/null +++ b/bindings/react-native/ios/tests/InboundEventBufferTests.swift @@ -0,0 +1,106 @@ +import XCTest +@testable import OfflineProtocol + +/// Pins the semantics the inbound message hold depends on: every message +/// survives on its own key, arrival order holds, the cap is a real capacity, +/// and a session edge refuses what was in flight for the old one. +final class InboundEventBufferTests: XCTestCase { + + func testTwoMessagesWithDifferentIdsBothSurvive() { + let buffer = InboundEventBuffer() + let generation = buffer.currentGeneration() + XCTAssertTrue(buffer.hold(key: "message_received:m1", eventJson: "{\"message_id\":\"m1\"}", generation: generation)) + XCTAssertTrue(buffer.hold(key: "message_received:m2", eventJson: "{\"message_id\":\"m2\"}", generation: generation)) + + let drained = buffer.drain() + XCTAssertEqual(drained.map { $0.key }, ["message_received:m1", "message_received:m2"]) + XCTAssertEqual(drained.map { $0.eventJson }, ["{\"message_id\":\"m1\"}", "{\"message_id\":\"m2\"}"]) + XCTAssertTrue(buffer.isEmpty) + } + + func testReholdingAKeyReplacesInPlace() { + let buffer = InboundEventBuffer() + let generation = buffer.currentGeneration() + buffer.hold(key: "message_received:m1", eventJson: "old", generation: generation) + buffer.hold(key: "message_received:m2", eventJson: "{}", generation: generation) + buffer.hold(key: "message_received:m1", eventJson: "new", generation: generation) + + let drained = buffer.drain() + XCTAssertEqual(drained.map { $0.key }, ["message_received:m1", "message_received:m2"]) + XCTAssertEqual(drained[0].eventJson, "new") + } + + func testCapDropsTheOldest() { + let buffer = InboundEventBuffer(maxEntries: 256) + let generation = buffer.currentGeneration() + for i in 0..<257 { + buffer.hold(key: "message_received:m\(i)", eventJson: "{}", generation: generation) + } + XCTAssertEqual(buffer.count, 256) + let keys = buffer.drain().map { $0.key } + XCTAssertEqual(keys.first, "message_received:m1") + XCTAssertEqual(keys.last, "message_received:m256") + } + + func testRestorePutsUndeliveredBackAtTheHeadAndSkipsRelatchedKeys() { + let buffer = InboundEventBuffer() + let generation = buffer.currentGeneration() + buffer.hold(key: "message_received:m1", eventJson: "{}", generation: generation) + buffer.hold(key: "message_received:m2", eventJson: "{}", generation: generation) + let drained = buffer.drain() + buffer.hold(key: "message_received:m2", eventJson: "newer", generation: generation) + buffer.hold(key: "message_received:m3", eventJson: "{}", generation: generation) + + buffer.restore(drained) + + let keys = buffer.drain() + XCTAssertEqual(keys.map { $0.key }, ["message_received:m1", "message_received:m2", "message_received:m3"]) + XCTAssertEqual(keys[1].eventJson, "newer", "a key held again keeps the newer copy") + } + + func testGenerationBumpRefusesInFlightHoldsAndRestores() { + let buffer = InboundEventBuffer() + let stale = buffer.currentGeneration() + buffer.hold(key: "message_received:m1", eventJson: "{}", generation: stale) + let drained = buffer.drain() + + buffer.bumpGeneration() + + XCTAssertFalse(buffer.hold(key: "message_received:m2", eventJson: "{}", generation: stale)) + buffer.restore(drained) + XCTAssertTrue(buffer.isEmpty, "nothing from the old session may reach the new one") + XCTAssertTrue(buffer.hold(key: "message_received:m3", eventJson: "{}", generation: buffer.currentGeneration())) + } + + func testKeyIsDerivedFromTypeAndMessageId() { + let buffer = InboundEventBuffer() + XCTAssertEqual( + buffer.key(forEventJson: "{\"type\":\"message_received\",\"message_id\":\"abc\"}"), + "message_received:abc" + ) + XCTAssertEqual( + buffer.key(forEventJson: "{\"type\":\"file_received\",\"file_id\":\"f1\"}"), + "file_received:f1" + ) + XCTAssertEqual( + buffer.key(forEventJson: "{\"type\":\"message_decryption_failed\",\"message_id\":\"d1\"}"), + "message_decryption_failed:d1" + ) + XCTAssertNil(buffer.key(forEventJson: "{\"type\":\"internet_status_changed\"}"), + "a periodic event is never held") + XCTAssertNil(buffer.key(forEventJson: "not json")) + // No id at all: keyed by arrival rather than dropped. + let a = buffer.key(forEventJson: "{\"type\":\"message_received\"}") + let b = buffer.key(forEventJson: "{\"type\":\"message_received\"}") + XCTAssertNotNil(a) + XCTAssertNotEqual(a, b) + } + + func testBufferedTypesMatchTheContract() { + XCTAssertEqual( + InboundEventBuffer.bufferedEventTypes, + ["message_received", "file_received", "message_decryption_failed"] + ) + XCTAssertEqual(InboundEventBuffer.defaultMaxEntries, 256) + } +} diff --git a/bindings/react-native/js-ci-harness/one-shot-hold.test.js b/bindings/react-native/js-ci-harness/one-shot-hold.test.js index d1a6f620..18fcb0e4 100644 --- a/bindings/react-native/js-ci-harness/one-shot-hold.test.js +++ b/bindings/react-native/js-ci-harness/one-shot-hold.test.js @@ -1,6 +1,7 @@ #!/usr/bin/env node /** - * Behavioral tests for the JS-layer one-shot event hold (`src/index.ts`). + * Behavioral tests for the JS-layer one-shot event hold and the inbound + * message hold (`src/index.ts`). * * Drives the *real compiled* `OfflineProtocol` class against a stubbed native * module, so these assert what the code does rather than what it says. The @@ -407,6 +408,92 @@ test('no drop warning while the app has any listener registered', async () => { ); }); +// --------------------------------------------------------------------------- +// The inbound hold: messages that arrive before the app has a listener +// --------------------------------------------------------------------------- + +const RECEIVED_1 = { type: 'message_received', timestamp: 1, message_id: 'm1', sender: 'bob', content: 'one' }; +const RECEIVED_2 = { type: 'message_received', timestamp: 2, message_id: 'm2', sender: 'bob', content: 'two' }; +const DECRYPT_FAILED = { type: 'message_decryption_failed', timestamp: 3, message_id: 'm3', sender: 'bob', code: 'PENDING_QUEUE_DROPPED', reason: 'x' }; + +test("a message_received before on('message_received') is replayed exactly once", async () => { + const { sdk, bus } = newSdk(); + bus.emit(RECEIVED_1); + await settle(); + const seen = []; + sdk.on('message_received', (event) => seen.push(event)); + await settle(); + await settle(); + assert.deepEqual(seen.map((event) => event.message_id), ['m1']); +}); + +test('two held messages with different ids both survive, in arrival order', async () => { + const { sdk, bus } = newSdk(); + bus.emit(RECEIVED_1); + bus.emit(RECEIVED_2); + const seen = []; + sdk.on('message_received', (event) => seen.push(event)); + await settle(); + assert.deepEqual(seen.map((event) => event.message_id), ['m1', 'm2']); +}); + +test("a held inbound event of another type stays held for its own listener", async () => { + const { sdk, bus } = newSdk(); + bus.emit(RECEIVED_1); + bus.emit(DECRYPT_FAILED); + const received = []; + const failed = []; + sdk.on('message_received', (event) => received.push(event)); + await settle(); + assert.deepEqual(received.map((event) => event.message_id), ['m1']); + sdk.on('message_decryption_failed', (event) => failed.push(event)); + await settle(); + assert.deepEqual(failed.map((event) => event.message_id), ['m3']); +}); + +test("an 'all' listener collects every held inbound event", async () => { + const { sdk, bus } = newSdk(); + bus.emit(RECEIVED_1); + bus.emit(DECRYPT_FAILED); + const seen = []; + sdk.on('all', (event) => seen.push(event)); + await settle(); + assert.deepEqual(seen.map((event) => event.type), ['message_received', 'message_decryption_failed']); +}); + +test('a message delivered live is not also held', async () => { + const { sdk, bus } = newSdk(); + const seen = []; + sdk.on('message_received', (event) => seen.push(event)); + bus.emit(RECEIVED_1); + sdk.on('message_received', (event) => seen.push(event)); + await settle(); + assert.equal(seen.length, 1); +}); + +test('the inbound hold drops the oldest past 256 entries', async () => { + const { sdk, bus } = newSdk(); + for (let i = 0; i < 257; i += 1) { + bus.emit({ ...RECEIVED_1, message_id: `m${i}` }); + } + const seen = []; + sdk.on('message_received', (event) => seen.push(event)); + await settle(); + assert.equal(seen.length, 256); + assert.equal(seen[0].message_id, 'm1', 'the oldest is the one that goes'); + assert.equal(seen[255].message_id, 'm256'); +}); + +test('destroy() clears the inbound hold', async () => { + const { sdk, bus } = newSdk(); + bus.emit(RECEIVED_1); + await sdk.destroy(); + const seen = []; + sdk.on('message_received', (event) => seen.push(event)); + await settle(); + assert.equal(seen.length, 0, 'a message held for a torn-down account must not surface later'); +}); + // --------------------------------------------------------------------------- // Runner // --------------------------------------------------------------------------- diff --git a/bindings/react-native/src/constants.ts b/bindings/react-native/src/constants.ts index 3ea07f09..a4639faa 100644 --- a/bindings/react-native/src/constants.ts +++ b/bindings/react-native/src/constants.ts @@ -59,6 +59,39 @@ export const ONE_SHOT_EVENT_TYPES = [ 'mesh_stopped_by_user', ] as const; +/** + * The *inbound* event tags: events that each report one message the core has + * already taken responsibility for. + * + * By the time `message_received` (or `file_received`, or the + * `message_decryption_failed` that stands in for one) is emitted, the core + * has sent the delivery ACK, dedup-marked the id and dropped its queued copy + * — the sender will not resend, and nothing in the SDK will restate the + * event. A drop between the core and the app's handler is therefore a lost + * message, and the drop is ordinary: the app is backgrounded and the React + * instance is down, or a push injection delivers the ciphertext before the + * app has registered `on('message_received')`. Unlike the one-shot set these + * do not collapse per type — every message is its own fact — so the native + * buffers key them `type:message_id` and the JS-side hold is a FIFO, both + * capped at 256 entries. + * + * The same three tags are enrolled in the native buffers on both platforms + * (`OfflineProtocolModule.BUFFERED_INBOUND_EVENT_TYPES` in Kotlin, + * `InboundEventBuffer.bufferedEventTypes` in Swift), which hold them across + * the *native→JS* gap; this set drives the JS-side hold that covers the + * *JS→app-listener* gap (see `OfflineProtocol.on`). A Rust guard + * (`react_native_buffered_inbound_event_set_matches_native` in + * `crates/offline-protocol-uniffi`) pins the three definitions together. + */ +export const BUFFERED_INBOUND_EVENT_TYPES = [ + 'message_received', + 'file_received', + 'message_decryption_failed', +] as const; + +/** Most inbound events the JS-side hold keeps; the oldest is dropped past it. */ +export const MAX_PENDING_INBOUND_EVENTS = 256; + /** * The Headless JS task key the Android keep-alive uses to wake JavaScript after * a process kill (Android only; see `registerMeshWakeTask`). diff --git a/bindings/react-native/src/index.ts b/bindings/react-native/src/index.ts index 6e653358..07855ef5 100644 --- a/bindings/react-native/src/index.ts +++ b/bindings/react-native/src/index.ts @@ -67,6 +67,8 @@ import { LINKING_ERROR, MESH_WAKE_TASK_KEY, ONE_SHOT_EVENT_TYPES, + BUFFERED_INBOUND_EVENT_TYPES, + MAX_PENDING_INBOUND_EVENTS, } from './constants'; export * from './types'; @@ -100,6 +102,11 @@ const ONE_SHOT_EVENT_TYPE_SET: ReadonlySet = new Set( ONE_SHOT_EVENT_TYPES ); +/** Membership test for {@link BUFFERED_INBOUND_EVENT_TYPES}, built once. */ +const BUFFERED_INBOUND_EVENT_TYPE_SET: ReadonlySet = new Set( + BUFFERED_INBOUND_EVENT_TYPES +); + interface InitialRuntimeConfig { dors?: { preferOnline: boolean; @@ -296,6 +303,17 @@ export class OfflineProtocol { * native Android buffer cannot close it. */ private pendingOneShotEvents: Map = new Map(); + /** + * Inbound message events that reached this instance while no listener was + * registered for them, held for the first listener that is. + * + * A FIFO rather than a per-type map: every `message_received` is its own + * message, so collapsing would lose all but the last. Capped at + * {@link MAX_PENDING_INBOUND_EVENTS}, oldest dropped first. See + * {@link BUFFERED_INBOUND_EVENT_TYPES} for why these are held at all and + * {@link OfflineProtocol.on} for the replay. + */ + private pendingInboundEvents: ProtocolEvent[] = []; /** * Event types already reported as dropped-with-no-listeners, so the warning * in {@link OfflineProtocol.emitEvent} fires once per type rather than once @@ -407,7 +425,7 @@ export class OfflineProtocol { pendingQueue: { maxPendingPerPeer: encryptionSource?.pendingQueue?.maxPendingPerPeer ?? 64, maxPendingGlobal: encryptionSource?.pendingQueue?.maxPendingGlobal ?? 4096, - pendingTtlMs: encryptionSource?.pendingQueue?.pendingTtlMs ?? 1800000, + pendingTtlMs: encryptionSource?.pendingQueue?.pendingTtlMs ?? 86400000, overflowPolicy: encryptionSource?.pendingQueue?.overflowPolicy ?? 'drop_oldest', }, @@ -780,6 +798,20 @@ export class OfflineProtocol { return; } + if (BUFFERED_INBOUND_EVENT_TYPE_SET.has(event.type)) { + // Appended, never collapsed: each of these is one message the core has + // already ACKed and will never restate. Past the cap the oldest goes — + // the same rule the native buffers apply. + this.pendingInboundEvents.push(event); + if (this.pendingInboundEvents.length > MAX_PENDING_INBOUND_EVENTS) { + this.pendingInboundEvents.splice( + 0, + this.pendingInboundEvents.length - MAX_PENDING_INBOUND_EVENTS + ); + } + return; + } + // Everything else is periodic, re-derivable, or followed by another event // carrying the same state, so dropping it is correct — but dropping it // *silently* while the app has registered nothing at all is @@ -852,9 +884,47 @@ export class OfflineProtocol { } this.eventListeners.get(eventType)!.add(listener as EventListener); this.replayHeldOneShotEvents(eventType); + this.replayHeldInboundEvents(eventType); return this; } + /** + * Hands held inbound events matching [eventType] to the listeners + * registered for them, on the next microtask, in arrival order. + * + * Same shape as {@link replayHeldOneShotEvents} for the same reasons: + * entries leave the hold when the replay is scheduled (so several + * `on(...)` calls in one tick cannot each deliver them), and delivery goes + * back through {@link emitEvent} (so every listener registered by then is + * served, and a listener removed in the interim re-holds instead of losing + * a message). + */ + private replayHeldInboundEvents(eventType: EventType | "all"): void { + if (this.pendingInboundEvents.length === 0) { + return; + } + + let replay: ProtocolEvent[]; + if (eventType === "all") { + replay = this.pendingInboundEvents; + this.pendingInboundEvents = []; + } else { + replay = this.pendingInboundEvents.filter( + (event) => event.type === eventType + ); + if (replay.length === 0) { + return; + } + this.pendingInboundEvents = this.pendingInboundEvents.filter( + (event) => event.type !== eventType + ); + } + + Promise.resolve().then(() => { + replay.forEach((event) => this.emitEvent(event)); + }); + } + /** * Hands any held one-shot event matching [eventType] to the listeners * registered for it, on the next microtask. @@ -3427,6 +3497,10 @@ export class OfflineProtocol { // the native `destroy()` above, which an uncreated instance skips. await Promise.resolve(); this.pendingOneShotEvents.clear(); + // The inbound hold goes with it: the native subscription is gone, so + // nothing can arrive, and a message held for an account the app just + // tore down must not surface in whatever it constructs next. + this.pendingInboundEvents = []; } /** diff --git a/bindings/react-native/src/types.ts b/bindings/react-native/src/types.ts index b8b7fbb4..f6e73af2 100644 --- a/bindings/react-native/src/types.ts +++ b/bindings/react-native/src/types.ts @@ -619,7 +619,7 @@ export interface PendingQueueConfig { maxPendingPerPeer?: number; /** Global pending message cap (default: 4096) */ maxPendingGlobal?: number; - /** Pending message TTL in milliseconds (default: 1800000 — 30 min) */ + /** Pending message TTL in milliseconds (default: 86400000 — 24 h) */ pendingTtlMs?: number; /** Overflow behavior when limits are hit (default: 'drop_oldest') */ overflowPolicy?: OverflowPolicy; @@ -1153,10 +1153,13 @@ export interface MessageFailedEvent extends BaseEvent { } /** - * Machine-readable decryption failure codes. `PENDING_QUEUE_DROPPED` means the - * message was dropped from the pending-decryption queue (overflow or TTL - * expiry) before the sender's session became ready; it was ACKed on receipt, - * so the sender will not retransmit it. + * Machine-readable decryption failure codes. `PENDING_QUEUE_DROPPED` means an + * encrypted message — text or a media chunk — was evicted from the + * pending-decryption queue (overflow, TTL expiry, or aged out of the persisted + * queue across restarts) before the sender's session became ready. It is + * emitted for text as well as media. The frame was never ACKed, so a sender + * that is still retrying will resend it and the resend can still be delivered + * once the session confirms; treat it as "at risk", not as terminal. */ export type DecryptionFailureCode = | 'INVALID_PAYLOAD' diff --git a/crates/offline-protocol-reliability/src/deduplicator.rs b/crates/offline-protocol-reliability/src/deduplicator.rs index 5cf64162..2e96849c 100644 --- a/crates/offline-protocol-reliability/src/deduplicator.rs +++ b/crates/offline-protocol-reliability/src/deduplicator.rs @@ -9,6 +9,7 @@ use chrono::{DateTime, Utc}; use offline_protocol_core::MessageId; +use serde::{Deserialize, Serialize}; use std::collections::HashMap; /// Configuration for deduplication. @@ -40,8 +41,17 @@ impl Default for DeduplicatorConfig { fn default() -> Self { Self { // Exact-match mode by default: no false positives (Bloom can drop ~1% of legitimate messages). - max_tracked_messages: 1000, - retention_time_secs: 3600, // 1 hour + // + // Sized for the seen set to be persisted across restarts + // (`export_seen`/`import_seen`): a message can reach a receiver + // twice on two paths — a push-injected copy and the relay socket + // copy after a reconnect, hours apart if the app was closed in + // between — and the second copy is only recognisable as a + // duplicate while its id is still tracked. An hour did not cover + // "reopened the next day"; a day does, and 2000 ids at ~50 bytes + // each is a 100 KB ceiling. + max_tracked_messages: 2000, + retention_time_secs: 86_400, // 24 hours use_bloom_filter: false, bloom_filter_bits: 1 << 20, // ~1MB per filter (1,048,576 bits) bloom_hash_count: 7, // ~1% false positive rate when Bloom is enabled @@ -216,6 +226,17 @@ impl RotatingBloomFilter { } } +/// One id of the exact-mode seen set, as exported for persistence and +/// imported on the next launch. `seen_at_ms` is wall-clock milliseconds so +/// the retention window can be re-applied on import. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SeenId { + /// The message id. + pub id: String, + /// When the id was first seen, unix milliseconds. + pub seen_at_ms: i64, +} + /// Entry for a seen message (used in HashMap mode). #[derive(Debug, Clone)] struct SeenEntry { @@ -223,6 +244,11 @@ struct SeenEntry { seen_at: DateTime, /// When this entry was last accessed (for LRU eviction). last_accessed: DateTime, + /// Whether [`Deduplicator::export_seen`] includes this id. `false` for + /// an id tracked with [`Deduplicator::mark_seen_local`]: a sender's own + /// outgoing frame, which dedup only has to recognise while this process + /// lives. + exportable: bool, } /// Deduplicator for tracking seen messages and preventing duplicates. @@ -272,6 +298,45 @@ impl Deduplicator { } } + /// Applies a new configuration in place, carrying the tracked set across. + /// + /// Rebuilding from scratch on a runtime configuration change forgets every + /// id held, which was equivalent to a restart while the set lived in memory + /// only and is a loss now that it is persisted: the ids restored at launch + /// would be dropped, and the next export would overwrite the record with a + /// near-empty set — so the restart-time duplicate the record exists to + /// recognise would be processed after all. The exact-mode entries therefore + /// survive, re-bounded by the new configuration: ids past the new retention + /// window are dropped, and past the new cap the newest are kept, with each + /// entry's exportable flag intact so a carried-over + /// [`Self::mark_seen_local`] id still stays out of [`Self::export_seen`]. + /// + /// A change of mode carries nothing: a bloom filter has no ids to hand + /// over and cannot take ids in. + pub fn reconfigure(&mut self, config: DeduplicatorConfig) { + let previous = std::mem::take(&mut self.seen_messages); + let was_exact = self.bloom_filter.is_none(); + *self = Self::with_config(config); + if !was_exact || self.bloom_filter.is_some() { + return; + } + let retention = chrono::Duration::seconds(self.config.retention_time_secs as i64); + let cutoff = Utc::now() - retention; + let mut entries: Vec<(String, SeenEntry)> = previous + .into_iter() + .filter(|(_, entry)| entry.seen_at > cutoff) + .collect(); + entries.sort_by(|left, right| { + right + .1 + .seen_at + .cmp(&left.1.seen_at) + .then_with(|| left.0.cmp(&right.0)) + }); + entries.truncate(self.config.max_tracked_messages); + self.seen_messages = entries.into_iter().collect(); + } + /// Returns `true` when deduplication is exact (HashMap mode): a positive /// [`Self::is_duplicate`] answer is then authoritative. In bloom mode /// (`false`) a positive answer may be a false positive (~1% with default @@ -331,6 +396,29 @@ impl Deduplicator { /// /// Returns `true` if this is a new message, `false` if it was already seen. pub fn mark_seen(&mut self, message_id: MessageId) -> bool { + self.mark_seen_with(message_id, true) + } + + /// Marks a message as seen for the life of this process only. + /// + /// Identical to [`Self::mark_seen`] for [`Self::is_duplicate`], eviction + /// and retention, but the id is left out of [`Self::export_seen`]. For a + /// sender marking its own outgoing frame: the mark exists to stop a + /// relayed echo of that frame re-entering the receive path, which a + /// restart does not change, and exporting it would spend the persisted + /// record's cap on ids no peer will ever push at this node. A later + /// [`Self::mark_seen`] of an id already tracked here is a no-op and does + /// not make it exportable. + /// + /// # Returns + /// + /// Returns `true` if the message was newly marked, `false` if it was + /// already tracked. + pub fn mark_seen_local(&mut self, message_id: MessageId) -> bool { + self.mark_seen_with(message_id, false) + } + + fn mark_seen_with(&mut self, message_id: MessageId, exportable: bool) -> bool { let msg_id_str = message_id.as_str(); if let Some(ref mut bloom) = self.bloom_filter { @@ -370,6 +458,7 @@ impl Deduplicator { SeenEntry { seen_at: now, last_accessed: now, + exportable, }, ); @@ -399,6 +488,75 @@ impl Deduplicator { self.seen_messages.remove(&message_id.as_str()).is_some() } + /// Exports up to `max` tracked ids, newest first, for persistence. + /// + /// Exact (HashMap) mode only: a bloom filter has no ids to export, so + /// bloom mode returns an empty vector and the persisted set simply stays + /// empty. Newest first so that a cap smaller than the tracked set keeps + /// the ids most likely to be replayed — the recent ones. Ids tracked with + /// [`Self::mark_seen_local`] are left out. + pub fn export_seen(&self, max: usize) -> Vec { + if self.bloom_filter.is_some() { + return Vec::new(); + } + let mut entries: Vec = self + .seen_messages + .iter() + .filter(|(_, entry)| entry.exportable) + .map(|(id, entry)| SeenId { + id: id.clone(), + seen_at_ms: entry.seen_at.timestamp_millis(), + }) + .collect(); + entries.sort_by(|left, right| { + right + .seen_at_ms + .cmp(&left.seen_at_ms) + .then_with(|| left.id.cmp(&right.id)) + }); + entries.truncate(max); + entries + } + + /// Re-admits ids exported by [`Self::export_seen`] on a previous run. + /// + /// Skips ids already tracked (the live set wins), ids whose retention + /// window has closed as of `now`, and stops at `max_tracked_messages` so + /// a restore can never push the set over its cap or evict live entries. + /// Entries are admitted in the order given, so a caller that wants the + /// newest to survive a tight cap passes them newest first — which is the + /// order `export_seen` produces. Returns how many were admitted. A no-op + /// in bloom mode, which has nothing to import into. + pub fn import_seen(&mut self, entries: Vec, now: DateTime) -> usize { + if self.bloom_filter.is_some() { + return 0; + } + let retention = chrono::Duration::seconds(self.config.retention_time_secs as i64); + let cutoff = now - retention; + let mut imported = 0; + for entry in entries { + if self.seen_messages.len() >= self.config.max_tracked_messages { + break; + } + let Some(seen_at) = DateTime::::from_timestamp_millis(entry.seen_at_ms) else { + continue; + }; + if seen_at <= cutoff || self.seen_messages.contains_key(&entry.id) { + continue; + } + self.seen_messages.insert( + entry.id, + SeenEntry { + seen_at, + last_accessed: seen_at, + exportable: true, + }, + ); + imported += 1; + } + imported + } + /// Removes expired entries that exceed the retention time. /// /// # Returns @@ -547,13 +705,14 @@ mod tests { } #[test] - fn test_default_config_is_hashmap_and_capacity_1000() { + fn test_default_config_is_hashmap_and_capacity_2000() { let config = DeduplicatorConfig::default(); assert!( !config.use_bloom_filter, "default should be exact-match HashMap to avoid false positives" ); - assert_eq!(config.max_tracked_messages, 1000); + assert_eq!(config.max_tracked_messages, 2000); + assert_eq!(config.retention_time_secs, 86_400); let dedup = Deduplicator::new(); assert!(!dedup.is_bloom_filter_mode()); @@ -939,4 +1098,179 @@ mod tests { assert!(dedup.is_duplicate(&msg_c), "C should survive"); assert!(dedup.is_duplicate(&msg_d), "D was just inserted"); } + #[test] + fn test_export_import_round_trip_newest_first() { + let mut dedup = Deduplicator::new(); + let ids: Vec = (0..5).map(|_| MessageId::new()).collect(); + for id in &ids { + dedup.mark_seen(id.clone()); + } + let exported = dedup.export_seen(usize::MAX); + assert_eq!(exported.len(), 5); + assert!( + exported + .windows(2) + .all(|pair| pair[0].seen_at_ms >= pair[1].seen_at_ms), + "export is newest first" + ); + // A cap keeps the newest, not an arbitrary subset. + let capped = dedup.export_seen(2); + assert_eq!(capped.len(), 2); + assert_eq!(capped, exported[..2].to_vec()); + + let mut restored = Deduplicator::new(); + assert_eq!(restored.import_seen(exported.clone(), Utc::now()), 5); + for id in &ids { + assert!(restored.is_duplicate(id)); + } + // The seen_at survives the trip, so a later export agrees. + let mut again = restored.export_seen(usize::MAX); + let mut original = exported; + again.sort_by(|a, b| a.id.cmp(&b.id)); + original.sort_by(|a, b| a.id.cmp(&b.id)); + assert_eq!(again, original); + } + + #[test] + fn test_import_ignores_expired_existing_and_respects_cap() { + let config = DeduplicatorConfig { + max_tracked_messages: 3, + retention_time_secs: 3600, + ..Default::default() + }; + let mut dedup = Deduplicator::with_config(config); + let now = Utc::now(); + let live = MessageId::new(); + let expired = MessageId::new(); + let fresh: Vec = (0..3).map(|_| MessageId::new()).collect(); + dedup.mark_seen(live.clone()); + + let seen = |id: &MessageId, secs_ago: i64| SeenId { + id: id.as_str(), + seen_at_ms: (now - chrono::Duration::seconds(secs_ago)).timestamp_millis(), + }; + let entries = vec![ + // Expired: past the retention window as of `now`. + seen(&expired, 3601), + // Already tracked: the live entry wins and this does not count. + seen(&live, 10), + seen(&fresh[0], 5), + seen(&fresh[1], 4), + // Over the cap once the two above are in: refused. + seen(&fresh[2], 3), + ]; + assert_eq!(dedup.import_seen(entries, now), 2); + assert_eq!(dedup.tracked_count(), 3); + assert!(!dedup.is_duplicate(&expired)); + assert!(dedup.is_duplicate(&live)); + assert!(dedup.is_duplicate(&fresh[0])); + assert!(dedup.is_duplicate(&fresh[1])); + assert!(!dedup.is_duplicate(&fresh[2])); + } + + #[test] + fn test_local_marks_dedup_but_are_not_exported() { + let mut dedup = Deduplicator::new(); + let inbound = MessageId::new(); + let own = MessageId::new(); + assert!(dedup.mark_seen(inbound.clone())); + assert!(dedup.mark_seen_local(own.clone())); + // Both dedup in memory alike. + assert!(dedup.is_duplicate(&inbound)); + assert!(dedup.is_duplicate(&own)); + assert!(!dedup.mark_seen_local(own.clone()), "already tracked"); + // A later plain mark of a local id is a no-op and does not promote it. + assert!(!dedup.mark_seen(own.clone())); + // Only the inbound id reaches the export. + let exported = dedup.export_seen(usize::MAX); + assert_eq!(exported.len(), 1); + assert_eq!(exported[0].id, inbound.as_str()); + // A local id can still be released like any other. + assert!(dedup.unmark_seen(&own)); + assert!(!dedup.is_duplicate(&own)); + } + + #[test] + fn test_export_import_are_noops_in_bloom_mode() { + let config = DeduplicatorConfig { + use_bloom_filter: true, + ..Default::default() + }; + let mut dedup = Deduplicator::with_config(config); + dedup.mark_seen(MessageId::new()); + assert!(dedup.export_seen(usize::MAX).is_empty()); + let imported = MessageId::new(); + assert_eq!( + dedup.import_seen( + vec![SeenId { + id: imported.as_str(), + seen_at_ms: Utc::now().timestamp_millis(), + }], + Utc::now() + ), + 0 + ); + assert!(!dedup.is_duplicate(&imported)); + } + + #[test] + fn test_reconfigure_carries_tracked_ids_within_the_new_bounds() { + let mut dedup = Deduplicator::new(); + let now = Utc::now(); + let expired = MessageId::new(); + let oldest_live = MessageId::new(); + let inbound = MessageId::new(); + let own = MessageId::new(); + // Plant two ids with explicit ages, then two fresh ones. + assert_eq!( + dedup.import_seen( + vec![ + SeenId { + id: expired.as_str(), + seen_at_ms: (now - chrono::Duration::seconds(7200)).timestamp_millis(), + }, + SeenId { + id: oldest_live.as_str(), + seen_at_ms: (now - chrono::Duration::seconds(60)).timestamp_millis(), + }, + ], + now + ), + 2 + ); + assert!(dedup.mark_seen(inbound.clone())); + assert!(dedup.mark_seen_local(own.clone())); + + // Tighter retention drops the expired id; a cap of 2 keeps the two + // newest of what is left, whichever way they were marked. + dedup.reconfigure(DeduplicatorConfig { + max_tracked_messages: 2, + retention_time_secs: 3600, + ..Default::default() + }); + assert_eq!(dedup.tracked_count(), 2); + assert!( + !dedup.is_duplicate(&expired), + "past the new retention window" + ); + assert!(!dedup.is_duplicate(&oldest_live), "past the new cap"); + assert!(dedup.is_duplicate(&inbound)); + assert!(dedup.is_duplicate(&own)); + // The exportable flag rides along: the local mark stays local. + let exported = dedup.export_seen(usize::MAX); + assert_eq!(exported.len(), 1); + assert_eq!(exported[0].id, inbound.as_str()); + + // A mode change carries nothing either way. + dedup.reconfigure(DeduplicatorConfig { + use_bloom_filter: true, + ..Default::default() + }); + assert!(dedup.is_bloom_filter_mode()); + assert!(!dedup.is_duplicate(&inbound)); + dedup.mark_seen(MessageId::new()); + dedup.reconfigure(DeduplicatorConfig::default()); + assert!(!dedup.is_bloom_filter_mode()); + assert_eq!(dedup.tracked_count(), 0); + } } diff --git a/crates/offline-protocol-reliability/src/lib.rs b/crates/offline-protocol-reliability/src/lib.rs index 14b29f67..e9a72fc9 100644 --- a/crates/offline-protocol-reliability/src/lib.rs +++ b/crates/offline-protocol-reliability/src/lib.rs @@ -18,7 +18,9 @@ pub mod retry_queue; pub use ack_manager::{AckConfig, AckEvictionInfo, AckManager}; pub use ack_optimization::{AckOptimizationConfig, AckOptimizer, AggregatedAck, PiggybackAckData}; -pub use deduplicator::{Deduplicator, DeduplicatorConfig, DeduplicatorMode, DeduplicatorStats}; +pub use deduplicator::{ + Deduplicator, DeduplicatorConfig, DeduplicatorMode, DeduplicatorStats, SeenId, +}; pub use error::{Error, Result}; pub use relay_seen::{ RelaySeenCache, RelaySeenConfig, SeenOutcome, DEFAULT_RELAY_SEEN_CAPACITY, diff --git a/crates/offline-protocol-transport/src/nostr_crypto.rs b/crates/offline-protocol-transport/src/nostr_crypto.rs index 194c5402..e6367bca 100644 --- a/crates/offline-protocol-transport/src/nostr_crypto.rs +++ b/crates/offline-protocol-transport/src/nostr_crypto.rs @@ -836,20 +836,20 @@ impl NostrEvent { /// again on the next connect. That duplicate is expected; the alternative /// (`since + 1`) would drop any event sharing that exact second. /// -/// **How much of the replayed overlap dedup actually absorbs is bounded, and -/// the bound is smaller than the overlap.** The engine's deduplicator retains -/// ids for `DeduplicatorConfig::retention_time_secs` (1 h by default) and at -/// most `max_tracked_messages` of them, while `since` reaches +/// **The replayed overlap is absorbed by dedup, within its cap.** The engine's +/// deduplicator retains ids for `DeduplicatorConfig::retention_time_secs` +/// (24 h by default, persisted across restarts) and at most +/// `max_tracked_messages` of them, while `since` reaches /// `NOSTR_CREATED_AT_JITTER_SECS + NOSTR_CLOCK_SKEW_MARGIN_SECS` (1 h 5 min) -/// below the mark. A reconnect after longer than the retention window — the -/// ordinary case for a mobile app reopened the next day — therefore re-processes -/// the overlap rather than deduplicating it. That is a cost, not a loss: a -/// replayed ciphertext whose ratchet generation is spent fails closed as -/// `Decryption` and is dropped, a past-epoch one triggers at most one -/// rate-limited re-key, and a group copy TTLs out of the pending buffer. The -/// engine pins this relationship in -/// `nostr_replay_overlap_exceeds_dedup_retention` so the two constants cannot -/// drift apart from this note. +/// below the mark — so a reconnect inside the retention window, including a +/// mobile app reopened the next day, deduplicates the overlap rather than +/// re-processing it. Past the window (or past the id cap) the overlap is +/// re-processed, which is a cost and not a loss: a replayed ciphertext whose +/// ratchet generation is spent fails closed as `Decryption` and is dropped, a +/// past-epoch one triggers at most one rate-limited re-key, and a group copy +/// TTLs out of the pending buffer. The engine pins this relationship in +/// `nostr_replay_overlap_fits_inside_dedup_retention` so the two constants +/// cannot drift apart from this note. pub fn create_subscription_message( pubkey_hex: &str, subscription_id: &str, diff --git a/crates/offline-protocol-uniffi/src/lib.rs b/crates/offline-protocol-uniffi/src/lib.rs index 5634bff2..b2e21ae2 100644 --- a/crates/offline-protocol-uniffi/src/lib.rs +++ b/crates/offline-protocol-uniffi/src/lib.rs @@ -1817,7 +1817,7 @@ impl Default for PendingQueueConfig { Self { max_pending_per_peer: 64, max_pending_global: 4096, - // Mirrors the core default (30 min); see DEFAULT_PENDING_TTL_MS in + // Mirrors the core default (24 h); see DEFAULT_PENDING_TTL_MS in // offline-protocol/src/config.rs for the deferred-ACK rationale. pending_ttl_ms: DEFAULT_PENDING_TTL_MS, overflow_policy: OverflowPolicy::DropOldest, @@ -4352,7 +4352,12 @@ impl OfflineProtocol { /// /// `reason` should carry platform-specific error context so reliability /// telemetry can classify root causes more accurately. + /// + /// The bridges also report the relay's `MessageSent { pushed: true }` here, + /// as the `relay_pushed` token. That parks the frame in the core but is not + /// a failed send, so it is kept out of the transport's delivery metrics. pub fn internet_send_failed_with_reason(&self, message_id: String, reason: Option) { + let scores_carrier = CoreProtocol::send_report_is_carrier_failure(reason.as_deref()); let mut protocol = self.lock_inner_recovering(); if let Err(err) = protocol.on_transport_send_failed_via( &message_id, @@ -4365,6 +4370,9 @@ impl OfflineProtocol { "Failed to apply welcome lifecycle transport failure" ); } + if !scores_carrier { + return; + } if let Some(transport_arc) = protocol .transport_manager() .get_transport(CoreTransportType::Internet) @@ -8723,6 +8731,71 @@ mod tests { .into_bytes() } + /// The bridges report the relay's `MessageSent { pushed: true }` through + /// `internet_send_failed_with_reason` as `relay_pushed`. The relay took that + /// frame, so the report must stay out of the Internet transport's failure + /// accounting: the frame still awaits the bridge's write confirmation + /// afterwards. A real failure report for the same frame still reaches it. + #[test] + fn test_internet_relay_push_report_is_not_scored_as_a_send_failure() { + let protocol = OfflineProtocol::new(create_test_config()).unwrap(); + protocol.start().unwrap(); + protocol.internet_status_changed(true).unwrap(); + let internet = || { + protocol + .lock_inner_recovering() + .transport_manager() + .get_transport(CoreTransportType::Internet) + .expect("the Internet transport is registered") + }; + let awaiting_confirmation = || { + internet() + .as_any() + .downcast_ref::() + .expect("the Internet transport is an InternetTransport") + .pending_confirmation_count() + }; + + let message = offline_protocol_core::Message::new( + offline_protocol_core::UserId::new("user123").unwrap(), + offline_protocol_core::UserId::new("bob").unwrap(), + offline_protocol_core::AppId::new("test-app").unwrap(), + "pushed while bob was offline", + ); + internet() + .send(&message) + .expect("queueing to the Internet transport"); + let frame = protocol + .internet_get_next_message() + .expect("the platform takes the frame"); + assert_eq!(frame.message_id, message.id.as_str()); + assert_eq!( + awaiting_confirmation(), + 1, + "precondition: the frame awaits its write confirmation" + ); + + protocol.internet_send_failed_with_reason( + frame.message_id.clone(), + Some("relay_pushed".to_string()), + ); + assert_eq!( + awaiting_confirmation(), + 1, + "a push is not a failed send, so the transport must not count it" + ); + + protocol.internet_send_failed_with_reason( + frame.message_id, + Some("recipient_unreachable: Recipient is offline".to_string()), + ); + assert_eq!( + awaiting_confirmation(), + 0, + "a real failure report still reaches the transport" + ); + } + /// Redundant same-state reports (e.g. bridge auth refresh) must not emit /// phantom TransportSwitched events; only real transitions do. #[test] @@ -12146,6 +12219,257 @@ mod tests { } } + /// The buffered-inbound event set agrees across TypeScript, Kotlin and + /// Swift, and each layer's hold is wired to a flush. + /// + /// `message_received`, `file_received` and `message_decryption_failed` + /// each report one message the core has already ACKed, dedup-marked and + /// dropped its queued copy of, so nothing will ever restate them and a + /// drop between the core and the app's handler is a lost message. Three + /// definitions and three holds have to agree for that not to happen — + /// the native buffers (Kotlin `BUFFERED_INBOUND_EVENT_TYPES`, Swift + /// `InboundEventBuffer.bufferedEventTypes`) cover the native→JS gap and + /// `BUFFERED_INBOUND_EVENT_TYPES` in `src/constants.ts` the + /// JS→app-listener gap — and every half fails silently, so they are + /// pinned here the way [`react_native_one_shot_event_set_matches_native`] + /// pins the one-shot set. The set is asserted exactly: enrolling a + /// periodic event here would replay stale state. + #[test] + fn react_native_buffered_inbound_event_set_matches_native() { + const INBOUND_TAGS: [&str; 3] = [ + "message_received", + "file_received", + "message_decryption_failed", + ]; + + let rn_dir = + std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../bindings/react-native"); + let read = |rel: &str| -> String { + let path = rn_dir.join(rel); + std::fs::read_to_string(&path) + .unwrap_or_else(|e| panic!("cannot read {}: {e}", path.display())) + }; + fn declared_between(source: &str, start: &str, end: &str) -> Vec { + let region = source + .split_once(start) + .unwrap_or_else(|| panic!("expected {start:?} in source")) + .1 + .split_once(end) + .unwrap_or_else(|| panic!("expected {end:?} after {start:?}")) + .0; + region + .lines() + .map(str::trim) + .filter(|l| !l.is_empty() && !l.starts_with("//")) + .map(|l| { + l.trim_end_matches(',') + .trim_matches(|c| c == '\'' || c == '"') + .to_string() + }) + .collect() + } + + // --- TypeScript: exactly these tags, in this order ------------------ + let constants_ts = read("src/constants.ts"); + assert_eq!( + declared_between( + &constants_ts, + "export const BUFFERED_INBOUND_EVENT_TYPES = [", + "] as const;" + ), + INBOUND_TAGS.to_vec(), + "src/constants.ts BUFFERED_INBOUND_EVENT_TYPES must hold exactly the inbound tags" + ); + assert!( + constants_ts.contains("export const MAX_PENDING_INBOUND_EVENTS = 256;"), + "the JS-side inbound hold must be capped at 256 like the native buffers" + ); + + // --- Kotlin --------------------------------------------------------- + let kotlin = read("android/src/main/java/com/offlineprotocol/OfflineProtocolModule.kt"); + assert_eq!( + declared_between( + &kotlin, + "private val BUFFERED_INBOUND_EVENT_TYPES: Set = setOf(", + ")" + ), + INBOUND_TAGS.to_vec(), + "OfflineProtocolModule.kt BUFFERED_INBOUND_EVENT_TYPES must match src/constants.ts" + ); + let kotlin_code = rn_source_code_only( + "android/src/main/java/com/offlineprotocol/OfflineProtocolModule.kt", + ); + assert!( + kotlin_code.contains("INBOUND_EVENT_BUFFER_CAPACITY = 256") + && kotlin_code + .contains("StickyEventBuffer(maxEntries = INBOUND_EVENT_BUFFER_CAPACITY)"), + "the Android inbound buffer must be a 256-entry StickyEventBuffer of its own" + ); + assert!( + kotlin_code.contains( + "if (!sendEvent(EVENT_NAME, eventParams(eventJson))) { \ + holdInboundEventIfBuffered(eventJson) }" + ), + "the Android core event callback must hold a buffered inbound event whenever the \ + emit refuses it, not only when the JS gate read shut before the emit" + ); + assert!( + kotlin_code.contains("inboundEvents.send(\"$type:$id\", eventJson)"), + "Android must key held inbound events type:message_id so every message survives" + ); + for call in [ + "inboundEvents.flush()", + "inboundEvents.beginSession()", + "inboundEvents.endSession()", + ] { + assert!( + kotlin_code.contains(call), + "OfflineProtocolModule.kt must drive {call} alongside the one-shot buffer" + ); + } + + // --- Swift ---------------------------------------------------------- + let swift_buffer = read("ios/InboundEventBuffer.swift"); + assert_eq!( + declared_between( + &swift_buffer, + "static let bufferedEventTypes: Set = [", + "]" + ), + INBOUND_TAGS.to_vec(), + "ios/InboundEventBuffer.swift bufferedEventTypes must match src/constants.ts" + ); + assert!( + swift_buffer.contains("static let defaultMaxEntries = 256"), + "the iOS inbound buffer must be capped at 256" + ); + let swift_module = rn_source_code_only("ios/OfflineProtocolModule.swift"); + assert!( + swift_module.contains("emitter.holdInboundEventIfBuffered(eventJson, generation: generation)"), + "EventCallbackImpl.onEvent must hold a buffered inbound event when sendEventToJS refuses it" + ); + for site in [ + "override func startObserving() { hasListeners = true DispatchQueue.main.async { [weak self] in self?.flushInboundEvents() } }", + "super.addListener(eventName) DispatchQueue.main.async { [weak self] in self?.flushInboundEvents() }", + "@objc private func applicationDidBecomeActive() { flushInboundEvents() }", + ] { + assert!( + swift_module.contains(site), + "OfflineProtocolModule.swift must flush held inbound events from every trigger; \ + missing: {site}" + ); + } + assert_eq!( + swift_module.matches("inboundEvents.bumpGeneration()").count(), + 2, + "OfflineProtocolModule.swift must bump the inbound generation on create() and destroy()" + ); + assert!( + read("MeshSdk.podspec").contains("\"ios/InboundEventBuffer.swift\","), + "MeshSdk.podspec must ship ios/InboundEventBuffer.swift — the pod lists sources \ + explicitly, so a missing entry is a link error in every consuming app" + ); + + // --- The JS hold and replay ---------------------------------------- + let index_ts = rn_source_code_only("src/index.ts"); + assert!( + index_ts.contains("BUFFERED_INBOUND_EVENT_TYPE_SET.has(event.type)") + && index_ts.contains("this.pendingInboundEvents.push(event)"), + "src/index.ts emitEvent must hold an inbound event it could not deliver to any app \ + listener" + ); + assert!( + index_ts.contains("this.replayHeldInboundEvents(eventType);"), + "src/index.ts on() must replay held inbound events to a new listener" + ); + } + + /// A `MessageSent { pushed: true }` parks the frame on both bridges. + /// + /// The relay has no store-and-forward: when the recipient is not on it, + /// the ciphertext goes out in a push notification and the sender is told + /// `MessageSent` — which, without this, resolved the frame as accepted + /// and left it awaiting an ACK from a peer who may never receive the + /// push. Both managers must read `pushed` right where they resolve the + /// relay acceptance and, when it is set, report that one id to the core + /// under the exact `relay_pushed` token and watch the recipient, mirroring + /// `handleRecipientUnreachable`. The token is the core's + /// `SEND_FAIL_REASON_RELAY_PUSHED`, and it must not be a + /// `recipient_unreachable` tail: that prefix fast-fails connection + /// requests and fails Welcomes, both of which the push may have + /// delivered. Neither platform can test the call site itself (see the + /// ordering guard below for why), so the source is pinned here. + #[test] + fn react_native_relay_parks_a_pushed_message_sent() { + let swift = rn_source_code_only("ios/InternetManager.swift"); + let kotlin = + rn_source_code_only("android/src/main/java/com/offlineprotocol/InternetManager.kt"); + + for (label, code) in [("ios", &swift), ("android", &kotlin)] { + let resolve = code + .find("resolveOnRelayAccepted(") + .unwrap_or_else(|| panic!("{label} InternetManager must resolve relay acceptance")); + let pushed = code + .find("\"pushed\"") + .unwrap_or_else(|| panic!("{label} InternetManager must read MessageSent.pushed")); + let park = code + .find("parkPushedMessage(") + .unwrap_or_else(|| panic!("{label} InternetManager must park a pushed message")); + assert!( + resolve < pushed && pushed < park, + "{label} InternetManager must read `pushed` next to resolveOnRelayAccepted and \ + park through parkPushedMessage right after it — the frame is out of the \ + in-flight tracker either way, so the park is the only thing left that \ + stops its ACK budget burning against an offline peer" + ); + // The definition is the last occurrence (both managers define it + // below the call site, next to handleRecipientUnreachable); the + // body checks below start there so they read the park, not the + // DeliveryError handler above it. + let definition = code + .rfind("parkPushedMessage(") + .expect("parkPushedMessage definition"); + assert!( + definition > park, + "{label} parkPushedMessage must be defined below its call site" + ); + let park_body = &code[definition..]; + let fail = park_body + .find("internetSendFailedWithReason(") + .expect("parkPushedMessage must fail the id into the core"); + let watch = park_body + .find("presenceWatch.watch(") + .expect("parkPushedMessage must presence-watch the recipient"); + let presence = park_body + .find("internetPeerPresence(") + .expect("parkPushedMessage must feed an offline presence to the core"); + assert!( + fail < watch && watch < presence, + "{label} parkPushedMessage must park first, then watch, then feed presence — \ + the order handleRecipientUnreachable uses" + ); + // The reason passed to that call: the exact token, and nothing + // under the recipient_unreachable prefix. + let report = &park_body[fail..watch]; + assert!( + report.contains("\"relay_pushed\""), + "{label} parkPushedMessage must report the id under the exact relay_pushed \ + token the core parks a plain DM on" + ); + assert!( + !report.contains("recipient_unreachable"), + "{label} parkPushedMessage must not use the recipient_unreachable prefix: the \ + core fast-fails connection requests and fails Welcomes on it, both of which \ + the push may have delivered" + ); + assert!( + !park_body[..fail].contains("drainRecipient("), + "{label} parkPushedMessage must fail only the pushed id, never drain the \ + recipient's other in-flight frames" + ); + } + } + /// The relay connection proves its address before it sends anything else. /// /// The relay attributes each inbound frame by whatever the connection has diff --git a/crates/offline-protocol/src/config.rs b/crates/offline-protocol/src/config.rs index 2373d91b..46a2d207 100644 --- a/crates/offline-protocol/src/config.rs +++ b/crates/offline-protocol/src/config.rs @@ -46,18 +46,25 @@ pub struct PendingQueueConfig { pub overflow_policy: OverflowPolicy, } -/// Default TTL for the pending-decryption queue (30 minutes, in ms). +/// Default TTL for the pending-decryption queue (24 hours, in ms). /// /// Under the deferred-ACK model an undecryptable message is no longer ACKed on /// receipt, so this queue is the primary recovery window before the session /// confirms — a 2-minute window was too short for a peer whose Welcome is slow -/// to arrive/adopt. Memory stays bounded by the per-peer/global byte caps plus -/// the `DropOldest` overflow policy; a longer TTL only lets entries linger -/// within those caps, it does not raise the ceiling. +/// to arrive/adopt, and 30 minutes was too short once the queue became +/// durable: with a relay that pushes ciphertext without store-and-forward +/// there is no second copy, so a frame that arrives while the recipient's +/// session is still being set up has to outlive the handshake, which for a +/// phone that is opened once a day means a day. Memory stays bounded by the +/// per-peer/global byte caps plus the `DropOldest` overflow policy; a longer +/// TTL only lets entries linger within those caps, it does not raise the +/// ceiling. The TTL is measured on an `Instant` and so restarts with the +/// process; the persisted copy is bounded separately by +/// `PENDING_DECRYPT_PERSISTED_MAX_AGE_MS` (7 days). /// /// The UniFFI `PendingQueueConfig` default mirrors this value — reference this /// constant there rather than re-hardcoding it. -pub const DEFAULT_PENDING_TTL_MS: u64 = 1_800_000; +pub const DEFAULT_PENDING_TTL_MS: u64 = 86_400_000; fn default_max_pending_bytes_per_peer() -> usize { 4 * 1024 * 1024 @@ -74,7 +81,7 @@ impl Default for PendingQueueConfig { max_pending_global: 4096, max_pending_bytes_per_peer: default_max_pending_bytes_per_peer(), max_pending_bytes_global: default_max_pending_bytes_global(), - // 30 minutes; see DEFAULT_PENDING_TTL_MS for the deferred-ACK + // 24 hours; see DEFAULT_PENDING_TTL_MS for the deferred-ACK // rationale and the FFI-mirror contract. pending_ttl_ms: DEFAULT_PENDING_TTL_MS, overflow_policy: OverflowPolicy::DropOldest, @@ -1498,7 +1505,9 @@ mod tests { assert_eq!(reliability.retry.outbox_max_lifetime_ms, 604_800_000); // 5 min ceiling — also mirrored by the RN bridge fallbacks. assert_eq!(reliability.retry.max_delay_ms, 300_000); - assert_eq!(reliability.dedup.max_tracked_messages, 1000); + // Sized for the persisted seen set: see `DeduplicatorConfig::default`. + assert_eq!(reliability.dedup.max_tracked_messages, 2000); + assert_eq!(reliability.dedup.retention_time_secs, 86_400); } #[test] @@ -1933,6 +1942,70 @@ mod tests { } } + /// Drift guard: the React Native `updateDedupConfig` fallbacks match the + /// core dedup defaults. The TypeScript layer forwards only the fields the + /// app set, so a `reliability.dedup` section that sets one field takes the + /// other from these literals, and a fallback that disagrees with the + /// documented default silently changes it for that app. The id cap sat at + /// 10000 against a documented default of 1000, then 2000, that way. + #[test] + fn rn_bridge_dedup_fallbacks_match_rust_default() { + let defaults = ProtocolConfig::new("test-app", "user123").reliability.dedup; + let rn_root = + std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../bindings/react-native"); + let kotlin = + rn_root.join("android/src/main/java/com/offlineprotocol/OfflineProtocolModule.kt"); + let swift = rn_root.join("ios/OfflineProtocolModule.swift"); + let expected = [ + ( + kotlin.clone(), + format!( + "json.optLong(\"maxTrackedMessages\", {})", + defaults.max_tracked_messages + ), + ), + ( + kotlin, + format!( + "json.optLong(\"retentionTimeSecs\", {})", + defaults.retention_time_secs + ), + ), + ( + swift.clone(), + format!( + "(config[\"maxTrackedMessages\"] as? NSNumber)?.uint64Value ?? {}", + defaults.max_tracked_messages + ), + ), + ( + swift, + format!( + "(config[\"retentionTimeSecs\"] as? NSNumber)?.uint64Value ?? {}", + defaults.retention_time_secs + ), + ), + ]; + + let Some(sources) = expected + .iter() + .map(|(path, _)| std::fs::read_to_string(path).ok()) + .collect::>>() + else { + eprintln!("bindings tree not present, skipping RN dedup fallback drift check"); + return; + }; + + for ((path, fallback), source) in expected.iter().zip(&sources) { + let squeezed = source.split_whitespace().collect::>().join(" "); + assert!( + squeezed.contains(fallback.as_str()), + "RN dedup fallback drifted from the core default: expected `{fallback}` in {}", + path.display() + ); + } + } + /// Drift guard: every public surface that documents /// [`EncryptionConfig::crypto_recovery_enabled`] must state that the re-key /// trigger is unauthenticated, and must not carry the retired claim that it @@ -2020,7 +2093,7 @@ mod tests { } } - /// Renders `1800000` as `1_800_000`, the form Kotlin and Swift use. + /// Renders `86400000` as `86_400_000`, the form Kotlin and Swift use. fn format_underscored(value: u64) -> String { let digits = value.to_string(); let mut out = String::new(); diff --git a/crates/offline-protocol/src/events.rs b/crates/offline-protocol/src/events.rs index 9051cf49..cc7b7ae5 100644 --- a/crates/offline-protocol/src/events.rs +++ b/crates/offline-protocol/src/events.rs @@ -456,19 +456,24 @@ pub enum DecryptionFailureCode { IdentityMismatch, /// Cryptographic operation failed. CryptoFailure, - /// A media chunk was evicted from the pending-decryption queue (overflow or - /// TTL expiry) before the sender's session became ready, so the file - /// transfer it belongs to is currently stalled. + /// An encrypted message — a text frame or a media chunk — was evicted from + /// the pending-decryption queue (overflow or TTL expiry, or it aged out of + /// the persisted queue across restarts) before the sender's session became + /// ready. For a media chunk the file transfer it belongs to is currently + /// stalled; for a text message the message itself is gone locally. /// /// Under the deferred-ACK model this is **advisory, not terminal**: the - /// evicted chunk was never ACKed, so the sender keeps retransmitting and a - /// later resend re-enters the queue and can still complete the transfer once - /// the session confirms. Treat this as "the transfer is stalled and may need - /// a resend", not "the transfer has permanently failed" — the terminal - /// failure signal for media is `FileReceiveFailed`. The same is true of a - /// hard decrypt failure while `crypto_recovery_enabled`, which surfaces - /// under its own code but is equally un-ACKed and equally recoverable by a - /// resend; see the `MessageDecryptionFailed` docs. + /// evicted frame was never ACKed, so a sender still retrying resends it and + /// a later resend re-enters the queue and can still complete once the + /// session confirms. Treat this as "at risk, may need a resend", not + /// "permanently failed" — the terminal failure signal for media is + /// `FileReceiveFailed`. It is emitted for text as well as for chunks: a + /// relay that pushes ciphertext without store-and-forward has no second + /// copy, so a silent text eviction was a lost message with no signal to the + /// app. The same is true of a hard decrypt failure while + /// `crypto_recovery_enabled`, which surfaces under its own code but is + /// equally un-ACKed and equally recoverable by a resend; see the + /// `MessageDecryptionFailed` docs. PendingQueueDropped, /// Failure class is unknown. Unknown, diff --git a/crates/offline-protocol/src/group_mesh.rs b/crates/offline-protocol/src/group_mesh.rs index f546ea5d..2a92b4ed 100644 --- a/crates/offline-protocol/src/group_mesh.rs +++ b/crates/offline-protocol/src/group_mesh.rs @@ -2388,10 +2388,15 @@ impl OfflineProtocol { /// enter the transport deduplicator (the platform bridges mint a fresh /// envelope UUID per injected message), so `from_str` fails or the /// unmark is a no-op there — both harmless. + /// + /// The transport-level release goes through `unmark_seen_persisted`, not + /// the deduplicator directly: the seen set is persisted, and an unmark it + /// never counted leaves the id in the stored record, so the next launch + /// restores it and swallows the redelivery after all. pub(crate) fn release_replay_protection(&mut self, message_id: &str) { self.group_mesh.message_dedup.remove(message_id); if let Ok(envelope_id) = MessageId::from_str(message_id) { - self.deduplicator.unmark_seen(&envelope_id); + self.unmark_seen_persisted(&envelope_id); } } diff --git a/crates/offline-protocol/src/protocol/blocking.rs b/crates/offline-protocol/src/protocol/blocking.rs index aa9b10b8..ecb862cb 100644 --- a/crates/offline-protocol/src/protocol/blocking.rs +++ b/crates/offline-protocol/src/protocol/blocking.rs @@ -10,7 +10,8 @@ use tracing::{debug, info}; impl OfflineProtocol { /// Blocks a user. Messages from this user will be silently dropped /// (no ACK sent, no event emitted). The blocked user receives no - /// notification. + /// notification. Encrypted frames of theirs already parked awaiting a + /// session are discarded, persisted records included. /// /// Blocking is idempotent — calling this for an already-blocked user /// succeeds silently. @@ -42,6 +43,23 @@ impl OfflineProtocol { self.persist_blocked_user(user_id); + // Frames this user sent before the block may be parked in the + // pending-decryption queue, each with a persisted record. The block is + // otherwise applied only when that queue drains, which never happens + // for a blocked peer, so the records would be restored on every launch + // for up to seven days and each eviction would report a + // `PendingQueueDropped` naming the user just blocked. Nothing new is + // parked after this: the receive path drops a blocked sender's frames + // before they reach the queue. + let discarded = self.discard_pending_decryption_for_peer(user_id); + if discarded > 0 { + debug!( + user_id = %user_id, + count = discarded, + "Discarded pending decryption queue for blocked user" + ); + } + info!(user_id = %user_id, "User blocked"); if let Ok(state) = lock_shared_state(&self.shared_state) { @@ -147,13 +165,11 @@ impl OfflineProtocol { // 4. Drain any inbound messages sitting in the pending decryption // queue (encrypted messages received before the session was ready). - let drained = self - .pending_queue - .drain_for_peer(&self.config.encryption.pending_queue, user_id); - if !drained.is_empty() { + let drained = self.discard_pending_decryption_for_peer(user_id); + if drained > 0 { debug!( user_id = %user_id, - count = drained.len(), + count = drained, "Drained pending decryption queue for unblocked user" ); } diff --git a/crates/offline-protocol/src/protocol/config_accessors.rs b/crates/offline-protocol/src/protocol/config_accessors.rs index 3f7843bf..79c800ec 100644 --- a/crates/offline-protocol/src/protocol/config_accessors.rs +++ b/crates/offline-protocol/src/protocol/config_accessors.rs @@ -11,8 +11,7 @@ use crate::{Error, ProtocolConfig, Result, TransportManager}; use offline_protocol_core::{MessageId, ServiceDescriptor}; use offline_protocol_mls::MlsManager; use offline_protocol_reliability::{ - AckConfig, AckManager, Deduplicator, DeduplicatorConfig, DeduplicatorStats, RetryConfig, - RetryQueue, + AckConfig, AckManager, DeduplicatorConfig, DeduplicatorStats, RetryConfig, RetryQueue, }; use offline_protocol_router::{DorsConfig, RelayConfig}; use offline_protocol_services::MeshServices; @@ -195,7 +194,15 @@ impl OfflineProtocol { /// Updates the deduplication configuration at runtime. /// - /// Note: This clears the deduplication cache and applies the new config. + /// The tracked set is carried across, re-bounded by the new configuration + /// (`Deduplicator::reconfigure`), rather than cleared. Clearing it was a + /// silent restart while the set lived in memory only; with the set + /// persisted it would discard the ids restored at launch — and the React + /// Native layer applies `reliability.dedup` through this method on every + /// `start()`, right after that restore — so the socket copy of a message + /// already consumed from a push injection would reach a spent ratchet + /// generation after all. The change is counted for persistence so the next + /// batch write re-states the record under the new bounds. /// /// Validated the same way its two siblings are, and for a reason that only /// appeared once they were. Both of those run the *whole* @@ -211,8 +218,9 @@ impl OfflineProtocol { let mut candidate = self.config.clone(); candidate.reliability.dedup = config.clone(); candidate.validate()?; - self.deduplicator = Deduplicator::with_config(config.clone()); + self.deduplicator.reconfigure(config.clone()); self.config.reliability.dedup = config; + self.note_dedup_change(); Ok(()) } diff --git a/crates/offline-protocol/src/protocol/data_sync.rs b/crates/offline-protocol/src/protocol/data_sync.rs index e1399da0..4ea7160c 100644 --- a/crates/offline-protocol/src/protocol/data_sync.rs +++ b/crates/offline-protocol/src/protocol/data_sync.rs @@ -761,7 +761,7 @@ impl OfflineProtocol { if self.deduplicator.is_duplicate(&message.id) { return; } - self.deduplicator.mark_seen(message.id.clone()); + self.deduplicator.mark_seen_local(message.id.clone()); // Tier 2: keep the plaintext so a resend after a re-key seals against // the peer's current epoch instead of replaying ciphertext they can diff --git a/crates/offline-protocol/src/protocol/decryption_queue.rs b/crates/offline-protocol/src/protocol/decryption_queue.rs index 8506633e..9162a872 100644 --- a/crates/offline-protocol/src/protocol/decryption_queue.rs +++ b/crates/offline-protocol/src/protocol/decryption_queue.rs @@ -50,6 +50,38 @@ pub(crate) struct DroppedPendingMessage { pub(crate) reason: &'static str, } +/// What [`PendingDecryptionQueue::enqueue_via`] did with the incoming frame, +/// plus every frame it dropped along the way. +/// +/// `admitted` is what the persistence hook keys on: only a frame the queue now +/// holds has a record worth writing. It is `false` both when the frame was +/// refused (it is then `refused`) and when the id was already queued, the +/// resend case, where the original copy stays authoritative and nothing +/// changed in memory or on disk. +/// +/// `dropped` and `refused` are kept apart because they differ on disk. Every +/// entry in `dropped` was admitted earlier, so it may have a persisted record +/// to delete. The refused frame was never admitted: on the live path it has no +/// record, and deleting one anyway would cost a storage round trip per frame +/// during exactly the flood that fills the queue. On the restore path it came +/// off disk and does have one, so that caller deletes it. +pub(crate) struct EnqueueOutcome { + pub(crate) admitted: bool, + pub(crate) dropped: Vec, + pub(crate) refused: Option, +} + +/// How `enqueue_via_inner` resolved the incoming frame. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Admission { + /// The queue now holds the frame. + Admitted, + /// The id was already queued; the original copy stays. + AlreadyQueued, + /// The queue could not take the frame under its caps or overflow policy. + Refused, +} + #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum QueueLimit { PerPeer, @@ -112,7 +144,7 @@ pub struct PendingQueueMetrics { /// Encapsulates the bounded pending decryption queue: encrypted messages /// received before the MLS session is ready, with per-peer and global limits, /// TTL expiration, and configurable overflow policies. -#[derive(Default)] +#[derive(Clone, Default)] pub(crate) struct PendingDecryptionQueue { /// Per-peer FIFO queues of encrypted messages. queues: HashMap>, @@ -646,23 +678,51 @@ impl PendingDecryptionQueue { sender: &str, message: &Message, ) -> Vec { - self.enqueue_via(config, sender, message, None) + let outcome = self.enqueue_via(config, sender, message, None); + let mut dropped = outcome.dropped; + dropped.extend(outcome.refused); + dropped } /// Enqueues an encrypted message that arrived before the MLS session was /// ready, recording the transport it arrived on so the drain can ACK it /// directly. /// - /// Returns every message dropped in the process — TTL-expired entries, - /// entries evicted to make room, or the incoming message itself when it - /// could not be admitted — so the protocol layer can surface the loss. + /// Returns whether the frame was admitted, every queued entry dropped in + /// the process (TTL-expired, or evicted to make room), and the incoming + /// frame itself when it could not be admitted, so the protocol layer can + /// surface the loss and keep the persisted copy in step. See + /// [`EnqueueOutcome`] for why the refused frame is reported apart. pub(crate) fn enqueue_via( &mut self, config: &PendingQueueConfig, sender: &str, message: &Message, arrival_transport: Option, - ) -> Vec { + ) -> EnqueueOutcome { + let mut dropped = Vec::new(); + let admission = + self.enqueue_via_inner(config, sender, message, arrival_transport, &mut dropped); + EnqueueOutcome { + admitted: admission == Admission::Admitted, + dropped, + refused: (admission == Admission::Refused).then(|| DroppedPendingMessage { + message: message.clone(), + reason: DropReason::OverflowDropNewest.as_str(), + }), + } + } + + /// The body of [`Self::enqueue_via`]. Only entries that were already queued + /// go onto `dropped`; the incoming frame's fate is the return value. + fn enqueue_via_inner( + &mut self, + config: &PendingQueueConfig, + sender: &str, + message: &Message, + arrival_transport: Option, + dropped: &mut Vec, + ) -> Admission { self.metrics.pending_messages_received_total = self .metrics .pending_messages_received_total @@ -670,7 +730,7 @@ impl PendingDecryptionQueue { let incoming_message_id = message.id.as_str(); let now = Instant::now(); - let mut dropped = self.prune_expired_for_peer(config, sender, now); + dropped.extend(self.prune_expired_for_peer(config, sender, now)); dropped.extend(self.prune_expired_global_front(config, now, 64)); // Idempotent by message id: under the deferred-ACK model the sender @@ -685,7 +745,7 @@ impl PendingDecryptionQueue { .iter() .any(|entry| entry.message_id == incoming_message_id) }) { - return dropped; + return Admission::AlreadyQueued; } let per_peer_limit = config.max_pending_per_peer; @@ -711,11 +771,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } let peer_len = self.queues.get(sender).map(VecDeque::len).unwrap_or(0); @@ -730,11 +786,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } OverflowPolicy::DropOldest => { let evicted_sequence = self @@ -772,11 +824,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } } } @@ -796,11 +844,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } if self.peer_bytes_for(sender) + incoming_bytes > per_peer_bytes_limit { @@ -813,11 +857,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } // DropOldest: evict from the peer's front until the incoming // message fits. Terminates: each eviction shrinks the peer's byte @@ -858,11 +898,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } } } @@ -882,11 +918,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } OverflowPolicy::DropOldest => { while self.total >= global_limit { @@ -921,11 +953,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } if self.total_bytes + incoming_bytes > global_bytes_limit { @@ -941,11 +969,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } while self.total_bytes + incoming_bytes > global_bytes_limit { match self.evict_global_oldest( @@ -976,11 +1000,7 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); - return dropped; + return Admission::Refused; } } @@ -1037,13 +1057,10 @@ impl PendingDecryptionQueue { &incoming_message_id, overflow_policy, ); - dropped.push(DroppedPendingMessage { - message: message.clone(), - reason: DropReason::OverflowDropNewest.as_str(), - }); + return Admission::Refused; } - dropped + Admission::Admitted } /// Drains all pending messages for a peer, updating bookkeeping. diff --git a/crates/offline-protocol/src/protocol/message_dispatch.rs b/crates/offline-protocol/src/protocol/message_dispatch.rs index 50b8adca..e71a4312 100644 --- a/crates/offline-protocol/src/protocol/message_dispatch.rs +++ b/crates/offline-protocol/src/protocol/message_dispatch.rs @@ -383,9 +383,11 @@ impl OfflineProtocol { // Drain the inbound pending decryption queue though: // unlike the outbound side those really are // ciphertexts sealed to the session just deleted, so - // they can never decrypt. - self.pending_queue - .drain_for_peer(&self.config.encryption.pending_queue, sender); + // they can never decrypt. Their persisted records go + // too: left on disk, the next launch restores them and + // drains them into the replacement session as + // spurious decrypt failures. + self.discard_pending_decryption_for_peer(sender); // Allow fresh key exchange self.key_package_sent_to.remove(sender); } diff --git a/crates/offline-protocol/src/protocol/mod.rs b/crates/offline-protocol/src/protocol/mod.rs index 0aa803df..2211b7e8 100644 --- a/crates/offline-protocol/src/protocol/mod.rs +++ b/crates/offline-protocol/src/protocol/mod.rs @@ -769,6 +769,16 @@ pub struct OfflineProtocol { /// next merge with any peer). last_persisted_lamport: u64, + /// Receive-side changes to the deduplicator's seen set since it was last + /// written. Drives the batched write in `persist_dedup_seen_if_due`; + /// see `DedupSeenRecord` for why the set is persisted at all. + dedup_dirty: u32, + + /// When the seen set was last written (or restored), for the time half of + /// the batching rule. Starts at construction so the first write waits for + /// either the interval or the dirty threshold like every later one. + dedup_last_persist: Instant, + /// The Nostr receive watermark last written to storage, or `None` if this /// session has neither written nor restored one. Debounces /// `persist_nostr_watermark()` the same way `last_persisted_lamport` @@ -882,6 +892,10 @@ impl Drop for OfflineProtocol { // protocol is dropped without an explicit stop() call. self.flush_lamport_clock(); self.flush_nostr_watermark(); + // And the deduplicator's seen set, whose write is batched the same + // way: a copy of a message received in the last few seconds before + // the drop would otherwise be re-processed by the next launch. + self.flush_dedup_seen(); // Same reason: edits batch before they reach a record, so without a // flush here the debounce window between an edit and its delta is a // window in which work is lost. @@ -1069,6 +1083,8 @@ impl OfflineProtocol { transport_status_snapshot: HashMap::new(), device_capability_snapshot: None, last_persisted_lamport: 0, + dedup_dirty: 0, + dedup_last_persist: Instant::now(), last_persisted_nostr_watermark: None, nostr_publication_slots: Vec::new(), nostr_published_slots: HashSet::new(), @@ -1169,6 +1185,10 @@ impl OfflineProtocol { let previous_state_record_cipher = self.state_record_cipher.take(); let previous_pending_messages = self.pending_encrypted_messages.clone(); let previous_pending_message_expiry = self.next_pending_message_expiry; + // The inbound half of the same transaction: the restore re-admits + // parked frames from the store being attached, so a rollback has to put + // back what the queue held before, not an empty queue. + let previous_pending_queue = self.pending_queue.clone(); // Populated by restore steps that run before other fallible ones, so // they belong in the transaction like everything else: a failed init // must not leave a key-package cache, parked media descriptors, or an @@ -1273,7 +1293,7 @@ impl OfflineProtocol { // The launch's whole durable-delete allowance, in one place because the // bound is on the launch. A device-barrier storm kills *this call*, not // any single walk in it, so no walk may hand itself a pool — see - // `PruneAllowance::pool`. Three of them, and the ceiling is their sum: + // `PruneAllowance::pool`. Four of them, and the ceiling is their sum: // // - `advisory_prunes` is shared by the `ADVISORY_PRUNE_WALKS` walks // whose prunes are caches or advisory signals. They draw in the order @@ -1287,16 +1307,23 @@ impl OfflineProtocol { // diagnostic or a *delivery* rather than a cache eviction, and // neither may be held hostage to a key-package flood in a category it // has nothing to do with. + // - `inbound_prunes` is private to the two receive-side walks for the + // same reason: the pending-decryption walk holds ciphertext the app + // is told about when it is lost. It draws first, and the seen-set + // restore takes whatever is left for its one possible delete. // // See `storage::MAX_RESTORE_PRUNE_DELETES`. let mut advisory_prunes = PruneAllowance::pool(); let mut pending_prunes = PruneAllowance::pool(); let mut outbox_prunes = PruneAllowance::pool(); + let mut inbound_prunes = PruneAllowance::pool(); // Restore state from previous session let restore_result = (|| { self.restore_pending_messages(&mut pending_prunes)?; + self.restore_pending_decrypt_entries(&mut inbound_prunes); self.restore_lamport_clock(); + self.restore_dedup_seen(&mut inbound_prunes); self.restore_encryption_capable_peers(); self.restore_blocked_users()?; self.restore_session_states_from_manager(manager.clone(), &mut advisory_prunes)?; @@ -1321,6 +1348,13 @@ impl OfflineProtocol { self.state_record_cipher = previous_state_record_cipher; self.pending_encrypted_messages = previous_pending_messages; self.next_pending_message_expiry = previous_pending_message_expiry; + // Back to what the queue held before this call. Emptying it lost + // frames that were never persisted (no store was attached, or the + // best-effort write failed). Keeping what the restore added would + // leave entries sourced from the store the rollback just detached: + // draining one deletes nothing on disk, so the next launch restores + // it and drains it again into a spent ratchet generation. + self.pending_queue = previous_pending_queue; // `deferred_restore_settlements` is deliberately left alone — see // the comment where the other baselines are captured. self.pending_key_packages = previous_pending_key_packages; @@ -1416,12 +1450,15 @@ impl OfflineProtocol { self.restore_nostr_watermark(); self.restore_nostr_publication_slots(); self.restore_nostr_discovery_claim(); - // Same three pools as `initialize_mls_inner`, for the same reason. + // Same four pools as `initialize_mls_inner`, for the same reason. let mut advisory_prunes = PruneAllowance::pool(); let mut pending_prunes = PruneAllowance::pool(); let mut outbox_prunes = PruneAllowance::pool(); + let mut inbound_prunes = PruneAllowance::pool(); self.restore_pending_messages(&mut pending_prunes)?; + self.restore_pending_decrypt_entries(&mut inbound_prunes); self.restore_lamport_clock(); + self.restore_dedup_seen(&mut inbound_prunes); self.restore_encryption_capable_peers(); self.restore_blocked_users()?; self.restore_outbox(&mut outbox_prunes)?; @@ -1991,6 +2028,9 @@ impl OfflineProtocol { // Same for the Nostr receive watermark: an un-flushed tail costs the // next launch a wider replay window. self.flush_nostr_watermark(); + // And the deduplicator's seen set: an un-flushed tail is a message + // whose second copy the next launch fails to recognise. + self.flush_dedup_seen(); // Answer any lookup still in flight while the transport is still up to // take the cancellations. Nothing pumps relays or sweeps deadlines once // stopped, so a resolution left here is an event that never comes. @@ -3276,6 +3316,9 @@ impl OfflineProtocol { self.run_throttled_reconciliation("process_tick"); let _ = self.prune_expired_pending_global_front(Instant::now(), 256); + // Batched like the Lamport clock: the seen set reaches disk on a + // change threshold or a time cadence, never per message. + self.persist_dedup_seen_if_due(); self.pump_media_transfers(); self.refresh_nostr_key_package_slots(); self.refresh_nostr_discovery_claim(); diff --git a/crates/offline-protocol/src/protocol/pending_queue.rs b/crates/offline-protocol/src/protocol/pending_queue.rs index 67b2b507..6ceb4a91 100644 --- a/crates/offline-protocol/src/protocol/pending_queue.rs +++ b/crates/offline-protocol/src/protocol/pending_queue.rs @@ -2,7 +2,7 @@ //! need access to the broader [`OfflineProtocol`] state (shared state, MLS //! decryption, lamport clock). -use super::decryption_queue::DroppedPendingMessage; +use super::decryption_queue::{DroppedPendingMessage, PendingDecryptMessage}; use super::{lock_shared_state, ChunkOutcome, InternalMessageResult, OfflineProtocol}; use crate::events::{DecryptionFailureCode, Event}; use chrono::Utc; @@ -29,6 +29,11 @@ impl OfflineProtocol { /// entry so the drain can send the deferred delivery ACK directly instead of /// relying on the sender's next resend (see the deferred-acknowledgement /// atom in `docs/state-machines/delivery-and-acks.md`). + /// + /// An admitted frame is also written to protocol-state storage + /// (`PendingDecryptRecord`), stamped with its first receipt, so it survives + /// a restart; see [`Self::enqueue_restored_pending_decryption`] for the + /// way back in. pub(super) fn enqueue_pending_decryption_via( &mut self, sender: &str, @@ -36,10 +41,50 @@ impl OfflineProtocol { arrival_transport: Option, ) { let config = &self.config.encryption.pending_queue; - let dropped = self + let outcome = self .pending_queue .enqueue_via(config, sender, message, arrival_transport); - self.report_dropped_pending_media(dropped); + // Only entries the queue let go can have records. A refused incoming + // frame was never written, so it is reported but not deleted. + self.delete_pending_decrypt_entries_from_storage( + outcome.dropped.iter().map(|entry| &entry.message.id), + ); + let events = Self::pending_drop_events(outcome.dropped.into_iter().chain(outcome.refused)); + self.emit_pending_drop_events(events); + if outcome.admitted { + self.persist_pending_decrypt_entry( + sender, + message, + Utc::now().timestamp_millis(), + arrival_transport, + ); + } + } + + /// Re-admits a frame read back from storage: the in-memory enqueue with + /// the same overflow handling as the live path, but **no** write, since the + /// record on disk is already the durable copy and keeps its first-receipt + /// timestamp. + /// + /// Returns every frame the enqueue let go, the refused incoming one + /// included. Each came off disk, so unlike on the live path each still has + /// a record, and neither the delete nor the report happens here: the + /// restore walk charges every delete to the launch budget and reports a + /// drop only together with its delete (see + /// `restore_pending_decrypt_entries`). A restore also runs before + /// `start()`, so the walk settles those reports through + /// `settle_restored_message_failures` rather than emitting them. + pub(crate) fn enqueue_restored_pending_decryption( + &mut self, + sender: &str, + message: &offline_protocol_core::Message, + received_via: Option, + ) -> Vec { + let config = &self.config.encryption.pending_queue; + let outcome = self + .pending_queue + .enqueue_via(config, sender, message, received_via); + outcome.dropped.into_iter().chain(outcome.refused).collect() } pub(super) fn prune_expired_pending_global_front( @@ -52,45 +97,113 @@ impl OfflineProtocol { .pending_queue .prune_expired_global_front(config, now, max_evictions); let count = expired.len(); - self.report_dropped_pending_media(expired); + self.report_dropped_pending(expired); count } - /// Surfaces pending-queue evictions of encrypted media chunks so an app - /// can react to a stalled transfer instead of watching it hang silently. - /// (The chunk is still encrypted at this point, so the file_id cannot be - /// named here.) + /// Drains a peer's parked frames without processing them, deleting each + /// one's persisted record. For the paths where every queued frame from a + /// peer is being discarded at once: the session reset on unblock, and the + /// peer-requested session reset in `message_dispatch`. Returns how many + /// were discarded. /// - /// Under the deferred-ACK model this is **advisory, not terminal**: an - /// evicted chunk was never ACKed, so the sender keeps retransmitting and a - /// later resend re-enters the queue and can still complete the transfer - /// once the session confirms. The event says the transfer is *stalled*, not - /// that it has failed — the terminal media signal is `FileReceiveFailed`. + /// Every discard must come through here rather than calling + /// `drain_for_peer` directly, or the records outlive their entries. + pub(super) fn discard_pending_decryption_for_peer(&mut self, peer_id: &str) -> usize { + let config = self.config.encryption.pending_queue.clone(); + let drained = self.pending_queue.drain_for_peer(&config, peer_id); + self.delete_pending_decrypt_entries_from_storage(drained.iter().map(|e| &e.message.id)); + drained.len() + } + + /// Takes a peer's parked frames out of the queue for processing, deleting + /// their persisted records first: whatever the drain does with a frame — + /// surface it, consume it, or drop it as undecryptable — the queue no + /// longer holds it, and a record that outlived its entry would be + /// restored and re-drained on the next launch, where a frame whose ratchet + /// generation was spent by this drain surfaces as a spurious decrypt + /// failure. A frame the handler re-queues during the drain (still + /// session-not-ready) goes through the live enqueue and is re-persisted + /// there. + fn take_pending_decryption_for_drain(&mut self, sender: &str) -> Vec { + let config = self.config.encryption.pending_queue.clone(); + let drained = self.pending_queue.drain_for_peer(&config, sender); + self.delete_pending_decrypt_entries_from_storage(drained.iter().map(|e| &e.message.id)); + drained + } + + /// Surfaces every pending-queue eviction as a `PendingQueueDropped` + /// decryption failure so an app can react — a stalled media transfer, or a + /// text message that will only arrive if the sender resends — instead of + /// watching the gap silently. (The frame is still encrypted at this point, + /// so neither the file_id nor the text can be named here.) /// - /// Dropped text messages keep their existing metrics-only handling: the - /// pending message queue was sized for them, they recover on the sender's - /// next resend, and their loss is already tracked via `PendingQueueMetrics`. - fn report_dropped_pending_media(&mut self, dropped: Vec) { - for entry in dropped { - if entry.message.content_type != ContentType::FileChunk { - continue; - } - warn!( - sender = %entry.message.sender, - message_id = %entry.message.id, - reason = entry.reason, - "Encrypted media chunk evicted from pending queue; its file transfer is stalled until the sender resends" - ); - if let Ok(state) = lock_shared_state(&self.shared_state) { - state.emit_event(Event::message_decryption_failed( - entry.message.id.clone(), - entry.message.sender.as_str().to_string(), - DecryptionFailureCode::PendingQueueDropped, + /// Under the deferred-ACK model this is **advisory, not terminal**: an + /// evicted frame was never ACKed, so a sender still retrying resends it, and + /// the resend re-enters the queue and can still complete once the session + /// confirms. The event says the message is *at risk*, not that it has + /// failed — for media the terminal signal is `FileReceiveFailed`. It is + /// emitted for text as well as for chunks: text drops used to be + /// metrics-only, which left an app with no way to distinguish "the sender + /// went quiet" from "the SDK evicted their message", and a relay that + /// pushes ciphertext without store-and-forward has no second copy to fall + /// back on, so silence there was a lost message. + fn report_dropped_pending(&mut self, dropped: Vec) { + // A dropped frame's record goes with it: the on-disk copy exists only + // to mirror what the queue holds, and a record that outlived its entry + // would be restored, re-dropped and re-reported on the next launch. + self.delete_pending_decrypt_entries_from_storage(dropped.iter().map(|e| &e.message.id)); + let events = Self::pending_drop_events(dropped); + self.emit_pending_drop_events(events); + } + + /// Logs each dropped frame and builds its `PendingQueueDropped` event + /// without emitting it. Split from the emit because the restore path has + /// to defer its events until the event pipeline is live. + pub(super) fn pending_drop_events( + dropped: impl IntoIterator, + ) -> Vec { + dropped + .into_iter() + .map(|entry| { + let is_media_chunk = entry.message.content_type == ContentType::FileChunk; + let reason = if is_media_chunk { format!( "encrypted media chunk evicted from pending queue ({}); its file transfer is stalled until the sender resends", entry.reason - ), - )); + ) + } else { + format!( + "encrypted message evicted from pending queue ({}); recoverable only if the sender resends", + entry.reason + ) + }; + warn!( + sender = %entry.message.sender, + message_id = %entry.message.id, + content_type = %entry.message.content_type, + reason = entry.reason, + "Encrypted message evicted from pending queue; recoverable only if the sender resends" + ); + Event::message_decryption_failed( + entry.message.id.clone(), + entry.message.sender.as_str().to_string(), + DecryptionFailureCode::PendingQueueDropped, + reason, + ) + }) + .collect() + } + + /// Emits events built by [`Self::pending_drop_events`], taking the + /// shared-state lock once for the batch. + fn emit_pending_drop_events(&self, events: Vec) { + if events.is_empty() { + return; + } + if let Ok(state) = lock_shared_state(&self.shared_state) { + for event in events { + state.emit_event(event); } } } @@ -133,8 +246,8 @@ impl OfflineProtocol { let expired = self .pending_queue .prune_expired_for_peer(&config, sender, Instant::now()); - self.report_dropped_pending_media(expired); - let drained = self.pending_queue.drain_for_peer(&config, sender); + self.report_dropped_pending(expired); + let drained = self.take_pending_decryption_for_drain(sender); if drained.is_empty() { return; @@ -177,7 +290,7 @@ impl OfflineProtocol { // now on its arrival transport so the sender can stop // retrying without a further resend. ChunkOutcome::Handled => { - self.deduplicator.mark_seen(msg.id.clone()); + self.mark_seen_persisted(msg.id.clone()); self.ack_drained_message(&msg, received_via); } // Not delivered, recoverable by a resend. Two shapes, both @@ -261,7 +374,7 @@ impl OfflineProtocol { // delivery) or re-decrypted (an MLS replay the ratchet // would reject). This is the counterpart to the unmark // in the receive loop's `Deferred` arm. - self.deduplicator.mark_seen(msg.id.clone()); + self.mark_seen_persisted(msg.id.clone()); // ACK on drain: the message is delivered locally now, so // send the deferred delivery ACK directly on its arrival @@ -279,7 +392,7 @@ impl OfflineProtocol { // path; re-mark so a resend is deduped rather than // reprocessed, and ACK it (control messages are // delivery-sensitive, exactly like the live path). - self.deduplicator.mark_seen(msg.id.clone()); + self.mark_seen_persisted(msg.id.clone()); self.ack_drained_message(&msg, received_via); debug!(message_id = %msg.id, "Delayed message was consumed internally"); } diff --git a/crates/offline-protocol/src/protocol/receive.rs b/crates/offline-protocol/src/protocol/receive.rs index 6f23425a..c6391fb0 100644 --- a/crates/offline-protocol/src/protocol/receive.rs +++ b/crates/offline-protocol/src/protocol/receive.rs @@ -204,7 +204,7 @@ impl OfflineProtocol { continue; } - self.deduplicator.mark_seen(message.id.clone()); + self.mark_seen_persisted(message.id.clone()); // Handle internal MLS messages let mut was_decrypted = false; @@ -240,7 +240,7 @@ impl OfflineProtocol { // duplicate re-ACK path above and leak that presence // anyway; reprocessing a replay costs no more than a // fresh forged message. - self.deduplicator.unmark_seen(&message.id); + self.unmark_seen_persisted(&message.id); continue; } InternalMessageResult::Deferred => { @@ -264,7 +264,7 @@ impl OfflineProtocol { // re-ACKed. For the other two, recovery is that // resend itself — re-sealed against a live // generation by Tier 2. - self.deduplicator.unmark_seen(&message.id); + self.unmark_seen_persisted(&message.id); continue; } InternalMessageResult::Decrypted(plaintext) => { @@ -300,7 +300,7 @@ impl OfflineProtocol { message.sender.as_str(), "Inbound plaintext message rejected by encryption policy", ); - self.deduplicator.unmark_seen(&message.id); + self.unmark_seen_persisted(&message.id); continue; } @@ -320,7 +320,7 @@ impl OfflineProtocol { if message.content_type == ContentType::FileChunk { match self.handle_incoming_file_chunk_via(&message, Some(transport_used)) { ChunkOutcome::Deferred => { - self.deduplicator.unmark_seen(&message.id); + self.unmark_seen_persisted(&message.id); } // Plaintext chunk rejected by encryption policy: // withhold the ACK and unmark the id, exactly like @@ -330,7 +330,7 @@ impl OfflineProtocol { // replay re-enter the gate rather than hit the // duplicate re-ACK path. ChunkOutcome::Rejected => { - self.deduplicator.unmark_seen(&message.id); + self.unmark_seen_persisted(&message.id); } ChunkOutcome::Handled => { if message.requires_ack { diff --git a/crates/offline-protocol/src/protocol/send.rs b/crates/offline-protocol/src/protocol/send.rs index 5ee0e442..9acd09fa 100644 --- a/crates/offline-protocol/src/protocol/send.rs +++ b/crates/offline-protocol/src/protocol/send.rs @@ -11,8 +11,8 @@ use super::{ MAX_PENDING_EXPIRIES_PER_PASS, MAX_PENDING_MESSAGES_GLOBAL, MAX_PENDING_MESSAGES_PER_PEER, MAX_PENDING_MESSAGE_BYTES_GLOBAL, MAX_PENDING_MESSAGE_BYTES_PER_PEER, MAX_READ_RECEIPT_IDS, MAX_RICH_EXTRAS_BYTES, MLS_ENVELOPE_COMPACT_V1, PENDING_CONNECTION_REQUEST_TTL, - RICH_PAYLOAD_V1, SEND_FAIL_REASON_RECIPIENT_UNREACHABLE, SEND_FAIL_REASON_TRANSPORT, - WELCOME_NO_CARRIER_RETRY_SECS, WELCOME_UNREACHABLE_RETRY_CAP_SECS, + RICH_PAYLOAD_V1, SEND_FAIL_REASON_RECIPIENT_UNREACHABLE, SEND_FAIL_REASON_RELAY_PUSHED, + SEND_FAIL_REASON_TRANSPORT, WELCOME_NO_CARRIER_RETRY_SECS, WELCOME_UNREACHABLE_RETRY_CAP_SECS, }; use super::{classify_transport_send_error, send_failure_token}; use crate::constants::{ @@ -364,7 +364,7 @@ impl OfflineProtocol { return Err(crate::Error::Other("Duplicate message".to_string())); } - self.deduplicator.mark_seen(message_id.clone()); + self.deduplicator.mark_seen_local(message_id.clone()); let previous_transport = self.transport_manager.current_transport(); @@ -454,7 +454,7 @@ impl OfflineProtocol { return Err(crate::Error::Other("Duplicate message".to_string())); } - self.deduplicator.mark_seen(message_id.clone()); + self.deduplicator.mark_seen_local(message_id.clone()); let previous_transport = self.transport_manager.current_transport(); let send_result = self @@ -522,7 +522,7 @@ impl OfflineProtocol { return Err(crate::Error::Other("Duplicate message".to_string())); } - self.deduplicator.mark_seen(message_id.clone()); + self.deduplicator.mark_seen_local(message_id.clone()); let previous_transport = self.transport_manager.current_transport(); let send_result = self.transport_manager.send(&message); @@ -616,7 +616,7 @@ impl OfflineProtocol { // Mark seen so a bridge that passes the frame through verbatim (an // adapter without a translator) cannot have the relay echo it back // into our own receive path. - self.deduplicator.mark_seen(message_id.clone()); + self.deduplicator.mark_seen_local(message_id.clone()); self.transport_manager .send_via_transport(&message, TransportType::Internet)?; @@ -666,7 +666,7 @@ impl OfflineProtocol { if self.deduplicator.is_duplicate(&message.id) { return; } - self.deduplicator.mark_seen(message.id.clone()); + self.deduplicator.mark_seen_local(message.id.clone()); let previous_transport = self.transport_manager.current_transport(); match self.transport_manager.send(&message) { @@ -2891,6 +2891,7 @@ impl OfflineProtocol { last_sent_at: Utc::now(), last_transport: None, reseal: staged_reseal, + relay_pushed: false, }); // Persist newly-created main-outbox entries so they survive a restart. // media entries are intentionally not persisted. @@ -2932,6 +2933,7 @@ impl OfflineProtocol { last_sent_at: now, last_transport: transport, reseal: staged_reseal, + relay_pushed: false, }); entry.message = message.clone(); @@ -3637,6 +3639,14 @@ impl OfflineProtocol { let transport_error = transport_error .as_deref() .map(classify_transport_send_error); + // A relay push is not a failure, so it is handled before either typed + // branch below can read it as one: a connection request would be + // fast-failed as undeliverable and a Welcome moved to `Failed`, both + // for a frame the push may have delivered. + if transport_error == Some(SEND_FAIL_REASON_RELAY_PUSHED) { + self.park_relay_pushed_dm(message_id, carrier); + return Ok(()); + } // Connection requests first: the relay's "recipient offline" // DeliveryError is the only fast, authoritative failure signal a // request gets (there is no ACK from an offline peer — the app @@ -3779,6 +3789,16 @@ impl OfflineProtocol { }; let recipient = entry.message.recipient.as_str().to_string(); let attempt_count = entry.attempt_count; + // A verdict for a frame the relay has already pushed is the answer to + // our own probe (`OutboxEntry::relay_pushed`): the relay refuses to + // notify twice, so it says `DeliveryError` for a message that may be + // sitting delivered on the recipient's device. Everything below still + // applies (the recipient is not on the relay, so record it, back the + // probe off, and re-park), except the app-facing event: telling the + // app the message is undeliverable would be exactly the claim + // `park_relay_pushed_dm` exists not to make. Media is never pushed + // and never parked, so the flag is only ever read off the main outbox. + let relay_pushed = !is_media && entry.relay_pushed; // The verdict, recorded as a fact before anything acts on it. Parking // below reacts to *this* message; the fact is what lets the next send // to the same recipient skip a carrier that has already said no. @@ -3805,18 +3825,25 @@ impl OfflineProtocol { // targets (e.g. an ordinary DM to a confirmed peer). Done here, after // the `entry` borrow of `self.outbox` has ended. self.note_confirmation_probe_unreachable(&recipient); - warn!( - message_id = %message_id, - file_id = ?file_id, - parked = !is_media, - "Recipient unreachable for in-flight message (non-terminal)" - ); - self.emit_event(Event::message_undeliverable( - parsed_id.clone(), - recipient.clone(), - reason, - file_id, - )); + if relay_pushed { + debug!( + message_id = %message_id, + "Recipient unreachable for a relay-pushed DM: re-parking without notifying the app" + ); + } else { + warn!( + message_id = %message_id, + file_id = ?file_id, + parked = !is_media, + "Recipient unreachable for in-flight message (non-terminal)" + ); + self.emit_event(Event::message_undeliverable( + parsed_id.clone(), + recipient.clone(), + reason, + file_id, + )); + } if let Some(frame) = media_frame { // The recipient is not on the relay, but they may be a few devices // away. A chunk keeps its pending ACK (media is never parked), so @@ -3837,6 +3864,89 @@ impl OfflineProtocol { self.park_unreachable_dm(&parsed_id, &recipient, attempt_count); } + /// Whether a platform send report also belongs in the carrier's delivery + /// metrics (`Transport::report_send_failure`). + /// + /// Every report does except `relay_pushed`. The bridges hand the relay's + /// `MessageSent { pushed: true }` to [`Self::on_transport_send_failed_via`] + /// because that is their one call for "the relay answered about this id", + /// but the relay took the frame into a device push, which is not a failed + /// send. A wrapper that scores the report without asking here records a + /// failure whenever the relay's answer beats the bridge's own write + /// confirmation, against the carrier the router ranks by that ratio. + pub fn send_report_is_carrier_failure(reason: Option<&str>) -> bool { + reason.map(classify_transport_send_error) != Some(SEND_FAIL_REASON_RELAY_PUSHED) + } + + /// Handles the relay's `MessageSent { pushed: true }` answer: the recipient + /// has no live socket, and the frame went out in a push notification. + /// + /// A plain DM is parked exactly as for a `DeliveryError`. The relay has no + /// store-and-forward, so if the push is lost nothing will re-deliver the + /// frame when the recipient reconnects, and without the park the missing + /// ACK burns the retry budget to a terminal `message_failed`. If the push + /// did deliver, the recipient's acknowledgement settles the parked entry + /// (`settle_parked_dm_from_ack`). + /// + /// Two differences from the `DeliveryError` path, both because the push + /// may have delivered the frame: + /// + /// - **Plain DMs only.** A connection request keeps its typed tracking and + /// a Welcome its lifecycle, both awaiting the acknowledgement or session + /// confirmation a delivered push produces. `is_parkable_plain_dm` + /// excludes both, and media chunks are never in `outbox`. + /// - **No `MessageUndeliverable`, now or on any later probe.** The app + /// would be told a message is undeliverable when it may already have + /// arrived. The park's timed probe resends the same id, and the relay + /// answers a retry of a pushed message with `DeliveryError` rather than + /// a second notification, which the bridges can only report as + /// `recipient_unreachable`. So the entry is flagged + /// (`OutboxEntry::relay_pushed`, persisted) and + /// [`Self::handle_recipient_unreachable_for_message`] re-parks a flagged + /// entry silently. Without the flag the guarantee would last exactly + /// one probe interval. + fn park_relay_pushed_dm(&mut self, message_id: &str, carrier: Option) { + let Ok(parsed_id) = MessageId::from_str(message_id) else { + return; + }; + if !self.is_parkable_plain_dm(&parsed_id) { + return; + } + let Some(entry) = self.outbox.get_mut(&parsed_id) else { + return; + }; + let recipient = entry.message.recipient.as_str().to_string(); + let attempt_count = entry.attempt_count; + let newly_pushed = !entry.relay_pushed; + entry.relay_pushed = true; + if newly_pushed { + // Re-persisted on the flip only: the entry was written when it + // was created, and the flag has to survive a restart for the + // silent re-park to hold on the first probe after one. + if let Some(entry) = self.outbox.get(&parsed_id) { + self.persist_outbox_entry(entry); + } + } + // The relay has said this recipient is not on it, which is the same + // fact a `DeliveryError` records, and the same thing the probe backoff + // keys on. + if let Some(carrier) = carrier { + self.reachability.record( + &recipient, + carrier, + Claim::Unreachable, + FactSource::GatewayVerdict, + Instant::now(), + ); + } + self.note_confirmation_probe_unreachable(&recipient); + debug!( + message_id = %message_id, + "Relay pushed a DM to an offline recipient; parking it" + ); + self.park_unreachable_dm(&parsed_id, &recipient, attempt_count); + } + /// The park action shared by the relay-verdict path /// ([`Self::handle_recipient_unreachable_for_message`]) and the /// exhausted-probe path ([`Self::try_repark_exhausted_dm`]): drops the @@ -3866,8 +3976,18 @@ impl OfflineProtocol { /// fresh `DeliveryError` re-enters this park, escalating the interval /// (15s → 600s cap) and re-emitting the non-terminal /// [`Event::MessageUndeliverable`]; - /// - the relay's push fallback succeeds → no verdict is returned at all - /// and the probe becomes an ordinary in-flight send on the ACK ladder; + /// - the relay's push fallback succeeds → the relay answers + /// `MessageSent { pushed: true }`, which parks the message again + /// ([`Self::park_relay_pushed_dm`]) and escalates the interval. That + /// happens once per push: the relay remembers which + /// `(sender, recipient, message_id)` triples it has pushed, for a day, + /// and answers a repeat inside that window with `DeliveryError` + /// (`already_pushed`) instead of a second notification. The outbox id + /// is stable across every probe, so each later rung earns that + /// `DeliveryError`, which re-parks the entry silently + /// (`OutboxEntry::relay_pushed`) rather than re-emitting + /// [`Event::MessageUndeliverable`] for a frame the push may have + /// delivered; /// - the peer is back → the probe *is* the delivery, which beats waiting /// for any presence edge. /// diff --git a/crates/offline-protocol/src/protocol/storage.rs b/crates/offline-protocol/src/protocol/storage.rs index 913a30ed..58524ba1 100644 --- a/crates/offline-protocol/src/protocol/storage.rs +++ b/crates/offline-protocol/src/protocol/storage.rs @@ -3,23 +3,30 @@ use super::state_crypto::{StateRecordCipher, SEALED_RECORD_OVERHEAD, STATE_RECORD_KEY_BYTES}; use super::{ lifetime_expired, storage_keys, MediaTransferDescriptor, OfflineProtocol, OutboxEntry, - PeerCapabilities, PendingMessage, PendingMessageRecord, ReceivedKeyPackage, SessionState, - WelcomeDeliveryState, WelcomeLifecycleRecord, DATA_GROUP_V1, DATA_MEDIA_V1, DATA_SYNC_V1, - MAX_BLOCKED_USERS, MAX_KEY_PACKAGE_SENT_TO, MAX_MIGRATED_PENDING_WRITES_PER_LAUNCH, - MAX_PENDING_KEY_PACKAGES, MAX_PENDING_MESSAGES_GLOBAL, MAX_PENDING_MESSAGES_PER_PEER, - MAX_PENDING_MESSAGE_BYTES_GLOBAL, MAX_PENDING_MESSAGE_BYTES_PER_PEER, - MAX_PERSISTED_CAPABILITY_VERSIONS, MAX_PROTOCOL_STATE_RECORD_BYTES, MLS_ENVELOPE_COMPACT_V1, + PeerCapabilities, PendingDecryptRecord, PendingMessage, PendingMessageRecord, + ReceivedKeyPackage, SessionState, WelcomeDeliveryState, WelcomeLifecycleRecord, DATA_GROUP_V1, + DATA_MEDIA_V1, DATA_SYNC_V1, MAX_BLOCKED_USERS, MAX_KEY_PACKAGE_SENT_TO, + MAX_MIGRATED_PENDING_WRITES_PER_LAUNCH, MAX_PENDING_KEY_PACKAGES, MAX_PENDING_MESSAGES_GLOBAL, + MAX_PENDING_MESSAGES_PER_PEER, MAX_PENDING_MESSAGE_BYTES_GLOBAL, + MAX_PENDING_MESSAGE_BYTES_PER_PEER, MAX_PERSISTED_CAPABILITY_VERSIONS, + MAX_PROTOCOL_STATE_RECORD_BYTES, MLS_ENVELOPE_COMPACT_V1, PENDING_DECRYPT_RECORD_VERSION, RICH_PAYLOAD_V1, WELCOME_LIFECYCLE_TTL_SECS, }; +use super::{ + DedupSeenRecord, DEDUP_PERSIST_DIRTY_THRESHOLD, DEDUP_PERSIST_INTERVAL, + DEDUP_SEEN_RECORD_VERSION, MAX_PERSISTED_DEDUP_IDS, +}; use crate::constants::{MAX_MEDIA_DESCRIPTORS, MAX_OUTBOX_ENTRIES}; +use crate::events::DecryptionFailureCode; use crate::{Error, Event, ProtocolStateError, ProtocolStateResult, ProtocolStateStorage, Result}; use chrono::{Duration as ChronoDuration, Utc}; -use offline_protocol_core::{LamportClock, MessageId}; +use offline_protocol_core::{LamportClock, Message, MessageId}; use offline_protocol_mls::{MlsManager, MlsStorage}; use offline_protocol_transport::{NostrKeypair, NostrTransport, TransportType}; use serde::de::DeserializeOwned; use std::collections::{HashMap, HashSet}; use std::sync::{Arc, RwLock}; +use std::time::Instant; use tracing::{debug, info, warn}; use zeroize::Zeroizing; @@ -45,6 +52,11 @@ pub(crate) enum StateCategory { /// because reading it means decrypting it. PendingMessages, PendingMessageEntries, + /// Inbound ciphertext parked before its session was ready. Post-split + /// only, like [`Self::StateAdoption`] — deliberately absent from + /// [`storage_keys::ADOPTABLE_STATE_KEY_TYPES`], which has no pre-split + /// data to inherit for it. + PendingDecryptEntries, Outbox, MediaDescriptors, PeerKeyPackages, @@ -54,6 +66,10 @@ pub(crate) enum StateCategory { BlockedUsers, BothCreateAwaitingDecrypt, LamportClock, + /// The deduplicator's seen set. Post-split only, like + /// [`Self::NostrWatermark`], and absent from + /// [`storage_keys::ADOPTABLE_STATE_KEY_TYPES`] for the same reason. + DedupSeenIds, /// The Nostr receive watermark. Post-split only, like [`Self::StateAdoption`] /// — deliberately absent from [`storage_keys::ADOPTABLE_STATE_KEY_TYPES`], /// which has no pre-split data to inherit for it. @@ -98,6 +114,7 @@ impl StateCategory { Some(match key_type { storage_keys::PENDING_MESSAGES => Self::PendingMessages, storage_keys::PENDING_MESSAGE_ENTRIES => Self::PendingMessageEntries, + storage_keys::PENDING_DECRYPT_ENTRIES => Self::PendingDecryptEntries, storage_keys::OUTBOX => Self::Outbox, storage_keys::MEDIA_DESCRIPTORS => Self::MediaDescriptors, storage_keys::PEER_KEY_PACKAGES => Self::PeerKeyPackages, @@ -107,6 +124,7 @@ impl StateCategory { storage_keys::BLOCKED_USERS => Self::BlockedUsers, storage_keys::BOTH_CREATE_AWAITING_DECRYPT => Self::BothCreateAwaitingDecrypt, storage_keys::LAMPORT_CLOCK => Self::LamportClock, + storage_keys::DEDUP_SEEN_IDS => Self::DedupSeenIds, storage_keys::NOSTR_WATERMARK => Self::NostrWatermark, storage_keys::NOSTR_KEY_PACKAGE_SLOTS => Self::NostrKeyPackageSlots, storage_keys::NOSTR_DISCOVERY_CLAIM => Self::NostrDiscoveryClaim, @@ -133,6 +151,7 @@ impl StateCategory { pub(crate) const ALL: &'static [Self] = &[ Self::PendingMessages, Self::PendingMessageEntries, + Self::PendingDecryptEntries, Self::Outbox, Self::MediaDescriptors, Self::PeerKeyPackages, @@ -142,6 +161,7 @@ impl StateCategory { Self::BlockedUsers, Self::BothCreateAwaitingDecrypt, Self::LamportClock, + Self::DedupSeenIds, Self::NostrWatermark, Self::NostrKeyPackageSlots, Self::NostrDiscoveryClaim, @@ -161,6 +181,7 @@ impl StateCategory { match self { Self::PendingMessages => storage_keys::PENDING_MESSAGES, Self::PendingMessageEntries => storage_keys::PENDING_MESSAGE_ENTRIES, + Self::PendingDecryptEntries => storage_keys::PENDING_DECRYPT_ENTRIES, Self::Outbox => storage_keys::OUTBOX, Self::MediaDescriptors => storage_keys::MEDIA_DESCRIPTORS, Self::PeerKeyPackages => storage_keys::PEER_KEY_PACKAGES, @@ -170,6 +191,7 @@ impl StateCategory { Self::BlockedUsers => storage_keys::BLOCKED_USERS, Self::BothCreateAwaitingDecrypt => storage_keys::BOTH_CREATE_AWAITING_DECRYPT, Self::LamportClock => storage_keys::LAMPORT_CLOCK, + Self::DedupSeenIds => storage_keys::DEDUP_SEEN_IDS, Self::NostrWatermark => storage_keys::NOSTR_WATERMARK, Self::NostrKeyPackageSlots => storage_keys::NOSTR_KEY_PACKAGE_SLOTS, Self::NostrDiscoveryClaim => storage_keys::NOSTR_DISCOVERY_CLAIM, @@ -193,6 +215,13 @@ impl StateCategory { /// per-recipient predecessor [`storage_keys::PENDING_MESSAGES`]: original /// plaintext, plus rich extras that can include /// `MediaMetadata::encryption_key`/`iv`. + /// - [`storage_keys::PENDING_DECRYPT_ENTRIES`]: inbound frames parked + /// before their session was ready. The body is MLS ciphertext, but the + /// envelope around it is not: sender, recipient, app id, metadata and + /// any outer `media_metadata` are in the clear, and a sealed record is + /// also the only thing stopping a container write from *substituting* + /// a frame — an AEAD makes an edited record unopenable, and unopenable + /// is dropped. /// - [`storage_keys::OUTBOX`]: the outgoing `Message` — ciphertext for /// encrypted sends, but plaintext when the app opted out of encryption, /// and its outer `media_metadata` carries the cloud-media secrets on the @@ -234,6 +263,14 @@ impl StateCategory { /// pointing at this address forever. Deleting the record instead is the /// benign direction and is not prevented (nothing sealing does can), /// which is why the claim is also republished on every launch. + /// - [`storage_keys::DEDUP_SEEN_IDS`]: sealed for the confidentiality of + /// timing rather than content. The record holds up to 2000 inbound + /// message ids, each with the millisecond it arrived, for a day. The wire + /// shows each id in transit; at rest the record is a day-long timeline of + /// when this install received messages, and the threat model lists + /// delivery metadata as an asset. Failing closed is cheap: with the key + /// unavailable the set is not written, and the next launch starts it + /// empty, which costs only the restart-time dedup the record provides. /// /// Everything else is advertised capability versions, a small state enum, a /// logical clock, a coarse wall-clock mark, or a value-less marker whose @@ -256,6 +293,7 @@ impl StateCategory { match self { Self::PendingMessages | Self::PendingMessageEntries + | Self::PendingDecryptEntries | Self::Outbox | Self::MediaDescriptors | Self::PeerKeyPackages @@ -264,7 +302,8 @@ impl StateCategory { | Self::DataDocs | Self::DataDeltaLog | Self::DataSpaces - | Self::DataSync => true, + | Self::DataSync + | Self::DedupSeenIds => true, Self::PeerCapabilities | Self::SessionStates | Self::WelcomeLifecycles @@ -418,6 +457,17 @@ pub(crate) enum RestorableRecord { /// overflow is the safer failure. See the rationale there. pub(super) const MAX_RESTORE_KEYS_PER_CATEGORY: usize = 4 * MAX_PENDING_MESSAGES_GLOBAL; +/// How long a parked inbound frame may sit on disk, measured from its first +/// receipt, before restore drops it instead of re-queuing it. +/// +/// The in-memory TTL (`PendingQueueConfig::pending_ttl_ms`) restarts with every +/// process, because it is measured on an `Instant`; without a second bound a +/// frame whose session never confirms would be restored on every launch +/// forever. Seven days is well past any handshake that is still going to +/// succeed, and past the sender's own retry budget, so a record older than +/// this is one nobody is still trying to deliver. +pub(crate) const PENDING_DECRYPT_PERSISTED_MAX_AGE_MS: i64 = 7 * 24 * 60 * 60 * 1000; + /// Restore-walk bound for the outbox. /// /// The live insert path hard-caps the outbox at [`MAX_OUTBOX_ENTRIES`] @@ -544,10 +594,18 @@ pub(super) const MAX_PENDING_RESTORE_ENTRIES: usize = 4 * MAX_PENDING_MESSAGES_G /// a live id for, and starving the pending walk defers every diagnostic it /// owes. Neither may be held hostage to a key-package flood. /// +/// The inbound walks get a fourth pool for the same reason: +/// [`OfflineProtocol::restore_pending_decrypt_entries`] holds ciphertext the +/// app is told about when it is lost, and [`OfflineProtocol::restore_dedup_seen`] +/// shares the pool after it for its one possible delete. Both refuse rather +/// than count, since every delete they make is advisory. The pending-decryption +/// walk reports a drop only together with its delete, so a refusal defers both +/// halves to a later launch at once. +/// /// # The derived launch ceiling /// -/// Three pools, so `3 × MAX_RESTORE_PRUNE_DELETES` is the whole launch's -/// allowance. All three are constructed side by side by `initialize_mls` — see +/// Four pools, so `4 × MAX_RESTORE_PRUNE_DELETES` is the whole launch's +/// allowance. All four are constructed side by side by `initialize_mls` — see /// [`PruneAllowance::pool`] for why none of them may be allocated inside the /// walk that spends it — and /// `test_one_launch_cannot_exceed_the_derived_restore_delete_ceiling` pins the @@ -580,7 +638,7 @@ pub(super) const MAX_PENDING_RESTORE_ENTRIES: usize = 4 * MAX_PENDING_MESSAGES_G /// by its own truncated-and-resumable pass rather than by this constant. It is /// a one-time upgrade sweep on a different provider, so it is deliberately /// outside the pools — but a reader deriving "the most barriers one launch can -/// issue" should count it separately rather than reading `3 ×` as the total. +/// issue" should count it separately rather than reading `4 ×` as the total. /// /// # This bounds deletes, and deletes are not the only durable cost /// @@ -773,7 +831,7 @@ impl PruneAllowance { /// it. The two constructors had identical bodies, which was the tell. /// /// So every pool is constructed by the caller that owns the launch — - /// `initialize_mls` builds all three side by side — and threaded in. The + /// `initialize_mls` builds all four side by side — and threaded in. The /// launch ceiling reads off that one call site, and /// `test_one_launch_cannot_exceed_the_derived_restore_delete_ceiling` pins /// it. @@ -798,6 +856,18 @@ impl PruneAllowance { PruneBudget::new(&mut self.remaining, ceiling, true) } + /// A refusing budget on a pool that no advisory walk shares. + /// + /// [`Self::refusing`] reserves [`MIN_ADVISORY_PRUNE_DELETES`] for each + /// advisory walk still to come, which on a private pool reserves for walks + /// that never arrive: the first draw on a fresh pool would get half of it. + /// This reserves nothing, so a walk that owns its pool can spend all of it, + /// and a later walk on the same pool gets whatever is left. + pub(super) fn refusing_private(&mut self) -> PruneBudget<'_> { + let ceiling = self.remaining; + PruneBudget::new(&mut self.remaining, ceiling, true) + } + /// A budget for a settlement-paired walk, which counts every delete but /// never refuses one. /// @@ -4363,6 +4433,305 @@ impl OfflineProtocol { } } + // ======================================================================== + // PENDING DECRYPTION QUEUE PERSISTENCE + // ======================================================================== + + /// Writes one parked inbound frame under its own message id. + /// + /// Best-effort, like [`Self::persist_pending_message`]: a failed write is + /// logged and the entry still lives in the in-memory queue, it just will + /// not survive a restart. Called only for a frame the queue *admitted* — + /// a frame it refused (overflow, oversized) has nothing to persist, and + /// a resend of an id already queued is a no-op in memory and on disk + /// (the original record is authoritative and keeps its first-receipt + /// timestamp). + pub(crate) fn persist_pending_decrypt_entry( + &self, + peer_id: &str, + message: &Message, + first_received_at_ms: i64, + received_via: Option, + ) { + let Some(storage) = &self.protocol_state_storage else { + return; + }; + let record = PendingDecryptRecord { + version: PENDING_DECRYPT_RECORD_VERSION, + peer_id: peer_id.to_string(), + message: message.clone(), + first_received_at_ms, + received_via, + }; + let data = match serde_json::to_vec(&record) { + Ok(data) => data, + Err(e) => { + warn!( + peer_id = %peer_id, + message_id = %message.id, + error = %e, + "Failed to serialize pending decryption entry" + ); + return; + } + }; + if let Err(e) = self.write_state_record( + storage.as_ref(), + storage_keys::PENDING_DECRYPT_ENTRIES, + &message.id.as_str(), + &data, + ) { + warn!( + peer_id = %peer_id, + message_id = %message.id, + error = %e, + "Failed to persist pending decryption entry" + ); + } + } + + /// Removes one persisted parked frame. Logged rather than swallowed for + /// the reason [`Self::delete_pending_message_from_storage`] is: a delete + /// that silently failed is a record the next launch restores and drains + /// again, and a re-drain of a frame whose ratchet generation was already + /// spent surfaces as a spurious decrypt failure. + pub(crate) fn delete_pending_decrypt_entry_from_storage(&self, message_id: &MessageId) { + let Some(storage) = &self.protocol_state_storage else { + return; + }; + if let Err(e) = storage.delete(storage_keys::PENDING_DECRYPT_ENTRIES, &message_id.as_str()) + { + warn!( + message_id = %message_id, + error = %e, + "Failed to clear persisted pending decryption entry" + ); + } + } + + /// Removes the persisted copies of a batch of parked frames. + pub(crate) fn delete_pending_decrypt_entries_from_storage<'a>( + &self, + message_ids: impl IntoIterator, + ) { + for message_id in message_ids { + self.delete_pending_decrypt_entry_from_storage(message_id); + } + } + + /// Test-only: the ids currently persisted for the pending-decryption + /// queue, in store order. + #[cfg(test)] + pub(crate) fn persisted_pending_decrypt_ids(&self) -> Vec { + let Some(storage) = self.protocol_state_storage.as_ref() else { + return Vec::new(); + }; + Self::list_state_keys(storage.as_ref(), storage_keys::PENDING_DECRYPT_ENTRIES) + .unwrap_or_default() + } + + /// Restores the pending-decryption queue from its per-message records. + /// + /// Each record that opens, parses, and is younger than + /// [`PENDING_DECRYPT_PERSISTED_MAX_AGE_MS`] is pushed back through the + /// in-memory enqueue — oldest first by `first_received_at_ms`, so the + /// per-peer FIFO and the overflow policy see the same order they would + /// have on the live path — **without** being re-persisted: the record on + /// disk is already the durable copy and keeps its first-receipt timestamp. + /// A restored entry gets a fresh `Instant::now()` as its in-memory + /// `received_at`, so its TTL restarts with the process; the 7-day age + /// bound is what keeps a frame from being restored forever. + /// + /// Restored ids are deliberately **not** dedup-marked. The live path + /// unmarks a deferred frame on receipt so the sender's resend re-enters + /// the queue (idempotent by id) instead of hitting the duplicate re-ACK + /// path, and a restore must leave that invariant where it found it. + /// + /// Every delete the walk causes (unreadable, corrupt, unknown version, + /// expired, or dropped by the in-memory caps) is advisory: a record left on + /// disk one launch longer is re-walked and dropped then. So the walk draws + /// on a pool of its own with refusing semantics rather than the shared + /// advisory pool: it holds inbound ciphertext the app is told about when + /// lost, and that must not be held hostage to a key-package flood in a + /// category it has nothing to do with, the same argument the outbound + /// pending walk makes. It draws first; [`Self::restore_dedup_seen`] shares + /// the pool after it. + /// + /// An expired record is settled with a `PendingQueueDropped` decryption + /// failure carrying the reason `expired_persisted`, through the deferred + /// settlement path so it reaches the app once the event pipeline is live. + /// A restored frame the in-memory caps drop (a lowered cap, or more on disk + /// than memory admits) is settled the same way: emitted directly, it would + /// reach an app that has not subscribed yet, or no callback at all. + /// + /// Either one is reported only on the launch that deletes its record. A + /// delete the budget refuses leaves the record on disk **unreported**, and + /// a later launch walks it again and owns both halves then, which is the + /// rule the outbound walk's capacity drain follows. Reporting it anyway + /// would repeat an aged-out report on every launch until the delete lands, + /// and for a frame the caps dropped it would claim a loss that a later + /// launch contradicts by admitting the frame. + /// Infallible by design: this queue holds nothing the rest of `initialize_mls` + /// depends on, so a listing failure is logged and the queue simply starts + /// empty this session. + pub(crate) fn restore_pending_decrypt_entries(&mut self, allowance: &mut PruneAllowance) { + let Some(storage) = self.protocol_state_storage.clone() else { + return; + }; + let key_ids = + match Self::list_state_keys(storage.as_ref(), storage_keys::PENDING_DECRYPT_ENTRIES) { + Ok(keys) => keys, + Err(e) => { + warn!(error = %e, "Failed to list pending decryption entries from storage"); + return; + } + }; + let listed = key_ids.len(); + if listed > MAX_RESTORE_KEYS_PER_CATEGORY { + warn!( + listed, + cap = MAX_RESTORE_KEYS_PER_CATEGORY, + "Pending decryption entries listed more records than any legitimate run can produce; ignoring the tail" + ); + } + + let now_ms = Utc::now().timestamp_millis(); + let mut budget = allowance.refusing_private(); + let mut restored: Vec = Vec::new(); + let mut settlements: Vec = Vec::new(); + for key_id in key_ids.into_iter().take(MAX_RESTORE_KEYS_PER_CATEGORY) { + let data = match self.read_state_record_detailed_budgeted( + storage.as_ref(), + storage_keys::PENDING_DECRYPT_ENTRIES, + &key_id, + Some(&mut budget), + ) { + Ok(StateRecord::Present(data)) => data, + Ok(StateRecord::Missing | StateRecord::Unreadable | StateRecord::Unavailable) + | Err(_) => continue, + }; + + let record = match serde_json::from_slice::(&data) { + Ok(record) if record.version == PENDING_DECRYPT_RECORD_VERSION => record, + Ok(record) => { + warn!( + key_id = %key_id, + version = record.version, + "Dropping pending decryption entry with an unknown record version" + ); + if budget.claim() { + self.delete_pending_decrypt_key(&key_id); + } + continue; + } + Err(e) => { + warn!( + key_id = %key_id, + error = %e, + "Dropping corrupted pending decryption entry" + ); + if budget.claim() { + self.delete_pending_decrypt_key(&key_id); + } + continue; + } + }; + + // The record is keyed by the message id it holds; a record filed + // under some other id is not one this SDK wrote. + if record.message.id.as_str() != key_id { + warn!( + key_id = %key_id, + message_id = %record.message.id, + "Dropping pending decryption entry whose key does not match its message id" + ); + if budget.claim() { + self.delete_pending_decrypt_key(&key_id); + } + continue; + } + + if now_ms.saturating_sub(record.first_received_at_ms) + >= PENDING_DECRYPT_PERSISTED_MAX_AGE_MS + { + // Reported with its delete or not at all this launch; see the + // doc comment above. + if !budget.claim() { + continue; + } + debug!( + key_id = %key_id, + peer_id = %record.peer_id, + "Dropping pending decryption entry that aged out on disk" + ); + self.delete_pending_decrypt_key(&key_id); + settlements.push(Event::message_decryption_failed( + record.message.id.clone(), + record.message.sender.as_str().to_string(), + DecryptionFailureCode::PendingQueueDropped, + "encrypted message evicted from pending queue (expired_persisted); \ + recoverable only if the sender resends" + .to_string(), + )); + continue; + } + + restored.push(record); + } + + // Oldest first, so the in-memory FIFO and the overflow policy see the + // order the live path would have produced. Ties (same millisecond) + // fall back to the id so the order is stable across launches. + restored.sort_by(|left, right| { + left.first_received_at_ms + .cmp(&right.first_received_at_ms) + .then_with(|| left.message.id.as_str().cmp(&right.message.id.as_str())) + }); + + let count = restored.len(); + for record in restored { + let dropped = self.enqueue_restored_pending_decryption( + &record.peer_id, + &record.message, + record.received_via, + ); + // A frame the caps let go is already out of memory but came off + // disk, so its record goes, charged like every other delete here. + // One the budget cannot fund stays on disk unreported. + let mut deleted = Vec::with_capacity(dropped.len()); + for entry in dropped { + if budget.claim() { + self.delete_pending_decrypt_entry_from_storage(&entry.message.id); + deleted.push(entry); + } + } + settlements.extend(Self::pending_drop_events(deleted)); + } + self.settle_restored_message_failures(settlements); + + if budget.exhausted { + warn!( + deleted = budget.spent, + budget = MAX_RESTORE_PRUNE_DELETES, + "Pending decryption prune hit its share of the launch delete budget; the rest is left on disk for a later launch" + ); + } + + if count > 0 { + info!(count, "Restored pending decryption entries from storage"); + } + } + + /// Deletes a pending-decryption record by key (restore-internal, mirrors + /// [`Self::delete_media_descriptor_key`]). + fn delete_pending_decrypt_key(&self, key_id: &str) { + if let Some(storage) = &self.protocol_state_storage { + if let Err(e) = storage.delete(storage_keys::PENDING_DECRYPT_ENTRIES, key_id) { + warn!(key_id = %key_id, error = %e, "Failed to delete pending decryption entry"); + } + } + } + // ======================================================================== // TELEMETRY SCRUB-SECRET PERSISTENCE // ======================================================================== @@ -4874,6 +5243,176 @@ impl OfflineProtocol { } } +impl OfflineProtocol { + // ======================================================================== + // DEDUPLICATOR SEEN-SET PERSISTENCE + // ======================================================================== + + /// Records one change to the deduplicator's seen set — a mark or an + /// unmark on the receive path — for the batched write below. + /// + /// Only receive-side changes are counted. The ids a sender marks for its + /// own outgoing frames are not persisted: they exist to stop a relayed + /// echo of our own message being re-processed, which a restart does not + /// change, and persisting them would spend the record's cap on ids no + /// peer will ever push at us. That holds because the send paths mark + /// through `Deduplicator::mark_seen_local`, which `export_seen` leaves + /// out; a send-side `mark_seen` would put the id in the next record and + /// is pinned against by + /// `test_dedup_seen_set_excludes_own_outgoing_ids`. + pub(crate) fn note_dedup_change(&mut self) { + self.dedup_dirty = self.dedup_dirty.saturating_add(1); + } + + /// Marks an inbound id as seen and counts the change for persistence. + pub(crate) fn mark_seen_persisted(&mut self, message_id: MessageId) -> bool { + let fresh = self.deduplicator.mark_seen(message_id); + if fresh { + self.note_dedup_change(); + } + fresh + } + + /// Forgets an inbound id and counts the change for persistence, so a + /// record written later does not resurrect an id the receive path + /// deliberately released (a deferred or rejected frame whose resend must + /// re-enter processing). + pub(crate) fn unmark_seen_persisted(&mut self, message_id: &MessageId) -> bool { + let removed = self.deduplicator.unmark_seen(message_id); + if removed { + self.note_dedup_change(); + } + removed + } + + /// Batched seen-set persistence, called from every `process()` tick. + /// + /// Writes when [`DEDUP_PERSIST_DIRTY_THRESHOLD`] changes have accumulated, + /// or when any change is at least [`DEDUP_PERSIST_INTERVAL`] old — so a + /// burst reaches disk quickly and a trickle reaches it within seconds, + /// while a quiet node writes nothing. The unconditional counterpart is + /// [`Self::flush_dedup_seen`]. + pub(crate) fn persist_dedup_seen_if_due(&mut self) { + if self.dedup_dirty == 0 { + return; + } + let due = self.dedup_dirty >= DEDUP_PERSIST_DIRTY_THRESHOLD + || self.dedup_last_persist.elapsed() >= DEDUP_PERSIST_INTERVAL; + if due { + self.write_dedup_seen_to_storage(); + } + } + + /// Writes the seen set regardless of the batching state. Called on + /// `stop()` and on drop so a shutdown loses nothing to the debounce. + pub(crate) fn flush_dedup_seen(&mut self) { + if self.dedup_dirty == 0 { + return; + } + self.write_dedup_seen_to_storage(); + } + + fn write_dedup_seen_to_storage(&mut self) { + let Some(storage) = self.protocol_state_storage.clone() else { + return; + }; + let record = DedupSeenRecord { + version: DEDUP_SEEN_RECORD_VERSION, + entries: self.deduplicator.export_seen(MAX_PERSISTED_DEDUP_IDS), + }; + let data = match serde_json::to_vec(&record) { + Ok(data) => data, + Err(e) => { + warn!(error = %e, "Failed to serialize deduplicator seen set"); + return; + } + }; + if let Err(e) = self.write_state_record( + storage.as_ref(), + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + &data, + ) { + warn!(error = %e, "Failed to persist deduplicator seen set"); + return; + } + // Reset only on success: a failed write leaves the changes dirty so + // the next tick retries rather than declaring them durable. + self.dedup_dirty = 0; + self.dedup_last_persist = Instant::now(); + } + + /// Restores the seen set from storage, next to the Lamport clock. + /// + /// Ids already tracked in memory win, ids past the retention window as + /// of now are skipped, and the configured cap is respected — all inside + /// `Deduplicator::import_seen`. A record that does not parse, or carries a + /// version this build does not know, is deleted and the set starts empty: + /// nothing is owed to anyone for it, since the worst case is the one + /// restart-time duplicate this record exists to prevent. + /// + /// Every delete this makes is charged to `allowance`, the inbound pool it + /// shares with [`Self::restore_pending_decrypt_entries`], because the + /// launch ceiling covers every durable delete on the restore path. That + /// includes the reader's own: the record is sealed, so one that will not + /// open (a regenerated record key) is dropped inside the read. A refused + /// delete costs nothing: the set starts empty either way, and the next + /// write replaces the record. + pub(crate) fn restore_dedup_seen(&mut self, allowance: &mut PruneAllowance) { + let Some(storage) = self.protocol_state_storage.clone() else { + return; + }; + let mut budget = allowance.refusing_private(); + let data = match self.read_state_record_detailed_budgeted( + storage.as_ref(), + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + Some(&mut budget), + ) { + Ok(StateRecord::Present(data)) => data, + Ok(StateRecord::Missing | StateRecord::Unreadable | StateRecord::Unavailable) + | Err(_) => return, + }; + let record = match serde_json::from_slice::(&data) { + Ok(record) if record.version == DEDUP_SEEN_RECORD_VERSION => record, + Ok(record) => { + warn!( + version = record.version, + "Dropping deduplicator seen set with an unknown record version" + ); + if budget.claim() { + let _ = storage.delete( + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + ); + } + return; + } + Err(e) => { + warn!(error = %e, "Dropping corrupted deduplicator seen set"); + if budget.claim() { + let _ = storage.delete( + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + ); + } + return; + } + }; + let listed = record.entries.len(); + let imported = self.deduplicator.import_seen(record.entries, Utc::now()); + if imported > 0 { + debug!( + imported, + listed, "Restored deduplicator seen set from storage" + ); + } + // What is in memory now matches disk (or is a subset disk already + // holds), so nothing is dirty until the next receive. + self.dedup_last_persist = Instant::now(); + } +} + #[cfg(test)] mod category_registration_tests { use super::*; diff --git a/crates/offline-protocol/src/protocol/tests/mod.rs b/crates/offline-protocol/src/protocol/tests/mod.rs index 424fb085..0797750e 100644 --- a/crates/offline-protocol/src/protocol/tests/mod.rs +++ b/crates/offline-protocol/src/protocol/tests/mod.rs @@ -8619,6 +8619,7 @@ fn test_unreachable_media_chunk_resolves_file_id() { last_sent_at: chrono::Utc::now(), last_transport: Some(TransportType::BLE), reseal: None, + relay_pushed: false, }, ); protocol @@ -8713,14 +8714,16 @@ fn test_unreachable_dm_internet_only_parks_with_probe() { assert!(protocol.outbox.contains_key(&message_id)); } +/// The relay's `MessageSent { pushed: true }` reaches the core as the bridge's +/// `relay_pushed` on the Internet carrier. For a plain DM it must take the +/// same park as a DeliveryError (pending ACK dropped, outbox entry kept, +/// probe scheduled) and put the recipient on the presence watch list, because +/// a relay without store-and-forward has just said the only copy in flight is +/// a push notification. It must not tell the app the message is +/// undeliverable: the push may have delivered it. #[test] -fn test_unreachable_dm_internet_only_probe_escalates_and_resets_on_edge() { - // The internet-only probe shares the mesh ladder: each consecutive - // verdict doubles the interval (15s -> 30s, 600s cap) so a peer that - // stays offline cannot become an unbounded resend loop into the relay, - // and any reachability edge resets the escalation. +fn test_relay_pushed_verdict_parks_dm_and_watches_recipient() { let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); - let mock_transport = MockTransport::new(TransportType::Internet); mock_transport.start().unwrap(); protocol @@ -8728,117 +8731,231 @@ fn test_unreachable_dm_internet_only_probe_escalates_and_resets_on_edge() { .add_transport(TransportType::Internet, Box::new(mock_transport)); protocol.start().unwrap(); + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let events_handle = Arc::clone(&events); + protocol.on_event(move |event| { + events_handle.lock().unwrap().push(event); + }); + let message_id = protocol .send_message("bob", "hello", None::, None::) .unwrap(); + assert!(protocol.ack_manager.is_waiting_for_ack(&message_id)); + assert!( + !protocol.presence_watch_peers().contains(&"bob".to_string()), + "an in-flight (ACK-pending) recipient is not watched" + ); protocol - .on_transport_send_failed( + .on_transport_send_failed_via( &message_id.as_str(), - Some("recipient_unreachable: peer offline".to_string()), + Some("relay_pushed".to_string()), + Some(TransportType::Internet), ) .unwrap(); - assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&1)); - let first_delay = protocol - .retry_queue - .time_until_next_retry() - .expect("first park must schedule a probe"); - assert!(first_delay > chrono::Duration::seconds(10)); - assert!(first_delay <= chrono::Duration::seconds(15)); - // Second consecutive verdict doubles the interval (15s -> 30s). - protocol - .on_transport_send_failed( - &message_id.as_str(), - Some("recipient_unreachable: peer offline".to_string()), - ) - .unwrap(); - assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&2)); - let second_delay = protocol - .retry_queue - .time_until_next_retry() - .expect("second park must re-schedule the probe"); - assert!(second_delay > chrono::Duration::seconds(20)); - assert!(second_delay <= chrono::Duration::seconds(30)); + assert!( + !events + .lock() + .unwrap() + .iter() + .any(|e| matches!(e, Event::MessageUndeliverable { .. })), + "a pushed DM may have been delivered, so it must not be reported undeliverable" + ); - // A relay presence-online answer is the internet-only reachability edge - // (the mesh path uses on_neighbor_discovered): it re-drives the parked DM - // and resets the escalation counter. - protocol.on_peer_presence("bob", true, None); - assert_eq!( - protocol.dm_unreachable_parks.get("bob"), - None, - "A presence-online edge must reset the escalation" + assert!( + protocol.outbox.contains_key(&message_id), + "a pushed DM must stay in the outbox" + ); + assert!( + !protocol.ack_manager.is_waiting_for_ack(&message_id), + "the park must drop the pending ACK so no budget burns against an offline peer" + ); + assert!( + protocol.retry_queue.contains(&message_id.as_str()), + "the park must schedule a reachability probe" + ); + assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&1)); + assert!( + protocol.presence_watch_peers().contains(&"bob".to_string()), + "the pushed DM's recipient must be presence-watched" ); } +/// The park's timed probe resends the same id, and the relay answers a retry +/// of a message it has already pushed with `DeliveryError` (`already_pushed`) +/// rather than a second notification. The bridges can only report that as +/// `recipient_unreachable`, so without the entry remembering it was pushed +/// the first probe would emit the `MessageUndeliverable` the push park exists +/// not to emit. The verdict must still re-park (the recipient is not on the +/// relay), just silently; a DM that was never pushed keeps the event. #[test] -fn test_unreachable_dm_internet_only_reemits_undeliverable_per_verdict() { - // Deliberate contract: unlike a mesh probe (which can never earn a relay - // verdict), an internet probe re-earns a DeliveryError while the peer - // stays offline, so MessageUndeliverable repeats — once per verdict. It - // is a repeatable status signal, never a terminal one; the terminal - // signal remains MessageFailed, which must NOT fire here. +fn test_probe_verdict_after_relay_push_reparks_without_undeliverable() { let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); + let mock_transport = MockTransport::new(TransportType::Internet); + mock_transport.start().unwrap(); + protocol + .transport_manager_mut() + .add_transport(TransportType::Internet, Box::new(mock_transport)); + protocol.start().unwrap(); let events: Arc>> = Arc::new(Mutex::new(Vec::new())); let events_handle = Arc::clone(&events); protocol.on_event(move |event| { events_handle.lock().unwrap().push(event); }); + let undeliverable_ids = || -> Vec { + events + .lock() + .unwrap() + .iter() + .filter_map(|e| match e { + Event::MessageUndeliverable { message_id, .. } => Some(message_id.clone()), + _ => None, + }) + .collect() + }; - let mock_transport = MockTransport::new(TransportType::Internet); - mock_transport.start().unwrap(); + let pushed_id = protocol + .send_message("bob", "hello", None::, None::) + .unwrap(); protocol - .transport_manager_mut() - .add_transport(TransportType::Internet, Box::new(mock_transport)); - protocol.start().unwrap(); + .on_transport_send_failed_via( + &pushed_id.as_str(), + Some("relay_pushed".to_string()), + Some(TransportType::Internet), + ) + .unwrap(); + assert!( + protocol.outbox.get(&pushed_id).unwrap().relay_pushed, + "the park must mark the entry as pushed" + ); + assert!(undeliverable_ids().is_empty()); - let message_id = protocol - .send_message("bob", "hello", None::, None::) + // The probe's verdict: the relay refuses to push twice. + protocol + .on_transport_send_failed_via( + &pushed_id.as_str(), + Some("recipient_unreachable: Recipient is offline; push already sent".to_string()), + Some(TransportType::Internet), + ) + .unwrap(); + assert!( + undeliverable_ids().is_empty(), + "a probe verdict for a pushed DM must not tell the app it is undeliverable" + ); + assert!( + protocol.outbox.contains_key(&pushed_id), + "the verdict still re-parks: the entry stays" + ); + assert!(!protocol.ack_manager.is_waiting_for_ack(&pushed_id)); + assert!( + protocol.retry_queue.contains(&pushed_id.as_str()), + "the verdict still re-parks: a probe is rescheduled" + ); + assert_eq!( + protocol.dm_unreachable_parks.get("bob"), + Some(&2), + "the verdict still re-parks: the interval escalates" + ); + + // Control: a DM the relay never pushed keeps the advisory event. + let plain_id = protocol + .send_message("carol", "hello", None::, None::) + .unwrap(); + protocol + .on_transport_send_failed_via( + &plain_id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + Some(TransportType::Internet), + ) .unwrap(); + assert_eq!( + undeliverable_ids(), + vec![plain_id.as_str()], + "an un-pushed DM's unreachable verdict is still reported" + ); +} - for _ in 0..2 { +/// The pushed mark rides in the persisted outbox record: the relay remembers +/// a push for a day, so the first probe after an app restart earns the same +/// `DeliveryError` and must stay silent too. +#[test] +fn test_relay_pushed_mark_survives_restart() { + let storage = Arc::new(InMemoryStorage::new()); + let message_id = { + let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); + let mock_transport = MockTransport::new(TransportType::Internet); + mock_transport.start().unwrap(); protocol - .on_transport_send_failed( + .transport_manager_mut() + .add_transport(TransportType::Internet, Box::new(mock_transport)); + protocol + .enable_message_persistence_for_test(storage.clone()) + .unwrap(); + protocol.start().unwrap(); + let message_id = protocol + .send_message("bob", "hello", None::, None::) + .unwrap(); + protocol + .on_transport_send_failed_via( &message_id.as_str(), - Some("recipient_unreachable: peer offline".to_string()), + Some("relay_pushed".to_string()), + Some(TransportType::Internet), ) .unwrap(); - } + message_id + }; - let events = events.lock().unwrap(); - let undeliverable = events - .iter() - .filter(|e| matches!(e, Event::MessageUndeliverable { .. })) - .count(); - assert_eq!( - undeliverable, 2, - "Each verdict must re-emit the non-terminal undeliverable signal" - ); + let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); + let mock_transport = MockTransport::new(TransportType::Internet); + mock_transport.start().unwrap(); + protocol + .transport_manager_mut() + .add_transport(TransportType::Internet, Box::new(mock_transport)); + protocol + .enable_message_persistence_for_test(storage.clone()) + .unwrap(); assert!( - !events - .iter() - .any(|e| matches!(e, Event::MessageFailed { .. })), - "Repeated verdicts must never settle the message terminally" + protocol + .outbox + .get(&message_id) + .expect("the parked DM is restored") + .relay_pushed, + "the pushed mark must be restored with the entry" ); -} - -#[test] -fn test_internet_only_probe_exhaustion_reparks_not_terminal() { - // The hoisted counter is what arms try_repark_exhausted_dm on this - // branch. A probe that succeeds locally but is never answered burns the - // ACK budget; with a live counter that exhaustion must re-park (keeping - // the outbox entry) rather than settle terminally — otherwise unifying - // the branches would have traded a stalled message for a failed one. - let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); + protocol.start().unwrap(); let events: Arc>> = Arc::new(Mutex::new(Vec::new())); let events_handle = Arc::clone(&events); protocol.on_event(move |event| { events_handle.lock().unwrap().push(event); }); + protocol + .on_transport_send_failed_via( + &message_id.as_str(), + Some("recipient_unreachable: Recipient is offline; push already sent".to_string()), + Some(TransportType::Internet), + ) + .unwrap(); + assert!( + !events + .lock() + .unwrap() + .iter() + .any(|e| matches!(e, Event::MessageUndeliverable { .. })), + "the first probe after a restart must stay silent for a pushed DM" + ); + assert!(protocol.outbox.contains_key(&message_id)); +} +/// A pushed connection request may have been delivered by the push, so the +/// `relay_pushed` answer must not fast-fail it the way a DeliveryError does: +/// no `ConnectionRequestUndeliverable`, and the request keeps its typed +/// tracking and its pending ACK so the recipient's answer still settles it. +#[test] +fn test_relay_pushed_does_not_fail_a_connection_request() { + let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); let mock_transport = MockTransport::new(TransportType::Internet); mock_transport.start().unwrap(); protocol @@ -8846,216 +8963,181 @@ fn test_internet_only_probe_exhaustion_reparks_not_terminal() { .add_transport(TransportType::Internet, Box::new(mock_transport)); protocol.start().unwrap(); - let message_id = protocol - .send_message("bob", "hello", None::, None::) - .unwrap(); - protocol - .on_transport_send_failed( - &message_id.as_str(), - Some("recipient_unreachable: peer offline".to_string()), - ) + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let events_handle = Arc::clone(&events); + protocol.on_event(move |event| { + events_handle.lock().unwrap().push(event); + }); + + let sent_id = protocol + .send_connection_request("bob", "Alice", None, None) .unwrap(); - assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&1)); + let awaiting_before = protocol.ack_manager.is_waiting_for_ack(&sent_id); - // Drive the exhaustion path directly: the probe's ACK budget ran out. protocol - .handle_max_retries_exceeded(&message_id, 10) + .on_transport_send_failed_via( + &sent_id.as_str(), + Some("relay_pushed".to_string()), + Some(TransportType::Internet), + ) .unwrap(); + let captured = events.lock().unwrap(); assert!( - protocol.outbox.contains_key(&message_id), - "Exhaustion with a live park counter must re-park, not evict" + !captured.iter().any(|e| matches!( + e, + Event::ConnectionRequestUndeliverable { .. } | Event::MessageUndeliverable { .. } + )), + "a pushed connection request must not be reported undeliverable" ); + drop(captured); assert!( - !events - .lock() - .unwrap() - .iter() - .any(|e| matches!(e, Event::MessageFailed { .. })), - "Re-parked exhaustion must not surface a terminal message_failed" + protocol + .pending_connection_requests + .contains_key(&sent_id.as_str()), + "the request keeps its typed tracking" ); assert_eq!( - protocol.dm_unreachable_parks.get("bob"), - Some(&2), - "The re-park escalates the interval like any other park" + protocol.ack_manager.is_waiting_for_ack(&sent_id), + awaiting_before, + "the request is not parked: its ACK state is untouched" ); } +/// A pushed Welcome may have been delivered by the push, so the `relay_pushed` +/// answer must leave its lifecycle alone: no move to `Failed`, no refunded +/// attempt, no `WelcomeSendFailed`. Session confirmation, or the existing +/// confirmation and rescue machinery, decides what became of it. #[test] -fn test_delivery_ack_redrives_the_peers_other_parked_dms() { - // The park counter is per-peer while the probes are per-message, so a - // burst of DMs to an offline peer escalates the shared ladder once per - // park: message 1 probes at 15s but message 3 is already scheduled at 60s, - // and a burst of seven lands the last at the 600s cap. Clearing the - // counter on delivery is therefore not enough on its own — delivery of any - // one of them proves the rest can go NOW, and on a consumer that never - // polls presence this ACK is the only edge that will ever say so. - let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); +fn test_relay_pushed_leaves_a_welcome_lifecycle_alone() { + let mut config = create_test_config(); + config.encryption.enabled = true; + config.encryption.store_pending = true; - let mock_transport = MockTransport::new(TransportType::Internet); - mock_transport.start().unwrap(); - let internet_handle = mock_transport.clone(); + let storage = Arc::new(InMemoryStorage::new()); + let mut protocol = OfflineProtocol::new(config).unwrap(); + protocol.initialize_mls_for_test(storage).unwrap(); + + let internet = MockTransport::new(TransportType::Internet); + internet.start().unwrap(); protocol .transport_manager_mut() - .add_transport(TransportType::Internet, Box::new(mock_transport)); + .add_transport(TransportType::Internet, Box::new(internet)); protocol.start().unwrap(); - let ids: Vec<_> = (0..3) - .map(|i| { - protocol - .send_message( - "bob", - &format!("hello {}", i), - None::, - None::, - ) - .unwrap() - }) - .collect(); - for id in &ids { - protocol - .on_transport_send_failed( - &id.as_str(), - Some("recipient_unreachable: peer offline".to_string()), - ) - .unwrap(); - } - assert_eq!( - protocol.dm_unreachable_parks.get("bob"), - Some(&3), - "each park escalates the shared per-peer ladder" + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let events_handle = Arc::clone(&events); + protocol.on_event(move |event| { + events_handle.lock().unwrap().push(event); + }); + + let bob_storage = Arc::new(crate::mls::InMemoryStorage::new()); + let bob_manager = crate::test_identity::manager_for("bob", bob_storage); + let bob_key_package = bob_manager.get_or_create_key_package().unwrap(); + protocol.pending_key_packages.insert( + id("bob"), + ReceivedKeyPackage { + key_package_data: bob_key_package.key_package_data, + local_expires_at_ms: Utc::now().timestamp_millis() as u64 + 60_000, + }, ); - for id in &ids { - assert!( - !protocol.ack_manager.is_waiting_for_ack(id), - "a parked DM holds no pending ACK" - ); - } + let _ = protocol + .send_message(&id("bob"), "hello", None::, None::) + .unwrap(); - // Message 0's probe fires and registers its ACK (what the retry queue does - // on a probe send); the peer answers it. The ACK deliberately carries no - // ACK_TRANSPORT_KEY label: the re-drive override must come from the - // sender's own record of the delivered send's transport - // (`OutboxEntry::last_transport` — Internet here, and available), never - // from the peer-supplied label, which decodes any absent or unknown value - // to BLE and would let a peer steer the sibling sends. - let probed = protocol.outbox.get(&ids[0]).unwrap().message.clone(); - protocol.ensure_ack_registration(&probed).unwrap(); - let sends_before = internet_handle.sent_messages().len(); + let before = protocol.welcome_lifecycles.get(&id("bob")).unwrap().clone(); + let welcome_id = before.welcome_message.id.as_str().to_string(); - let ack = Message::builder( - UserId::new("bob").unwrap(), - UserId::new("user123").unwrap(), - AppId::new("test-app").unwrap(), - ) - .content(String::new()) - .metadata(ACK_FOR_KEY, ids[0].as_str()) - .build(); - protocol.handle_ack_message(&ack); + protocol + .on_transport_send_failed_via( + &welcome_id, + Some("relay_pushed".to_string()), + Some(TransportType::Internet), + ) + .unwrap(); + let after = protocol.welcome_lifecycles.get(&id("bob")).unwrap(); assert_eq!( - protocol.dm_unreachable_parks.get("bob"), - None, - "delivery proves reachability: the escalation starts over" + after.state, before.state, + "the lifecycle state is untouched" ); + assert_eq!(after.attempt, before.attempt, "no attempt is refunded"); + assert_eq!(after.unreachable_parks, before.unreachable_parks); + assert_eq!(after.last_reason_code, before.last_reason_code); assert!( - internet_handle.sent_messages().len() > sends_before, - "the siblings must go out now, not wait out their escalated timers" + !events + .lock() + .unwrap() + .iter() + .any(|e| matches!(e, Event::WelcomeSendFailed { .. })), + "a pushed Welcome must not be reported as a failed send" ); - for id in &ids[1..] { - assert!( - protocol.ack_manager.is_waiting_for_ack(id), - "each sibling must be re-driven with a fresh ACK budget" - ); - assert!( - protocol.outbox.contains_key(id), - "re-driving must not settle the sibling" - ); - } } +/// The watch list is the SDK's own "who am I waiting to hear about": a parked +/// DM's recipient appears on it, and disappears once a presence-online answer +/// re-drives the message — the re-driven send registers a fresh pending ACK, +/// which is what takes the recipient back off the list. #[test] -fn test_delivery_ack_redrive_ignores_peer_label_and_falls_back_to_dors() { - // Two hostile-to-override conditions at once: the delivering transport - // recorded on the outbox entry is gone by ACK time (last_transport = BLE, - // never attached), and the ACK carries a garbage ACK_TRANSPORT_KEY label - // (peer-supplied; decodes to BLE). The re-drive must ignore the label - // entirely and, with its own last_transport unavailable, fall back to - // DORS — a missing carrier costs routing freedom, never the re-drive. +fn test_parked_dm_recipient_is_watched_until_presence_online_redrives_it() { let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); - let mock_transport = MockTransport::new(TransportType::Internet); mock_transport.start().unwrap(); - let internet_handle = mock_transport.clone(); + let handle = mock_transport.clone(); protocol .transport_manager_mut() .add_transport(TransportType::Internet, Box::new(mock_transport)); protocol.start().unwrap(); - let ids: Vec<_> = (0..2) - .map(|i| { - protocol - .send_message( - "bob", - &format!("hello {}", i), - None::, - None::, - ) - .unwrap() - }) - .collect(); - for id in &ids { - protocol - .on_transport_send_failed( - &id.as_str(), - Some("recipient_unreachable: peer offline".to_string()), - ) - .unwrap(); - } - assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&2)); - - // The carrier that delivered message 0 has since dropped. - protocol.outbox.get_mut(&ids[0]).unwrap().last_transport = Some(TransportType::BLE); - - let probed = protocol.outbox.get(&ids[0]).unwrap().message.clone(); - protocol.ensure_ack_registration(&probed).unwrap(); - let sends_before = internet_handle.sent_messages().len(); + let message_id = protocol + .send_message("bob", "hello", None::, None::) + .unwrap(); + protocol + .on_transport_send_failed_via( + &message_id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + Some(TransportType::Internet), + ) + .unwrap(); + assert!( + protocol.presence_watch_peers().contains(&"bob".to_string()), + "a parked DM's recipient is watched" + ); + handle.clear_sent_messages(); - let ack = Message::builder( - UserId::new("bob").unwrap(), - UserId::new("user123").unwrap(), - AppId::new("test-app").unwrap(), - ) - .content(String::new()) - .metadata(ACK_FOR_KEY, ids[0].as_str()) - .metadata(crate::constants::ACK_TRANSPORT_KEY, "garbage") - .build(); - protocol.handle_ack_message(&ack); + protocol.on_peer_presence("bob", true, None); - assert_eq!(protocol.dm_unreachable_parks.get("bob"), None); assert!( - internet_handle.sent_messages().len() > sends_before, - "the sibling must be re-driven over DORS's pick, not stranded on a \ - carrier this device does not have" + handle.sent_messages().iter().any(|m| m.id == message_id), + "presence-online must re-drive the parked DM over the carrier that answered" ); assert!( - protocol.ack_manager.is_waiting_for_ack(&ids[1]), - "the sibling must be re-driven with a fresh ACK budget" + protocol.ack_manager.is_waiting_for_ack(&message_id), + "the re-driven send registers a fresh pending ACK" + ); + assert!( + !protocol.presence_watch_peers().contains(&"bob".to_string()), + "a recipient with only ACK-pending traffic is no longer watched" + ); + assert!( + !protocol.dm_unreachable_parks.contains_key("bob"), + "the reachability edge clears the park counter" ); } #[test] -fn test_unreachable_dm_mesh_probe_escalates_and_resets_on_edge() { - // With a local mesh carrier up the peer may be a room away, so the park - // keeps a timed reachability probe whose interval escalates per - // consecutive unreachable park and resets on a reachability edge. +fn test_unreachable_dm_internet_only_probe_escalates_and_resets_on_edge() { + // The internet-only probe shares the mesh ladder: each consecutive + // verdict doubles the interval (15s -> 30s, 600s cap) so a peer that + // stays offline cannot become an unbounded resend loop into the relay, + // and any reachability edge resets the escalation. let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); - let mock_transport = MockTransport::new(TransportType::BLE); + let mock_transport = MockTransport::new(TransportType::Internet); mock_transport.start().unwrap(); protocol .transport_manager_mut() - .add_transport(TransportType::BLE, Box::new(mock_transport)); + .add_transport(TransportType::Internet, Box::new(mock_transport)); protocol.start().unwrap(); let message_id = protocol @@ -9076,7 +9158,7 @@ fn test_unreachable_dm_mesh_probe_escalates_and_resets_on_edge() { assert!(first_delay > chrono::Duration::seconds(10)); assert!(first_delay <= chrono::Duration::seconds(15)); - // Second consecutive park doubles the interval (15s -> 30s). + // Second consecutive verdict doubles the interval (15s -> 30s). protocol .on_transport_send_failed( &message_id.as_str(), @@ -9091,20 +9173,32 @@ fn test_unreachable_dm_mesh_probe_escalates_and_resets_on_edge() { assert!(second_delay > chrono::Duration::seconds(20)); assert!(second_delay <= chrono::Duration::seconds(30)); - // A per-peer reachability edge resets the escalation counter (the BLE - // mock accepts the re-driven send, so the edge genuinely takes). - protocol.on_neighbor_discovered("bob"); - assert_eq!(protocol.dm_unreachable_parks.get("bob"), None); + // A relay presence-online answer is the internet-only reachability edge + // (the mesh path uses on_neighbor_discovered): it re-drives the parked DM + // and resets the escalation counter. + protocol.on_peer_presence("bob", true, None); + assert_eq!( + protocol.dm_unreachable_parks.get("bob"), + None, + "A presence-online edge must reset the escalation" + ); } #[test] -fn test_unreachable_media_chunk_is_not_parked() { - // Media chunks keep the normal retry machinery: their offline story is - // retry exhaustion -> transfer abort -> persisted descriptor -> - // MediaResendRequired with app-resupplied bytes. Parking would strand - // chunk bytes in memory for the outbox lifetime instead. +fn test_unreachable_dm_internet_only_reemits_undeliverable_per_verdict() { + // Deliberate contract: unlike a mesh probe (which can never earn a relay + // verdict), an internet probe re-earns a DeliveryError while the peer + // stays offline, so MessageUndeliverable repeats — once per verdict. It + // is a repeatable status signal, never a terminal one; the terminal + // signal remains MessageFailed, which must NOT fire here. let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let events_handle = Arc::clone(&events); + protocol.on_event(move |event| { + events_handle.lock().unwrap().push(event); + }); + let mock_transport = MockTransport::new(TransportType::Internet); mock_transport.start().unwrap(); protocol @@ -9112,254 +9206,398 @@ fn test_unreachable_media_chunk_is_not_parked() { .add_transport(TransportType::Internet, Box::new(mock_transport)); protocol.start().unwrap(); - let chunk_message = signed_frame(&id("user123"), "bob", "chunk-bytes"); - let chunk_id = chunk_message.id.clone(); - protocol.media_outbox.insert( - chunk_id.clone(), - OutboxEntry { - message: chunk_message, - attempt_count: 1, - first_sent_at: chrono::Utc::now(), - last_sent_at: chrono::Utc::now(), - last_transport: Some(TransportType::Internet), - reseal: None, - }, - ); - protocol - .ack_manager - .register_pending_ack(chunk_id.clone(), None) + let message_id = protocol + .send_message("bob", "hello", None::, None::) .unwrap(); - protocol - .on_transport_send_failed( - &chunk_id.as_str(), - Some("recipient_unreachable: peer offline".to_string()), - ) - .unwrap(); + for _ in 0..2 { + protocol + .on_transport_send_failed( + &message_id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + ) + .unwrap(); + } - assert!( - protocol.ack_manager.is_waiting_for_ack(&chunk_id), - "Media chunk must keep its pending ACK (not parked)" + let events = events.lock().unwrap(); + let undeliverable = events + .iter() + .filter(|e| matches!(e, Event::MessageUndeliverable { .. })) + .count(); + assert_eq!( + undeliverable, 2, + "Each verdict must re-emit the non-terminal undeliverable signal" + ); + assert!( + !events + .iter() + .any(|e| matches!(e, Event::MessageFailed { .. })), + "Repeated verdicts must never settle the message terminally" ); - assert!(protocol.media_outbox.contains_key(&chunk_id)); - assert!(protocol.dm_unreachable_parks.is_empty()); } -// ============================================================================ -// MESH FALLBACK ON THE RELAY'S UNREACHABLE VERDICT -// -// An infrastructure carrier being up says nothing about whether a particular -// recipient is on it, but it is what the send-time reachability check asks — -// so an online device used to keep every frame to itself, whatever the relay -// then said about the peer. These cover the one per-peer reachability fact -// that does arrive, the relay's `recipient_unreachable` verdict, driving the -// mesh hand-off the check cannot. -// ============================================================================ - -/// A device with a working relay connection and one neighbor standing next to -/// it — `carol`, who is not the recipient of anything here and can only carry. -/// -/// The radio refuses recipients it holds no link to, the way BLE does, so a -/// frame for a distant peer really does leave over the relay rather than being -/// swallowed by the mesh carrier. -fn online_device_with_a_neighbor() -> (OfflineProtocol, MockTransport, MockTransport) { - let mut protocol = OfflineProtocol::new(create_relay_test_config_for_user("user123")).unwrap(); +#[test] +fn test_internet_only_probe_exhaustion_reparks_not_terminal() { + // The hoisted counter is what arms try_repark_exhausted_dm on this + // branch. A probe that succeeds locally but is never answered burns the + // ACK budget; with a live counter that exhaustion must re-park (keeping + // the outbox entry) rather than settle terminally — otherwise unifying + // the branches would have traded a stalled message for a failed one. + let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); - let internet = MockTransport::new(TransportType::Internet); - internet.start().unwrap(); - let ble = MockTransport::new(TransportType::BLE); - ble.start().unwrap(); - ble.set_reject_unknown_recipients(true); - ble.add_connected_peer("carol", -55); + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let events_handle = Arc::clone(&events); + protocol.on_event(move |event| { + events_handle.lock().unwrap().push(event); + }); + let mock_transport = MockTransport::new(TransportType::Internet); + mock_transport.start().unwrap(); protocol .transport_manager_mut() - .add_transport(TransportType::Internet, Box::new(internet.clone())); - protocol - .transport_manager_mut() - .add_transport(TransportType::BLE, Box::new(ble.clone())); + .add_transport(TransportType::Internet, Box::new(mock_transport)); protocol.start().unwrap(); - (protocol, internet, ble) -} - -/// The acknowledgement a recipient sends back, as it looks arriving here. -fn delivery_ack_frame(from: &str, to: &str, acked: &MessageId) -> Message { - Message::builder( - UserId::new(from).unwrap(), - UserId::new(to).unwrap(), - AppId::new("test-app").unwrap(), - ) - .content(String::new()) - .requires_ack(false) - .metadata(ACK_FOR_KEY, acked.as_str()) - .metadata(ACK_HOP_COUNT_KEY, "2") - .metadata(ACK_TRANSPORT_KEY, "ble") - .build() -} - -#[test] -fn test_unreachable_verdict_offers_a_parked_dm_to_the_mesh() { - // The mixed neighborhood: this device is online, the recipient is not on - // the relay, and somebody who might reach them is standing right here. - let (mut protocol, internet, ble) = online_device_with_a_neighbor(); - let message_id = protocol .send_message("bob", "hello", None::, None::) .unwrap(); - - assert_eq!( - internet.sent_messages().len(), - 1, - "with the relay up the frame goes to the relay" - ); - assert!( - ble.peer_sends().is_empty(), - "and the mesh is not spent before the relay has said anything about this peer" - ); - protocol .on_transport_send_failed( &message_id.as_str(), Some("recipient_unreachable: peer offline".to_string()), ) .unwrap(); + assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&1)); - let handed = ble.peer_sends(); - assert_eq!( - handed.len(), - 1, - "the relay declaring this peer unreachable is the per-peer fact the \ - send-time check cannot have; the neighbors must be asked" - ); - assert_eq!(handed[0].0, "carol"); - assert_eq!(handed[0].1.id, message_id); + // Drive the exhaustion path directly: the probe's ACK budget ran out. + protocol + .handle_max_retries_exceeded(&message_id, 10) + .unwrap(); - // A neighbor taking a copy is not proof of arrival, so the park stands. - assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&1)); - assert!(protocol.outbox.contains_key(&message_id)); assert!( - !protocol.ack_manager.is_waiting_for_ack(&message_id), - "parking still drops the pending ACK" + protocol.outbox.contains_key(&message_id), + "Exhaustion with a live park counter must re-park, not evict" + ); + assert!( + !events + .lock() + .unwrap() + .iter() + .any(|e| matches!(e, Event::MessageFailed { .. })), + "Re-parked exhaustion must not surface a terminal message_failed" + ); + assert_eq!( + protocol.dm_unreachable_parks.get("bob"), + Some(&2), + "The re-park escalates the interval like any other park" ); } #[test] -fn test_a_later_park_offers_the_dm_to_whoever_is_around_then() { - // Neighbors come and go. A DM parked while nobody was in range must be - // offered again to whoever turns up, not only on the first park — the - // offer belongs to the park action, not to the first verdict. - let mut protocol = OfflineProtocol::new(create_relay_test_config_for_user("user123")).unwrap(); +fn test_delivery_ack_redrives_the_peers_other_parked_dms() { + // The park counter is per-peer while the probes are per-message, so a + // burst of DMs to an offline peer escalates the shared ladder once per + // park: message 1 probes at 15s but message 3 is already scheduled at 60s, + // and a burst of seven lands the last at the 600s cap. Clearing the + // counter on delivery is therefore not enough on its own — delivery of any + // one of them proves the rest can go NOW, and on a consumer that never + // polls presence this ACK is the only edge that will ever say so. + let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); - let internet = MockTransport::new(TransportType::Internet); - internet.start().unwrap(); - let ble = MockTransport::new(TransportType::BLE); - ble.start().unwrap(); - ble.set_reject_unknown_recipients(true); - protocol - .transport_manager_mut() - .add_transport(TransportType::Internet, Box::new(internet)); + let mock_transport = MockTransport::new(TransportType::Internet); + mock_transport.start().unwrap(); + let internet_handle = mock_transport.clone(); protocol .transport_manager_mut() - .add_transport(TransportType::BLE, Box::new(ble.clone())); + .add_transport(TransportType::Internet, Box::new(mock_transport)); protocol.start().unwrap(); - let message_id = protocol - .send_message("bob", "hello", None::, None::) - .unwrap(); - protocol - .on_transport_send_failed( - &message_id.as_str(), - Some("recipient_unreachable: peer offline".to_string()), - ) - .unwrap(); - assert!( - ble.peer_sends().is_empty(), - "nobody is around to carry it yet" + let ids: Vec<_> = (0..3) + .map(|i| { + protocol + .send_message( + "bob", + &format!("hello {}", i), + None::, + None::, + ) + .unwrap() + }) + .collect(); + for id in &ids { + protocol + .on_transport_send_failed( + &id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + ) + .unwrap(); + } + assert_eq!( + protocol.dm_unreachable_parks.get("bob"), + Some(&3), + "each park escalates the shared per-peer ladder" ); + for id in &ids { + assert!( + !protocol.ack_manager.is_waiting_for_ack(id), + "a parked DM holds no pending ACK" + ); + } - // Someone walks up, and the probe earns a second verdict. - ble.add_connected_peer("carol", -55); - protocol - .on_transport_send_failed( - &message_id.as_str(), - Some("recipient_unreachable: peer offline".to_string()), - ) - .unwrap(); + // Message 0's probe fires and registers its ACK (what the retry queue does + // on a probe send); the peer answers it. The ACK deliberately carries no + // ACK_TRANSPORT_KEY label: the re-drive override must come from the + // sender's own record of the delivered send's transport + // (`OutboxEntry::last_transport` — Internet here, and available), never + // from the peer-supplied label, which decodes any absent or unknown value + // to BLE and would let a peer steer the sibling sends. + let probed = protocol.outbox.get(&ids[0]).unwrap().message.clone(); + protocol.ensure_ack_registration(&probed).unwrap(); + let sends_before = internet_handle.sent_messages().len(); + + let ack = Message::builder( + UserId::new("bob").unwrap(), + UserId::new("user123").unwrap(), + AppId::new("test-app").unwrap(), + ) + .content(String::new()) + .metadata(ACK_FOR_KEY, ids[0].as_str()) + .build(); + protocol.handle_ack_message(&ack); - let handed = ble.peer_sends(); - assert_eq!(handed.len(), 1, "the second park must ask the new neighbor"); - assert_eq!(handed[0].1.id, message_id); assert_eq!( protocol.dm_unreachable_parks.get("bob"), - Some(&2), - "and the escalation continues as before" + None, + "delivery proves reachability: the escalation starts over" + ); + assert!( + internet_handle.sent_messages().len() > sends_before, + "the siblings must go out now, not wait out their escalated timers" ); + for id in &ids[1..] { + assert!( + protocol.ack_manager.is_waiting_for_ack(id), + "each sibling must be re-driven with a fresh ACK budget" + ); + assert!( + protocol.outbox.contains_key(id), + "re-driving must not settle the sibling" + ); + } } #[test] -fn test_a_parked_dm_is_settled_by_an_acknowledgement_carried_back() { - // The other half of the offer. Parking removed the pending ACK, so the - // answer to a mesh-carried delivery lands on the branch that used to drop - // it — leaving a delivered, read message being probed until its outbox - // lifetime ran out. Without this the offer would be worse than useless. - let (mut protocol, _internet, ble) = online_device_with_a_neighbor(); +fn test_delivery_ack_redrive_ignores_peer_label_and_falls_back_to_dors() { + // Two hostile-to-override conditions at once: the delivering transport + // recorded on the outbox entry is gone by ACK time (last_transport = BLE, + // never attached), and the ACK carries a garbage ACK_TRANSPORT_KEY label + // (peer-supplied; decodes to BLE). The re-drive must ignore the label + // entirely and, with its own last_transport unavailable, fall back to + // DORS — a missing carrier costs routing freedom, never the re-drive. + let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); - let events: Arc>> = Arc::new(Mutex::new(Vec::new())); - let events_handle = Arc::clone(&events); - protocol.on_event(move |event| { - events_handle.lock().unwrap().push(event); - }); + let mock_transport = MockTransport::new(TransportType::Internet); + mock_transport.start().unwrap(); + let internet_handle = mock_transport.clone(); + protocol + .transport_manager_mut() + .add_transport(TransportType::Internet, Box::new(mock_transport)); + protocol.start().unwrap(); + + let ids: Vec<_> = (0..2) + .map(|i| { + protocol + .send_message( + "bob", + &format!("hello {}", i), + None::, + None::, + ) + .unwrap() + }) + .collect(); + for id in &ids { + protocol + .on_transport_send_failed( + &id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + ) + .unwrap(); + } + assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&2)); + + // The carrier that delivered message 0 has since dropped. + protocol.outbox.get_mut(&ids[0]).unwrap().last_transport = Some(TransportType::BLE); + + let probed = protocol.outbox.get(&ids[0]).unwrap().message.clone(); + protocol.ensure_ack_registration(&probed).unwrap(); + let sends_before = internet_handle.sent_messages().len(); + + let ack = Message::builder( + UserId::new("bob").unwrap(), + UserId::new("user123").unwrap(), + AppId::new("test-app").unwrap(), + ) + .content(String::new()) + .metadata(ACK_FOR_KEY, ids[0].as_str()) + .metadata(crate::constants::ACK_TRANSPORT_KEY, "garbage") + .build(); + protocol.handle_ack_message(&ack); + + assert_eq!(protocol.dm_unreachable_parks.get("bob"), None); + assert!( + internet_handle.sent_messages().len() > sends_before, + "the sibling must be re-driven over DORS's pick, not stranded on a \ + carrier this device does not have" + ); + assert!( + protocol.ack_manager.is_waiting_for_ack(&ids[1]), + "the sibling must be re-driven with a fresh ACK budget" + ); +} + +#[test] +fn test_unreachable_dm_mesh_probe_escalates_and_resets_on_edge() { + // With a local mesh carrier up the peer may be a room away, so the park + // keeps a timed reachability probe whose interval escalates per + // consecutive unreachable park and resets on a reachability edge. + let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); + + let mock_transport = MockTransport::new(TransportType::BLE); + mock_transport.start().unwrap(); + protocol + .transport_manager_mut() + .add_transport(TransportType::BLE, Box::new(mock_transport)); + protocol.start().unwrap(); let message_id = protocol .send_message("bob", "hello", None::, None::) .unwrap(); + protocol .on_transport_send_failed( &message_id.as_str(), Some("recipient_unreachable: peer offline".to_string()), ) .unwrap(); - assert!(protocol.outbox.contains_key(&message_id)); + assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&1)); + let first_delay = protocol + .retry_queue + .time_until_next_retry() + .expect("first park must schedule a probe"); + assert!(first_delay > chrono::Duration::seconds(10)); + assert!(first_delay <= chrono::Duration::seconds(15)); - // Bob answers, and the answer is carried back to us by carol. - ble.queue_message_from( - delivery_ack_frame("bob", "user123", &message_id), - "carol".to_string(), - ); - assert!( - protocol.receive_message().is_none(), - "an acknowledgement is not app traffic" - ); + // Second consecutive park doubles the interval (15s -> 30s). + protocol + .on_transport_send_failed( + &message_id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + ) + .unwrap(); + assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&2)); + let second_delay = protocol + .retry_queue + .time_until_next_retry() + .expect("second park must re-schedule the probe"); + assert!(second_delay > chrono::Duration::seconds(20)); + assert!(second_delay <= chrono::Duration::seconds(30)); - assert!( - !protocol.outbox.contains_key(&message_id), - "the message was delivered; it must not keep being probed" - ); - assert!( - !protocol.retry_queue.contains(&message_id.as_str()), - "and its reachability probe must be gone with it" + // A per-peer reachability edge resets the escalation counter (the BLE + // mock accepts the re-driven send, so the edge genuinely takes). + protocol.on_neighbor_discovered("bob"); + assert_eq!(protocol.dm_unreachable_parks.get("bob"), None); +} + +#[test] +fn test_unreachable_media_chunk_is_not_parked() { + // Media chunks keep the normal retry machinery: their offline story is + // retry exhaustion -> transfer abort -> persisted descriptor -> + // MediaResendRequired with app-resupplied bytes. Parking would strand + // chunk bytes in memory for the outbox lifetime instead. + let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); + + let mock_transport = MockTransport::new(TransportType::Internet); + mock_transport.start().unwrap(); + protocol + .transport_manager_mut() + .add_transport(TransportType::Internet, Box::new(mock_transport)); + protocol.start().unwrap(); + + let chunk_message = signed_frame(&id("user123"), "bob", "chunk-bytes"); + let chunk_id = chunk_message.id.clone(); + protocol.media_outbox.insert( + chunk_id.clone(), + OutboxEntry { + message: chunk_message, + attempt_count: 1, + first_sent_at: chrono::Utc::now(), + last_sent_at: chrono::Utc::now(), + last_transport: Some(TransportType::Internet), + reseal: None, + relay_pushed: false, + }, ); - let captured = events.lock().unwrap(); + protocol + .ack_manager + .register_pending_ack(chunk_id.clone(), None) + .unwrap(); + + protocol + .on_transport_send_failed( + &chunk_id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + ) + .unwrap(); + assert!( - captured.iter().any(|event| matches!( - event, - Event::MessageDelivered { message_id: delivered, .. } if *delivered == message_id.as_str() - )), - "the app must be told it was delivered: {:?}", - *captured + protocol.ack_manager.is_waiting_for_ack(&chunk_id), + "Media chunk must keep its pending ACK (not parked)" ); + assert!(protocol.media_outbox.contains_key(&chunk_id)); + assert!(protocol.dm_unreachable_parks.is_empty()); } -/// The acknowledgement a leaf node sends back, which carries one entry. +// ============================================================================ +// MESH FALLBACK ON THE RELAY'S UNREACHABLE VERDICT +// +// An infrastructure carrier being up says nothing about whether a particular +// recipient is on it, but it is what the send-time reachability check asks — +// so an online device used to keep every frame to itself, whatever the relay +// then said about the peer. These cover the one per-peer reachability fact +// that does arrive, the relay's `recipient_unreachable` verdict, driving the +// mesh hand-off the check cannot. +// ============================================================================ + +/// A device with a working relay connection and one neighbor standing next to +/// it — `carol`, who is not the recipient of anything here and can only carry. /// -/// `offline-protocol-leaf` writes only [`ACK_FOR_KEY`]. The hop count and the -/// carrier are deliberately absent: a device is a direct peer, so its hop count -/// is zero, and it does not own its radio, so naming one would be firmware's -/// guess crossing the wire as fact. Both of this engine's defaults for the -/// missing entries are therefore already right for a leaf, which is the fact -/// this pins. -fn leaf_delivery_ack_frame(from: &str, to: &str, acked: &MessageId) -> Message { +/// The radio refuses recipients it holds no link to, the way BLE does, so a +/// frame for a distant peer really does leave over the relay rather than being +/// swallowed by the mesh carrier. +fn online_device_with_a_neighbor() -> (OfflineProtocol, MockTransport, MockTransport) { + let mut protocol = OfflineProtocol::new(create_relay_test_config_for_user("user123")).unwrap(); + + let internet = MockTransport::new(TransportType::Internet); + internet.start().unwrap(); + let ble = MockTransport::new(TransportType::BLE); + ble.start().unwrap(); + ble.set_reject_unknown_recipients(true); + ble.add_connected_peer("carol", -55); + + protocol + .transport_manager_mut() + .add_transport(TransportType::Internet, Box::new(internet.clone())); + protocol + .transport_manager_mut() + .add_transport(TransportType::BLE, Box::new(ble.clone())); + protocol.start().unwrap(); + + (protocol, internet, ble) +} + +/// The acknowledgement a recipient sends back, as it looks arriving here. +fn delivery_ack_frame(from: &str, to: &str, acked: &MessageId) -> Message { Message::builder( UserId::new(from).unwrap(), UserId::new(to).unwrap(), @@ -9368,42 +9606,217 @@ fn leaf_delivery_ack_frame(from: &str, to: &str, acked: &MessageId) -> Message { .content(String::new()) .requires_ack(false) .metadata(ACK_FOR_KEY, acked.as_str()) + .metadata(ACK_HOP_COUNT_KEY, "2") + .metadata(ACK_TRANSPORT_KEY, "ble") .build() } -/// A leaf's answer settles a message here, and the app is told what it means. -/// -/// Until issue 402 a leaf owed no acknowledgement, so every frame sent to one -/// ran the retry ladder to exhaustion and ended in `MessageFailed` for a -/// command the device had carried out. This is the engine's half of the fix, -/// and it is a separate test from the leaf's because neither crate can see the -/// other: the leaf builds the frame, this reads it, and the only thing they -/// share is `ACK_FOR_KEY`, which now has one declaration for exactly that -/// reason. #[test] -fn test_an_acknowledgement_carrying_only_ack_for_settles_the_message() { - let (mut protocol, _internet, ble) = online_device_with_a_neighbor(); - - let events: Arc>> = Arc::new(Mutex::new(Vec::new())); - let events_handle = Arc::clone(&events); - protocol.on_event(move |event| { - events_handle.lock().unwrap().push(event); - }); +fn test_unreachable_verdict_offers_a_parked_dm_to_the_mesh() { + // The mixed neighborhood: this device is online, the recipient is not on + // the relay, and somebody who might reach them is standing right here. + let (mut protocol, internet, ble) = online_device_with_a_neighbor(); let message_id = protocol .send_message("bob", "hello", None::, None::) .unwrap(); + + assert_eq!( + internet.sent_messages().len(), + 1, + "with the relay up the frame goes to the relay" + ); + assert!( + ble.peer_sends().is_empty(), + "and the mesh is not spent before the relay has said anything about this peer" + ); + protocol .on_transport_send_failed( &message_id.as_str(), Some("recipient_unreachable: peer offline".to_string()), ) .unwrap(); - assert!(protocol.outbox.contains_key(&message_id)); - ble.queue_message_from( - leaf_delivery_ack_frame("bob", "user123", &message_id), - "bob".to_string(), + let handed = ble.peer_sends(); + assert_eq!( + handed.len(), + 1, + "the relay declaring this peer unreachable is the per-peer fact the \ + send-time check cannot have; the neighbors must be asked" + ); + assert_eq!(handed[0].0, "carol"); + assert_eq!(handed[0].1.id, message_id); + + // A neighbor taking a copy is not proof of arrival, so the park stands. + assert_eq!(protocol.dm_unreachable_parks.get("bob"), Some(&1)); + assert!(protocol.outbox.contains_key(&message_id)); + assert!( + !protocol.ack_manager.is_waiting_for_ack(&message_id), + "parking still drops the pending ACK" + ); +} + +#[test] +fn test_a_later_park_offers_the_dm_to_whoever_is_around_then() { + // Neighbors come and go. A DM parked while nobody was in range must be + // offered again to whoever turns up, not only on the first park — the + // offer belongs to the park action, not to the first verdict. + let mut protocol = OfflineProtocol::new(create_relay_test_config_for_user("user123")).unwrap(); + + let internet = MockTransport::new(TransportType::Internet); + internet.start().unwrap(); + let ble = MockTransport::new(TransportType::BLE); + ble.start().unwrap(); + ble.set_reject_unknown_recipients(true); + protocol + .transport_manager_mut() + .add_transport(TransportType::Internet, Box::new(internet)); + protocol + .transport_manager_mut() + .add_transport(TransportType::BLE, Box::new(ble.clone())); + protocol.start().unwrap(); + + let message_id = protocol + .send_message("bob", "hello", None::, None::) + .unwrap(); + protocol + .on_transport_send_failed( + &message_id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + ) + .unwrap(); + assert!( + ble.peer_sends().is_empty(), + "nobody is around to carry it yet" + ); + + // Someone walks up, and the probe earns a second verdict. + ble.add_connected_peer("carol", -55); + protocol + .on_transport_send_failed( + &message_id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + ) + .unwrap(); + + let handed = ble.peer_sends(); + assert_eq!(handed.len(), 1, "the second park must ask the new neighbor"); + assert_eq!(handed[0].1.id, message_id); + assert_eq!( + protocol.dm_unreachable_parks.get("bob"), + Some(&2), + "and the escalation continues as before" + ); +} + +#[test] +fn test_a_parked_dm_is_settled_by_an_acknowledgement_carried_back() { + // The other half of the offer. Parking removed the pending ACK, so the + // answer to a mesh-carried delivery lands on the branch that used to drop + // it — leaving a delivered, read message being probed until its outbox + // lifetime ran out. Without this the offer would be worse than useless. + let (mut protocol, _internet, ble) = online_device_with_a_neighbor(); + + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let events_handle = Arc::clone(&events); + protocol.on_event(move |event| { + events_handle.lock().unwrap().push(event); + }); + + let message_id = protocol + .send_message("bob", "hello", None::, None::) + .unwrap(); + protocol + .on_transport_send_failed( + &message_id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + ) + .unwrap(); + assert!(protocol.outbox.contains_key(&message_id)); + + // Bob answers, and the answer is carried back to us by carol. + ble.queue_message_from( + delivery_ack_frame("bob", "user123", &message_id), + "carol".to_string(), + ); + assert!( + protocol.receive_message().is_none(), + "an acknowledgement is not app traffic" + ); + + assert!( + !protocol.outbox.contains_key(&message_id), + "the message was delivered; it must not keep being probed" + ); + assert!( + !protocol.retry_queue.contains(&message_id.as_str()), + "and its reachability probe must be gone with it" + ); + let captured = events.lock().unwrap(); + assert!( + captured.iter().any(|event| matches!( + event, + Event::MessageDelivered { message_id: delivered, .. } if *delivered == message_id.as_str() + )), + "the app must be told it was delivered: {:?}", + *captured + ); +} + +/// The acknowledgement a leaf node sends back, which carries one entry. +/// +/// `offline-protocol-leaf` writes only [`ACK_FOR_KEY`]. The hop count and the +/// carrier are deliberately absent: a device is a direct peer, so its hop count +/// is zero, and it does not own its radio, so naming one would be firmware's +/// guess crossing the wire as fact. Both of this engine's defaults for the +/// missing entries are therefore already right for a leaf, which is the fact +/// this pins. +fn leaf_delivery_ack_frame(from: &str, to: &str, acked: &MessageId) -> Message { + Message::builder( + UserId::new(from).unwrap(), + UserId::new(to).unwrap(), + AppId::new("test-app").unwrap(), + ) + .content(String::new()) + .requires_ack(false) + .metadata(ACK_FOR_KEY, acked.as_str()) + .build() +} + +/// A leaf's answer settles a message here, and the app is told what it means. +/// +/// Until issue 402 a leaf owed no acknowledgement, so every frame sent to one +/// ran the retry ladder to exhaustion and ended in `MessageFailed` for a +/// command the device had carried out. This is the engine's half of the fix, +/// and it is a separate test from the leaf's because neither crate can see the +/// other: the leaf builds the frame, this reads it, and the only thing they +/// share is `ACK_FOR_KEY`, which now has one declaration for exactly that +/// reason. +#[test] +fn test_an_acknowledgement_carrying_only_ack_for_settles_the_message() { + let (mut protocol, _internet, ble) = online_device_with_a_neighbor(); + + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let events_handle = Arc::clone(&events); + protocol.on_event(move |event| { + events_handle.lock().unwrap().push(event); + }); + + let message_id = protocol + .send_message("bob", "hello", None::, None::) + .unwrap(); + protocol + .on_transport_send_failed( + &message_id.as_str(), + Some("recipient_unreachable: peer offline".to_string()), + ) + .unwrap(); + assert!(protocol.outbox.contains_key(&message_id)); + + ble.queue_message_from( + leaf_delivery_ack_frame("bob", "user123", &message_id), + "bob".to_string(), ); assert!( protocol.receive_message().is_none(), @@ -9545,6 +9958,7 @@ fn test_unreachable_media_chunk_is_offered_to_the_mesh() { last_sent_at: chrono::Utc::now(), last_transport: Some(TransportType::Internet), reseal: None, + relay_pushed: false, }, ); protocol @@ -9965,6 +10379,7 @@ fn test_unpark_cancel_spares_connection_request_ack() { last_sent_at: chrono::Utc::now(), last_transport: Some(TransportType::Internet), reseal: None, + relay_pushed: false, }, ); protocol @@ -10143,6 +10558,7 @@ fn test_unpark_cancel_spares_welcome_ack() { last_sent_at: chrono::Utc::now(), last_transport: Some(TransportType::Internet), reseal: None, + relay_pushed: false, }, ); protocol @@ -10353,6 +10769,7 @@ fn test_cleanup_outbox_absolute_lifetime_cap_is_terminal_in_process() { last_sent_at: chrono::Utc::now(), // fresh: probe just sent last_transport: None, reseal: None, + relay_pushed: false, }, ) }; @@ -17299,178 +17716,1384 @@ fn test_encrypted_message_group_not_found_is_queued_with_typed_classification() let message = signed_frame(&id("sender123"), &id("user123"), &encrypted_content); - let result = protocol.process_internal_message(&message); + let result = protocol.process_internal_message(&message); + + assert!(matches!(result, Some(InternalMessageResult::Deferred))); + assert!(protocol.pending_queue.contains_peer(&id("sender123"))); + assert_eq!(protocol.pending_queue.peer_queue_len(&id("sender123")), 1); +} + +#[test] +fn test_pending_queue_stress_memory_plateaus_with_unfinished_handshake() { + let mut config = create_test_config(); + config.encryption.enabled = true; + config.encryption.pending_queue.max_pending_per_peer = 32; + config.encryption.pending_queue.max_pending_global = 64; + config.encryption.pending_queue.pending_ttl_ms = 60_000; + config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropOldest; + + let mut protocol = OfflineProtocol::new(config).unwrap(); + for idx in 0..10_000 { + let msg = pending_test_message(&id("sender123"), &format!("encrypted-{idx}")); + protocol.enqueue_pending_decryption("sender123", &msg); + } + + assert_eq!(protocol.pending_queue.total(), 32); + assert_eq!( + protocol.pending_queue.metrics().pending_messages_current, + 32 + ); + assert_eq!( + *protocol + .pending_queue + .metrics() + .pending_messages_per_peer + .get("sender123") + .unwrap(), + 32 + ); +} + +#[test] +fn test_pending_queue_sustained_mixed_invalid_and_early_encrypted_is_bounded() { + let mut config = create_test_config(); + config.encryption.enabled = true; + config.encryption.pending_queue.max_pending_per_peer = 16; + config.encryption.pending_queue.max_pending_global = 32; + config.encryption.pending_queue.pending_ttl_ms = 60_000; + config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropOldest; + + let mut protocol = OfflineProtocol::new(config).unwrap(); + protocol + .initialize_mls_for_test(Arc::new(crate::mls::InMemoryStorage::new())) + .unwrap(); + + let early_slot = + offline_protocol_mls::GroupId::for_session(&id("user123"), &id("sender123")).unwrap(); + let sender_address = id("sender123"); + let valid_early_encrypted = format!( + "{}{{\"group_id\":\"{early_slot}\",\"message_type\":\"Application\",\"epoch\":0,\"ciphertext\":[1,2,3],\"sender_id\":\"{sender_address}\",\"timestamp_ms\":12345}}", + internal_prefixes::ENCRYPTED + ); + let malformed_variants = [ + format!("{}{{", internal_prefixes::ENCRYPTED), + format!("{}{{\"group_id\":\"bad\"", internal_prefixes::ENCRYPTED), + format!("{}[]", internal_prefixes::ENCRYPTED), + format!( + "{}{{\"ciphertext\":\"not-array\"}}", + internal_prefixes::ENCRYPTED + ), + ]; + + let mut early_count: u64 = 0; + let mut invalid_count: u64 = 0; + for idx in 0..10_000 { + let is_invalid = idx % 5 == 0; + let content = if is_invalid { + invalid_count += 1; + malformed_variants[(idx % malformed_variants.len()) as usize].as_str() + } else { + early_count += 1; + valid_early_encrypted.as_str() + }; + + let message = signed_frame(&id("sender123"), &id("user123"), content); + let result = protocol.process_internal_message(&message); + // Both shapes defer, but for different reasons — and only one of them + // queues. A valid-but-early encrypted message is queued and not ACKed + // (it becomes decryptable once the session lands). A malformed payload + // is *not* queued — it can never become parseable — but is still not + // ACKed, so the sender's resend stays the recovery path. The metrics + // assertions below are what pin that split: only the early messages + // count toward `pending_messages_received_total`, so a malformed flood + // cannot grow the queue. + assert!(matches!(result, Some(InternalMessageResult::Deferred))); + } + + let per_peer_limit = protocol + .config + .encryption + .pending_queue + .max_pending_per_peer; + let global_limit = protocol.config.encryption.pending_queue.max_pending_global; + assert!(protocol.pending_queue.total() <= global_limit); + assert!(protocol.pending_queue.peer_queue_len("sender123") <= per_peer_limit); + + let metrics = protocol.pending_queue_metrics(); + assert_eq!(metrics.pending_messages_received_total, early_count); + assert_eq!( + metrics.pending_messages_current, + protocol.pending_queue.total() + ); + assert!(metrics.pending_messages_dropped_overflow_total > 0); + assert_eq!(early_count + invalid_count, 10_000); +} + +#[test] +fn test_pending_queue_flood_respects_per_peer_fairness() { + let mut config = create_test_config(); + config.encryption.enabled = true; + config.encryption.pending_queue.max_pending_per_peer = 3; + config.encryption.pending_queue.max_pending_global = 6; + config.encryption.pending_queue.pending_ttl_ms = 60_000; + config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropOldest; + + let mut protocol = OfflineProtocol::new(config).unwrap(); + + for idx in 0..100 { + let msg = pending_test_message(&id("noisy-peer"), &format!("noisy-{idx}")); + protocol.enqueue_pending_decryption("noisy-peer", &msg); + } + for idx in 0..3 { + let msg = pending_test_message(&id("peer-a"), &format!("a-{idx}")); + protocol.enqueue_pending_decryption("peer-a", &msg); + let msg = pending_test_message(&id("peer-b"), &format!("b-{idx}")); + protocol.enqueue_pending_decryption("peer-b", &msg); + } + + assert!(protocol.pending_queue.total() <= 6); + assert!(protocol.pending_queue.peer_queue_len("noisy-peer") <= 3); + assert!(protocol.pending_queue.contains_peer("peer-a")); + assert!(protocol.pending_queue.contains_peer("peer-b")); +} + +#[test] +fn test_pending_queue_drop_newest_policy_enforced_for_per_peer_limit() { + let mut config = create_test_config(); + config.encryption.enabled = true; + config.encryption.pending_queue.max_pending_per_peer = 1; + config.encryption.pending_queue.max_pending_global = 10; + config.encryption.pending_queue.pending_ttl_ms = 60_000; + config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropNewest; + + let mut protocol = OfflineProtocol::new(config).unwrap(); + let first = pending_test_message(&id("peer-a"), "first"); + let second = pending_test_message(&id("peer-a"), "second"); + protocol.enqueue_pending_decryption("peer-a", &first); + protocol.enqueue_pending_decryption("peer-a", &second); + + assert_eq!(protocol.pending_queue.peer_queue_len("peer-a"), 1); + assert_eq!( + protocol + .pending_queue + .peek_entry("peer-a", 0) + .unwrap() + .message + .content, + "first" + ); + assert_eq!( + protocol + .pending_queue + .metrics() + .pending_messages_dropped_overflow_total, + 1 + ); +} + +#[test] +fn test_pending_queue_overflow_emits_pending_queue_dropped_for_text() { + // Text frames evicted from the pending-decryption queue used to be + // metrics-only; only media chunks surfaced `PendingQueueDropped`. With a + // relay that pushes ciphertext without store-and-forward there is no + // second copy, so a silent text eviction was a lost message with no + // signal to the app. Overflowing the per-peer cap must now surface the + // evicted text frame's id under the same code, attributed to its sender. + let mut config = create_test_config(); + config.encryption.enabled = true; + config.encryption.pending_queue.max_pending_per_peer = 1; + config.encryption.pending_queue.max_pending_global = 10; + config.encryption.pending_queue.pending_ttl_ms = 60_000; + config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropOldest; + + let mut protocol = OfflineProtocol::new(config).unwrap(); + let dropped: Arc>> = + Arc::new(Mutex::new(Vec::new())); + let dropped_clone = dropped.clone(); + protocol.on_event(move |event| { + if let Event::MessageDecryptionFailed { + message_id, + sender, + code, + reason, + } = event + { + dropped_clone + .lock() + .unwrap() + .push((message_id, sender, code, reason)); + } + }); + + let first = pending_test_message(&id("peer-a"), "first"); + let second = pending_test_message(&id("peer-a"), "second"); + assert_ne!(first.content_type, ContentType::FileChunk); + protocol.enqueue_pending_decryption("peer-a", &first); + assert!(dropped.lock().unwrap().is_empty()); + protocol.enqueue_pending_decryption("peer-a", &second); + + let events = dropped.lock().unwrap(); + assert_eq!(events.len(), 1, "exactly the evicted frame is reported"); + let (message_id, sender, code, reason) = &events[0]; + assert_eq!(*message_id, first.id.as_str()); + assert_eq!(sender, &id("peer-a")); + assert_eq!(*code, DecryptionFailureCode::PendingQueueDropped); + assert!( + reason.contains("overflow_drop_oldest") && reason.contains("sender resends"), + "reason must carry the machine-readable drop cause, got {reason:?}" + ); + assert!( + !reason.contains("media chunk"), + "text evictions must not be described as media, got {reason:?}" + ); +} + +// --------------------------------------------------------------------------- +// Pending-decryption queue persistence (`pending_decrypt_entries`) +// --------------------------------------------------------------------------- + +/// A bob whose protocol-state storage is `storage`, so a second call with the +/// same handle is a restart against the same on-disk state. +fn pending_decrypt_bob(storage: Arc) -> OfflineProtocol { + let mut config = create_test_config_for_user("bob"); + config.encryption.enabled = true; + config.encryption.store_pending = true; + let mut bob = OfflineProtocol::new(config).unwrap(); + bob.initialize_mls_for_test(storage).unwrap(); + bob +} + +/// Alice's side of a handshake with `bob`: a Welcome for bob and one message +/// encrypted to him, both as wire frames, with the Welcome deliberately held +/// back so the message is queued as session-not-ready. +fn pending_decrypt_alice_frames(bob: &OfflineProtocol) -> (Message, Message) { + let alice_manager = + crate::test_identity::manager_for("alice", Arc::new(crate::mls::InMemoryStorage::new())); + let bob_key_package = { + let manager = bob.mls_manager.as_ref().unwrap().read().unwrap(); + manager.get_or_create_key_package().unwrap() + }; + alice_manager + .import_key_package(&id("bob"), &bob_key_package.key_package_data) + .unwrap(); + let welcome = alice_manager.create_session(&id("bob")).unwrap(); + let encrypted = alice_manager + .encrypt_for_user(&id("bob"), b"survived-the-restart") + .unwrap(); + let encrypted_wire = signed_frame( + &id("alice"), + &id("bob"), + &format!( + "{}{}", + internal_prefixes::ENCRYPTED, + serde_json::to_string(&encrypted).unwrap() + ), + ); + let welcome_wire = signed_frame( + &id("alice"), + &id("bob"), + &format!( + "{}{}", + internal_prefixes::WELCOME, + serde_json::to_string(&welcome).unwrap() + ), + ); + (encrypted_wire, welcome_wire) +} + +/// The headline case: a frame parked before its Welcome arrives survives an +/// app restart, and the Welcome that arrives *after* the restart still drains +/// and delivers it — the message a push-only relay would otherwise have lost. +#[test] +fn test_pending_decrypt_entry_survives_restart_and_drains_on_welcome() { + let storage = Arc::new(InMemoryStorage::new()); + let mut bob = pending_decrypt_bob(storage.clone()); + let (encrypted_wire, welcome_wire) = pending_decrypt_alice_frames(&bob); + let message_id = encrypted_wire.id.clone(); + + let result = bob.process_internal_message_via(&encrypted_wire, Some(TransportType::BLE)); + assert!(matches!(result, Some(InternalMessageResult::Deferred))); + assert!(bob.pending_queue.contains_peer(&id("alice"))); + assert_eq!( + bob.persisted_pending_decrypt_ids(), + vec![message_id.as_str()], + "an admitted frame is persisted under its own id" + ); + drop(bob); + + let mut bob = pending_decrypt_bob(storage.clone()); + assert_eq!( + bob.pending_queue.peer_queue_len(&id("alice")), + 1, + "the parked frame must be restored on the next launch" + ); + let restored = bob.pending_queue.peek_entry(&id("alice"), 0).unwrap(); + assert_eq!(restored.message.id, message_id); + assert_eq!( + restored.received_via, + Some(TransportType::BLE), + "the arrival transport rides in the record so the drain can still ACK directly" + ); + assert!( + !bob.deduplicator.is_duplicate(&message_id), + "a restored id is not dedup-marked: the sender's resend must still re-enter the queue" + ); + assert_eq!( + bob.persisted_pending_decrypt_ids(), + vec![message_id.as_str()], + "restore must not rewrite (or delete) a record it re-admitted" + ); + + let welcome_result = bob.process_internal_message(&welcome_wire); + assert!(matches!( + welcome_result, + Some(InternalMessageResult::Consumed) + )); + assert!(!bob.pending_queue.contains_peer(&id("alice"))); + let delivered = bob + .receive_message() + .expect("the restored frame must surface once the session confirms"); + assert_eq!(delivered.content, "survived-the-restart"); + assert_eq!( + delivered + .metadata + .get("delayed_decrypt") + .map(String::as_str), + Some("true") + ); + assert!( + bob.persisted_pending_decrypt_ids().is_empty(), + "a drained frame's record is deleted" + ); + assert!( + bob.deduplicator.is_duplicate(&message_id), + "delivery re-marks the id like the live drain does" + ); +} + +/// The drain deletes the record whatever it does with the frame — here a +/// frame that decrypts nowhere (no session, plain content) and is simply +/// dropped by the drain must still not come back on the next launch. +#[test] +fn test_pending_decrypt_drained_entry_is_deleted_even_when_undeliverable() { + let storage = Arc::new(InMemoryStorage::new()); + let mut bob = pending_decrypt_bob(storage.clone()); + let frame = pending_test_message(&id("alice"), "opaque"); + bob.enqueue_pending_decryption(&id("alice"), &frame); + assert_eq!(bob.persisted_pending_decrypt_ids().len(), 1); + + bob.process_pending_decryption(&id("alice")); + assert!(!bob.pending_queue.contains_peer(&id("alice"))); + assert!(bob.persisted_pending_decrypt_ids().is_empty()); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert!( + !bob.pending_queue.contains_peer(&id("alice")), + "a drained frame must not be restored" + ); +} + +#[test] +fn test_pending_decrypt_overflow_dropped_entry_is_deleted() { + let storage = Arc::new(InMemoryStorage::new()); + let mut config = create_test_config_for_user("bob"); + config.encryption.enabled = true; + config.encryption.pending_queue.max_pending_per_peer = 1; + config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropOldest; + let mut bob = OfflineProtocol::new(config).unwrap(); + bob.initialize_mls_for_test(storage.clone()).unwrap(); + + let first = pending_test_message(&id("alice"), "first"); + let second = pending_test_message(&id("alice"), "second"); + bob.enqueue_pending_decryption(&id("alice"), &first); + assert_eq!(bob.persisted_pending_decrypt_ids(), vec![first.id.as_str()]); + bob.enqueue_pending_decryption(&id("alice"), &second); + + assert_eq!(bob.pending_queue.peer_queue_len(&id("alice")), 1); + assert_eq!( + bob.persisted_pending_decrypt_ids(), + vec![second.id.as_str()], + "the evicted frame's record goes with it; the survivor's stays" + ); + + // A refused frame (DropNewest) never had a record and must not gain one. + bob.config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropNewest; + let third = pending_test_message(&id("alice"), "third"); + bob.enqueue_pending_decryption(&id("alice"), &third); + assert_eq!( + bob.persisted_pending_decrypt_ids(), + vec![second.id.as_str()] + ); +} + +/// The in-memory TTL restarts with the process, so the persisted copy needs +/// its own bound: a record past `PENDING_DECRYPT_PERSISTED_MAX_AGE_MS` is +/// dropped on restore, its record deleted, and the app told under the same +/// code an in-memory eviction uses. +#[test] +fn test_pending_decrypt_record_past_persisted_max_age_is_dropped_with_event() { + use super::storage::PENDING_DECRYPT_PERSISTED_MAX_AGE_MS; + + let storage = Arc::new(InMemoryStorage::new()); + let bob = pending_decrypt_bob(storage.clone()); + let stale = pending_test_message(&id("alice"), "stale"); + let fresh = pending_test_message(&id("alice"), "fresh"); + let now_ms = Utc::now().timestamp_millis(); + let eight_days = PENDING_DECRYPT_PERSISTED_MAX_AGE_MS + 24 * 60 * 60 * 1000; + bob.persist_pending_decrypt_entry(&id("alice"), &stale, now_ms - eight_days, None); + bob.persist_pending_decrypt_entry(&id("alice"), &fresh, now_ms - 1_000, None); + assert_eq!(bob.persisted_pending_decrypt_ids().len(), 2); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert_eq!( + bob.pending_queue.peer_queue_len(&id("alice")), + 1, + "only the fresh record is restored" + ); + assert_eq!( + bob.pending_queue + .peek_entry(&id("alice"), 0) + .unwrap() + .message + .id, + fresh.id + ); + assert_eq!( + bob.persisted_pending_decrypt_ids(), + vec![fresh.id.as_str()], + "the aged-out record is deleted" + ); + // Restore runs before the event pipeline is live, so the settlement is + // parked for start() like every other restore-time terminal event. + let expired: Vec<_> = bob + .deferred_restore_settlements + .iter() + .filter_map(|event| match event { + Event::MessageDecryptionFailed { + message_id, + sender, + code, + reason, + } => Some((message_id.clone(), sender.clone(), *code, reason.clone())), + _ => None, + }) + .collect(); + assert_eq!(expired.len(), 1, "exactly the aged-out frame is reported"); + let (message_id, sender, code, reason) = &expired[0]; + assert_eq!(*message_id, stale.id.as_str()); + assert_eq!(sender, &id("alice")); + assert_eq!(*code, DecryptionFailureCode::PendingQueueDropped); + assert!( + reason.contains("expired_persisted"), + "the reason names the persisted-age bound, got {reason:?}" + ); +} + +#[test] +fn test_pending_decrypt_corrupt_record_is_deleted_on_restore() { + let storage = Arc::new(InMemoryStorage::new()); + let bob = pending_decrypt_bob(storage.clone()); + let good = pending_test_message(&id("alice"), "good"); + bob.persist_pending_decrypt_entry(&id("alice"), &good, Utc::now().timestamp_millis(), None); + // Sealed like a real record (so it opens) but not a record. + let state_storage = bob.protocol_state_storage.clone().unwrap(); + bob.write_state_record( + state_storage.as_ref(), + storage_keys::PENDING_DECRYPT_ENTRIES, + "not-a-record", + b"{\"version\":1,\"peer_id\":", + ) + .unwrap(); + // A well-formed record filed under a key that is not its message id is + // not one this SDK wrote either. + let mislabeled = pending_test_message(&id("alice"), "mislabeled"); + let record = serde_json::to_vec(&PendingDecryptRecord { + version: PENDING_DECRYPT_RECORD_VERSION, + peer_id: id("alice"), + message: mislabeled, + first_received_at_ms: Utc::now().timestamp_millis(), + received_via: None, + }) + .unwrap(); + bob.write_state_record( + state_storage.as_ref(), + storage_keys::PENDING_DECRYPT_ENTRIES, + "wrong-key", + &record, + ) + .unwrap(); + assert_eq!(bob.persisted_pending_decrypt_ids().len(), 3); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert_eq!(bob.pending_queue.peer_queue_len(&id("alice")), 1); + assert_eq!( + bob.persisted_pending_decrypt_ids(), + vec![good.id.as_str()], + "records that do not parse, or lie about their key, are deleted" + ); + assert!( + bob.deferred_restore_settlements.is_empty(), + "a corrupt record names no message, so there is nothing to settle" + ); +} + +/// The sender's resend after the receiver restarted: the restored copy is +/// authoritative, so the resend is a no-op in memory and on disk — one entry, +/// one record, and the record keeps its first-receipt timestamp. +#[test] +fn test_pending_decrypt_reenqueue_after_restore_keeps_a_single_copy() { + let storage = Arc::new(InMemoryStorage::new()); + let mut bob = pending_decrypt_bob(storage.clone()); + let frame = pending_test_message(&id("alice"), "resent"); + bob.enqueue_pending_decryption(&id("alice"), &frame); + let state_storage = bob.protocol_state_storage.clone().unwrap(); + let original_record = bob + .read_state_record( + state_storage.as_ref(), + storage_keys::PENDING_DECRYPT_ENTRIES, + &frame.id.as_str(), + ) + .unwrap() + .unwrap(); + drop(bob); + + let mut bob = pending_decrypt_bob(storage); + assert_eq!(bob.pending_queue.peer_queue_len(&id("alice")), 1); + bob.enqueue_pending_decryption_via(&id("alice"), &frame, Some(TransportType::Internet)); + + assert_eq!(bob.pending_queue.peer_queue_len(&id("alice")), 1); + assert_eq!(bob.pending_queue.total(), 1); + assert_eq!(bob.persisted_pending_decrypt_ids(), vec![frame.id.as_str()]); + let state_storage = bob.protocol_state_storage.clone().unwrap(); + let record_after = bob + .read_state_record( + state_storage.as_ref(), + storage_keys::PENDING_DECRYPT_ENTRIES, + &frame.id.as_str(), + ) + .unwrap() + .unwrap(); + assert_eq!( + record_after, original_record, + "a resend of a queued id must not rewrite its record" + ); +} + +/// Unblocking a peer resets their session and discards what they had parked; +/// the records go too, so the discarded frames do not come back on restart. +#[test] +fn test_pending_decrypt_records_are_deleted_when_peer_queue_is_discarded() { + let storage = Arc::new(InMemoryStorage::new()); + let mut bob = pending_decrypt_bob(storage.clone()); + bob.enqueue_pending_decryption(&id("alice"), &pending_test_message(&id("alice"), "a")); + bob.enqueue_pending_decryption(&id("carol"), &pending_test_message(&id("carol"), "c")); + assert_eq!(bob.persisted_pending_decrypt_ids().len(), 2); + + assert_eq!(bob.discard_pending_decryption_for_peer(&id("alice")), 1); + assert!(!bob.pending_queue.contains_peer(&id("alice"))); + assert_eq!(bob.persisted_pending_decrypt_ids().len(), 1); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert!(!bob.pending_queue.contains_peer(&id("alice"))); + assert!(bob.pending_queue.contains_peer(&id("carol"))); +} + +/// Blocking a peer discards the frames already parked for them, records +/// included. The block is otherwise applied only when the queue drains, which +/// never happens for a blocked peer, so the records were restored on every +/// launch for up to seven days, and each eviction reported a +/// `PendingQueueDropped` naming the peer the user had blocked. +#[test] +fn test_block_user_discards_persisted_pending_decrypt_records() { + let storage = Arc::new(InMemoryStorage::new()); + let mut bob = pending_decrypt_bob(storage.clone()); + bob.enqueue_pending_decryption(&id("alice"), &pending_test_message(&id("alice"), "a")); + bob.enqueue_pending_decryption(&id("carol"), &pending_test_message(&id("carol"), "c")); + assert_eq!( + bob.persisted_pending_decrypt_ids().len(), + 2, + "precondition: both parked frames are persisted" + ); + + bob.block_user(&id("alice")).unwrap(); + + assert!( + !bob.pending_queue.contains_peer(&id("alice")), + "the blocked peer's frames leave the queue" + ); + assert_eq!( + bob.persisted_pending_decrypt_ids().len(), + 1, + "and their records leave the store" + ); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert!( + bob.is_user_blocked(&id("alice")), + "precondition: the block itself survives the restart" + ); + assert!( + !bob.pending_queue.contains_peer(&id("alice")), + "nothing parked for the blocked peer comes back on the next launch" + ); + assert!( + bob.pending_queue.contains_peer(&id("carol")), + "another peer's parked frame is untouched" + ); +} + +/// A peer-requested session reset discards the frames parked for the session +/// it deletes, and their records go with them. Left on disk, the next launch +/// restores frames sealed to a dead session and drains them into the +/// replacement one as spurious decrypt failures. +#[test] +fn test_session_reset_deletes_persisted_pending_decrypt_records() { + let (mut alice, _alice_h) = make_encrypted_protocol("alice"); + let (mut bob, _bob_h) = make_encrypted_protocol("bob"); + alice.start().unwrap(); + bob.start().unwrap(); + establish_confirmed_session(&mut alice, &id("alice"), &mut bob, &id("bob")); + + alice.enqueue_pending_decryption(&id("bob"), &pending_test_message(&id("bob"), "old-epoch")); + alice.enqueue_pending_decryption( + &id("carol"), + &pending_test_message(&id("carol"), "unrelated"), + ); + assert_eq!( + alice.persisted_pending_decrypt_ids().len(), + 2, + "precondition: both parked frames are persisted" + ); + + feed_session_reset_key_package(&mut alice, &mut bob, &id("bob")); + + assert!( + !alice.pending_queue.contains_peer(&id("bob")), + "precondition: the reset drained bob's parked frames" + ); + assert_eq!( + alice.persisted_pending_decrypt_ids().len(), + 1, + "the reset must delete the records of the frames it discarded" + ); + assert!( + alice.pending_queue.contains_peer(&id("carol")), + "another peer's parked frame is untouched" + ); +} + +// --------------------------------------------------------------------------- +// Deduplicator seen-set persistence (`dedup_seen_ids`) +// --------------------------------------------------------------------------- + +/// Releasing a dropped group entry's replay protection must reach the +/// persisted seen set. A write that landed while the entry was buffered holds +/// the id; if the release is not counted, nothing rewrites that record, the +/// next launch restores the id, and the sender's redelivery is swallowed and +/// re-ACKed as delivered for the whole retention window. +#[test] +fn test_released_replay_protection_is_not_restored_from_the_seen_set() { + let storage = Arc::new(InMemoryStorage::new()); + let mut bob = pending_decrypt_bob(storage.clone()); + bob.start().unwrap(); + let envelope = MessageId::new(); + assert!(bob.mark_seen_persisted(envelope.clone())); + // A batched write lands while the entry is still buffered. + bob.flush_dedup_seen(); + // The buffered entry is then dropped undelivered. + bob.release_replay_protection(&envelope.as_str()); + assert!(!bob.deduplicator.is_duplicate(&envelope)); + bob.stop().unwrap(); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert!( + !bob.deduplicator.is_duplicate(&envelope), + "a released id must not come back from the persisted seen set" + ); +} + +/// A protocol-state store that counts writes to one category, for pinning +/// the seen-set batching rule. +struct DedupWriteCountingStorage { + inner: Arc, + dedup_writes: Mutex, +} + +impl crate::ProtocolStateStorage for DedupWriteCountingStorage { + fn store(&self, key_type: &str, key_id: &str, data: &[u8]) -> crate::ProtocolStateResult<()> { + if key_type == storage_keys::DEDUP_SEEN_IDS { + *self.dedup_writes.lock().unwrap() += 1; + } + self.inner + .store(key_type, key_id, data) + .map_err(crate::protocol::map_test_storage_error) + } + + fn load(&self, key_type: &str, key_id: &str) -> crate::ProtocolStateResult>> { + self.inner + .load(key_type, key_id) + .map_err(crate::protocol::map_test_storage_error) + } + + fn delete(&self, key_type: &str, key_id: &str) -> crate::ProtocolStateResult<()> { + self.inner + .delete(key_type, key_id) + .map_err(crate::protocol::map_test_storage_error) + } + + fn list_keys(&self, key_type: &str) -> crate::ProtocolStateResult> { + self.inner + .list_keys(key_type) + .map_err(crate::protocol::map_test_storage_error) + } +} + +/// The restart case the record exists for: an id marked on the receive path +/// is still a duplicate on the next launch, so the socket copy of a message +/// the app already consumed from a push injection is deduped rather than +/// sent to a ratchet whose generation it already spent. +#[test] +fn test_dedup_seen_set_survives_stop_and_restart() { + let storage = Arc::new(InMemoryStorage::new()); + let mut bob = pending_decrypt_bob(storage.clone()); + bob.start().unwrap(); + let seen = MessageId::new(); + let released = MessageId::new(); + assert!(bob.mark_seen_persisted(seen.clone())); + assert!(bob.mark_seen_persisted(released.clone())); + assert!(bob.unmark_seen_persisted(&released)); + // Nothing has been written yet: the write is batched, and `stop()` is + // what flushes it. + let state_storage = bob.protocol_state_storage.clone().unwrap(); + assert!(bob + .read_state_record( + state_storage.as_ref(), + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + ) + .unwrap() + .is_none()); + bob.stop().unwrap(); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert!( + bob.deduplicator.is_duplicate(&seen), + "an id marked before the restart must still read as a duplicate" + ); + assert!( + !bob.deduplicator.is_duplicate(&released), + "an id the receive path released must not be resurrected by the record" + ); + assert_eq!(bob.dedup_dirty, 0, "a restore is not a change to persist"); +} + +/// The React Native layer applies `reliability.dedup` through +/// `update_dedup_config` on every `start()`, after the seen set has been +/// restored. The update must carry the restored ids across, not rebuild an +/// empty deduplicator: otherwise the restart-time duplicate the record exists +/// to recognise is processed after all, and the next batch write overwrites +/// the record with the near-empty set. +#[test] +fn test_dedup_seen_set_survives_a_runtime_config_update() { + let storage = Arc::new(InMemoryStorage::new()); + let mut bob = pending_decrypt_bob(storage.clone()); + bob.start().unwrap(); + let seen = MessageId::new(); + assert!(bob.mark_seen_persisted(seen.clone())); + bob.stop().unwrap(); + drop(bob); + + let mut bob = pending_decrypt_bob(storage.clone()); + bob.start().unwrap(); + assert!( + bob.deduplicator.is_duplicate(&seen), + "precondition: restored" + ); + + let mut updated = bob.config.reliability.dedup.clone(); + updated.max_tracked_messages += 1; + bob.update_dedup_config(updated).unwrap(); + + assert!( + bob.deduplicator.is_duplicate(&seen), + "a runtime config update must not discard the restored seen set" + ); + assert!( + bob.dedup_dirty > 0, + "the update is a change to persist: the record is re-stated under the new bounds" + ); + bob.stop().unwrap(); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert!( + bob.deduplicator.is_duplicate(&seen), + "the record written after the update still holds the id" + ); +} + +/// A sender's own outgoing ids dedup a relayed echo in memory but never +/// reach the persisted record: the send paths mark through +/// `mark_seen_local`, so the record's cap is spent on inbound ids only. A +/// send-side `mark_seen` would put the id in the next write and fail this. +#[test] +fn test_dedup_seen_set_excludes_own_outgoing_ids() { + let mut config = create_test_config(); + config.encryption.enabled = false; + let mut protocol = OfflineProtocol::new(config).unwrap(); + let storage = Arc::new(TestProtocolStateStorage { + storage: Arc::new(InMemoryStorage::new()), + }); + protocol.protocol_state_storage = Some(storage.clone()); + // The seen set is sealed, so a write needs the record key. + protocol.state_record_cipher = Some(state_crypto::StateRecordCipher::new( + &[7u8; state_crypto::STATE_RECORD_KEY_BYTES], + )); + protocol.start().unwrap(); + + let own = protocol + .send_message(&id("bob"), "outgoing", None, None::) + .unwrap(); + let inbound = MessageId::new(); + assert!(protocol.mark_seen_persisted(inbound.clone())); + assert!( + protocol.deduplicator.is_duplicate(&own), + "the sender still recognises its own id while the process lives" + ); + protocol.stop().unwrap(); + + let data = protocol + .read_state_record( + storage.as_ref(), + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + ) + .unwrap() + .expect("stop() flushes the seen set"); + let record: super::DedupSeenRecord = serde_json::from_slice(&data).unwrap(); + let ids: Vec<&str> = record.entries.iter().map(|e| e.id.as_str()).collect(); + assert!( + ids.contains(&inbound.as_str().as_str()), + "the inbound id is persisted" + ); + assert!( + !ids.contains(&own.as_str().as_str()), + "an outgoing id must not be persisted: {ids:?}" + ); +} + +/// The batching rule: nothing per message, one write once 32 changes have +/// accumulated, and one write for any smaller batch once the interval has +/// elapsed on a `process()` tick. +#[test] +fn test_dedup_seen_set_persistence_is_batched() { + use super::{DEDUP_PERSIST_DIRTY_THRESHOLD, DEDUP_PERSIST_INTERVAL}; + + let storage = Arc::new(DedupWriteCountingStorage { + inner: Arc::new(InMemoryStorage::new()), + dedup_writes: Mutex::new(0), + }); + let mut protocol = OfflineProtocol::new(create_test_config()).unwrap(); + protocol.protocol_state_storage = Some(storage.clone()); + // The seen set is sealed, so a write needs the record key. + protocol.state_record_cipher = Some(state_crypto::StateRecordCipher::new( + &[7u8; state_crypto::STATE_RECORD_KEY_BYTES], + )); + protocol.start().unwrap(); + let writes = || *storage.dedup_writes.lock().unwrap(); + + // 31 marks: under the threshold and inside the interval, so the tick + // writes nothing. + for _ in 0..(DEDUP_PERSIST_DIRTY_THRESHOLD - 1) { + protocol.mark_seen_persisted(MessageId::new()); + } + protocol.process().unwrap(); + assert_eq!( + writes(), + 0, + "a batch under the threshold waits for the interval" + ); + assert_eq!(protocol.dedup_dirty, DEDUP_PERSIST_DIRTY_THRESHOLD - 1); + + // The interval elapses: the next tick writes the batch, once. + protocol.dedup_last_persist = Instant::now() - DEDUP_PERSIST_INTERVAL - Duration::from_secs(1); + protocol.process().unwrap(); + assert_eq!( + writes(), + 1, + "the tick after the interval writes exactly once" + ); + assert_eq!(protocol.dedup_dirty, 0); + protocol.process().unwrap(); + assert_eq!(writes(), 1, "a clean set is not rewritten"); + + // The threshold: 32 changes write on the very next tick with no wait. + for _ in 0..DEDUP_PERSIST_DIRTY_THRESHOLD { + protocol.mark_seen_persisted(MessageId::new()); + } + protocol.process().unwrap(); + assert_eq!(writes(), 2, "reaching the threshold writes without waiting"); + + // A re-mark of a tracked id is not a change. + let tracked = MessageId::new(); + protocol.mark_seen_persisted(tracked.clone()); + let dirty = protocol.dedup_dirty; + assert!(!protocol.mark_seen_persisted(tracked)); + assert_eq!(protocol.dedup_dirty, dirty); + + // stop() flushes whatever is dirty, unconditionally. + protocol.stop().unwrap(); + assert_eq!(writes(), 3, "stop flushes the remaining batch"); +} + +/// A record this build cannot read is dropped and the set starts empty; the +/// restore never fails `initialize_mls` over it. +#[test] +fn test_dedup_seen_set_corrupt_record_is_dropped() { + let storage = Arc::new(InMemoryStorage::new()); + let bob = pending_decrypt_bob(storage.clone()); + let state_storage = bob.protocol_state_storage.clone().unwrap(); + bob.write_state_record( + state_storage.as_ref(), + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + b"{\"version\":1,\"entries\":[", + ) + .unwrap(); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert_eq!(bob.deduplicator.tracked_count(), 0); + let state_storage = bob.protocol_state_storage.clone().unwrap(); + assert!( + bob.read_state_record( + state_storage.as_ref(), + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + ) + .unwrap() + .is_none(), + "the unreadable record is deleted rather than re-examined every launch" + ); +} + +/// The seen set is sealed at rest. Its ids and receipt times are a day-long +/// timeline of when this install received messages, so the stored bytes must +/// carry neither in the clear, and the restore must still open them. +#[test] +fn test_dedup_seen_set_is_sealed_at_rest() { + let storage = Arc::new(InMemoryStorage::new()); + let mut bob = pending_decrypt_bob(storage.clone()); + bob.start().unwrap(); + let seen = MessageId::new(); + assert!(bob.mark_seen_persisted(seen.clone())); + bob.stop().unwrap(); + + let state_storage = bob.protocol_state_storage.clone().unwrap(); + let raw = state_storage + .load( + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + ) + .unwrap() + .expect("stop() flushes the seen set"); + let id_text = seen.as_str(); + assert!( + !raw.windows(id_text.len()) + .any(|window| window == id_text.as_bytes()), + "the stored seen set must not carry an id in the clear" + ); + assert!( + serde_json::from_slice::(&raw).is_err(), + "the stored bytes must be a sealed record, not the plaintext JSON" + ); + drop(bob); + + let bob = pending_decrypt_bob(storage); + assert!( + bob.deduplicator.is_duplicate(&seen), + "the restore opens the sealed record" + ); +} + +/// A restored frame the in-memory caps drop is settled like a record that aged +/// out on disk: its record is deleted and its `PendingQueueDropped` waits for +/// `start()`, instead of being emitted during `initialize_mls` into an app that +/// has not subscribed. Both overflow policies, because they drop through +/// different paths: `DropOldest` evicts the queued frame, `DropNewest` refuses +/// the incoming one, and a refused restored frame has a record too. +#[test] +fn test_pending_decrypt_restore_overflow_is_settled_after_start() { + use crate::config::OverflowPolicy; + + for drop_oldest in [true, false] { + let storage = Arc::new(InMemoryStorage::new()); + let seeder = pending_decrypt_bob(storage.clone()); + let now_ms = Utc::now().timestamp_millis(); + let older = pending_test_message(&id("alice"), "older"); + let newer = pending_test_message(&id("alice"), "newer"); + seeder.persist_pending_decrypt_entry(&id("alice"), &older, now_ms - 2_000, None); + seeder.persist_pending_decrypt_entry(&id("alice"), &newer, now_ms - 1_000, None); + drop(seeder); + + let mut config = create_test_config_for_user("bob"); + config.encryption.enabled = true; + config.encryption.store_pending = true; + config.encryption.pending_queue.max_pending_per_peer = 1; + config.encryption.pending_queue.overflow_policy = if drop_oldest { + OverflowPolicy::DropOldest + } else { + OverflowPolicy::DropNewest + }; + let mut bob = OfflineProtocol::new(config).unwrap(); + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let events_handle = Arc::clone(&events); + bob.on_event(move |event| { + events_handle.lock().unwrap().push(event); + }); + bob.initialize_mls_for_test(storage).unwrap(); + + let (dropped, kept) = if drop_oldest { + (&older, &newer) + } else { + (&newer, &older) + }; + let dropped_ids = |events: &[Event]| -> Vec { + events + .iter() + .filter_map(|event| match event { + Event::MessageDecryptionFailed { + message_id, + code: DecryptionFailureCode::PendingQueueDropped, + .. + } => Some(message_id.clone()), + _ => None, + }) + .collect() + }; + assert_eq!(bob.pending_queue.peer_queue_len(&id("alice")), 1); + assert_eq!( + bob.persisted_pending_decrypt_ids(), + vec![kept.id.as_str()], + "drop_oldest={drop_oldest}: the dropped frame's record is deleted" + ); + assert!( + dropped_ids(&events.lock().unwrap()).is_empty(), + "drop_oldest={drop_oldest}: nothing may reach the app during initialize_mls" + ); + assert_eq!( + dropped_ids(&bob.deferred_restore_settlements), + vec![dropped.id.as_str()], + "drop_oldest={drop_oldest}: the drop waits for start()" + ); + + bob.start().unwrap(); + assert_eq!( + dropped_ids(&events.lock().unwrap()), + vec![dropped.id.as_str()], + "drop_oldest={drop_oldest}: start() delivers it" + ); + } +} + +/// Every drop a restore makes is charged to the inbound pool and reported only +/// together with its delete. +/// +/// A frame the in-memory caps turn away on restore came off disk, so its record +/// has to go. Deleted unbudgeted, one launch could issue a delete per over-cap +/// record on top of the pool, and `MAX_RESTORE_PRUNE_DELETES` bounds the whole +/// launch because each delete costs a directory flush. Reported without its +/// delete, the app would be told a frame was dropped that the next launch +/// restores and may admit, and an aged-out record would be reported once per +/// launch until its delete landed. So a record the pool cannot fund stays on +/// disk unreported, and a later launch reports it exactly once. +#[test] +fn test_pending_decrypt_restore_drops_are_budgeted_and_reported_with_their_delete() { + use super::storage::{MAX_RESTORE_PRUNE_DELETES, PENDING_DECRYPT_PERSISTED_MAX_AGE_MS}; + use crate::config::OverflowPolicy; + use std::collections::HashSet; + + const SEED: usize = MAX_RESTORE_PRUNE_DELETES + 88; + + // One launch of bob over the same two stores, counting the deletes it + // issues. A per-peer cap of one drops every restored frame but one. + let launch = + |secure: &Arc, state: &Arc, policy: OverflowPolicy| { + let counting = Arc::new(DeleteCountingStorage::new(state.clone())); + let state_handle: Arc = counting.clone(); + let secure_handle: Arc = secure.clone(); + let mut config = create_test_config_for_user("bob"); + config.encryption.enabled = true; + config.encryption.store_pending = true; + config.encryption.pending_queue.max_pending_per_peer = 1; + config.encryption.pending_queue.overflow_policy = policy; + let mut bob = OfflineProtocol::new(config).unwrap(); + bob.initialize_mls(secure_handle, state_handle).unwrap(); + (bob, counting) + }; + // The ids a launch reported as dropped, parked for start(). + let reported = |bob: &OfflineProtocol| -> HashSet { + bob.deferred_restore_settlements + .iter() + .filter_map(|event| match event { + Event::MessageDecryptionFailed { + message_id, + code: DecryptionFailureCode::PendingQueueDropped, + .. + } => Some(message_id.clone()), + _ => None, + }) + .collect() + }; + let on_disk = |bob: &OfflineProtocol| -> HashSet { + bob.persisted_pending_decrypt_ids().into_iter().collect() + }; + + // Frames the caps drop, under both overflow policies: the evicted older + // entries and the refused incoming ones take the same path. + for policy in [OverflowPolicy::DropOldest, OverflowPolicy::DropNewest] { + let secure = Arc::new(InMemoryStorage::new()); + let state = Arc::new(InMemoryStorage::new()); + let (seeder, _) = launch(&secure, &state, policy); + let now_ms = Utc::now().timestamp_millis(); + for i in 0..SEED { + let message = pending_test_message(&id("alice"), &format!("frame {i}")); + seeder.persist_pending_decrypt_entry( + &id("alice"), + &message, + now_ms - (SEED - i) as i64, + None, + ); + } + assert_eq!(on_disk(&seeder).len(), SEED, "{policy:?}: precondition"); + drop(seeder); - assert!(matches!(result, Some(InternalMessageResult::Deferred))); - assert!(protocol.pending_queue.contains_peer(&id("sender123"))); - assert_eq!(protocol.pending_queue.peer_queue_len(&id("sender123")), 1); -} + let (bob, counting) = launch(&secure, &state, policy); + let first = reported(&bob); + let left = on_disk(&bob); + assert_eq!(bob.pending_queue.peer_queue_len(&id("alice")), 1); + assert_eq!( + counting.deletes_for(storage_keys::PENDING_DECRYPT_ENTRIES), + MAX_RESTORE_PRUNE_DELETES, + "{policy:?}: every over-cap delete is charged to the inbound pool" + ); + assert_eq!( + first.len(), + MAX_RESTORE_PRUNE_DELETES, + "{policy:?}: a drop is reported only with its delete" + ); + assert!( + first.is_disjoint(&left), + "{policy:?}: nothing reported is still on disk" + ); + assert_eq!( + first.len() + left.len(), + SEED, + "{policy:?}: every record is either reported and deleted, or still on disk" + ); + drop(bob); -#[test] -fn test_pending_queue_stress_memory_plateaus_with_unfinished_handshake() { - let mut config = create_test_config(); - config.encryption.enabled = true; - config.encryption.pending_queue.max_pending_per_peer = 32; - config.encryption.pending_queue.max_pending_global = 64; - config.encryption.pending_queue.pending_ttl_ms = 60_000; - config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropOldest; + let (bob, counting) = launch(&secure, &state, policy); + let second = reported(&bob); + assert_eq!( + counting.deletes_for(storage_keys::PENDING_DECRYPT_ENTRIES), + SEED - 1 - MAX_RESTORE_PRUNE_DELETES, + "{policy:?}: the next launch deletes what the first could not fund" + ); + assert_eq!(second.len(), SEED - 1 - MAX_RESTORE_PRUNE_DELETES); + assert!( + first.is_disjoint(&second), + "{policy:?}: no drop is reported on two launches" + ); + assert_eq!( + on_disk(&bob).len(), + 1, + "{policy:?}: only the admitted frame is left" + ); + } - let mut protocol = OfflineProtocol::new(config).unwrap(); - for idx in 0..10_000 { - let msg = pending_test_message(&id("sender123"), &format!("encrypted-{idx}")); - protocol.enqueue_pending_decryption("sender123", &msg); + // Records that aged out on disk follow the same rule. + let secure = Arc::new(InMemoryStorage::new()); + let state = Arc::new(InMemoryStorage::new()); + let (seeder, _) = launch(&secure, &state, OverflowPolicy::DropOldest); + let aged_out_at = Utc::now().timestamp_millis() - PENDING_DECRYPT_PERSISTED_MAX_AGE_MS - 60_000; + for i in 0..SEED { + let message = pending_test_message(&id("alice"), &format!("stale {i}")); + seeder.persist_pending_decrypt_entry(&id("alice"), &message, aged_out_at, None); } + drop(seeder); - assert_eq!(protocol.pending_queue.total(), 32); + let (bob, counting) = launch(&secure, &state, OverflowPolicy::DropOldest); + let first = reported(&bob); + let left = on_disk(&bob); assert_eq!( - protocol.pending_queue.metrics().pending_messages_current, - 32 + counting.deletes_for(storage_keys::PENDING_DECRYPT_ENTRIES), + MAX_RESTORE_PRUNE_DELETES ); assert_eq!( - *protocol - .pending_queue - .metrics() - .pending_messages_per_peer - .get("sender123") - .unwrap(), - 32 + first.len(), + MAX_RESTORE_PRUNE_DELETES, + "an aged-out record is reported only with its delete" ); + assert!(first.is_disjoint(&left)); + assert_eq!(left.len(), SEED - MAX_RESTORE_PRUNE_DELETES); + drop(bob); + + let (bob, _) = launch(&secure, &state, OverflowPolicy::DropOldest); + let second = reported(&bob); + assert_eq!(second.len(), SEED - MAX_RESTORE_PRUNE_DELETES); + assert!( + first.is_disjoint(&second), + "an aged-out record is reported on one launch only" + ); + assert!(on_disk(&bob).is_empty()); } +/// A frame the queue refuses on the live path was never written, so refusing +/// it must not issue a storage delete. Under a flood of frames the caps turn +/// away, one delete per refusal is a storage round trip per inbound frame, +/// under the protocol lock, for a record that does not exist. Every refusal is +/// still reported to the app. #[test] -fn test_pending_queue_sustained_mixed_invalid_and_early_encrypted_is_bounded() { - let mut config = create_test_config(); +fn test_pending_decrypt_refused_live_frame_issues_no_delete() { + let secure = Arc::new(InMemoryStorage::new()); + let counting = Arc::new(DeleteCountingStorage::new(Arc::new(InMemoryStorage::new()))); + let state_handle: Arc = counting.clone(); + let secure_handle: Arc = secure.clone(); + let mut config = create_test_config_for_user("bob"); config.encryption.enabled = true; - config.encryption.pending_queue.max_pending_per_peer = 16; - config.encryption.pending_queue.max_pending_global = 32; - config.encryption.pending_queue.pending_ttl_ms = 60_000; - config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropOldest; - - let mut protocol = OfflineProtocol::new(config).unwrap(); - protocol - .initialize_mls_for_test(Arc::new(crate::mls::InMemoryStorage::new())) - .unwrap(); - - let early_slot = - offline_protocol_mls::GroupId::for_session(&id("user123"), &id("sender123")).unwrap(); - let sender_address = id("sender123"); - let valid_early_encrypted = format!( - "{}{{\"group_id\":\"{early_slot}\",\"message_type\":\"Application\",\"epoch\":0,\"ciphertext\":[1,2,3],\"sender_id\":\"{sender_address}\",\"timestamp_ms\":12345}}", - internal_prefixes::ENCRYPTED - ); - let malformed_variants = [ - format!("{}{{", internal_prefixes::ENCRYPTED), - format!("{}{{\"group_id\":\"bad\"", internal_prefixes::ENCRYPTED), - format!("{}[]", internal_prefixes::ENCRYPTED), - format!( - "{}{{\"ciphertext\":\"not-array\"}}", - internal_prefixes::ENCRYPTED - ), - ]; - - let mut early_count: u64 = 0; - let mut invalid_count: u64 = 0; - for idx in 0..10_000 { - let is_invalid = idx % 5 == 0; - let content = if is_invalid { - invalid_count += 1; - malformed_variants[(idx % malformed_variants.len()) as usize].as_str() - } else { - early_count += 1; - valid_early_encrypted.as_str() - }; + config.encryption.pending_queue.max_pending_per_peer = 1; + config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropNewest; + let mut bob = OfflineProtocol::new(config).unwrap(); + bob.initialize_mls(secure_handle, state_handle).unwrap(); + let events: Arc>> = Arc::new(Mutex::new(Vec::new())); + let events_handle = Arc::clone(&events); + bob.on_event(move |event| { + events_handle.lock().unwrap().push(event); + }); + let deletes_before = counting.deletes_for(storage_keys::PENDING_DECRYPT_ENTRIES); - let message = signed_frame(&id("sender123"), &id("user123"), content); - let result = protocol.process_internal_message(&message); - // Both shapes defer, but for different reasons — and only one of them - // queues. A valid-but-early encrypted message is queued and not ACKed - // (it becomes decryptable once the session lands). A malformed payload - // is *not* queued — it can never become parseable — but is still not - // ACKed, so the sender's resend stays the recovery path. The metrics - // assertions below are what pin that split: only the early messages - // count toward `pending_messages_received_total`, so a malformed flood - // cannot grow the queue. - assert!(matches!(result, Some(InternalMessageResult::Deferred))); + let admitted = pending_test_message(&id("alice"), "admitted"); + bob.enqueue_pending_decryption(&id("alice"), &admitted); + for n in 0..16 { + let refused = pending_test_message(&id("alice"), &format!("refused {n}")); + bob.enqueue_pending_decryption(&id("alice"), &refused); } - let per_peer_limit = protocol - .config - .encryption - .pending_queue - .max_pending_per_peer; - let global_limit = protocol.config.encryption.pending_queue.max_pending_global; - assert!(protocol.pending_queue.total() <= global_limit); - assert!(protocol.pending_queue.peer_queue_len("sender123") <= per_peer_limit); - - let metrics = protocol.pending_queue_metrics(); - assert_eq!(metrics.pending_messages_received_total, early_count); + assert_eq!(bob.pending_queue.peer_queue_len(&id("alice")), 1); assert_eq!( - metrics.pending_messages_current, - protocol.pending_queue.total() + bob.persisted_pending_decrypt_ids(), + vec![admitted.id.as_str()] ); - assert!(metrics.pending_messages_dropped_overflow_total > 0); - assert_eq!(early_count + invalid_count, 10_000); + assert_eq!( + counting.deletes_for(storage_keys::PENDING_DECRYPT_ENTRIES) - deletes_before, + 0, + "a refused live frame has no record, so refusing it must not issue a delete" + ); + let reported = events + .lock() + .unwrap() + .iter() + .filter(|event| { + matches!( + event, + Event::MessageDecryptionFailed { + code: DecryptionFailureCode::PendingQueueDropped, + .. + } + ) + }) + .count(); + assert_eq!(reported, 16, "every refusal is still reported"); } +/// A failed `initialize_mls` puts the inbound pending-decryption queue back to +/// what it held before the call. Emptying it lost a frame that was never +/// persisted. Keeping what the restore added left a frame sourced from the +/// store the rollback detached, whose drain deletes nothing on disk. #[test] -fn test_pending_queue_flood_respects_per_peer_fairness() { - let mut config = create_test_config(); - config.encryption.enabled = true; - config.encryption.pending_queue.max_pending_per_peer = 3; - config.encryption.pending_queue.max_pending_global = 6; - config.encryption.pending_queue.pending_ttl_ms = 60_000; - config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropOldest; - - let mut protocol = OfflineProtocol::new(config).unwrap(); - - for idx in 0..100 { - let msg = pending_test_message(&id("noisy-peer"), &format!("noisy-{idx}")); - protocol.enqueue_pending_decryption("noisy-peer", &msg); +fn test_initialize_mls_restore_failure_rolls_back_pending_decryption_queue() { + struct FailingOutboxListStorage { + inner: Arc, } - for idx in 0..3 { - let msg = pending_test_message(&id("peer-a"), &format!("a-{idx}")); - protocol.enqueue_pending_decryption("peer-a", &msg); - let msg = pending_test_message(&id("peer-b"), &format!("b-{idx}")); - protocol.enqueue_pending_decryption("peer-b", &msg); + impl MlsStorage for FailingOutboxListStorage { + fn store( + &self, + key_type: &str, + key_id: &str, + data: &[u8], + ) -> offline_protocol_mls::storage::StorageResult<()> { + self.inner.store(key_type, key_id, data) + } + fn load( + &self, + key_type: &str, + key_id: &str, + ) -> offline_protocol_mls::storage::StorageResult>> { + self.inner.load(key_type, key_id) + } + fn delete( + &self, + key_type: &str, + key_id: &str, + ) -> offline_protocol_mls::storage::StorageResult<()> { + self.inner.delete(key_type, key_id) + } + fn list_keys( + &self, + key_type: &str, + ) -> offline_protocol_mls::storage::StorageResult> { + if key_type == storage_keys::OUTBOX { + return Err(offline_protocol_mls::StorageError::LoadFailed( + "forced outbox restore failure".to_string(), + )); + } + self.inner.list_keys(key_type) + } } - assert!(protocol.pending_queue.total() <= 6); - assert!(protocol.pending_queue.peer_queue_len("noisy-peer") <= 3); - assert!(protocol.pending_queue.contains_peer("peer-a")); - assert!(protocol.pending_queue.contains_peer("peer-b")); -} + // A parked frame on disk, which the restore re-admits before the outbox + // step fails. + let backing = Arc::new(InMemoryStorage::new()); + let on_disk = pending_test_message(&id("alice"), "on-disk"); + { + let mut seeder = pending_decrypt_bob(backing.clone()); + seeder.enqueue_pending_decryption(&id("alice"), &on_disk); + assert_eq!( + seeder.persisted_pending_decrypt_ids(), + vec![on_disk.id.as_str()] + ); + } -#[test] -fn test_pending_queue_drop_newest_policy_enforced_for_per_peer_limit() { - let mut config = create_test_config(); + let mut config = create_test_config_for_user("bob"); config.encryption.enabled = true; - config.encryption.pending_queue.max_pending_per_peer = 1; - config.encryption.pending_queue.max_pending_global = 10; - config.encryption.pending_queue.pending_ttl_ms = 60_000; - config.encryption.pending_queue.overflow_policy = crate::config::OverflowPolicy::DropNewest; + config.encryption.store_pending = true; + let mut bob = OfflineProtocol::new(config).unwrap(); + // Parked before any store is attached, so it exists only in memory. + let in_memory = pending_test_message(&id("carol"), "in-memory"); + bob.enqueue_pending_decryption(&id("carol"), &in_memory); - let mut protocol = OfflineProtocol::new(config).unwrap(); - let first = pending_test_message(&id("peer-a"), "first"); - let second = pending_test_message(&id("peer-a"), "second"); - protocol.enqueue_pending_decryption("peer-a", &first); - protocol.enqueue_pending_decryption("peer-a", &second); + let result = bob.initialize_mls_for_test(Arc::new(FailingOutboxListStorage { inner: backing })); + assert!( + result.is_err(), + "precondition: the outbox restore step fails" + ); - assert_eq!(protocol.pending_queue.peer_queue_len("peer-a"), 1); assert_eq!( - protocol - .pending_queue - .peek_entry("peer-a", 0) - .unwrap() - .message - .content, - "first" + bob.pending_queue.peer_queue_len(&id("carol")), + 1, + "a frame that only ever lived in memory must survive the rollback" ); - assert_eq!( - protocol - .pending_queue - .metrics() - .pending_messages_dropped_overflow_total, - 1 + assert!( + !bob.pending_queue.contains_peer(&id("alice")), + "a frame the failed restore re-admitted must not outlive the store it came from" ); } @@ -18982,13 +20605,13 @@ fn test_one_launch_cannot_exceed_the_derived_restore_delete_ceiling() { // The invariant the whole `PruneAllowance` split exists for, and the one a // per-pool test cannot see: `MAX_RESTORE_PRUNE_DELETES` bounds a *launch*, // because a device-barrier storm kills the synchronous `initialize_mls` - // call rather than any single walk in it. Three pools, so the ceiling is - // `3 × MAX_RESTORE_PRUNE_DELETES` — and the point is that adding a seventh + // call rather than any single walk in it. Four pools, so the ceiling is + // `4 × MAX_RESTORE_PRUNE_DELETES` — and the point is that adding another // walk, or letting one allocate its own pool again, moves this number // without moving any per-walk assertion. // // Every restore category that can delete is seeded past its own pool here, - // so all three pools bind at once: + // so all four pools bind at once: // // - advisory (shared): session states, peer key packages, peer // capabilities, Welcome lifecycles, media descriptors — 200 potential @@ -18999,7 +20622,10 @@ fn test_one_launch_cannot_exceed_the_derived_restore_delete_ceiling() { // rather than left out — a walk that starts deleting would move the // total, and this assertion is what would catch it. // - pending messages (private): 600 potential, 512 funded; - // - outbox (private): 600 potential, 512 funded. + // - outbox (private): 600 potential, 512 funded; + // - inbound (private): 600 pending-decryption records plus one corrupt + // seen set, 512 funded. The pending-decryption walk draws first and + // takes the whole pool, so the seen-set delete is the one refused. use super::storage::MAX_RESTORE_PRUNE_DELETES; const ADVISORY_SEED: usize = 200; @@ -19026,7 +20652,23 @@ fn test_one_launch_cannot_exceed_the_derived_restore_delete_ceiling() { b"not a sealed record", ) .unwrap(); + backing + .store( + storage_keys::PENDING_DECRYPT_ENTRIES, + &format!("in{i:05}"), + b"not a sealed record", + ) + .unwrap(); } + // A sealed category, so these bytes will not open: the reader drops the + // record, and that delete is charged to the inbound pool like the rest. + backing + .store( + storage_keys::DEDUP_SEEN_IDS, + storage_keys::DEDUP_SEEN_IDS_ID, + b"{not json", + ) + .unwrap(); for i in 0..ADVISORY_SEED { backing .store( @@ -19082,6 +20724,8 @@ fn test_one_launch_cannot_exceed_the_derived_restore_delete_ceiling() { + counting.deletes_for(storage_keys::MEDIA_DESCRIPTORS); let pending = counting.deletes_for(storage_keys::PENDING_MESSAGES); let outbox = counting.deletes_for(storage_keys::OUTBOX); + let inbound = counting.deletes_for(storage_keys::PENDING_DECRYPT_ENTRIES) + + counting.deletes_for(storage_keys::DEDUP_SEEN_IDS); assert_eq!( advisory, @@ -19099,8 +20743,13 @@ fn test_one_launch_cannot_exceed_the_derived_restore_delete_ceiling() { "so does the outbox walk — its walk bound alone is not the ceiling" ); assert_eq!( - advisory + pending + outbox, - 3 * MAX_RESTORE_PRUNE_DELETES, + inbound, MAX_RESTORE_PRUNE_DELETES, + "the two inbound walks share one private pool, and the first may spend \ + all of it rather than reserving for advisory walks that never draw on it" + ); + assert_eq!( + advisory + pending + outbox + inbound, + 4 * MAX_RESTORE_PRUNE_DELETES, "the launch ceiling is the sum of the pools, and nothing else on this \ path may issue an unbudgeted delete" ); @@ -19536,6 +21185,7 @@ fn test_outbox_capacity_prune_stays_inside_the_launch_budget() { last_sent_at: base + ChronoDuration::seconds(i as i64), last_transport: None, reseal: None, + relay_pushed: false, }, ); } @@ -19615,6 +21265,7 @@ fn test_outbox_absolute_expiry_prune_stays_inside_the_launch_budget() { last_sent_at: stale, last_transport: None, reseal: None, + relay_pushed: false, }, ); } @@ -23402,6 +25053,7 @@ fn test_restore_outbox_skips_corrupted_entries() { last_sent_at: chrono::Utc::now(), last_transport: None, reseal: None, + relay_pushed: false, }, ); storage @@ -23449,6 +25101,7 @@ fn test_restore_outbox_prunes_overflow() { last_sent_at: base + ChronoDuration::seconds(i as i64), last_transport: None, reseal: None, + relay_pushed: false, }; if i == 0 { oldest_id = Some(entry.message.id.as_str()); @@ -23493,6 +25146,7 @@ fn test_restore_outbox_refreshes_expired_ttl_carrier_relative() { last_sent_at: old, last_transport: None, reseal: None, + relay_pushed: false, }, ); @@ -23643,6 +25297,7 @@ fn test_restore_outbox_prune_keeps_fresh_over_refreshed_stale() { last_sent_at: now - ChronoDuration::seconds((i + 1) as i64), last_transport: None, reseal: None, + relay_pushed: false, }, ); } @@ -23656,6 +25311,7 @@ fn test_restore_outbox_prune_keeps_fresh_over_refreshed_stale() { last_sent_at: now - ChronoDuration::hours(2), last_transport: None, reseal: None, + relay_pushed: false, }; lapsed_ids.push(entry.message.id.as_str().to_string()); store_outbox_entry(&storage, &entry); @@ -24219,6 +25875,7 @@ fn test_cleanup_outbox_media_expiry_does_not_emit_message_failed() { last_sent_at: chrono::Utc::now() - ChronoDuration::seconds(1), last_transport: None, reseal: None, + relay_pushed: false, }, ); @@ -24255,6 +25912,7 @@ fn test_restore_outbox_drops_absolutely_expired() { last_sent_at: now - ChronoDuration::seconds(5), last_transport: None, reseal: None, + relay_pushed: false, }, ); @@ -24269,6 +25927,7 @@ fn test_restore_outbox_drops_absolutely_expired() { last_sent_at: now - ChronoDuration::seconds(2), last_transport: None, reseal: None, + relay_pushed: false, }, ); @@ -24373,6 +26032,7 @@ fn test_flush_outbox_for_peer_includes_media_outbox() { last_sent_at: chrono::Utc::now(), last_transport: None, reseal: None, + relay_pushed: false, }, ); @@ -24438,6 +26098,7 @@ fn test_flush_outbox_all_includes_media_outbox() { last_sent_at: chrono::Utc::now(), last_transport: None, reseal: None, + relay_pushed: false, }, ); @@ -26258,7 +27919,7 @@ fn nostr_watermark_is_a_noop_without_a_nostr_transport() { } #[test] -fn nostr_replay_overlap_exceeds_dedup_retention() { +fn nostr_replay_overlap_fits_inside_dedup_retention() { // Drift guard between two constants that live in different crates and are // only related here, in the crate that owns both the Nostr transport and // the deduplicator. @@ -26267,17 +27928,17 @@ fn nostr_replay_overlap_exceeds_dedup_retention() { // is mandatory — the sender writes `created_at`, so an event published now // can be stamped a jitter window in the past, and a `since` sitting at the // mark would filter out the very events the query exists to fetch. Dedup is - // what would otherwise make the resulting duplicates free, and it does not - // reach far enough: ids are retained for `retention_time_secs`, which is - // *shorter* than the overlap, so a reconnect after longer than the - // retention window re-processes rather than absorbs it. + // what makes the resulting duplicates free, and since the seen set became + // persistent (24 h retention, restored across restarts) it reaches far + // enough: ids outlive the overlap, so a reconnect inside the retention + // window — an app reopened the next day included — absorbs it. // - // That residual is documented in `docs/nostr.md`, the CHANGELOG, and - // `create_subscription_message`. This test exists so those notes cannot go - // stale: if someone raises dedup retention past the overlap (or shrinks the - // overlap below retention), the residual is gone and this fails — at which - // point the fix is to update those three notes to claim full absorption, - // not to weaken this assertion. + // That absorption is documented in `docs/nostr.md`, the 0.26.0 CHANGELOG + // entry, and `create_subscription_message`. This test exists so those notes + // cannot go stale: if someone shrinks dedup retention below the overlap (or + // widens the overlap past retention), the residual is back and this fails — + // at which point the fix is to update those three notes to state the + // residual again, not to weaken this assertion. let overlap_secs = offline_protocol_transport::constants::NOSTR_CREATED_AT_JITTER_SECS + offline_protocol_transport::constants::NOSTR_CLOCK_SKEW_MARGIN_SECS; @@ -26290,12 +27951,12 @@ fn nostr_replay_overlap_exceeds_dedup_retention() { .retention_time_secs as i64; assert!( - overlap_secs > retention_secs, - "the documented replay residual is gone: the {overlap_secs}s Nostr replay \ - overlap now fits inside the {retention_secs}s dedup retention window, so \ - duplicates ARE fully absorbed. Update docs/nostr.md, the CHANGELOG entry, \ - and the create_subscription_message doc comment, which all state the \ - opposite." + retention_secs >= overlap_secs, + "the replay residual is back: the {overlap_secs}s Nostr replay overlap no \ + longer fits inside the {retention_secs}s dedup retention window, so a \ + reconnect after longer than retention re-processes its overlap. Update \ + docs/nostr.md, the CHANGELOG entry, and the create_subscription_message \ + doc comment, which all claim absorption." ); } @@ -35687,6 +37348,7 @@ fn test_relay_unreachable_reason_is_classified_and_still_parks_the_dm() { last_sent_at: chrono::Utc::now(), last_transport: Some(TransportType::BLE), reseal: None, + relay_pushed: false, }, ); diff --git a/crates/offline-protocol/src/protocol/types.rs b/crates/offline-protocol/src/protocol/types.rs index 26f7499a..2abbeb32 100644 --- a/crates/offline-protocol/src/protocol/types.rs +++ b/crates/offline-protocol/src/protocol/types.rs @@ -201,6 +201,26 @@ pub(crate) const WELCOME_PRESENCE_RESCUE_MAX_SECS: i64 = 600; /// calling `internet_send_failed_with_reason` — keep them in sync. pub(crate) const SEND_FAIL_REASON_RECIPIENT_UNREACHABLE: &str = "recipient_unreachable"; +/// The relay accepted a frame for a recipient with no live socket and handed +/// it to a push notification (`MessageSent { pushed: true }`). Not a failure: +/// the push may deliver it. What it does establish is that the relay, which +/// has no store-and-forward, holds no copy for when the recipient reconnects. +/// +/// Deliberately its own token rather than a `recipient_unreachable` tail. +/// That prefix fast-fails connection requests and moves Welcomes to `Failed`, +/// both wrong for a frame that may have arrived. This token parks plain DMs +/// only (`park_relay_pushed_dm`), and connection requests and Welcomes stay +/// on their ordinary path. +/// +/// Cross-layer contract: the React Native platform bridges +/// (`InternetManager.kt` / `InternetManager.swift`) and the Python relay +/// client (`internet_manager.py`) pass this exact literal to +/// `internet_send_failed_with_reason`; pinned on the React Native side by +/// `react_native_relay_parks_a_pushed_message_sent` and on the Python side by +/// `TestMessageSentPushed`. That call must not also score the report as a send +/// failure; see `OfflineProtocol::send_report_is_carrier_failure`. +pub(crate) const SEND_FAIL_REASON_RELAY_PUSHED: &str = "relay_pushed"; + /// Fallback token for a send failure that classifies as nothing more specific. pub(crate) const SEND_FAIL_REASON_TRANSPORT: &str = "transport_send_failed"; /// A Welcome was written to a carrier that never confirmed it. @@ -216,6 +236,7 @@ pub(crate) const SEND_FAIL_REASON_CONFIRM_TIMEOUT: &str = "send_confirmation_tim /// `every_send_failure_token_classifies_to_itself` fails otherwise. pub(crate) const SEND_FAIL_REASON_TOKENS: &[&str] = &[ SEND_FAIL_REASON_RECIPIENT_UNREACHABLE, + SEND_FAIL_REASON_RELAY_PUSHED, SEND_FAIL_REASON_TRANSPORT, SEND_FAIL_REASON_CONFIRM_TIMEOUT, "transport_not_connected", @@ -358,6 +379,48 @@ pub(crate) const RECONCILIATION_THROTTLE_MS: u64 = 2_000; /// is only used for causal ordering and the gap is absorbed on the next /// merge with any peer. pub(crate) const LAMPORT_PERSIST_INTERVAL: u64 = 64; + +/// Most seen ids one `DedupSeenRecord` carries. Matches the default +/// `max_tracked_messages`; a larger configured tracker persists its newest +/// ids only, which are the ones a replay is most likely to repeat. +pub(crate) const MAX_PERSISTED_DEDUP_IDS: usize = 2000; + +/// Seen-set changes that force a write before the time cadence elapses. A +/// burst of inbound traffic is exactly when the set is worth having on disk, +/// and 32 changes at ~60 bytes each is well under the cost of one write. +pub(crate) const DEDUP_PERSIST_DIRTY_THRESHOLD: u32 = 32; + +/// How long a dirty seen set may wait for the next `process()` tick before it +/// is written. Bounds what a crash loses to a few seconds of receipts. +pub(crate) const DEDUP_PERSIST_INTERVAL: std::time::Duration = std::time::Duration::from_secs(5); + +/// The deduplicator's seen set as persisted under +/// [`storage_keys::DEDUP_SEEN_IDS`]. +/// +/// The deduplicator was in-memory only, so after a restart the socket copy of +/// a message the app had already consumed from a push injection was not +/// recognised as a duplicate: it went to the ratchet, whose generation for +/// that message was already spent, and surfaced as a decryption failure — a +/// spurious signal the app counts toward its split-brain breaker, which then +/// tears down a healthy session. Persisting the ids makes the second copy +/// recognisable across the restart. Ids and timestamps only; nothing here is +/// content. +/// +/// `version` is the forward-compatibility hinge: an unknown version is dropped +/// rather than guessed at, and the set simply starts empty. +#[derive(Serialize, Deserialize)] +pub(crate) struct DedupSeenRecord { + #[serde(default = "dedup_seen_record_version")] + pub(crate) version: u8, + pub(crate) entries: Vec, +} + +/// The only record version this build writes or reads. +pub(crate) const DEDUP_SEEN_RECORD_VERSION: u8 = 1; + +fn dedup_seen_record_version() -> u8 { + DEDUP_SEEN_RECORD_VERSION +} /// Seconds the Nostr receive watermark must advance before it is written back /// to protocol-state storage. Same debounce role as /// [`LAMPORT_PERSIST_INTERVAL`]: every inbound relay event moves the mark, and @@ -1558,6 +1621,48 @@ pub(crate) struct PendingMessageRecord { pub(crate) message: PendingMessage, } +/// One inbound encrypted frame parked in the pending-decryption queue, as +/// persisted under [`storage_keys::PENDING_DECRYPT_ENTRIES`]. +/// +/// The queue itself is in-memory only (`PendingDecryptionQueue`); this record +/// is what lets an entry survive an app restart. Without it a frame that +/// arrived before its session was ready was lost the moment the process died +/// — and with a relay that pushes ciphertext without store-and-forward there +/// is no second copy to ask for, so a restart during a slow handshake was a +/// silently lost message. +/// +/// Keyed by message id, like [`PendingMessageRecord`], so a drain, a prune or +/// an overflow drop deletes exactly the record it settled. `peer_id` rides +/// inside the record because the in-memory queue is a per-peer map rebuilt +/// from records the store enumerates in no particular order. +/// +/// `first_received_at_ms` is wall-clock, not the `Instant` the in-memory entry +/// carries: an `Instant` does not survive a restart, so on restore the entry is +/// re-stamped with a fresh `Instant::now()` (its in-memory TTL restarts) and +/// this field bounds the *total* time on disk instead — +/// `PENDING_DECRYPT_PERSISTED_MAX_AGE_MS`. +/// +/// `version` is a forward-compatibility hinge: a reader that sees a version it +/// does not know treats the record as corrupt and drops it rather than +/// guessing at fields. Records written before the field existed default to 1. +#[derive(Serialize, Deserialize)] +pub(crate) struct PendingDecryptRecord { + #[serde(default = "pending_decrypt_record_version")] + pub(crate) version: u8, + pub(crate) peer_id: String, + pub(crate) message: Message, + pub(crate) first_received_at_ms: i64, + #[serde(default)] + pub(crate) received_via: Option, +} + +/// The only record version this build writes or reads. +pub(crate) const PENDING_DECRYPT_RECORD_VERSION: u8 = 1; + +fn pending_decrypt_record_version() -> u8 { + PENDING_DECRYPT_RECORD_VERSION +} + impl PendingMessage { /// Recomputes [`Self::serialized_bytes`] from the current field values. /// @@ -1702,6 +1807,10 @@ pub(crate) mod storage_keys { /// per-recipient layout could only report the loss per peer, because every /// id was inside the record that would not open. pub const PENDING_MESSAGE_ENTRIES: &str = "pending_message_entries"; + /// Inbound ciphertext parked in the pending-decryption queue, one record + /// per message keyed by message id — the receive-side mirror of + /// [`PENDING_MESSAGE_ENTRIES`]. See `PendingDecryptRecord`. + pub const PENDING_DECRYPT_ENTRIES: &str = "pending_decrypt_entries"; /// Key type for persisted per-peer MLS session confirmation state. pub const SESSION_STATES: &str = "session_states"; /// Key type for persisted per-peer received key packages (survives restart). @@ -1728,6 +1837,11 @@ pub(crate) mod storage_keys { pub const LAMPORT_CLOCK: &str = "lamport_clock"; /// Key ID for the single Lamport clock entry. pub const LAMPORT_CLOCK_ID: &str = "current"; + /// The deduplicator's exact-mode seen set, one record for the whole set. + /// See `DedupSeenRecord`. + pub const DEDUP_SEEN_IDS: &str = "dedup_seen_ids"; + /// Key ID for the single seen-set record. + pub const DEDUP_SEEN_IDS_ID: &str = "current"; /// Key type for the durable record that a peer has proved it runs MLS. /// /// Successor to the `tofu_keys` category, which stored a pinned public key @@ -2056,6 +2170,24 @@ pub(crate) struct OutboxEntry { /// verbatim. #[serde(skip)] pub(crate) reseal: Option, + /// The relay has handed this frame to a device push at least once + /// (`MessageSent { pushed: true }`, see [`SEND_FAIL_REASON_RELAY_PUSHED`]). + /// + /// Sticky for the entry's lifetime, and persisted, because of what a later + /// relay verdict for the same id means. The relay remembers which + /// `(sender, recipient, message_id)` triples it has pushed and answers a + /// retry of one with `DeliveryError` (`already_pushed`) instead of a + /// second notification, and the bridges cannot tell that verdict from a + /// plain "recipient offline": both reach the core as + /// `recipient_unreachable`. So the reachability probe a park schedules + /// earns a `DeliveryError` fifteen seconds after every push, and without + /// this flag the app would be told `MessageUndeliverable` about a message + /// the push may already have delivered. With it, that verdict still + /// re-parks the entry (the recipient is not on the relay) but emits + /// nothing app-facing. A legacy record restores as `false`, so the first + /// probe after upgrading emits once; nothing is lost. + #[serde(default)] + pub(crate) relay_pushed: bool, } #[derive(Clone)] @@ -2353,6 +2485,44 @@ mod send_failure_classification_tests { } } + /// The relay-pushed answer is its own token, not a `recipient_unreachable` + /// tail. The bridges pass this literal (pinned on their side by + /// `react_native_relay_parks_a_pushed_message_sent`), and reading it under + /// the unreachable prefix would fast-fail connection requests and fail + /// Welcomes that the push may have delivered. + #[test] + fn relay_pushed_is_its_own_token() { + assert_eq!(SEND_FAIL_REASON_RELAY_PUSHED, "relay_pushed"); + assert!(!SEND_FAIL_REASON_RELAY_PUSHED.starts_with(SEND_FAIL_REASON_RECIPIENT_UNREACHABLE)); + assert_eq!( + classify_transport_send_error("relay_pushed"), + SEND_FAIL_REASON_RELAY_PUSHED + ); + } + + /// A push is parked in the core but is not a failed send, so the one + /// report that must stay out of a carrier's failure accounting is the + /// exact token. Anything else, including relay text that merely contains + /// it, still counts. + #[test] + fn only_a_relay_push_report_is_kept_out_of_carrier_failures() { + use crate::OfflineProtocol; + assert!(!OfflineProtocol::send_report_is_carrier_failure(Some( + "relay_pushed" + ))); + for reason in [ + Some("recipient_unreachable: Recipient is offline"), + Some("Internet transport send failed"), + Some("relay_pushed: extra"), + None, + ] { + assert!( + OfflineProtocol::send_report_is_carrier_failure(reason), + "{reason:?} is a send failure" + ); + } + } + /// Anything unrecognized fails closed to the fallback, never to itself. #[test] fn unknown_text_falls_back_and_is_never_echoed() { diff --git a/docs/android-integration.md b/docs/android-integration.md index 36e667b0..f8700578 100644 --- a/docs/android-integration.md +++ b/docs/android-integration.md @@ -78,7 +78,7 @@ class MainActivity : AppCompatActivity() { storePending = true, maxPendingPerPeer = 100.toULong(), maxPendingGlobal = 1_000.toULong(), - pendingTtlMs = 1_800_000.toULong(), // 30 min (the SDK default) + pendingTtlMs = 86_400_000.toULong(), // 24 h (the SDK default) overflowPolicy = OverflowPolicy.DROP_OLDEST, // These 9 use their defaults: requireEncryption (true), // maxGroupMembers (256u), groupRelayEnabled (true), diff --git a/docs/api-reference.md b/docs/api-reference.md index 8691c38b..b851a052 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -181,7 +181,7 @@ interface EncryptionConfig { pendingQueue?: { maxPendingPerPeer?: number; // Default: 64 maxPendingGlobal?: number; // Default: 4096 - pendingTtlMs?: number; // Default: 1800000 (30 min) + pendingTtlMs?: number; // Default: 86400000 (24 h) overflowPolicy?: 'drop_oldest' | 'drop_newest'; // Default: drop_oldest }; compactEnvelopeEnabled?: boolean; // Default: true diff --git a/docs/bridges/README.md b/docs/bridges/README.md index 17f7e84b..a01bda02 100644 --- a/docs/bridges/README.md +++ b/docs/bridges/README.md @@ -126,8 +126,8 @@ literal `8 * 1024 * 1024` in each. There is no per-language test for it, so a binding edited alone fails the Rust suite rather than its own. See [S6](swift.md#s6-secure-storage). -The one-shot event tag list and the mesh wake task key are pinned the same way, -by Rust guards that read the binding sources. +The one-shot event tag list, the buffered inbound event set and the mesh wake +task key are pinned the same way, by Rust guards that read the binding sources. **The relay address-proof signing domain** is the fifth, and it is the one set pinned by both mechanisms at once. The Swift and Kotlin @@ -271,6 +271,25 @@ Which one applies depends on **where the event fires** relative to subscription, not on what the event means. A replay-on-subscribe mechanism alone does not fix a one-shot that fires before any subscriber exists. +A third shape exists for one narrow class and must not be folded into either of +the two above: + +- **A held inbound buffer** for events that each report one message the core + has already taken responsibility for: `message_received`, `file_received` + and the `message_decryption_failed` that stands in for one. By the time they + are emitted the core has ACKed the message, dedup-marked its id and dropped + its queued copy, so nothing will restate them and a drop is a lost message. + They do not collapse per type, since every message is its own fact, so the + hold is keyed `type:message_id`, kept in arrival order, and capped at a real + capacity (256, oldest dropped) rather than a backstop. Each bridge holds + them across the native-to-JS gap (Kotlin `BUFFERED_INBOUND_EVENT_TYPES`, + Swift `InboundEventBuffer`) and the TypeScript layer across the JS-to-listener + gap, all flushed on subscribe and on foreground and scoped to the session + that produced them. The three definitions are pinned together by + `react_native_buffered_inbound_event_set_matches_native`. Enrolling a + periodic event there would replay stale state; collapsing it into the + one-shot map would keep one message of many. + ## C11. A storage adapter is a supported extension point, and is verified The SDK persists two separate things: MLS and identity secrets, through diff --git a/docs/changelog/0.20.md b/docs/changelog/0.20.md index ca221749..fb1f53aa 100644 --- a/docs/changelog/0.20.md +++ b/docs/changelog/0.20.md @@ -276,7 +276,7 @@ and unreleased changes, is [CHANGELOG.md](../../CHANGELOG.md). **Residuals, stated plainly.** The mark can only be as good as what has been received. A relay that truncates a reconnect's history at `limit` returns its *newest* events, so a device coming back to more than 500 stored events advances past ones it never saw. The hour-plus overlap bounds which are at risk — only those already older than jitter + skew at the moment of truncation. That truncation is also reachable on purpose: the routing tag is public and only *decodability* gates the watermark (parsing a `Message` needs no signature), so a sustained flood of decodable junk can both crowd real events out of a truncated query and advance the mark past them. What keeps that recoverable is that Nostr is not the only path — ACK-gated messages sit in the sender's outbox for 7 days and are retransmitted with a fresh `created_at`, which lands above any watermark — so the cost is delay on a Nostr-only route, not loss. - The replayed overlap is also **not fully deduplicated**, which is a cost rather than a correctness issue but worth stating: message-id dedup retains ids for an hour by default while `since` reaches back an hour and five minutes, so a reconnect after longer than the retention window (an app reopened the next day) re-processes its overlap instead of absorbing it. A replayed ciphertext whose ratchet generation is spent fails closed and is dropped, a past-epoch one triggers at most one rate-limited re-key, and a replayed group copy TTLs out of the pending buffer. The two constants are pinned against each other by a test so this note cannot silently go stale. + The replayed overlap is also **not fully deduplicated**, which is a cost rather than a correctness issue but worth stating: message-id dedup retains ids for an hour by default while `since` reaches back an hour and five minutes, so a reconnect after longer than the retention window (an app reopened the next day) re-processes its overlap instead of absorbing it. A replayed ciphertext whose ratchet generation is spent fails closed and is dropped, a past-epoch one triggers at most one rate-limited re-key, and a replayed group copy TTLs out of the pending buffer. The two constants are pinned against each other by a test so this note cannot silently go stale. *(Superseded in 0.26.0: dedup retention is 24 hours and the seen set is persisted across restarts, so the overlap is now absorbed within the id cap.)* And the watermark bounds *replay*, not metadata exposure: the envelope is still published in cleartext, which the sealed gift wrap addresses separately. diff --git a/docs/configuration.md b/docs/configuration.md index 0f8a1178..e3d87da7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -290,7 +290,7 @@ Controls automatic MLS end-to-end encryption. See [MLS Integration Guide](./mls- | `cryptoRecoveryEnabled` | boolean | true | Recover an undecryptable 1:1 message instead of dropping it and ACKing anyway (kill switch — see [Crypto-Failure Recovery](#crypto-failure-recovery)) | | `pendingQueue.maxPendingPerPeer` | number | 64 | Max inbound encrypted messages held per peer awaiting session readiness | | `pendingQueue.maxPendingGlobal` | number | 4096 | Max inbound encrypted messages held across all peers | -| `pendingQueue.pendingTtlMs` | number | 1800000 | TTL for held encrypted messages (30 minutes) | +| `pendingQueue.pendingTtlMs` | number | 86400000 | TTL for held encrypted messages (24 hours; the queue is also persisted, see `docs/mls-integration.md`) | | `pendingQueue.overflowPolicy` | string | `drop_oldest` | Overflow action: `drop_oldest` or `drop_newest` | `pendingQueue` bounds the **inbound** pending-*decryption* queue — messages that @@ -568,8 +568,14 @@ per-member delivery path. **Dedup Config**: | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `maxTrackedMessages` | number | 1000 | Max message IDs to track (must be > 0) | -| `retentionTimeSecs` | number | 3600 | Retention time (1 hour; must be > 0) | +| `maxTrackedMessages` | number | 2000 | Max message IDs to track (must be > 0) | +| `retentionTimeSecs` | number | 86400 | Retention time (24 hours; must be > 0) | + +The seen set is persisted (up to 2000 ids, newest first, sealed like the pending +queues) and restored on the next launch. A message can reach the device twice on +two paths, a push injection and then the relay socket after a reconnect, and the +second copy is still recognised as a duplicate across an app restart. The +retention window is applied again on import. Both fields are now **rejected at `0`**. Neither failed safe: at `maxTrackedMessages: 0` the exact-match tracker evicts on every insert, so it @@ -889,7 +895,7 @@ config = ProtocolConfig( require_encryption=True, max_pending_per_peer=64, max_pending_global=4096, - pending_ttl_ms=1_800_000, # 30 min (the SDK default) + pending_ttl_ms=86_400_000, # 24 h (the SDK default) overflow_policy=OverflowPolicy.DROP_OLDEST, ) ``` diff --git a/docs/ios-integration.md b/docs/ios-integration.md index b9d67589..86b29618 100644 --- a/docs/ios-integration.md +++ b/docs/ios-integration.md @@ -117,7 +117,7 @@ final class MeshController { storePending: true, maxPendingPerPeer: 100, maxPendingGlobal: 1000, - pendingTtlMs: 1_800_000, // 30 min (the SDK default) + pendingTtlMs: 86_400_000, // 24 h (the SDK default) overflowPolicy: .dropOldest // These 9 use their defaults: requireEncryption (true), // maxGroupMembers (256), groupRelayEnabled (true), diff --git a/docs/mesh.md b/docs/mesh.md index 96e8ab06..2b6d5009 100644 --- a/docs/mesh.md +++ b/docs/mesh.md @@ -648,8 +648,8 @@ const config = { }, dedup: { useBloomFilter: false, // HashMap mode (default); set true for bloom filter - maxTrackedMessages: 1000, // HashMap mode capacity - retentionTimeSecs: 3600, // 1 hour retention + maxTrackedMessages: 2000, // HashMap mode capacity (persisted across restarts) + retentionTimeSecs: 86400, // 24 hour retention }, }, }; diff --git a/docs/mls-integration.md b/docs/mls-integration.md index 93766168..9a8a3e6a 100644 --- a/docs/mls-integration.md +++ b/docs/mls-integration.md @@ -111,16 +111,21 @@ const protocol = new OfflineProtocol({ | `requireEncryption` | `true` | Enforce encrypted delivery (send fails closed if encryption cannot be applied) | | `pendingQueue.maxPendingPerPeer` | `64` | Per-peer cap for encrypted messages received before session readiness | | `pendingQueue.maxPendingGlobal` | `4096` | Global cap for encrypted messages received before session readiness | -| `pendingQueue.pendingTtlMs` | `1800000` | TTL (30 min) for encrypted messages held before session readiness | +| `pendingQueue.pendingTtlMs` | `86400000` | TTL (24 h) for encrypted messages held before session readiness | | `pendingQueue.overflowPolicy` | `drop_oldest` | Overflow policy: `drop_oldest` or `drop_newest` | | `compactEnvelopeEnabled` | `true` | Emit the compact MLS envelope to recipients that advertise `env_versions` | | `richPayloadEnabled` | `true` | Seal rich extras inside the MLS ciphertext for recipients that advertise `rich_versions` | | `cryptoRecoveryEnabled` | `true` | Recover an undecryptable 1:1 message instead of dropping it and ACKing anyway ([below](#crypto-failure-recovery)) | -The `pendingTtlMs` default is 30 minutes, not the 2 minutes earlier releases -used: under the deferred-ACK model a message held here is not delivery-ACKed on -receipt, so this queue is the primary recovery window before the session -confirms. Memory stays bounded by the per-peer and global caps plus the +The `pendingTtlMs` default is 24 hours (it was 2 minutes, then 30 minutes, in +earlier releases): under the deferred-ACK model a message held here is not +delivery-ACKed on receipt, so this queue is the primary recovery window before +the session confirms. With a relay that pushes ciphertext without +store-and-forward there is no second copy to ask for, either. The queue is also +persisted (`pending_decrypt_entries`, sealed like the outbound pending queue), +so a frame survives an app restart; the in-memory TTL restarts with the +process and a persisted record is dropped after 7 days on disk, surfacing a +`PENDING_QUEUE_DROPPED` decryption failure with reason `expired_persisted`. Memory stays bounded by the per-peer and global caps plus the `drop_oldest` policy — a longer TTL lets entries linger within those caps, it does not raise the ceiling. @@ -463,9 +468,11 @@ A protocol-state provider is a byte store, not a trusted one. Store and return the bytes you are handed **verbatim** — do not inspect, re-encode, compress, or truncate them. -The SDK seals the record values that can carry message plaintext or media key -material — pending session messages, outbox entries, and media transfer -descriptors — with ChaCha20-Poly1305 under a per-install key kept in +The SDK seals the record values that can carry message plaintext, media key +material, or a timeline of when messages arrived: pending session messages, +outbox entries, media transfer descriptors, parked inbound ciphertext +(`pending_decrypt_entries`) and the deduplicator's seen set (`dedup_seen_ids`). +They are sealed with ChaCha20-Poly1305 under a per-install key kept in `MlsStorageProvider` (key type `protocol_state_record_key`). Each record's associated data binds it to its `(keyType, keyId)` slot, so a record cannot be moved between peers or categories by anyone with write access to the container. diff --git a/docs/nostr.md b/docs/nostr.md index f96e0787..f08b4b86 100644 --- a/docs/nostr.md +++ b/docs/nostr.md @@ -225,7 +225,7 @@ Bridges that still call the timestamp-less `nostrMessageReceived(senderId, data) **Two residuals worth knowing, neither of which loses messages:** -*The replayed overlap is not fully deduplicated.* Message-id dedup retains ids for an hour by default (`reliability.dedup.retention_time_secs`, and at most `max_tracked_messages` of them), while `since` reaches back an hour and five minutes. A reconnect sooner than the retention window has its overlap absorbed; a reconnect after longer — an app reopened the next day, the common case — re-processes it instead. That costs work, not correctness: a replayed ciphertext whose ratchet generation is spent fails closed and is dropped, a past-epoch one triggers at most one rate-limited re-key, and a replayed group copy TTLs out of the pending buffer. +*The replayed overlap is deduplicated only inside the retention window and the id cap.* Message-id dedup retains ids for a day by default (`reliability.dedup.retention_time_secs`, and at most `max_tracked_messages` of them, persisted across restarts), while `since` reaches back an hour and five minutes. A reconnect sooner than the retention window has its overlap absorbed, including an app reopened the next day, now that the set survives the restart. A reconnect after longer, or after more traffic than the id cap holds, re-processes it instead. That costs work, not correctness: a replayed ciphertext whose ratchet generation is spent fails closed and is dropped, a past-epoch one triggers at most one rate-limited re-key, and a replayed group copy TTLs out of the pending buffer. *Junk can crowd out stored history.* The routing tag is `SHA-256(address)`, so anyone who knows an address can publish events to it, and only the *decodability* of a frame gates the watermark — parsing a `Message` needs no signature. An attacker who floods more than `limit` decodable events can therefore both push real messages out of a truncated initial query and advance the mark past them, leaving those below the next `since`. What makes this recoverable rather than terminal is that it is not the only delivery path: ACK-gated messages stay in the sender's outbox for 7 days and are retransmitted with a fresh `created_at`, which lands above any watermark. The exposure is delay on a Nostr-only route, and it needs a sustained flood rather than a single event. diff --git a/docs/react-native-integration.md b/docs/react-native-integration.md index 595fe5c4..6b430534 100644 --- a/docs/react-native-integration.md +++ b/docs/react-native-integration.md @@ -164,7 +164,7 @@ Four consequences worth designing against: - **Handlers must be idempotent.** Every one of these mechanisms can deliver the same fact more than once — Android by redelivering a held copy, iOS by restating on each foreground until the transport is re-enabled, the JS layer by replaying to a late listener. Set a flag; do not push a screen or fire a notification per event. - **They can arrive late.** Treat them as "this is true", not "this just happened" — reconcile against actual state rather than assuming the event is fresh. -- **Register listeners before `start()`.** The JS hold makes an `await` between construction and your first `on(...)` survivable, but only for these two tags — every other event in that window is dropped, correctly, because it is periodic or re-derivable. Registering synchronously right after construction keeps the window at zero and is still the right habit. (The SDK warns once per event type if events arrive while you have registered no listeners at all.) +- **Register listeners before `start()`.** The JS hold makes an `await` between construction and your first `on(...)` survivable, but only for these two tags and for the inbound message events in §6.4. Every other event in that window is dropped, correctly, because it is periodic or re-derivable. Registering synchronously right after construction keeps the window at zero and is still the right habit. (The SDK warns once per event type if events arrive while you have registered no listeners at all.) - **They are not durable.** No mechanism survives a process kill, and neither the Android hold nor the JS hold survives a JS reload. Persisting would not help: if the process was killed, the event was never generated in the first place. The JS hold is also cleared where continuing to hold would be *worse* than dropping — redelivering a stale one-shot is the same failure inverted, not a milder one. A held `mesh_stopped_by_user` replayed after you called `start()` would report a mesh that is coming up as down, with nothing to correct it, so `start()` discards anything no listener has claimed by then; `enableTransport('internet', ...)` discards a held `internet_session_superseded`, because that call is what clears the latch the event reports; and `destroy()` discards whatever is left, so an instance you destroy and start again cannot hand the previous session's event to the next session's first listener. @@ -199,7 +199,7 @@ Because `isInternetSuperseded()` reads the latch itself rather than a delivery, Android can kill your process while mesh is running — memory pressure is the usual reason, and a foreground service makes it less likely, not impossible. The keep-alive service is `START_STICKY`, so the system hands the service back afterwards, **but the SDK never re-creates the protocol from there.** By default, if nothing in the new process has brought a mesh back up by the time the re-delivered intent lands, the service stops itself, so no "Mesh Active" notification outlives the protocol it advertises. (An app that boots React Native from `Application.onCreate` can win that race and have a mesh running already — then the service keeps the notification it is holding for the mesh *your app* started.) -This is a decision, not a missing feature. A protocol re-created with no JavaScript context behind it is worse than one that is simply down: the receive path sends a delivery ACK *before* it emits `message_received`, that ACK makes the sender retire the message from its outbox, and the event is then dropped because nothing is subscribed. The message is gone, and its sender was told it arrived. Staying down keeps the failure recoverable — the sender's outbox holds for up to seven days, retries, parks, and pushes, and delivers once this device is genuinely running again. +This is a decision, not a missing feature. A protocol re-created with no JavaScript context behind it is worse than one that is simply down: the receive path sends a delivery ACK *before* it emits `message_received`, that ACK makes the sender retire the message from its outbox, and with no JavaScript behind the protocol nothing would ever take the event. The inbound hold in §6.4 bridges a short gap in memory, up to 256 events; it is not a place for messages to wait out a dead process. The message is gone, and its sender was told it arrived. Staying down keeps the failure recoverable: the sender's outbox holds for up to seven days, retries, parks, and pushes, and delivers once this device is genuinely running again. **You can opt in to having the mesh come back on its own** — see §6.3. It does not weaken any of the above: nothing native re-creates the protocol there either. It starts *JavaScript* first, so a receiver exists before a protocol does, and your own code decides what happens next. @@ -218,7 +218,7 @@ if (state !== ProtocolState.Running) { Two more things to know if you restart the SDK yourself: - **Never reuse a `destroy()`ed instance.** `destroy()` removes the event subscriptions, and only the constructor creates them — a destroyed instance that is `start()`ed again will run but deliver zero events. Construct a new `OfflineProtocol`. -- **Nothing is queued for you while the process is dead.** The one-shot event delivery described in §6.1 is in-memory on both platforms; a process kill loses it. That is not a gap — if the process was killed, the event was never generated. For the relay case there is a durable read regardless: `isInternetSuperseded()` reports the transport's own latch, so a restarted process that re-enables the relay and is displaced again learns it the same way. +- **Nothing is queued for you while the process is dead.** The one-shot event delivery described in §6.1 is in-memory on both platforms, and so is the inbound hold in §6.4. For a one-shot that is not a gap: if the process was killed, the event was never generated. For an inbound message it is one, which is why §6.4 says to persist messages in your handler. For the relay case there is a durable read regardless: `isInternetSuperseded()` reports the transport's own latch, so a restarted process that re-enables the relay and is displaced again learns it the same way. ### 6.3 Restoring the mesh automatically after a process kill (Android, opt-in) @@ -270,6 +270,22 @@ Three more things the task has to get right: --- +### 6.4 Inbound message events are held until you listen (both platforms) + +`message_received`, `file_received` and `message_decryption_failed` each report one message, and nothing restates them. By the time `message_received` or `file_received` is emitted, the core has already acknowledged the content, so its sender has retired it. Dropping one of these events because nothing was listening would lose a message, so the SDK holds them instead. + +- **Each native bridge** holds up to 256 of them while JavaScript cannot take them: no subscription yet, or no live React instance while the app is backgrounded. Entries are keyed by event type and message id, kept in arrival order, and redelivered on the next subscription or app foreground. +- **The JS layer** holds up to 256 that arrive before you have registered a listener for their type, and replays them in arrival order to the first matching `on(...)`, or to an `on('all', ...)` listener. +- **Held events belong to one session.** `destroy()` discards them on every layer, so a message held for an account you tore down never reaches the next one. + +Unlike the one-shot events in §6.1 these are not collapsed: every event is its own message, so you receive each one rather than the latest. + +Three limits to design around: + +- **The cap is real.** Past 256 held events the oldest is dropped. For `message_received` and `file_received` that is a lost message whose sender was told it arrived. Register those listeners synchronously after construction, before `start()`, so the hold stays empty in normal operation. +- **Nothing here is durable.** A JS reload or a process kill loses whatever is held. Persist a message in your handler, keyed by `message_id`, before doing anything slower with it. +- **Keep handlers idempotent.** Dedupe on `message_id`. The holds are built for at-least-once delivery, not exactly-once. + ## 7. Group Messaging (MLS-Encrypted Mesh Groups) Group features (create groups, send messages, manage members) are provided directly by the SDK's **mesh group methods**. Membership and message encryption are MLS, end to end, and handled entirely by the SDK — **there is no group server that can read your messages**, and no separate service for your app to run. Each method runs against your `OfflineProtocol` instance and delivers over whatever transports DORS has selected. diff --git a/docs/state-machines/outbox-and-retries.md b/docs/state-machines/outbox-and-retries.md index b3562dd7..8dab6e37 100644 --- a/docs/state-machines/outbox-and-retries.md +++ b/docs/state-machines/outbox-and-retries.md @@ -139,6 +139,21 @@ recovers through a descriptor-based resend request. When the relay reports a recipient unreachable, a direct message is **parked** rather than retried into a void. +A relay push is the second trigger. When the recipient has no live socket the +relay hands the ciphertext to a device push and answers +`MessageSent { pushed: true }`; it keeps no copy, so if the push is lost nothing +re-delivers the frame when the recipient reconnects. The bridges report that +answer to the core as `relay_pushed`, and the message parks exactly as below +with two differences, both because the push may have delivered it: only a plain +direct message parks (a connection request keeps its typed tracking and a +Welcome its lifecycle, both awaiting the answer a delivered push produces), and +**no `MessageUndeliverable` is emitted**, on the push or on any later probe of +that frame. The relay remembers what it has pushed for a day and answers a +repeat with `DeliveryError` rather than a second notification, which reaches +the core as an ordinary unreachable verdict. The outbox entry therefore carries +a persisted `relay_pushed` mark, and a verdict for a marked entry re-parks it +silently. Without the mark the silence would last exactly one probe interval. + A parked message is probed periodically with a backoff that widens from 15 seconds toward 10 minutes. When the peer returns, parked messages re-enter the queue. @@ -191,7 +206,9 @@ with their own timeouts and their own downgrade paths. When the recipient is not reachable and a push channel exists, the ciphertext travels in the push payload. The message stays in the outbox: a push is a wake -signal plus an opportunistic delivery, not an acknowledgement. +signal plus an opportunistic delivery, not an acknowledgement. The relay tells +the sender it pushed (`MessageSent { pushed: true }`), and the sender parks the +message on that answer; see [Parking](#parking). ## Restart behaviour