diff --git a/CHANGELOG.md b/CHANGELOG.md index dc0bb7bd..77ccc8e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -132,6 +132,15 @@ the client's answers were wrong on a device rather than merely different: through, and the queue now waits for a good one. `telemetryStats().dropped` counts events and only events, so a queued batch that cannot be opened on load is warned about rather than added to it in a different unit. +- **A 403 `telemetry_disabled` drops the queue.** The ingest answers with that + `error` code, rather than the `forbidden` a bad key draws, while the + application's telemetry toggle is off in the developer portal. The pipe + drops the refused batch and everything queued behind it, discards what it + collects afterwards instead of queuing it, and sends nothing more until the + next `enableTelemetry`, so a toggled-off stretch stays a gap on the + dashboard rather than filling in from the device's queue once the toggle + comes back. Every other 401 or 403 keeps its queue as above, and so does a + 403 from an ingest that predates the code, which answers `forbidden`. - **Telemetry session boundaries follow the process on Android**, through the set of started activities rather than `onHostPause`. `onHostPause` is `Activity.onPause`, which fires for a runtime permission dialog (including diff --git a/bindings/python/offline_protocol_sdk/protocol_manager.py b/bindings/python/offline_protocol_sdk/protocol_manager.py index 65b8eca5..09b1e066 100644 --- a/bindings/python/offline_protocol_sdk/protocol_manager.py +++ b/bindings/python/offline_protocol_sdk/protocol_manager.py @@ -527,8 +527,10 @@ def telemetry_stats(self) -> TelemetryStats | None: ``accepted_events`` is what the ingest reported accepting, which is what an invoice is reconciled against; ``dropped`` counts events - lost to the ring buffer, the queue caps, the six-day expiry or a - permanent rejection; ``last_error`` is the most recent send failure. + lost to the ring buffer, the queue caps, the six-day expiry, a + permanent rejection, or everything queued or collected after the + ingest reported the application's telemetry toggle off; + ``last_error`` is the most recent send failure. """ return self._protocol.telemetry_stats() diff --git a/bindings/react-native/src/index.ts b/bindings/react-native/src/index.ts index eb7afebe..6e653358 100644 --- a/bindings/react-native/src/index.ts +++ b/bindings/react-native/src/index.ts @@ -1955,10 +1955,11 @@ export class OfflineProtocol { * The telemetry pipe's counters, or `null` while telemetry is not * enabled. `acceptedEvents` is what the ingest reported accepting, which * is what an invoice is reconciled against; `dropped` counts events lost - * to the ring buffer, the durable queue's caps, the six-day expiry, or a - * permanent rejection; `lastError` is the most recent send failure, and - * clears once a batch is accepted, so it reports the current state rather - * than the high-water mark of a recovered outage. + * to the ring buffer, the durable queue's caps, the six-day expiry, a + * permanent rejection, or everything queued or collected after the ingest + * reported the application's telemetry toggle off; `lastError` is the most + * recent send failure, and clears once a batch is accepted, so it reports + * the current state rather than the high-water mark of a recovered outage. */ async telemetryStats(): Promise { const stats = await OfflineProtocolNativeModule.telemetryStats(); diff --git a/bindings/react-native/src/types.ts b/bindings/react-native/src/types.ts index 03527507..b8b7fbb4 100644 --- a/bindings/react-native/src/types.ts +++ b/bindings/react-native/src/types.ts @@ -3081,8 +3081,10 @@ export interface TelemetryConfig { * `sentEvents` counts events in batches the ingest answered 2xx; * `acceptedEvents` sums the `accepted` count the ingest reported, which is * what an invoice is reconciled against. `dropped` counts events lost to - * the ring buffer, the durable queue's caps, the six-day expiry, or a - * permanent rejection. `lastFlushAtMs` is when a batch was last accepted. + * the ring buffer, the durable queue's caps, the six-day expiry, a + * permanent rejection, or everything queued or collected after the ingest + * reported the application's telemetry toggle off. `lastFlushAtMs` is when + * a batch was last accepted. */ export interface TelemetryStats { buffered: number; diff --git a/crates/offline-protocol-uniffi/src/offline_protocol.udl b/crates/offline-protocol-uniffi/src/offline_protocol.udl index 25267737..3cc900d2 100644 --- a/crates/offline-protocol-uniffi/src/offline_protocol.udl +++ b/crates/offline-protocol-uniffi/src/offline_protocol.udl @@ -934,7 +934,8 @@ dictionary TelemetryConfig { // in batches the ingest answered 2xx; `accepted_events` sums the `accepted` // count the ingest reported, which is what an invoice is reconciled against. // `dropped` counts events lost to the ring buffer, the durable queue's caps, -// the six-day expiry, or a permanent rejection. +// the six-day expiry, a permanent rejection, or everything queued or +// collected after the ingest reported the application's telemetry toggle off. dictionary TelemetryStats { u64 buffered; u64 sent_events; diff --git a/crates/offline-protocol/src/telemetry/pipe/mod.rs b/crates/offline-protocol/src/telemetry/pipe/mod.rs index 1822d4d2..c3fbe43a 100644 --- a/crates/offline-protocol/src/telemetry/pipe/mod.rs +++ b/crates/offline-protocol/src/telemetry/pipe/mod.rs @@ -118,7 +118,9 @@ pub struct TelemetryStats { /// Events the ingest reported accepted (the `accepted` count of each /// 202 body, summed; a replay counts nothing new). pub accepted_events: u64, - /// Events lost: ring overflow, queue caps, TTL expiry, permanent 4xx. + /// Events lost: ring overflow, queue caps, TTL expiry, permanent 4xx, + /// and everything queued or collected once the developer portal switched + /// telemetry off for the application. /// /// Events, never records. This number is what reconciles an invoice /// against the device, so it counts in the unit the ingest meters. A @@ -188,6 +190,16 @@ pub(crate) struct PipeShared { /// `-1` until a batch has been accepted. last_flush_at_ms: AtomicI64, last_error: Mutex>, + /// Set once the ingest has answered `telemetry_disabled`: the + /// application's toggle is off in the developer portal. From then on the + /// ring is emptied into nothing at every cycle rather than cut into + /// batches, and a durable queue adopted afterwards is emptied as it is + /// adopted, so nothing the pipe holds or adopts is left on disk for a + /// later pipe to send, and the gap the toggle promises holds at the + /// device. Read on the cycle and at adoption, never on the emit path, + /// which keeps its one atomic load. Cleared only by a fresh pipe, which is + /// what the next `enable_telemetry` builds. + server_disabled: AtomicBool, } fn lock(mutex: &Mutex) -> MutexGuard<'_, T> { @@ -229,7 +241,13 @@ impl PipeShared { self.adopt_pending_backend(); let drained = lock(&self.pipeline).drain(); - if !drained.is_empty() { + if self.server_disabled.load(Ordering::Relaxed) { + // Counted as dropped, in events, the unit the rest of `dropped` + // uses: the toggle is off, and these are the events it says are + // not kept. + self.uploader_dropped + .fetch_add(drained.len() as u64, Ordering::Relaxed); + } else if !drained.is_empty() { let device_id = if self.include_device_id { lock(&self.device_id).clone() } else { @@ -280,6 +298,15 @@ impl PipeShared { if report.batches_sent > 0 { self.last_flush_at_ms.store(now, Ordering::Relaxed); } + if report.disabled && !self.server_disabled.swap(true, Ordering::Relaxed) { + tracing::warn!( + target: PIPE_LOG_TARGET, + dropped_events = report.dropped_events, + "hosted telemetry is switched off for this application in the developer \ + portal; the queue was dropped and nothing more is collected until \ + enable_telemetry" + ); + } // Mirrored, not accumulated: the uploader clears its own error when a // batch is accepted, and a stats reader asking "is telemetry working" // after a recovered outage must not still be shown the outage. A @@ -305,6 +332,20 @@ impl PipeShared { Attach::Busy(backend) => Some((backend, true)), Attach::Failed(backend) => Some((backend, false)), }; + if refused.is_none() && self.server_disabled.load(Ordering::Relaxed) { + // The toggle is off, and adoption is the one way a batch can still + // enter the store: the cycle no longer cuts any, but adopting + // loads whatever the queue on that storage already holds, an + // earlier launch's batches included. The halted uploader returns + // before it reads the queue, so a batch kept here would sit on + // disk until the next `enable_telemetry` resent it, which is the + // backfill the toggle promises not to make. + let mut dropped = 0u64; + while let Some(batch) = store.pop_front() { + dropped += u64::from(batch.event_count); + } + self.uploader_dropped.fetch_add(dropped, Ordering::Relaxed); + } store.flush_index(); self.durable.store(store.is_durable(), Ordering::Relaxed); self.store_dropped @@ -606,6 +647,7 @@ impl TelemetryPipe { store_dropped: AtomicU64::new(0), last_flush_at_ms: AtomicI64::new(-1), last_error: Mutex::new(None), + server_disabled: AtomicBool::new(false), }); // The host's state at enable time, recorded rather than reported: // there is no session to close yet, and the seed exists so that a diff --git a/crates/offline-protocol/src/telemetry/pipe/tests/mod.rs b/crates/offline-protocol/src/telemetry/pipe/tests/mod.rs index 379c3245..04e2963e 100644 --- a/crates/offline-protocol/src/telemetry/pipe/tests/mod.rs +++ b/crates/offline-protocol/src/telemetry/pipe/tests/mod.rs @@ -38,11 +38,14 @@ impl FakeClock { } } -/// A client that captures every body and answers with a fixed status. +/// A client that captures every body and answers with a fixed status. A 2xx +/// carries the ingest's accepted count; anything else carries the ingest's +/// error shape, with `error_code` as the code (`forbidden` when unset). #[derive(Clone)] pub(crate) struct CapturingClient { pub(crate) bodies: Arc>>, pub(crate) status: Arc, + pub(crate) error_code: Arc>>, } impl CapturingClient { @@ -50,6 +53,7 @@ impl CapturingClient { Self { bodies: Arc::default(), status: Arc::new(AtomicI64::new(202)), + error_code: Arc::default(), } } @@ -74,10 +78,21 @@ impl HttpClient for CapturingClient { .ok() .and_then(|v| v["events"].as_array().map(|e| e.len())) .unwrap_or(0); + let body = if (200..300).contains(&status) { + format!(r#"{{"accepted":{accepted},"rejected":0,"rejected_reasons":[]}}"#) + } else { + let code = self + .error_code + .lock() + .unwrap() + .clone() + .unwrap_or_else(|| "forbidden".to_string()); + format!(r#"{{"error":"{code}","message":"{code}"}}"#) + }; Ok(Response { status: status as u16, retry_after: None, - body: format!(r#"{{"accepted":{accepted},"rejected":0,"rejected_reasons":[]}}"#), + body, }) } } diff --git a/crates/offline-protocol/src/telemetry/pipe/tests/scenarios.rs b/crates/offline-protocol/src/telemetry/pipe/tests/scenarios.rs index 3d3dcc35..27c0f27c 100644 --- a/crates/offline-protocol/src/telemetry/pipe/tests/scenarios.rs +++ b/crates/offline-protocol/src/telemetry/pipe/tests/scenarios.rs @@ -556,3 +556,174 @@ fn a_detached_worker_finishing_an_upload_cannot_orphan_its_replacements_batch() "the batch the ingest accepted was queued again: {sessions:?}" ); } + +/// The toggle's gap holds at the device. A `403` whose `error` code is +/// `telemetry_disabled` drops the refused batch and everything queued behind +/// it, what is collected afterwards is discarded rather than queued, and a +/// fresh pipe over the same store has nothing from the off period to resend. +/// Contrast `a_batch_refused_for_a_bad_key_is_still_there_when_the_key_is_fixed`: +/// a bad key keeps its queue, because that is a fault the developer fixes. +#[test] +fn a_403_telemetry_disabled_drops_the_queue_and_discards_what_follows() { + let clock = FakeClock::at(1_000); + let storage = Arc::new(MemoryStorage::default()); + let client = CapturingClient::accepting(); + client.status.store(0, Ordering::SeqCst); // offline, so two batches queue up + let pipe = inline_pipe(&test_config(), &clock, client.clone(), sealed(&storage)); + emit_failed(&pipe, 1); + pipe.flush(); + clock.set(1_500); // inside the backoff: the second batch is cut, not sent + emit_failed(&pipe, 2); + pipe.flush(); + let queued = |pipe: &TelemetryPipe| crate::telemetry::pipe::lock(&pipe.shared().store).len(); + assert_eq!(queued(&pipe), 2); + assert!(client.bodies.lock().unwrap().is_empty()); + + // The developer switches the toggle off in the portal. + client.status.store(403, Ordering::SeqCst); + *client.error_code.lock().unwrap() = Some("telemetry_disabled".into()); + clock.set(3_000); + pipe.flush(); + assert_eq!(client.bodies.lock().unwrap().len(), 1, "one attempt"); + assert_eq!( + pipe.stats().last_error.as_deref(), + Some("ingest responded 403 (telemetry_disabled)") + ); + assert_eq!( + queued(&pipe), + 0, + "the refused batch and the one queued behind it are gone" + ); + assert_eq!(pipe.stats().dropped, 2); + + // Collected while off: discarded at the next flush, never queued or sent. + emit_failed(&pipe, 3); + clock.set(4_000); + pipe.flush(); + assert_eq!(queued(&pipe), 0); + assert_eq!(pipe.stats().dropped, 3); + assert_eq!( + client.bodies.lock().unwrap().len(), + 1, + "nothing more is sent" + ); + pipe.stop(FINAL_FLUSH_BUDGET); + + // The toggle comes back on and the app relaunches: a fresh pipe over the + // same protocol-state store has nothing from the off period to resend. + client.status.store(202, Ordering::SeqCst); + *client.error_code.lock().unwrap() = None; + let again = inline_pipe(&test_config(), &clock, client.clone(), sealed(&storage)); + // Without this, a fresh pipe that failed to adopt the store would run in + // memory, and "no backfill" below would hold with nothing proven. + assert!( + again.is_durable(), + "the fresh pipe adopted the durable queue" + ); + again.flush(); + assert_eq!(client.bodies.lock().unwrap().len(), 1, "no backfill"); + emit_failed(&again, 4); + again.flush(); + let bodies = client.bodies.lock().unwrap(); + assert_eq!(bodies.len(), 2, "and new events flow again"); + assert!( + bodies[1].contains("protocol.message.failed"), + "{}", + bodies[1] + ); +} + +/// A queue adopted after the refusal is emptied as it is adopted. The engine +/// starts every pipe in memory and hands the protocol-state store over once +/// `initialize_mls` has it, so a flush can reach the ingest first. The halted +/// uploader never reads the queue again, so without the sweep at adoption the +/// batches an earlier launch left on disk would stay there until the next +/// `enable_telemetry` resent them. +#[test] +fn a_queue_adopted_after_a_telemetry_disabled_refusal_is_dropped_too() { + let clock = FakeClock::at(1_000); + let storage = Arc::new(MemoryStorage::default()); + let client = CapturingClient::accepting(); + + // An earlier launch, offline throughout, leaves one batch on disk. + client.status.store(0, Ordering::SeqCst); + let earlier = inline_pipe(&test_config(), &clock, client.clone(), sealed(&storage)); + emit_failed(&earlier, 1); + earlier.flush(); + earlier.stop(FINAL_FLUSH_BUDGET); + assert!(client.bodies.lock().unwrap().is_empty()); + + // This launch starts in memory, as `enable_telemetry` does, and reaches + // the ingest before storage attaches. The toggle is off. + client.status.store(403, Ordering::SeqCst); + *client.error_code.lock().unwrap() = Some("telemetry_disabled".into()); + let pipe = inline_pipe(&test_config(), &clock, client.clone(), Backend::Memory); + emit_failed(&pipe, 2); + pipe.flush(); + assert_eq!(client.bodies.lock().unwrap().len(), 1, "one attempt"); + assert_eq!(pipe.stats().dropped, 1); + + // `initialize_mls` attaches the store, and the pipe adopts the queue the + // earlier launch left there. + let Backend::Sealed { + storage: backing, + cipher, + } = sealed(&storage) + else { + unreachable!("sealed() builds a sealed backend"); + }; + pipe.attach_storage(backing, cipher); + // Without this, an adoption that never happened would leave the store + // empty too, and the assertions below would hold with nothing proven. + assert!(pipe.is_durable(), "the queue on disk was adopted"); + let queued = crate::telemetry::pipe::lock(&pipe.shared().store).len(); + assert_eq!(queued, 0, "and emptied as it was adopted"); + assert_eq!( + pipe.stats().dropped, + 2, + "the adopted batch counts as dropped" + ); + pipe.stop(FINAL_FLUSH_BUDGET); + + // The toggle comes back on: the next launch has nothing to resend. + client.status.store(202, Ordering::SeqCst); + *client.error_code.lock().unwrap() = None; + let again = inline_pipe(&test_config(), &clock, client.clone(), sealed(&storage)); + assert!( + again.is_durable(), + "the fresh pipe adopted the durable queue" + ); + again.flush(); + assert_eq!(client.bodies.lock().unwrap().len(), 1, "no backfill"); +} + +/// A refusal on the final flush drops the queue the same way. `stop` runs the +/// same cycle, and a batch it left on disk would be resent by the next +/// `enable_telemetry`. +#[test] +fn a_telemetry_disabled_refusal_on_the_final_flush_drops_the_queue_too() { + let clock = FakeClock::at(1_000); + let storage = Arc::new(MemoryStorage::default()); + let client = CapturingClient::accepting(); + client.status.store(403, Ordering::SeqCst); + *client.error_code.lock().unwrap() = Some("telemetry_disabled".into()); + let pipe = inline_pipe(&test_config(), &clock, client.clone(), sealed(&storage)); + emit_failed(&pipe, 1); + pipe.stop(FINAL_FLUSH_BUDGET); + assert_eq!( + client.bodies.lock().unwrap().len(), + 1, + "the final flush reached the ingest" + ); + assert_eq!(pipe.stats().dropped, 1); + + client.status.store(202, Ordering::SeqCst); + *client.error_code.lock().unwrap() = None; + let again = inline_pipe(&test_config(), &clock, client.clone(), sealed(&storage)); + assert!( + again.is_durable(), + "the fresh pipe adopted the durable queue" + ); + again.flush(); + assert_eq!(client.bodies.lock().unwrap().len(), 1, "no backfill"); +} diff --git a/crates/offline-protocol/src/telemetry/pipe/uploader.rs b/crates/offline-protocol/src/telemetry/pipe/uploader.rs index 580551bc..5f8ae03e 100644 --- a/crates/offline-protocol/src/telemetry/pipe/uploader.rs +++ b/crates/offline-protocol/src/telemetry/pipe/uploader.rs @@ -14,7 +14,8 @@ //! | 2xx | The ingest owns the batch; delete it, reset the backoff | //! | 3xx | Never followed, so it arrives as an answer; drop it, keep going | //! | 413 | The batch can never fit; drop it, keep going | -//! | 401, 403 | The key or the app id is wrong; keep the batch, record the error, stop until the next `enable_telemetry` | +//! | 403 with `error: telemetry_disabled` | The application's telemetry toggle is off in the developer portal; drop the batch and everything queued behind it, record the error, stop until the next `enable_telemetry`, and have the pipe discard what it collects meanwhile | +//! | any other 401, 403 | The key or the app id is wrong; keep the batch, record the error, stop until the next `enable_telemetry` | //! | 429 | Wait `Retry-After` when given, else back off | //! | 408, 5xx, no answer | Back off: 1 s doubling to 15 min, plus up to a second of jitter | //! | any other 4xx | The bytes will never be accepted; drop, keep going | @@ -48,6 +49,10 @@ pub(crate) const BATTERY_DEFER_BELOW: u8 = 15; pub(crate) const HEADER_APP_ID: &str = "X-Mesh-Analytics-App-Id"; pub(crate) const HEADER_IDEMPOTENCY: &str = "Idempotency-Key"; +/// The `error` code the ingest answers a 403 with while the application's +/// telemetry toggle is off in the developer portal. Every other 403 carries +/// `forbidden`, and is a bad key or a key for another app. +pub(crate) const ERROR_TELEMETRY_DISABLED: &str = "telemetry_disabled"; /// One HTTP request, as the client sees it. pub(crate) struct Request<'a> { @@ -83,6 +88,13 @@ pub(crate) enum Outcome { Drop, /// Never resend anything until re-enabled. AuthHalt, + /// The developer portal has hosted telemetry switched off for this + /// application (403 with `error: telemetry_disabled`). Drop the batch and + /// everything queued, and never resend anything until re-enabled. Unlike + /// a bad key this is not a fault the developer fixes and re-enables + /// through, and holding the events would resend the off period once the + /// toggle came back, which is the backfill the toggle promises not to make. + Disabled, } /// What one drain did, for the caller's stats. @@ -93,6 +105,9 @@ pub(crate) struct DrainReport { pub(crate) dropped_events: u64, pub(crate) batches_sent: u32, pub(crate) halted: bool, + /// The halt was a `telemetry_disabled`: the queue was dropped, and the + /// pipe should discard what it collects from here on. + pub(crate) disabled: bool, } pub(crate) struct Uploader { @@ -217,6 +232,21 @@ impl Uploader { report.halted = true; break; } + Outcome::Disabled => { + // The toggle is off in the portal. Nothing queued will be + // wanted when it comes back on, since resending it then + // is exactly the backfill the toggle promises not to + // make, so the whole queue goes, not just the head, and + // the halt stops every later send until the next enable. + while let Some(batch) = store.pop_front() { + report.dropped_events += u64::from(batch.event_count); + } + self.reset_backoff(); + self.auth_halted = true; + report.halted = true; + report.disabled = true; + break; + } Outcome::Retry { after_ms } => { self.schedule_backoff(now_ms, after_ms); break; @@ -288,6 +318,13 @@ pub(crate) fn classify_response( } *last_error = Some(format!("ingest responded {status}")); match status { + // Only a 403, and only with the code. A body that does not parse or + // names any other code is the ordinary halt, so an ingest that + // predates the code, or answers `forbidden`, keeps the queue as before. + 403 if has_error_code(&response.body, ERROR_TELEMETRY_DISABLED) => { + *last_error = Some(format!("ingest responded 403 ({ERROR_TELEMETRY_DISABLED})")); + Outcome::Disabled + } 401 | 403 => Outcome::AuthHalt, 408 | 500..=599 => Outcome::Retry { after_ms: None }, 429 => Outcome::Retry { @@ -308,6 +345,13 @@ pub(crate) fn classify_response( } } +/// Whether an ingest error body, `{"error": "...", "message": "..."}`, names +/// `code`. A body that is not JSON, or has no string `error`, names nothing. +fn has_error_code(body: &str, code: &str) -> bool { + serde_json::from_str::(body) + .is_ok_and(|value| value.get("error").and_then(serde_json::Value::as_str) == Some(code)) +} + /// The number of events the ingest counted for a 2xx answer. /// /// A 202 body carries `accepted`; a 200 replay carries `deduplicated: true` @@ -582,6 +626,75 @@ pub(crate) mod tests { } } + #[test] + fn a_403_telemetry_disabled_drops_the_whole_queue_and_halts() { + let body = r#"{"error":"telemetry_disabled","message":"forbidden: mesh telemetry is disabled for this application in the developer portal"}"#; + let (mut up, requests) = uploader(vec![answer(403, body), answer(202, "")]); + let mut store = test_store(Backend::Memory); + store.push(batch(1, 0)); + store.push(batch(4, 0)); + let report = up.drain(&mut store, 0, 8, None); + assert!(report.halted); + assert!(report.disabled); + assert!(up.is_halted()); + assert!( + store.is_empty(), + "the refused batch and the one queued behind it are both gone" + ); + assert_eq!(report.dropped_events, 5); + assert_eq!(requests.lock().unwrap().len(), 1, "nothing more is sent"); + assert!(up.next_retry_at_ms().is_none()); + assert_eq!( + up.last_error(), + Some("ingest responded 403 (telemetry_disabled)") + ); + // Still halted on a later wake, whatever has been queued since. + store.push(batch(1, 0)); + let again = up.drain(&mut store, 1_000_000, 8, None); + assert!(again.halted); + assert_eq!(requests.lock().unwrap().len(), 1); + } + + #[test] + fn only_a_403_carrying_the_telemetry_disabled_code_disables() { + let mut last = None; + let response = |status: u16, body: &str| Response { + status, + retry_after: None, + body: body.into(), + }; + let disabled = r#"{"error":"telemetry_disabled","message":"x"}"#; + assert_eq!( + classify_response(&response(403, disabled), 0, &mut last), + Outcome::Disabled + ); + assert_eq!( + last.as_deref(), + Some("ingest responded 403 (telemetry_disabled)") + ); + // The code on any other status is not the portal toggle. + assert_eq!( + classify_response(&response(401, disabled), 0, &mut last), + Outcome::AuthHalt + ); + assert_eq!(last.as_deref(), Some("ingest responded 401")); + // A 403 with the shared code, no body, or junk is the ordinary halt: + // an ingest that predates the code keeps the queue as it always did. + for body in [ + r#"{"error":"forbidden","message":"forbidden: api key revoked"}"#, + "", + "not json", + r#"{"message":"no code"}"#, + ] { + assert_eq!( + classify_response(&response(403, body), 0, &mut last), + Outcome::AuthHalt, + "{body:?}" + ); + assert_eq!(last.as_deref(), Some("ingest responded 403"), "{body:?}"); + } + } + #[test] fn retryable_answers_back_off_exponentially_with_jitter_to_a_15_minute_cap() { let client = ScriptedClient { diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index 3d9bd375..21ef89bc 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -2055,7 +2055,8 @@ Diagnostics screen shows what the free API can drive instead. **What the pipe does differently from the analytics package**, each a named test: the backoff caps at 15 min rather than 5; a 413 is dropped rather than split; a 401 or 403 halts sending until the next `enableTelemetry` rather than -dropping the batch and continuing; `Retry-After` is honoured; `Idempotency-Key` +dropping the batch and continuing, except a 403 `telemetry_disabled`, which +drops the queue because the portal toggle is off; `Retry-After` is honoured; `Idempotency-Key` and `User-Agent` headers are sent; requests time out at 15 s (10 s to connect); flushes are deferred below 15% battery unless charging; and `reason` goes up as the engine's raw token rather than a classification made diff --git a/docs/privacy.md b/docs/privacy.md index 2a64f662..8ca5ad35 100644 --- a/docs/privacy.md +++ b/docs/privacy.md @@ -119,7 +119,11 @@ batches may upload when telemetry is enabled again under the same `appId`, subject to the queue limits and the six-day expiry. It is cleared by uninstalling the app, by `wipePersistedState()`, and by enabling telemetry under a different `appId`, which discards it rather than uploading another -application's batches under your key. +application's batches under your key. It is also cleared when the ingest +reports that the application's telemetry toggle is off in the developer +portal: the SDK discards the queue, and whatever it collects until the next +`enableTelemetry`, rather than holding them for an upload once the toggle is +back on. ## What is never collected diff --git a/docs/security/threat-model.md b/docs/security/threat-model.md index 96eaeda0..76f5a36e 100644 --- a/docs/security/threat-model.md +++ b/docs/security/threat-model.md @@ -607,7 +607,10 @@ is the one exception, and its egress is bounded as follows. collection and initiates a final flush, waiting up to three seconds for shutdown. An in-flight request may complete afterward. Remaining persisted batches are retained for a later enablement, subject to queue limits and - expiry. + expiry, except after the ingest answers `telemetry_disabled` (the + application's toggle is off in the developer portal). That discards the + queue and whatever is collected until the next enablement, so a toggled-off + stretch is never uploaded later. The key that authenticates the stream is R13. diff --git a/docs/telemetry.md b/docs/telemetry.md index 5079166c..0683f908 100644 --- a/docs/telemetry.md +++ b/docs/telemetry.md @@ -171,8 +171,15 @@ deduplicates on `batch_id` for seven and a later replay would count twice. A 401 or 403 (a bad key, or a key for another app) keeps the batch, records the error in the stats, and stops sending until the next `enableTelemetry`: the key is a configuration fault you fix and re-enable through, so the queue -waits rather than losing what it holds. Any other 4xx drops the batch and -continues, and so does any 3xx, because the uploader follows no redirect. A +waits rather than losing what it holds. The one exception is a 403 whose +`error` code is `telemetry_disabled`, the ingest's answer while the +application's telemetry toggle is off in the developer portal. That drops the +refused batch and everything queued behind it, discards whatever is collected +afterwards at the next flush rather than queuing it, and sends nothing more +until the next `enableTelemetry`, so a toggled-off stretch is a gap on the +dashboard and the device holds nothing to fill it with later. Any other 4xx +drops the batch and continues, and so does any 3xx, because the uploader +follows no redirect. A followed 301, 302 or 303 would turn the `POST` into a `GET` at another location, where a 2xx would delete the batch unsent and count it as accepted. @@ -296,7 +303,7 @@ offline stretch collected. It is cleared by uninstalling the app, by | `buffered` | Events in the ring buffer, not yet cut into a batch | | `sentEvents` | Events in batches the ingest answered 2xx | | `acceptedEvents` | The `accepted` count the ingest reported, summed; a deduplicated replay adds nothing | -| `dropped` | Events lost: ring overflow, queue caps, the six-day expiry, or a permanent rejection. Counted in events, so it excludes records that would not open, which are counted in records and logged | +| `dropped` | Events lost: ring overflow, queue caps, the six-day expiry, a permanent rejection, or everything queued or collected after the ingest reported the application's telemetry toggle off. Counted in events, so it excludes records that would not open, which are counted in records and logged | | `sessionId` | The current session id | | `lastError` | The most recent send failure, or the configuration problem that halted sending | | `lastFlushAtMs` | When a batch was last accepted | @@ -392,6 +399,7 @@ is a portal feature, not an SDK one. The threat model records this as |---|---| | `apiKey` or `appId` empty or not printable ASCII, empty `appVersion`, zero or over-1 MiB `maxBatchBytes`, zero `maxBufferedRecords`, `flushIntervalMs` under 1000 | `enableTelemetry` rejects with `TelemetryConfigInvalid` naming the field | | Wrong key, or a key for another app | `lastError: "ingest responded 401"` (or 403); sending halts until the next `enableTelemetry` | +| The application's telemetry toggle is off in the developer portal | `lastError: "ingest responded 403 (telemetry_disabled)"`; the queue is dropped, later events are discarded, and sending halts until the next `enableTelemetry`. The ingest can go on refusing for up to 30 seconds after the toggle comes back on, so a launch inside that window is refused the same way and keeps nothing until the app next calls `enableTelemetry`, usually the launch after | | No network | `lastError` names the transport failure; batches wait, bounded by the caps and the six-day expiry | | Calling `flushTelemetry` or `telemetryStats` before `enableTelemetry` | A no-op, and `null` |