Release 3.8.1 (#107)

Outbound redaction searches only the text written in a request; thinkwatch-core v0.74.0.

Author: fylorn

CHANGELOG.md

@@ -11,6 +11,56 @@ target.
 
 ## [Unreleased]
 
+## [3.8.1] — 2026-10-13
+
+Outbound redaction searches only the text written in a request, located
+by the structure of the request's format, and no longer changes ids or
+other values the protocol needs as they are. The thinkwatch-core crates
+move from v0.73.0 to v0.74.0.
+
+### Read before upgrading
+
+- **Only the written text of a request is searched.** Fields outside it,
+  such as `metadata`, `user`, tool parameter schemas, response format
+  schemas and stop sequences, and blocks or fields the gateway does not
+  recognise, are no longer searched: a value there goes upstream as
+  written and is not reported. Hits inside ids and other protocol values
+  are no longer reported either, so fewer redaction events
+  (`gateway.redaction_replaced`, `gateway.redaction_flagged`) may be
+  written than before.
+- **Stored bodies are masked as before.** With `audit.body_redact_pii` on,
+  stored request and response bodies are still searched over their whole
+  text, fields outside the written text included, and so is the answer a
+  log row stores under a content rule that checks answers.
+
+### Fixed
+
+- **Redaction no longer changes ids or protocol values.** Before a request
+  left, the built-in rules searched the whole body, ids included, and
+  custom rules skipped only some identifiers, by key name. With outbound
+  redaction On and the Chinese mainland mobile number rule
+  (`cn-mobile-phone`, off by default) enabled, a Codex reasoning item id
+  (`rs_…`) holding a run of 11 digits was taken for a mobile number and
+  replaced with a placeholder. The upstream refused the request, and since
+  Codex sends its reasoning items back on every turn, every later turn of
+  that conversation was refused as well. Under Monitor the id was reported
+  as a hit.
+
+  The rules now search only the text written in the request: system
+  prompts, the messages of every role (earlier answers and text documents
+  included), reasoning text and summaries, tool-call arguments and inputs,
+  tool results and tool descriptions, as well as Chat Completions'
+  predicted output and the code a Gemini model executed and its output.
+  Ids, types, roles, names, the model, signatures, encrypted content,
+  media and settings are never searched. Each hop is searched in the
+  format it is sent in: the caller's when the request is forwarded, the
+  upstream's when it is converted.
+
+### Changed
+
+- thinkwatch-core crates (tw-bedrock, tw-breaker, tw-dialect, tw-guard)
+  v0.73.0 → v0.74.0.
+
 ## [3.8.0] — 2026-10-13
 
 The security page is organised by what each guard protects against: Secrets
@@ -2355,7 +2405,8 @@ unreleased builds should: stop the gateway, run `db/schema.sql`
 against PostgreSQL, restart against this tag. The schema is
 idempotent end-to-end, so the apply is safe to repeat.
 
-[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v3.8.0...HEAD
+[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v3.8.1...HEAD
+[3.8.1]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.8.1
 [3.8.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.8.0
 [3.7.1]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.7.1
 [3.7.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.7.0

Cargo.lock

@@ -10,7 +10,7 @@ members = [
 ]
 
 [workspace.package]
-version = "3.8.0"
+version = "3.8.1"
 edition = "2024"
 # The MSRV: the newest `rust-version` among the locked dependencies
 # (sqlx 0.9 and the aws-smithy crates under aws-sigv4). Without it,
@@ -70,10 +70,10 @@ opt-level = 3
 # never re-exported through a local shim. And the reverse: something only
 # this side uses (the at-rest crypto, IMDSv2 credentials, the gateway error)
 # lives here, not in core.
-tw-bedrock = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.73.0" }
-tw-breaker = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.73.0" }
-tw-dialect = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.73.0" }
-tw-guard = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.73.0" }
+tw-bedrock = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.74.0" }
+tw-breaker = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.74.0" }
+tw-dialect = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.74.0" }
+tw-guard = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.74.0" }
 
 # Web framework
 axum = { version = "0.8", features = ["macros", "ws"] }

Cargo.toml

@@ -24,12 +24,15 @@ use tw_guard::redact::rules::RuleSet;
 /// built-in rule's id or the custom rule's name. Nothing is kept to restore
 /// them: the result is write-only audit data.
 ///
-/// Found the way a request body is searched (`tw_guard::redact::flow::hits`):
-/// JSON escapes are read, a custom rule's match stays inside one JSON
-/// string so the result is still JSON, base64 payloads are left alone,
-/// and so are the gateway's own placeholders (a captured answer still
-/// carries them). The one base64 payload that is read is a compaction
-/// summary a conversion carries (see [`redact_carried_summaries`]).
+/// Found over the whole text (`tw_guard::redact::flow::hits`), not only
+/// the text written in a request the way a request on the wire is
+/// searched: what is stored is never sent, and a captured body may be an
+/// answer, a stream or a tool result. JSON escapes are read, a custom
+/// rule's match stays inside one JSON string so the result is still JSON,
+/// object keys and base64 payloads are left alone, and so are the
+/// gateway's own placeholders (a captured answer still carries them).
+/// The one base64 payload that is read is a compaction summary a
+/// conversion carries (see [`redact_carried_summaries`]).
 pub fn redact_blob(rules: &RuleSet, input: &str) -> String {
     if rules.is_empty() {
         return input.to_string();

crates/common/src/pii.rs

@@ -1490,7 +1490,10 @@ mod tests {
             {"role": "user", "content": "my email is alice@corp.example.org"}
         ]})
         .to_string();
-        let (_, ledger) = redaction.look(request.as_bytes());
+        let (_, ledger) = redaction.look(
+            request.as_bytes(),
+            crate::redaction::Shape::Generate(Dialect::Chat),
+        );
         let content = crate::content_filter::ContentFilter::new(&ContentPolicy {
             mode: Mode::Enforce,
             custom: vec![CustomContentRule {

crates/gateway/src/lifecycle/mod.rs

@@ -26,10 +26,10 @@
 //! The content filter reads the caller's text where the request's own
 //! format puts it, and a rule that deletes or replaces text does so there:
 //! the request goes on as the changed one, decoded again. Outbound redaction
-//! then searches the whole request, numbers what it finds once, and every
-//! hop — forwarded or converted — goes out through it (see
-//! `crate::redaction`). Both are thinkwatch-core's, shared with the
-//! desktop gateway.
+//! then searches the text written in the request, numbers what it finds
+//! once, and every hop — forwarded or converted — goes out through it,
+//! searched as the format it is sent in (see `crate::redaction`). Both
+//! are thinkwatch-core's, shared with the desktop gateway.
 
 use std::convert::Infallible;
 
@@ -61,7 +61,7 @@ use crate::guards::Caller;
 use crate::lifecycle::Completed;
 use crate::metadata::RequestMetadata;
 use crate::protocol::UpstreamProtocol;
-use crate::redaction::Redaction;
+use crate::redaction::{Redaction, Shape};
 use crate::router::RouteEntry;
 use tw_guard::redact::replace::Ledger;
 
@@ -446,7 +446,11 @@ impl Outbound {
             if let Some(stripped) = tw_dialect::convert::strip_carried(client, &bytes) {
                 bytes = stripped;
             }
-            let (bytes, ledger) = self.redaction.replace(bytes, &self.ledger);
+            // Sent in the caller's own format: its written text is where
+            // the caller's format puts it.
+            let (bytes, ledger) =
+                self.redaction
+                    .replace(bytes, &self.ledger, Shape::Generate(client));
             return Ok(Wire {
                 body: bytes,
                 path,
@@ -479,8 +483,11 @@ impl Outbound {
             body = serde_json::to_vec(&v).unwrap_or(body);
         }
         // The conversion moved the placeholders along with the text; one it
-        // assembled from two pieces is numbered here.
-        let (body, ledger) = self.redaction.replace(body, &self.ledger);
+        // assembled from two pieces is numbered here. The text is searched
+        // where the upstream's format puts it.
+        let (body, ledger) =
+            self.redaction
+                .replace(body, &self.ledger, Shape::Generate(protocol.dialect()));
         // With none of the caller's own, any breakpoints are the ones the
         // conversion added for Claude.
         let auto_cache = decoded.request.cache.is_empty()
@@ -763,21 +770,24 @@ async fn run(
     let decoded =
         tw_dialect::convert::decode(surface.dialect, &raw, path, internal_query(surface.dialect));
 
-    // 5. Outbound redaction: the whole request is searched and what is
-    //    found numbered once, in the order the caller wrote it.
+    // 5. Outbound redaction: the text written in the request — located by
+    //    the structure of the caller's format, never its ids, types or
+    //    names — is searched and what is found numbered once, in the order
+    //    the caller wrote it.
     //    Placeholders are stable per value, so two callers sending the same
     //    structure redact to the same bytes and share a cache slot; each
     //    restores their own values on the way out.
     //
     //    The audit row keeps what the caller wrote (their at-rest
     //    redaction setting applies on top).
-    let (findings, ledger) = guards.redaction.look(&body);
+    let shape = Shape::Generate(surface.dialect);
+    let (findings, ledger) = guards.redaction.look(&body, shape);
     crate::redaction::record(&state.audit, &caller, &guards.redaction, &findings);
     let request_for_audit = body.to_vec();
     let outbound_body = if ledger.is_empty() {
         raw
     } else {
-        let (replaced, _) = guards.redaction.replace(body.to_vec(), &ledger);
+        let (replaced, _) = guards.redaction.replace(body.to_vec(), &ledger, shape);
         serde_json::from_slice(&replaced).map_err(|_| {
             ctx.emit(GatewayError::TransformError(
                 "The request could not be read after redaction.".into(),
@@ -1227,7 +1237,7 @@ mod tests {
             mode: tw_guard::policy::Mode::Off,
             ..Default::default()
         });
-        let (_, ledger) = redaction.look(b"{}");
+        let (_, ledger) = redaction.look(b"{}", Shape::Generate(surface.dialect));
         Outbound {
             surface,
             path: path.into(),

crates/gateway/src/proxy/generate.rs

@@ -326,15 +326,21 @@ fn raw(f: &Frame) -> String {
 mod tests {
     use super::*;
 
+    /// The test requests are written as chat completions.
+    const CHAT: crate::redaction::Shape = crate::redaction::Shape::Generate(Dialect::Chat);
+
     /// A ledger that issued `<<TW_EMAIL_1>>` for `a@x.com`, or nothing.
     fn ctx(email: Option<&str>) -> Ledger {
         let r = crate::redaction::Redaction::new(&tw_guard::policy::RedactPolicy {
             mode: tw_guard::policy::Mode::Enforce,
             enable: vec!["email".into()],
             ..Default::default()
         });
-        let body = serde_json::json!({"text": email.unwrap_or("")}).to_string();
-        let ledger = r.look(body.as_bytes()).1;
+        let body = serde_json::json!({"messages": [
+            {"role": "user", "content": email.unwrap_or("")}
+        ]})
+        .to_string();
+        let ledger = r.look(body.as_bytes(), CHAT).1;
         assert_eq!(ledger.len(), usize::from(email.is_some()));
         ledger
     }
@@ -551,8 +557,8 @@ mod tests {
             mode: tw_guard::policy::Mode::Enforce,
             ..Default::default()
         });
-        let body = serde_json::json!({"text": PEM}).to_string();
-        let ledger = r.look(body.as_bytes()).1;
+        let body = serde_json::json!({"messages": [{"role": "user", "content": PEM}]}).to_string();
+        let ledger = r.look(body.as_bytes(), CHAT).1;
         assert_eq!(ledger.len(), 1);
         ledger
     }