fix(auth): give a key configured without a key_id the same id on every load

A key whose configuration carries no key_id -- every key from
AGENTFLOW_API_KEYS, and a key-file entry without one -- drew a random id
from generate_key_id on every load. A key-file entry kept its id only when
load() could write it back, and docker-compose.prod.yml mounts the key file
read-only, over a config/api_keys.yaml whose two entries carry no key_id.
The id names the key's Redis bucket, its api_usage rows and the admin views
keyed by id: replicas sharing Redis each kept their own bucket for the same
key (N replicas, N times its rpm), and every restart and reload started a
fresh bucket and split its usage history.

ensure_key_ids now derives the id -- <tenant>-<name>-<8 hex of the entry's
peppered lookup digest>, from its stored key_lookup or compute_key_lookup
over its plaintext -- and _legacy_env_keys runs environment keys through
it. A clash lengthens the digest prefix, so ids stay unique and stable. A
new pepper changes only an id derived from plaintext and never written back
-- every environment key, and a plaintext entry of a key file the process
cannot write; an entry with a stored key_lookup, and a derived id already
written to a writable key file, keep theirs. A legacy hash-only entry has
nothing to derive from and keeps a random id; generate_key_id and
create_key are unchanged.

tests/unit/test_auth_key_identity.py takes each scenario of
docs/specs/api-key-identity.md (nine of its twelve tests fail against the
code before this change), with false-reject controls that load the shipped
key file and a persisted id beside a hash-only entry. The new key_rotation
helpers are killed from test_key_rotation_mutation.py, the file the
mutation gate runs; its one test that pinned a random id for entries with a
plaintext key now builds hash-only entries, the case that keeps one. The
CHANGELOG's ten LF-only lines are now CRLF like the rest of the file.

docs/architecture.md and the reload requirement of
docs/specs/api-key-rate-limiting.md now name the one key a reload still
cannot keep: a legacy hash-only entry in a key file the process cannot
write, whose random id -- and so its bucket and window -- changes on every
load. docs/specs/api-key-identity.md already specified that id.

Reviewed by opus-cli, the writer's own engine: glm-5.3 was rate-limited
until 2026-09-14 and codex until 2026-10-05.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHUBaNNPWihtsgeFWnRmaS

Author: JuliaEdom

CHANGELOG.md

@@ -4,7 +4,28 @@ All notable changes to AgentFlow are documented in this file.
 
 ## [Unreleased]
 
-### Fixed — reloading the key store no longer resets rate-limit windows (T-46)
+### Fixed — a key configured without a key_id keeps the same id on every load
+
+A key whose configuration carries no `key_id` — every key from
+`AGENTFLOW_API_KEYS`, and a key-file entry without one — drew a random id on
+every `AuthManager.load()`. A key-file entry kept its id only when the id could
+be written back, and `docker-compose.prod.yml` mounts the key file read-only.
+The id names the key's Redis bucket (`kid:<key_id>`), its `api_usage` rows and
+the admin views keyed by id, so replicas sharing Redis each kept their own
+bucket for the same key — N replicas, N times its rpm — and every restart and
+reload started a fresh bucket and split the key's usage history. Such a key now
+gets `<tenant>-<name>-<first 8 hex of its key-lookup digest>`: the stored
+`key_lookup`, else the peppered `compute_key_lookup` of its plaintext, never
+the plaintext or an unpeppered hash of it. The same key gets the same id on
+every load, restart and replica that shares the pepper. Changing the pepper
+changes only an id derived from a plaintext key and never written back: every
+environment key, and a plaintext key-file entry in a file the process cannot
+write. A writable key file keeps the id written to it on the first load, and an
+entry with a stored `key_lookup` keeps the id derived from that digest. An id
+already taken lengthens the digest prefix. A legacy hash-only entry, with
+neither a plaintext key nor a `key_lookup`, still gets a random id.
+
+### Fixed — reloading the key store no longer resets rate-limit windows
 
 `AuthManager.load()` carried the in-memory rate-limit windows over by the
 plaintext key index (`keys_by_value`), but since audit S-6 every window is
@@ -14,9 +35,12 @@ emptied them all. `is_rate_limited()` and the in-memory secondary check in
 `check_rate_limit()` then handed every tenant a fresh budget, which during a
 Redis outage is the whole limit. Windows are now carried over by bucket name:
 a still-configured key keeps its window, a removed key loses it, and no
-plaintext key names a window at any point of the reload. Keys from
-`AGENTFLOW_API_KEYS` still get a new random `key_id` on every load, so their
-windows keep resetting until T-47.
+plaintext key names a window at any point of the reload. A window survives
+only while its key keeps its `key_id`, which a key configured without one —
+from `AGENTFLOW_API_KEYS`, or in a key file the process cannot write — now does
+too (entry above), with one exception: a legacy hash-only entry, with neither a
+plaintext key nor a `key_lookup`, in a key file the process cannot write still
+gets a new id, and so an emptied window, on every reload.
 
 ### Docs — the pre-push hedges outlived the push
 
@@ -605,16 +629,16 @@ applies `--ignore` only for `safety_id` values whose waiver scope matches that
 bucket. Expired waivers stop suppressing findings. A waiver whose scope is not
 scanned, or a duplicate `safety_id`, fails closed.
 
-### Documentation — root records archived (2026-09-02–2026-09-04)
-
-Twenty-one immutable tracked Markdown records moved from the repository root
-to `docs/evidence/records/` with unchanged filenames and SHA-256 digests.
-Living citations now use those paths. After all eight documentation-cleanup
-items closed, `plan_26_08_2026.md` moved to `docs/archive/plans/` with its
-closure evidence intact. The retired local 2026-04-17 benchmark baseline moved
-from the standalone `docs/benchmark-baseline-archive/` directory into the
-performance archive.
-
+### Documentation — root records archived (2026-09-02–2026-09-04)
+
+Twenty-one immutable tracked Markdown records moved from the repository root
+to `docs/evidence/records/` with unchanged filenames and SHA-256 digests.
+Living citations now use those paths. After all eight documentation-cleanup
+items closed, `plan_26_08_2026.md` moved to `docs/archive/plans/` with its
+closure evidence intact. The retired local 2026-04-17 benchmark baseline moved
+from the standalone `docs/benchmark-baseline-archive/` directory into the
+performance archive.
+
 ### Security — nltk 3.10.0 -> 3.10.3 in uv.lock (Dependabot GHSA-m4rf-3fr8-xwx3, GHSA-6hwm-xvph-95vm)
 
 - `uv lock --upgrade-package nltk` only; nltk is a transitive dependency of `llama-index-core` and is not part of the `cloud`/`postgres` export, so `requirements-docker.lock` is unchanged. Closes the critical (JVM argument injection in the Stanford wrappers) and high (uncontrolled `dot` search path) advisories GitHub reported on the default branch on 2026-09-01.

docs/architecture.md

@@ -134,7 +134,7 @@ See [Architecture Decision Records](decisions/) for detailed trade-off analysis.
 ## Security
 
 ### Implemented
-- **API authentication**: API key via `X-API-Key` header (set `AGENTFLOW_API_KEYS` env var). Requirements for a key's id: [API key identity](specs/api-key-identity.md)
+- **API authentication**: API key via `X-API-Key` header (set `AGENTFLOW_API_KEYS` env var). A key configured without a `key_id` (every environment key, and a key-file entry without one) gets an id derived from its key-lookup digest, stable across reloads, restarts and replicas (except a legacy hash-only entry, with neither a plaintext key nor a `key_lookup`: it has nothing to derive from and keeps a random id, which in a key file the process cannot write changes on every load); changing the pepper changes only an id derived from a plaintext key and never written back (every environment key, and a plaintext key-file entry in a file the process cannot write), while a writable key file keeps the id written to it on the first load and an entry with a stored `key_lookup` keeps the id derived from that digest. Requirements for a key's id: [API key identity](specs/api-key-identity.md)
 - **Transport gate (audit P2-3)**: `AGENTFLOW_PROFILE=production` refuses to boot over plaintext transport to an external ClickHouse/Redis/PostgreSQL (loopback exempt; deliberate exceptions named in `AGENTFLOW_INSECURE_TRANSPORT_OK`), and refuses a wildcard CORS origin outside demo mode. The ClickHouse client supports HTTPS with hostname verification and a private-CA bundle (`CLICKHOUSE_SECURE`, `CLICKHOUSE_CA_CERT`)
 - **Rate limiting**: Per-key sliding window with Redis backing when available and in-memory fallback for local/test, configurable via `AGENTFLOW_RATE_LIMIT_RPM` (default: 120/min). Requirements: [API key rate limiting](specs/api-key-rate-limiting.md)
 - **Health/docs exempt**: `/v1/health`, `/docs`, `/metrics` don't require auth

docs/specs/api-key-rate-limiting.md

@@ -20,7 +20,10 @@ id, and no bucket name SHALL contain a plaintext API key.
 Reloading the key configuration — `load()`, a SIGHUP reload, or the reload that
 ends every key create, rotate and revoke — SHALL keep the in-memory rate-limit
 window of every key that is still configured, and SHALL drop the window of a
-key the reload removed.
+key the reload removed. A key whose id changes on every load, a legacy
+hash-only entry in a key file the process cannot write (see
+[API key identity](api-key-identity.md)), gets a new bucket on every reload, so
+its window starts empty.
 
 ### Scenario: a full window survives a reload
 - **GIVEN** a key with `rate_limit_rpm: 1` that has already made its one request in the current window

src/agentflow_runtime/serving/api/auth/key_rotation.py

@@ -4,6 +4,7 @@
 import re
 import secrets
 import threading
+from collections.abc import Collection
 from datetime import UTC, datetime, timedelta
 
 try:
@@ -22,6 +23,42 @@
     is_permission_denied,
 )
 
+# A derived key_id ends in this many characters of the key's lookup digest: the
+# same shape as the 8 hex characters `generate_key_id` draws at random.
+KEY_ID_DIGEST_CHARS = 8
+
+
+def key_id_slug(value: str, fallback: str) -> str:
+    return "-".join(re.findall(r"[a-z0-9]+", value.lower())) or fallback
+
+
+def derived_key_id(item: TenantKey, existing_ids: Collection[str]) -> str | None:
+    """A stable key_id for an entry configured without one, or None.
+
+    The suffix is a prefix of the entry's peppered lookup digest -- its stored
+    `key_lookup`, else `compute_key_lookup` over its plaintext key -- so the
+    same key gets the same id on every load, restart and replica that shares
+    the pepper. Never the plaintext or an unpeppered hash of it: the id lands in
+    logs, Redis key names, usage rows and admin responses, and must not let a
+    guessed key be confirmed offline (audit FB-07).
+
+    An id already taken lengthens the prefix, so the later entry of a clashing
+    pair still gets the same id on every load. None when there is nothing to
+    derive from (a legacy hash-only entry) or the whole digest is taken.
+    """
+    if item.key_lookup is not None:
+        lookup = item.key_lookup
+    elif item.key is not None:
+        lookup = compute_key_lookup(item.key)
+    else:
+        return None
+    stem = f"{key_id_slug(item.tenant, 'tenant')}-{key_id_slug(item.name, 'agent')}-"
+    for size in range(KEY_ID_DIGEST_CHARS, len(lookup) + 1):
+        candidate = stem + lookup[:size]
+        if candidate not in existing_ids:
+            return candidate
+    return None
+
 
 class KeyRotator:
     def __init__(self, manager: AuthManager) -> None:
@@ -274,8 +311,8 @@ def generate_key_id(
         name: str,
         existing_ids: set[str] | None = None,
     ) -> str:
-        tenant_slug = re.sub(r"[^a-z0-9]+", "-", tenant.lower()).strip("-") or "tenant"
-        name_slug = re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-") or "agent"
+        tenant_slug = key_id_slug(tenant, "tenant")
+        name_slug = key_id_slug(name, "agent")
         seen_ids = set(existing_ids or ())
         while True:
             candidate = f"{tenant_slug}-{name_slug}-{secrets.token_hex(4)}"
@@ -288,7 +325,13 @@ def ensure_key_ids(self, config: ApiKeysConfig) -> bool:
         for index, item in enumerate(config.keys):
             if item.key_id is not None:
                 continue
-            key_id = self.generate_key_id(item.tenant, item.name, existing_ids)
+            # Derived, not drawn: an id that never reaches the file (a read-only
+            # key store, or AGENTFLOW_API_KEYS) must come out the same on the next
+            # load and in every replica. Only a legacy hash-only entry, with
+            # nothing to derive from, still gets a random one.
+            key_id = derived_key_id(item, existing_ids) or self.generate_key_id(
+                item.tenant, item.name, existing_ids
+            )
             existing_ids.add(key_id)
             config.keys[index] = item.model_copy(update={"key_id": key_id})
             changed = True

src/agentflow_runtime/serving/api/auth/manager.py

@@ -330,7 +330,7 @@ def load(self) -> None:
                     self._key_rotator.schedule_rotation_cleanup(item)
             # Carry windows over by bucket name (`_rate_limit_key`), never by the
             # plaintext `keys_by_value` index: a still-configured key keeps its
-            # window across the reload, a removed key's window is dropped. (T-46)
+            # window across the reload, a removed key's window is dropped.
             live_buckets = {self._rate_limit_key(item) for item in config.keys}
             self._rate_windows = defaultdict(
                 list,
@@ -704,7 +704,6 @@ def _legacy_env_keys(self) -> list[TenantKey]:
                 key, name = pair, "unnamed"
             items.append(
                 TenantKey(
-                    key_id=self._key_rotator.generate_key_id("default", name.strip(), set()),
                     key=key.strip(),
                     name=name.strip(),
                     tenant="default",
@@ -713,7 +712,12 @@ def _legacy_env_keys(self) -> list[TenantKey]:
                     created_at=datetime.now(UTC).date(),
                 )
             )
-        return items
+        # The same derivation as an id-less key-file entry: from the key's
+        # lookup digest, so the key keeps its id -- and its rate-limit bucket --
+        # across reloads, restarts and replicas.
+        config = ApiKeysConfig(keys=items)
+        self._key_rotator.ensure_key_ids(config)
+        return config.keys
 
     def _rate_limit_key(self, tenant_key: TenantKey) -> str:
         # This string becomes a Redis sorted-set key NAME (``rate_limiter`` calls