Skip to content

Commit e87ab88

Browse files
committed
fix: caching issues in config signing
lint lint
1 parent e3770e7 commit e87ab88

11 files changed

Lines changed: 211 additions & 113 deletions

File tree

airborne_dashboard/components/settings/integrity/signing-keys-card.tsx

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -228,9 +228,7 @@ export function SigningKeysCard({ canRead, canCreate, canUpdate }: SigningKeysCa
228228
{!signingKey.is_default && !signingKey.disabled && <Badge variant="outline">Active</Badge>}
229229
</div>
230230
</TableCell>
231-
<TableCell className="text-muted-foreground">
232-
{formatDate(signingKey.created_at)}
233-
</TableCell>
231+
<TableCell className="text-muted-foreground">{formatDate(signingKey.created_at)}</TableCell>
234232
<TableCell className="text-right">
235233
<KeyActions
236234
signingKey={signingKey}

airborne_docs/docs/dashboard/integrity.mdx

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,10 @@ Airborne signs every [release config](/docs/guides/verify-the-release-config-sig
99

1010
Each key is an **ECDSA P-256** keypair. The **private key never leaves the server**: it cannot be downloaded and is never returned by the API. The **public key** is downloadable at any time — that is the half your apps verify with.
1111

12-
:::info[You already have a key]
13-
Every application is given a keypair with the key ID `default` the moment it is created, and existing applications were backfilled with one. Signing is on out of the box; there is nothing to switch on.
12+
:::info[New applications get a key automatically]
13+
Every application is given a keypair with the key ID `default` when it is created, so signing is on from the first release config it serves.
14+
15+
Applications that **predate signing** need a one-time backfill: add `signingkeys` to [`MIGRATIONS_TO_RUN_ON_BOOT`](/docs/server/configuration#boot-time-migrations) and restart the server. It is idempotent, so it is safe to leave enabled. Until it runs, those applications serve release configs with **no** signature header — never an error in itself, but a client that requires a signature will reject them.
1416
:::
1517

1618
## The key list
@@ -32,6 +34,8 @@ The page lists every key the application owns:
3234

3335
A new key is **not** the default — it signs nothing until you promote it. That is deliberate: it gives you a window to distribute the new public key to your apps before anything starts signing with it.
3436

37+
The exception is an application's **first** key, which becomes the default the moment it is created — an application with keys but no default would have nothing to sign with. You only land here if the application has no keys at all, which normally means it predates signing and has not been backfilled.
38+
3539
Key IDs must be unique within the application and are immutable. They must:
3640

3741
- contain only lowercase letters (`a-z`), digits (`0-9`), and dashes (`-`);
@@ -46,13 +50,15 @@ This is the file you ship to your apps. See [Verify the release config signature
4650

4751
## Set the default
4852

49-
**Set as default** promotes a key. From that moment, every release-config response that does not name a specific key is signed with it, and the change takes effect immediately — there is no cache to wait out.
53+
**Set as default** promotes a key. From that moment, every release-config response that does not name a specific key is signed with it.
54+
55+
Airborne drops its own cached signatures and invalidates the application's release route at the CDN, so promoting a key does not leave clients on the 24-hour edge cache — they pick up the new `keyid` as soon as the invalidation propagates, usually within a minute.
5056

5157
Exactly one key is the default at any time; promoting a key demotes the previous one.
5258

5359
## Enable and disable
5460

55-
**Disable** takes a key out of service. Airborne will refuse to sign with it, and any request that names it in `X-Signing-Key-Id` is rejected with a **400**. Re-enabling restores it.
61+
**Disable** takes a key out of service. Airborne will refuse to sign with it, and any request that names it in `X-Signing-Key-Id` is rejected with a **400**. Disabling also invalidates the cached release configs that key signed, so responses carrying its `keyid` stop being served from the edge. Re-enabling restores it.
5662

5763
Two rules keep you from locking your own apps out:
5864

airborne_docs/docs/guides/verify-the-release-config-signature.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ Omit the header and Airborne signs with the application's **default** key. To pi
123123
X-Signing-Key-Id: release-signing-2026
124124
```
125125

126-
An unknown or disabled key ID — or one belonging to a different application — is rejected with **400**.
126+
Sending the header with an **empty** value is the same as omitting it: the default key signs. A *non-empty* key ID that is unknown or disabled — or that belongs to a different application — is rejected with **400**.
127127

128128
Key IDs contain only lowercase letters (`a-z`), digits (`0-9`), and single dashes. They cannot start or end with a dash.
129129

airborne_docs/openapi/airborne.openapi.json

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2888,13 +2888,13 @@
28882888
{
28892889
"name": "x-signing-key-id",
28902890
"in": "header",
2891-
"description": "ID of the signing key to sign the response with. This is the readable ID chosen\nwhen the key was created. Optional — when omitted, the application's default\nsigning key is used. A key ID that is invalid, unknown, disabled, or belongs to\nanother application is rejected with a 400.",
2891+
"description": "ID of the signing key to sign the response with. This is the readable ID chosen\nwhen the key was created. Optional — when omitted, or sent with an empty value,\nthe application's default signing key is used. A non-empty key ID that is\ninvalid, unknown, disabled, or belongs to another application is rejected with\na 400.",
28922892
"schema": {
28932893
"type": "string",
28942894
"maxLength": 50,
28952895
"minLength": 1,
28962896
"pattern": "^[a-z0-9]+(-[a-z0-9]+)*$",
2897-
"description": "ID of the signing key to sign the response with. This is the readable ID chosen\nwhen the key was created. Optional — when omitted, the application's default\nsigning key is used. A key ID that is invalid, unknown, disabled, or belongs to\nanother application is rejected with a 400."
2897+
"description": "ID of the signing key to sign the response with. This is the readable ID chosen\nwhen the key was created. Optional — when omitted, or sent with an empty value,\nthe application's default signing key is used. A non-empty key ID that is\ninvalid, unknown, disabled, or belongs to another application is rejected with\na 400."
28982898
}
28992899
}
29002900
],
@@ -3012,13 +3012,13 @@
30123012
{
30133013
"name": "x-signing-key-id",
30143014
"in": "header",
3015-
"description": "ID of the signing key to sign the response with. This is the readable ID chosen\nwhen the key was created. Optional — when omitted, the application's default\nsigning key is used. A key ID that is invalid, unknown, disabled, or belongs to\nanother application is rejected with a 400.",
3015+
"description": "ID of the signing key to sign the response with. This is the readable ID chosen\nwhen the key was created. Optional — when omitted, or sent with an empty value,\nthe application's default signing key is used. A non-empty key ID that is\ninvalid, unknown, disabled, or belongs to another application is rejected with\na 400.",
30163016
"schema": {
30173017
"type": "string",
30183018
"maxLength": 50,
30193019
"minLength": 1,
30203020
"pattern": "^[a-z0-9]+(-[a-z0-9]+)*$",
3021-
"description": "ID of the signing key to sign the response with. This is the readable ID chosen\nwhen the key was created. Optional — when omitted, the application's default\nsigning key is used. A key ID that is invalid, unknown, disabled, or belongs to\nanother application is rejected with a 400."
3021+
"description": "ID of the signing key to sign the response with. This is the readable ID chosen\nwhen the key was created. Optional — when omitted, or sent with an empty value,\nthe application's default signing key is used. A non-empty key ID that is\ninvalid, unknown, disabled, or belongs to another application is rejected with\na 400."
30223022
}
30233023
}
30243024
],

airborne_server/src/organisation/application/properties.rs

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ use crate::{
1818
middleware::auth::{require_org_and_app, AuthResponse},
1919
organisation::application::properties::types::ConfigProperty,
2020
release::utils::parse_kv_string,
21-
types as airborne_types,
21+
signing, types as airborne_types,
2222
types::{ABError, AppState},
2323
utils::document::{
2424
document_to_json_value, dotted_docs_to_nested, hashmap_to_json_value,
@@ -287,16 +287,25 @@ async fn put_properties_schema_api(
287287
}
288288

289289
let metadata_for_rollback = task_metadata.clone();
290-
match transaction::run_fail_end(tasks, move |success_indices| async move {
291-
rollback_config_update(
292-
success_indices,
293-
metadata_for_rollback,
294-
&state.superposition_client,
295-
)
296-
.await;
290+
// Hand the rollback its own client rather than the whole `AppState`, so
291+
// `state` is still available to invalidate caches once the writes land.
292+
let client_for_rollback = state.superposition_client.clone();
293+
let outcome = transaction::run_fail_end(tasks, move |success_indices| async move {
294+
rollback_config_update(success_indices, metadata_for_rollback, &client_for_rollback).await;
297295
})
298-
.await
299-
{
296+
.await;
297+
298+
// `config.properties` is part of the release config the serve path signs, but
299+
// editing a property does not mint a new `config.version` — and that version is
300+
// what the signature cache is keyed on. Drop the cached signatures so the next
301+
// serve re-signs the body it actually returns instead of handing back a
302+
// signature computed over the old one.
303+
//
304+
// Done regardless of outcome: a rollback can itself fail, so a failed request
305+
// is not a guarantee that nothing changed.
306+
signing::utils::invalidate_signature_cache(&state, &organisation, &application).await;
307+
308+
match outcome {
300309
Ok(values) => info!("All good: {:?}", values),
301310
Err(e) => match e {
302311
transaction::TxnError::Operation { source, .. } => return Err(source),

airborne_server/src/release.rs

Lines changed: 8 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -526,17 +526,7 @@ async fn create_release(
526526

527527
let response_resources = final_resources.unwrap_or_default();
528528

529-
let path = format!("/release/{}/{}*", organisation.clone(), application.clone());
530-
531-
if let Err(e) = utils::invalidate_cf(
532-
&state.cf_client,
533-
path,
534-
&state.env.cloudfront_distribution_id,
535-
)
536-
.await
537-
{
538-
info!("Failed to invalidate CloudFront cache: {:?}", e);
539-
}
529+
utils::invalidate_release_cache(&state, &organisation, &application).await;
540530

541531
let now = Utc::now();
542532
let nested_config_props_result = dotted_docs_to_nested(config_properties.clone());
@@ -991,17 +981,7 @@ async fn ramp_release(
991981

992982
info!("Successfully ramped experiment {}", experiment_id);
993983

994-
let path = format!("/release/{}/{}*", organisation.clone(), application.clone());
995-
996-
if let Err(e) = utils::invalidate_cf(
997-
&state.cf_client,
998-
path,
999-
&state.env.cloudfront_distribution_id,
1000-
)
1001-
.await
1002-
{
1003-
info!("Failed to invalidate CloudFront cache: {:?}", e);
1004-
}
984+
utils::invalidate_release_cache(&state, &organisation, &application).await;
1005985

1006986
Ok(Json(RampReleaseResponse {
1007987
success: true,
@@ -1139,17 +1119,7 @@ async fn conclude_release(
11391119
experiment_id, transformed_variant_id
11401120
);
11411121

1142-
let path = format!("/release/{}/{}*", organisation.clone(), application.clone());
1143-
1144-
if let Err(e) = utils::invalidate_cf(
1145-
&state.cf_client,
1146-
path,
1147-
&state.env.cloudfront_distribution_id,
1148-
)
1149-
.await
1150-
{
1151-
info!("Failed to invalidate CloudFront cache: {:?}", e);
1152-
}
1122+
utils::invalidate_release_cache(&state, &organisation, &application).await;
11531123

11541124
Ok(Json(ConcludeReleaseResponse {
11551125
success: true,
@@ -1492,10 +1462,11 @@ async fn serve_release_handler(
14921462
ABError::InternalServerError(format!("Failed to serialize release config: {e}"))
14931463
})?;
14941464

1495-
let requested_key_id = req
1496-
.headers()
1497-
.get(signing::utils::SIGNING_KEY_ID_HEADER)
1498-
.and_then(|value| value.to_str().ok());
1465+
let requested_key_id = signing::utils::requested_key_id(
1466+
req.headers()
1467+
.get(signing::utils::SIGNING_KEY_ID_HEADER)
1468+
.and_then(|value| value.to_str().ok()),
1469+
);
14991470

15001471
// config.version is a UUID minted with the resolved release variant. The
15011472
// signing helper combines it with the selected key so repeat serves can

airborne_server/src/release/utils.rs

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -493,6 +493,21 @@ pub async fn invalidate_cf(
493493
Ok(())
494494
}
495495

496+
/// Best-effort: drop the CDN's cached release-config responses for an application.
497+
pub async fn invalidate_release_cache(state: &AppState, organisation: &str, application: &str) {
498+
let path = format!("/release/{organisation}/{application}*");
499+
500+
if let Err(e) = invalidate_cf(
501+
&state.cf_client,
502+
path,
503+
&state.env.cloudfront_distribution_id,
504+
)
505+
.await
506+
{
507+
info!("Failed to invalidate CloudFront cache: {:?}", e);
508+
}
509+
}
510+
496511
pub async fn check_non_concluded_releases(
497512
superposition_org_id: String,
498513
dims: HashMap<String, Value>,

airborne_server/src/signing.rs

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ use airborne_authz_macros::authz;
2929

3030
use crate::{
3131
middleware::auth::{require_org_and_app, AuthResponse},
32+
release::utils as release_utils,
3233
signing::types::{CreateSigningKeyRequest, SigningKeyResponse, UpdateSigningKeyRequest},
3334
types as airborne_types,
3435
types::{ABError, AppState, ListResponse, WithHeaders},
@@ -104,6 +105,13 @@ async fn create_signing_key(
104105
// "no default key" result has to go.
105106
utils::invalidate_key_cache(&state, &organisation, &application, &key.name).await;
106107

108+
// That same first key flips the application from serving unsigned configs to
109+
// signed ones, and the edge is still holding the unsigned responses. A key
110+
// that did not become the default changes nothing about what is served.
111+
if key.is_default {
112+
release_utils::invalidate_release_cache(&state, &organisation, &application).await;
113+
}
114+
107115
Ok(WithHeaders::new(Json(SigningKeyResponse::from(key))).status(StatusCode::CREATED))
108116
}
109117

@@ -161,18 +169,26 @@ async fn update_signing_key(
161169
)?;
162170

163171
let key_id = utils::validate_key_id(&path.into_inner())?;
172+
let disabled = req.into_inner().disabled;
164173

165174
let key = utils::set_key_disabled(
166175
state.db_pool.clone(),
167176
organisation.clone(),
168177
application.clone(),
169178
key_id,
170-
req.into_inner().disabled,
179+
disabled,
171180
)
172181
.await?;
173182

174183
utils::invalidate_key_cache(&state, &organisation, &application, &key.name).await;
175184

185+
// Disabling a key has to stop its signatures from being served, and the edge
186+
// is still holding responses it signed. Enabling one invalidates nothing:
187+
// no cached response becomes wrong by a key becoming usable again.
188+
if disabled {
189+
release_utils::invalidate_release_cache(&state, &organisation, &application).await;
190+
}
191+
176192
Ok(Json(SigningKeyResponse::from(key)))
177193
}
178194

@@ -206,5 +222,9 @@ async fn set_default_signing_key(
206222

207223
utils::invalidate_key_cache(&state, &organisation, &application, &key.name).await;
208224

225+
// Every release config cached at the edge was signed by the key this one just
226+
// replaced, so it has to be re-fetched to pick up the new `keyid`.
227+
release_utils::invalidate_release_cache(&state, &organisation, &application).await;
228+
209229
Ok(Json(SigningKeyResponse::from(key)))
210230
}

0 commit comments

Comments
 (0)