Skip to content

Commit 1bd9e51

Browse files
authored
Merge pull request #33 from BotCoder254/feat/log-chunk-sections
Resolve secrets/vars expressions at dispatch + full detection state in the UI
2 parents 8e2f492 + e045958 commit 1bd9e51

17 files changed

Lines changed: 619 additions & 235 deletions

File tree

‎CLAUDE.md‎

Lines changed: 41 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -186,24 +186,47 @@ variables), and `environments` (distinct job `environment:` names, case-insensit
186186
deduped, slug-filtered so dynamic `${{ … }}` names are never stored). All scanners share
187187
`workflow_parse::scan_context_refs` (identifier `[A-Za-z_][A-Za-z0-9_]*` ≤200 B, dedupe,
188188
cap 100, sort). Two read endpoints compute "referenced but not configured" as a set
189-
difference at QUERY time (`db/workflows.rs::missing_secret_refs/workspace_var_refs/
190-
missing_environment_refs` — `jsonb_array_elements_text` LATERAL with a `jsonb_typeof`
191-
guard so pre-feature/failed-parse metadata reads as empty; anti-join semantics: a secret
192-
name counts as configured via workspace scope, same-repo repository scope, or any
193-
environment scope): `GET …/secrets/requirements` (`secrets.read`) and
194-
`GET …/environments/requirements` (`content.read`). Handlers re-filter names at read
195-
time against the configurable-name allow-lists (defense-in-depth against hand-edited
196-
metadata) and cap output (200 names, 20 refs/name, true `referenceCount`). YAML never
197-
mints rows — the UI offers one-click create through the ordinary RBAC'd endpoints:
198-
`DetectedRequirementsCard` (secrets page right rail; "Add" pre-fills `SecretFormDialog`
199-
via `presetName`/`presetRepository`, repo scope only when every reference shares one
200-
repo) and `DetectedEnvironmentsCard` (environments page; "Create" pre-fills
201-
`EnvironmentFormDialog` via `presetName`). Both requirements queries live under their
202-
feature's react-query prefix, so every mutation's existing invalidation clears satisfied
203-
warnings automatically. The environment detail page lists `boundWorkflows` (from
204-
`list_binding_environment`, case-insensitive) linking to the workflow detail pages.
205-
Detection refreshes on every repo sync; workflows synced before this feature light up
206-
after their next push/manual re-sync.
189+
difference at QUERY time (`db/workflows.rs::secret_ref_states/workspace_var_refs/
190+
environment_ref_states` — `jsonb_array_elements_text` LATERAL with a `jsonb_typeof`
191+
guard so pre-feature/failed-parse metadata reads as empty; configured semantics: a
192+
secret name counts as configured via workspace scope, same-repo repository scope, or
193+
any environment scope, and the LATERAL picks the highest-precedence match's id):
194+
`GET …/secrets/requirements` (`secrets.read`) and `GET …/environments/requirements`
195+
(`content.read`) return EVERY detected (name, repository) pair with
196+
`configured`/`configuredId`, not just missing ones. Handlers re-filter names at read
197+
time against charset allow-lists (defense-in-depth against hand-edited metadata;
198+
deliberately looser than the configurable-name rule so reserved-prefix refs like
199+
DOCKER_TOKEN stay visible — the client marks them unconfigurable via the shared
200+
`features/secrets/lib/secretNameRules.ts`) and cap output (200 entries, 20 refs each,
201+
true `referenceCount`). YAML never mints rows — the repo-grouped "Detected in
202+
workflows" cards (`DetectedRequirementsCard` on the secrets page,
203+
`DetectedEnvironmentsCard` on the environments page) show Configured entries as
204+
success badges linking to the detail pages and missing ones with one-click Add/Create
205+
through the ordinary RBAC'd endpoints (dialog `presetName`/`presetRepository`). Both
206+
requirements queries live under their feature's react-query prefix, so every
207+
mutation's existing invalidation updates the cards automatically. The workflow detail
208+
Metadata tab surfaces `secretRefs`/`varRefs`/`environments` (with a re-sync hint when
209+
the stored parse predates detection), the environment detail page lists
210+
`boundWorkflows` (from `list_binding_environment`, case-insensitive), and the pipeline
211+
Environment tab shows the job's `plan.environment` binding. Detection refreshes on
212+
every repo sync; `workflow_parse::PARSER_VERSION` is stamped into metadata and the
213+
sync skip-condition re-parses stored workflows once after any parser bump (bump it
214+
whenever parse output changes, or new metadata never reaches already-synced
215+
workflows).
216+
217+
**Dispatch-time expression resolution.** `workflow_parse::substitute_context_refs`
218+
resolves GitHub-style `${{ secrets.NAME }}` / `${{ vars.NAME }}` blocks (dot or
219+
bracket form; the trimmed inner expression must be exactly one such ref — compound
220+
expressions and other contexts pass through untouched, it is a substitutor, not an
221+
evaluator). `scheduler::dispatch` applies it to plan env VALUES and step `run`
222+
strings after decrypting the job's secrets: known secrets resolve to their value,
223+
unknown secrets and all vars resolve to "" (GitHub's unset semantics; no variables
224+
store yet). Substitution happens in dispatch memory only — stored plans keep the
225+
literals, so API responses never carry resolved values and reruns re-substitute with
226+
current secrets. Mask registration runs after substitution, and secret plaintexts are
227+
always masks, so a substituted-into-run secret still masks in logs. Secrets also
228+
continue to inject as env vars under their own names (precedence: plan env <
229+
workspace < repository < environment).
207230

208231
**Notification Center (operational inbox).** Notifications are a per-user, actionable
209232
PROJECTION of the immutable `audit_logs` ledger — the Activity Feed keeps the complete

‎backend/src/db/workflows.rs‎

Lines changed: 35 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -243,14 +243,16 @@ pub struct PushRunnableWorkflow {
243243
}
244244

245245
/// One workflow-YAML reference to a secret/var/environment name, with the
246-
/// repository and workflow it came from. Names only — never values.
246+
/// repository and workflow it came from and — when something satisfies it —
247+
/// the id of the configured row. Names and ids only, never values.
247248
#[derive(Debug, sqlx::FromRow)]
248249
pub struct RequirementRefRow {
249250
pub name: String,
250251
pub repository_id: Uuid,
251252
pub repository_name: String,
252253
pub workflow_id: Uuid,
253254
pub workflow_path: String,
255+
pub configured_id: Option<Uuid>,
254256
}
255257

256258
/// CTE expanding one metadata ref array across a workspace's workflows.
@@ -279,31 +281,37 @@ fn refs_cte(key: &str) -> String {
279281
)
280282
}
281283

282-
/// Secret names referenced by workflow YAML with no configured secret that
283-
/// could satisfy them: a name counts as configured when a workspace-scoped
284-
/// secret, a repository-scoped secret on the referencing repo, or an
285-
/// environment-scoped secret in any environment carries it (which environment
286-
/// applies is a dispatch-time question — "any environment" keeps the rule
287-
/// simple and matches how precedence is explained in the UI).
288-
pub async fn missing_secret_refs(
284+
/// Every secret name referenced by workflow YAML, with the id of the secret
285+
/// that would satisfy it when one exists: a name counts as configured for a
286+
/// referencing repo when a workspace-scoped secret, a repository-scoped
287+
/// secret on that repo, or an environment-scoped secret in any environment
288+
/// carries it (which environment applies is a dispatch-time question). The
289+
/// LATERAL picks the highest-precedence match so the UI can link straight
290+
/// to it.
291+
pub async fn secret_ref_states(
289292
pool: &PgPool,
290293
workspace_id: Uuid,
291294
) -> sqlx::Result<Vec<RequirementRefRow>> {
292295
let sql = refs_cte("secretRefs")
293296
+ r#"
294-
SELECT name, repository_id, repository_name, workflow_id, workflow_path
297+
SELECT refs.name, refs.repository_id, refs.repository_name,
298+
refs.workflow_id, refs.workflow_path, s.id AS configured_id
295299
FROM refs
296-
WHERE NOT EXISTS (
297-
SELECT 1 FROM secrets s
300+
LEFT JOIN LATERAL (
301+
SELECT s.id
302+
FROM secrets s
298303
WHERE s.workspace_id = $1
299304
AND s.name = refs.name
300305
AND (
301306
(s.repository_id IS NULL AND s.environment_id IS NULL)
302307
OR s.repository_id = refs.repository_id
303308
OR s.environment_id IS NOT NULL
304309
)
305-
)
306-
ORDER BY name, repository_name, workflow_path
310+
ORDER BY (s.environment_id IS NOT NULL) DESC,
311+
(s.repository_id IS NOT NULL) DESC
312+
LIMIT 1
313+
) s ON TRUE
314+
ORDER BY refs.name, refs.repository_name, refs.workflow_path
307315
LIMIT 1000
308316
"#;
309317
sqlx::query_as::<_, RequirementRefRow>(&sql)
@@ -313,14 +321,15 @@ pub async fn missing_secret_refs(
313321
}
314322

315323
/// Every `${{ vars.NAME }}` reference across the workspace — informational
316-
/// only (the platform doesn't manage plain variables), so no anti-join.
324+
/// only (the platform doesn't manage plain variables), never configured.
317325
pub async fn workspace_var_refs(
318326
pool: &PgPool,
319327
workspace_id: Uuid,
320328
) -> sqlx::Result<Vec<RequirementRefRow>> {
321329
let sql = refs_cte("varRefs")
322330
+ r#"
323-
SELECT name, repository_id, repository_name, workflow_id, workflow_path
331+
SELECT name, repository_id, repository_name, workflow_id, workflow_path,
332+
NULL::uuid AS configured_id
324333
FROM refs
325334
ORDER BY name, repository_name, workflow_path
326335
LIMIT 1000
@@ -331,21 +340,21 @@ pub async fn workspace_var_refs(
331340
.await
332341
}
333342

334-
/// Environment names bound by workflow YAML with no matching environment row
335-
/// (case-insensitive, like dispatch resolution).
336-
pub async fn missing_environment_refs(
343+
/// Every environment name bound by workflow YAML, with the id of the
344+
/// matching environment row when it exists (case-insensitive, like dispatch
345+
/// resolution).
346+
pub async fn environment_ref_states(
337347
pool: &PgPool,
338348
workspace_id: Uuid,
339349
) -> sqlx::Result<Vec<RequirementRefRow>> {
340350
let sql = refs_cte("environments")
341351
+ r#"
342-
SELECT name, repository_id, repository_name, workflow_id, workflow_path
352+
SELECT refs.name, refs.repository_id, refs.repository_name,
353+
refs.workflow_id, refs.workflow_path, e.id AS configured_id
343354
FROM refs
344-
WHERE NOT EXISTS (
345-
SELECT 1 FROM environments e
346-
WHERE e.workspace_id = $1 AND lower(e.name) = lower(refs.name)
347-
)
348-
ORDER BY name, repository_name, workflow_path
355+
LEFT JOIN environments e
356+
ON e.workspace_id = $1 AND lower(e.name) = lower(refs.name)
357+
ORDER BY refs.name, refs.repository_name, refs.workflow_path
349358
LIMIT 1000
350359
"#;
351360
sqlx::query_as::<_, RequirementRefRow>(&sql)
@@ -363,7 +372,8 @@ pub async fn list_binding_environment(
363372
) -> sqlx::Result<Vec<RequirementRefRow>> {
364373
let sql = refs_cte("environments")
365374
+ r#"
366-
SELECT name, repository_id, repository_name, workflow_id, workflow_path
375+
SELECT name, repository_id, repository_name, workflow_id, workflow_path,
376+
NULL::uuid AS configured_id
367377
FROM refs
368378
WHERE lower(refs.name) = lower($2)
369379
ORDER BY repository_name, workflow_path

‎backend/src/handlers/environments.rs‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -144,23 +144,24 @@ pub async fn summary(
144144

145145
/// GET /api/workspaces/{workspace_id}/environments/requirements
146146
///
147-
/// Environment names bound by workflow YAML (`environment:`) with no
148-
/// matching environment row — detected at sync time, computed as a set
149-
/// difference at query time. Deliberately never auto-created: YAML must not
150-
/// mint workspace resources past RBAC; the UI offers a one-click create
151-
/// through the ordinary RBAC'd endpoint instead.
147+
/// Every environment name bound by workflow YAML (`environment:`), detected
148+
/// at sync time, with its configured state — `configuredId` is the matching
149+
/// environment row's id when one exists. Missing names are deliberately
150+
/// never auto-created: YAML must not mint workspace resources past RBAC;
151+
/// the UI offers a one-click create through the ordinary RBAC'd endpoint
152+
/// instead.
152153
pub async fn requirements(
153154
State(state): State<AppState>,
154155
CurrentUser(user): CurrentUser,
155156
Path(workspace_id): Path<Uuid>,
156157
) -> AppResult<Json<serde_json::Value>> {
157158
authz::require_permission(&state.pool, user.id, workspace_id, authz::CONTENT_READ).await?;
158159

159-
let missing = db::workflows::missing_environment_refs(&state.pool, workspace_id).await?;
160+
let refs = db::workflows::environment_ref_states(&state.pool, workspace_id).await?;
160161
Ok(Json(json!({
161162
// Read-time re-filter (the secrets requirements pattern): only names
162-
// an environment row could actually take surface as creatable.
163-
"environments": super::secrets::group_requirements(missing, |name| {
163+
// an environment row could actually take surface.
164+
"environments": super::secrets::group_requirements(refs, |name| {
164165
validate_name(name).is_ok()
165166
}),
166167
})))

‎backend/src/handlers/secrets.rs‎

Lines changed: 51 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -228,65 +228,91 @@ pub(crate) fn valid_ref_ident(name: &str) -> bool {
228228
&& chars.all(|c| c.is_ascii_alphanumeric() || c == '_')
229229
}
230230

231-
/// Fold requirement rows into per-name entries: names sorted (BTreeMap),
232-
/// capped at 200 names and 20 references each — `referenceCount` carries the
233-
/// true total. `keep` is the read-time allow-list filter.
231+
/// Fold requirement rows into per-(repository, name) entries — the shape the
232+
/// repo-grouped detection cards render directly. Sorted by repository then
233+
/// name (BTreeMap key), capped at 200 entries and 20 workflow references
234+
/// each with `referenceCount` carrying the true total. `keep` is the
235+
/// read-time charset allow-list. `configuredId` is the covering secret's or
236+
/// matching environment's id (null when missing). Names and ids only, never
237+
/// values.
234238
pub(crate) fn group_requirements(
235239
rows: Vec<db::workflows::RequirementRefRow>,
236240
keep: impl Fn(&str) -> bool,
237241
) -> Vec<serde_json::Value> {
238-
const MAX_NAMES: usize = 200;
239-
const MAX_REFS_PER_NAME: usize = 20;
240-
241-
let mut grouped: std::collections::BTreeMap<String, (i64, Vec<serde_json::Value>)> =
242+
const MAX_ENTRIES: usize = 200;
243+
const MAX_REFS_PER_ENTRY: usize = 20;
244+
245+
struct Entry {
246+
repository_id: Uuid,
247+
configured_id: Option<Uuid>,
248+
count: i64,
249+
references: Vec<serde_json::Value>,
250+
}
251+
let mut grouped: std::collections::BTreeMap<(String, String), Entry> =
242252
std::collections::BTreeMap::new();
243253
for row in rows {
244254
if !keep(&row.name) {
245255
continue;
246256
}
247-
if !grouped.contains_key(&row.name) && grouped.len() >= MAX_NAMES {
257+
let key = (row.repository_name.clone(), row.name.clone());
258+
if !grouped.contains_key(&key) && grouped.len() >= MAX_ENTRIES {
248259
continue;
249260
}
250-
let entry = grouped.entry(row.name).or_default();
251-
entry.0 += 1;
252-
if entry.1.len() < MAX_REFS_PER_NAME {
253-
entry.1.push(json!({
254-
"repositoryId": row.repository_id,
255-
"repositoryName": row.repository_name,
261+
let entry = grouped.entry(key).or_insert_with(|| Entry {
262+
repository_id: row.repository_id,
263+
configured_id: None,
264+
count: 0,
265+
references: Vec::new(),
266+
});
267+
entry.count += 1;
268+
if entry.configured_id.is_none() {
269+
entry.configured_id = row.configured_id;
270+
}
271+
if entry.references.len() < MAX_REFS_PER_ENTRY {
272+
entry.references.push(json!({
256273
"workflowId": row.workflow_id,
257274
"workflowPath": row.workflow_path,
258275
}));
259276
}
260277
}
261278
grouped
262279
.into_iter()
263-
.map(|(name, (count, references))| {
264-
json!({ "name": name, "referenceCount": count, "references": references })
280+
.map(|((repository_name, name), entry)| {
281+
json!({
282+
"name": name,
283+
"repositoryId": entry.repository_id,
284+
"repositoryName": repository_name,
285+
"configured": entry.configured_id.is_some(),
286+
"configuredId": entry.configured_id,
287+
"referenceCount": entry.count,
288+
"references": entry.references,
289+
})
265290
})
266291
.collect()
267292
}
268293

269294
/// GET /api/workspaces/{workspace_id}/secrets/requirements
270295
///
271-
/// Workflow-declared requirements detected at sync time: secret names the
272-
/// YAML references (`${{ secrets.X }}`) with no configured secret that could
273-
/// satisfy them, plus `${{ vars.X }}` references (informational — the
274-
/// platform doesn't manage plain variables). Names only, never values.
296+
/// Workflow-declared requirements detected at sync time: every secret name
297+
/// the YAML references (`${{ secrets.X }}`) with its configured state and,
298+
/// when satisfied, the id of the covering secret; plus `${{ vars.X }}`
299+
/// references (informational — the platform doesn't manage plain
300+
/// variables). The charset filter is deliberately loose (`valid_ref_ident`,
301+
/// not the configurable-name rule) so reserved-prefix refs like
302+
/// DOCKER_TOKEN stay VISIBLE — the client marks them unconfigurable instead
303+
/// of silently hiding them. Names and ids only, never values.
275304
pub async fn requirements(
276305
State(state): State<AppState>,
277306
CurrentUser(user): CurrentUser,
278307
Path(workspace_id): Path<Uuid>,
279308
) -> AppResult<Json<serde_json::Value>> {
280309
authz::require_permission(&state.pool, user.id, workspace_id, authz::SECRETS_READ).await?;
281310

282-
let missing = db::workflows::missing_secret_refs(&state.pool, workspace_id).await?;
311+
let refs = db::workflows::secret_ref_states(&state.pool, workspace_id).await?;
283312
let vars = db::workflows::workspace_var_refs(&state.pool, workspace_id).await?;
284313

285314
Ok(Json(json!({
286-
// Only names that could actually become overup secrets surface as
287-
// missing — a lowercase or GITHUB_*-reserved ref is unconfigurable
288-
// here (GitHub folds case; we don't) and would be a dead-end button.
289-
"secrets": group_requirements(missing, |name| validate_name(name).is_ok()),
315+
"secrets": group_requirements(refs, valid_ref_ident),
290316
"vars": group_requirements(vars, valid_ref_ident),
291317
})))
292318
}

0 commit comments

Comments
 (0)