Skip to content

Commit 80a5598

Browse files
authored
Merge pull request #31 from BotCoder254/feat/log-chunk-sections
Add detected requirements for secrets, vars, and environments
2 parents a5e3383 + 9f296fd commit 80a5598

21 files changed

Lines changed: 888 additions & 53 deletions

‎CLAUDE.md‎

Lines changed: 45 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,34 @@ catalog (summary strip + URL-synced filters + keyset infinite scroll + detail pa
177177
audit history) plus a create/replace dialog that never echoes values and a security
178178
posture card that warns when `SECRETS_MASTER_KEY` is unset.
179179

180+
**Detected requirements (sync-time reference discovery).** Workflow parsing records what
181+
each workflow *expects* as structured metadata in `workflows.metadata` JSONB — no
182+
migration: `secretRefs` (`${{ secrets.NAME }}`, dot AND bracket `secrets['NAME']` forms,
183+
word-boundary checked so `mysecrets.` never matches, `GITHUB_TOKEN` excluded),
184+
`varRefs` (`${{ vars.NAME }}` — detect/display only; the platform doesn't manage plain
185+
variables), and `environments` (distinct job `environment:` names, case-insensitively
186+
deduped, slug-filtered so dynamic `${{ … }}` names are never stored). All scanners share
187+
`workflow_parse::scan_context_refs` (identifier `[A-Za-z_][A-Za-z0-9_]*` ≤200 B, dedupe,
188+
cap 100, sort). Two read endpoints compute "referenced but not configured" as a set
189+
difference at QUERY time (`db/workflows.rs::missing_secret_refs/workspace_var_refs/
190+
missing_environment_refs` — `jsonb_array_elements_text` LATERAL with a `jsonb_typeof`
191+
guard so pre-feature/failed-parse metadata reads as empty; anti-join semantics: a secret
192+
name counts as configured via workspace scope, same-repo repository scope, or any
193+
environment scope): `GET …/secrets/requirements` (`secrets.read`) and
194+
`GET …/environments/requirements` (`content.read`). Handlers re-filter names at read
195+
time against the configurable-name allow-lists (defense-in-depth against hand-edited
196+
metadata) and cap output (200 names, 20 refs/name, true `referenceCount`). YAML never
197+
mints rows — the UI offers one-click create through the ordinary RBAC'd endpoints:
198+
`DetectedRequirementsCard` (secrets page right rail; "Add" pre-fills `SecretFormDialog`
199+
via `presetName`/`presetRepository`, repo scope only when every reference shares one
200+
repo) and `DetectedEnvironmentsCard` (environments page; "Create" pre-fills
201+
`EnvironmentFormDialog` via `presetName`). Both requirements queries live under their
202+
feature's react-query prefix, so every mutation's existing invalidation clears satisfied
203+
warnings automatically. The environment detail page lists `boundWorkflows` (from
204+
`list_binding_environment`, case-insensitive) linking to the workflow detail pages.
205+
Detection refreshes on every repo sync; workflows synced before this feature light up
206+
after their next push/manual re-sync.
207+
180208
**Notification Center (operational inbox).** Notifications are a per-user, actionable
181209
PROJECTION of the immutable `audit_logs` ledger — the Activity Feed keeps the complete
182210
history, notifications hold only what a user should act on (OWASP's audit-vs-messaging
@@ -265,14 +293,18 @@ overup/
265293
│ │ ├── pipelines/ # execution ledger + live detail workspace: PipelineGraph,
266294
│ │ │ # ExecutionTimeline, LogViewer (xterm), tab panels,
267295
│ │ │ # usePipelineStream (WS), stores/logStore (zustand)
268-
│ │ ├── artifacts/ # workspace artifact catalog: summary strip, URL-synced
269-
│ │ │ # filters, keyset infinite scroll, provenance detail page
296+
│ │ ├── artifacts/ # workspace artifact catalog: summary strip (4 KPI cells —
297+
│ │ │ # the house norm), URL-synced filters, keyset infinite
298+
│ │ │ # scroll, provenance detail page
270299
│ │ ├── secrets/ # write-only encrypted secrets: catalog + posture column,
271-
│ │ │ # create/replace dialog (value never echoed), detail
272-
│ │ │ # page with audit history
300+
│ │ │ # detected-requirements card (missing secretRefs +
301+
│ │ │ # vars, one-click Add), create/replace dialog (value
302+
│ │ │ # never echoed), detail page with audit history
273303
│ │ ├── environments/ # deployment environments: catalog (summary strip +
274-
│ │ │ # URL-synced search + keyset infinite scroll), detail
275-
│ │ │ # page with scoped secrets + audit, create/edit dialog
304+
│ │ │ # URL-synced search + keyset infinite scroll +
305+
│ │ │ # detected-environments card w/ one-click create),
306+
│ │ │ # detail page with scoped secrets + bound workflows +
307+
│ │ │ # audit, create/edit dialog
276308
│ │ ├── notifications/ # operational inbox: NotificationBell (badge, popover/
277309
│ │ │ # bottom-sheet switch), NotificationPanel, history page,
278310
│ │ │ # PreferencesDialog, useNotificationStream (per-user WS)
@@ -558,6 +590,13 @@ into a traversal-safe tar streamed into the container via the Docker archive API
558590
`environment.created/updated/deleted` audit rows in-transaction, and a delete cascade
559591
audits every removed secret. A secret create with an `environmentId` from another
560592
workspace gets the same flat error as a bad `repositoryId` — no existence oracle
593+
- **Detected requirements are read-only projections of parser output**: sync-time
594+
scanners only store allow-listed identifiers (charset + length + caps) into
595+
`workflows.metadata`; the requirements endpoints re-filter names at read time against
596+
the same configurable-name rules, cap output (200 names / 20 refs each), ride the
597+
sibling read permissions (`secrets.read` / `content.read`), and expose names only —
598+
never values. YAML never mints secrets or environments: "Add"/"Create" go through the
599+
ordinary RBAC'd mutation endpoints
561600
- **Notifications are self-scoped by construction**: every read/mutation predicate pins
562601
`user_id = caller` in SQL (a forged notification id 404s flat), the live hub is keyed by
563602
the AUTHENTICATED user id (routing, not filtering, is the isolation), titles/bodies are

‎backend/src/db/workflows.rs‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,140 @@ pub struct PushRunnableWorkflow {
228228
pub raw_content: String,
229229
}
230230

231+
/// One workflow-YAML reference to a secret/var/environment name, with the
232+
/// repository and workflow it came from. Names only — never values.
233+
#[derive(Debug, sqlx::FromRow)]
234+
pub struct RequirementRefRow {
235+
pub name: String,
236+
pub repository_id: Uuid,
237+
pub repository_name: String,
238+
pub workflow_id: Uuid,
239+
pub workflow_path: String,
240+
}
241+
242+
/// CTE expanding one metadata ref array across a workspace's workflows.
243+
/// `key` is always a compile-time literal (`secretRefs`/`varRefs`/
244+
/// `environments`) — never user input; every user-facing value stays a bind.
245+
/// The `jsonb_typeof` guard makes workflows synced before a key existed (and
246+
/// failed parses, whose metadata is `{}`) read as empty rather than erroring.
247+
fn refs_cte(key: &str) -> String {
248+
format!(
249+
r#"
250+
WITH refs AS (
251+
SELECT ref.name AS name,
252+
r.id AS repository_id,
253+
r.full_name AS repository_name,
254+
w.id AS workflow_id,
255+
w.path AS workflow_path
256+
FROM workflows w
257+
JOIN repositories r ON r.id = w.repository_id
258+
CROSS JOIN LATERAL jsonb_array_elements_text(
259+
CASE WHEN jsonb_typeof(w.metadata->'{key}') = 'array'
260+
THEN w.metadata->'{key}' ELSE '[]'::jsonb END
261+
) AS ref(name)
262+
WHERE r.workspace_id = $1
263+
)
264+
"#
265+
)
266+
}
267+
268+
/// Secret names referenced by workflow YAML with no configured secret that
269+
/// could satisfy them: a name counts as configured when a workspace-scoped
270+
/// secret, a repository-scoped secret on the referencing repo, or an
271+
/// environment-scoped secret in any environment carries it (which environment
272+
/// applies is a dispatch-time question — "any environment" keeps the rule
273+
/// simple and matches how precedence is explained in the UI).
274+
pub async fn missing_secret_refs(
275+
pool: &PgPool,
276+
workspace_id: Uuid,
277+
) -> sqlx::Result<Vec<RequirementRefRow>> {
278+
let sql = refs_cte("secretRefs")
279+
+ r#"
280+
SELECT name, repository_id, repository_name, workflow_id, workflow_path
281+
FROM refs
282+
WHERE NOT EXISTS (
283+
SELECT 1 FROM secrets s
284+
WHERE s.workspace_id = $1
285+
AND s.name = refs.name
286+
AND (
287+
(s.repository_id IS NULL AND s.environment_id IS NULL)
288+
OR s.repository_id = refs.repository_id
289+
OR s.environment_id IS NOT NULL
290+
)
291+
)
292+
ORDER BY name, repository_name, workflow_path
293+
LIMIT 1000
294+
"#;
295+
sqlx::query_as::<_, RequirementRefRow>(&sql)
296+
.bind(workspace_id)
297+
.fetch_all(pool)
298+
.await
299+
}
300+
301+
/// Every `${{ vars.NAME }}` reference across the workspace — informational
302+
/// only (the platform doesn't manage plain variables), so no anti-join.
303+
pub async fn workspace_var_refs(
304+
pool: &PgPool,
305+
workspace_id: Uuid,
306+
) -> sqlx::Result<Vec<RequirementRefRow>> {
307+
let sql = refs_cte("varRefs")
308+
+ r#"
309+
SELECT name, repository_id, repository_name, workflow_id, workflow_path
310+
FROM refs
311+
ORDER BY name, repository_name, workflow_path
312+
LIMIT 1000
313+
"#;
314+
sqlx::query_as::<_, RequirementRefRow>(&sql)
315+
.bind(workspace_id)
316+
.fetch_all(pool)
317+
.await
318+
}
319+
320+
/// Environment names bound by workflow YAML with no matching environment row
321+
/// (case-insensitive, like dispatch resolution).
322+
pub async fn missing_environment_refs(
323+
pool: &PgPool,
324+
workspace_id: Uuid,
325+
) -> sqlx::Result<Vec<RequirementRefRow>> {
326+
let sql = refs_cte("environments")
327+
+ r#"
328+
SELECT name, repository_id, repository_name, workflow_id, workflow_path
329+
FROM refs
330+
WHERE NOT EXISTS (
331+
SELECT 1 FROM environments e
332+
WHERE e.workspace_id = $1 AND lower(e.name) = lower(refs.name)
333+
)
334+
ORDER BY name, repository_name, workflow_path
335+
LIMIT 1000
336+
"#;
337+
sqlx::query_as::<_, RequirementRefRow>(&sql)
338+
.bind(workspace_id)
339+
.fetch_all(pool)
340+
.await
341+
}
342+
343+
/// Workflows whose YAML binds the given environment name (case-insensitive)
344+
/// — feeds the environment detail page's "bound workflows" list.
345+
pub async fn list_binding_environment(
346+
pool: &PgPool,
347+
workspace_id: Uuid,
348+
name: &str,
349+
) -> sqlx::Result<Vec<RequirementRefRow>> {
350+
let sql = refs_cte("environments")
351+
+ r#"
352+
SELECT name, repository_id, repository_name, workflow_id, workflow_path
353+
FROM refs
354+
WHERE lower(refs.name) = lower($2)
355+
ORDER BY repository_name, workflow_path
356+
LIMIT 50
357+
"#;
358+
sqlx::query_as::<_, RequirementRefRow>(&sql)
359+
.bind(workspace_id)
360+
.bind(name)
361+
.fetch_all(pool)
362+
.await
363+
}
364+
231365
pub async fn list_jobs(pool: &PgPool, workflow_id: Uuid) -> sqlx::Result<Vec<WorkflowJobRow>> {
232366
sqlx::query_as::<_, WorkflowJobRow>(
233367
r#"

‎backend/src/handlers/environments.rs‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,30 @@ pub async fn summary(
142142
})))
143143
}
144144

145+
/// GET /api/workspaces/{workspace_id}/environments/requirements
146+
///
147+
/// Environment names bound by workflow YAML (`environment:`) with no
148+
/// matching environment row — detected at sync time, computed as a set
149+
/// difference at query time. Deliberately never auto-created: YAML must not
150+
/// mint workspace resources past RBAC; the UI offers a one-click create
151+
/// through the ordinary RBAC'd endpoint instead.
152+
pub async fn requirements(
153+
State(state): State<AppState>,
154+
CurrentUser(user): CurrentUser,
155+
Path(workspace_id): Path<Uuid>,
156+
) -> AppResult<Json<serde_json::Value>> {
157+
authz::require_permission(&state.pool, user.id, workspace_id, authz::CONTENT_READ).await?;
158+
159+
let missing = db::workflows::missing_environment_refs(&state.pool, workspace_id).await?;
160+
Ok(Json(json!({
161+
// Read-time re-filter (the secrets requirements pattern): only names
162+
// an environment row could actually take surface as creatable.
163+
"environments": super::secrets::group_requirements(missing, |name| {
164+
validate_name(name).is_ok()
165+
}),
166+
})))
167+
}
168+
145169
#[derive(Debug, Deserialize)]
146170
#[serde(rename_all = "camelCase")]
147171
pub struct AuditQuery {
@@ -175,9 +199,24 @@ pub async fn detail(
175199
.ok_or(AppError::NotFound)?;
176200
let audit =
177201
db::environments::list_audit(&state.pool, workspace_id, Some(environment_id), 20).await?;
202+
// Workflows whose YAML binds this environment (case-insensitive, like
203+
// dispatch resolution) — sync-time metadata, names/paths only.
204+
let bound = db::workflows::list_binding_environment(&state.pool, workspace_id, &meta.name)
205+
.await?
206+
.into_iter()
207+
.map(|row| {
208+
json!({
209+
"repositoryId": row.repository_id,
210+
"repositoryName": row.repository_name,
211+
"workflowId": row.workflow_id,
212+
"workflowPath": row.workflow_path,
213+
})
214+
})
215+
.collect::<Vec<_>>();
178216
Ok(Json(json!({
179217
"environment": EnvironmentResponse::from(meta),
180218
"audit": audit_events(audit),
219+
"boundWorkflows": bound,
181220
})))
182221
}
183222

‎backend/src/handlers/secrets.rs‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,80 @@ pub async fn summary(
217217
})))
218218
}
219219

220+
/// A well-formed `${{ … }}` reference identifier (`[A-Za-z_][A-Za-z0-9_]*`,
221+
/// ≤200 bytes). The parser only writes such names, but requirements are
222+
/// re-filtered at read time so legacy or hand-edited metadata can never
223+
/// smuggle arbitrary strings into a response.
224+
pub(crate) fn valid_ref_ident(name: &str) -> bool {
225+
let mut chars = name.chars();
226+
matches!(chars.next(), Some(c) if c.is_ascii_alphabetic() || c == '_')
227+
&& name.len() <= 200
228+
&& chars.all(|c| c.is_ascii_alphanumeric() || c == '_')
229+
}
230+
231+
/// Fold requirement rows into per-name entries: names sorted (BTreeMap),
232+
/// capped at 200 names and 20 references each — `referenceCount` carries the
233+
/// true total. `keep` is the read-time allow-list filter.
234+
pub(crate) fn group_requirements(
235+
rows: Vec<db::workflows::RequirementRefRow>,
236+
keep: impl Fn(&str) -> bool,
237+
) -> Vec<serde_json::Value> {
238+
const MAX_NAMES: usize = 200;
239+
const MAX_REFS_PER_NAME: usize = 20;
240+
241+
let mut grouped: std::collections::BTreeMap<String, (i64, Vec<serde_json::Value>)> =
242+
std::collections::BTreeMap::new();
243+
for row in rows {
244+
if !keep(&row.name) {
245+
continue;
246+
}
247+
if !grouped.contains_key(&row.name) && grouped.len() >= MAX_NAMES {
248+
continue;
249+
}
250+
let entry = grouped.entry(row.name).or_default();
251+
entry.0 += 1;
252+
if entry.1.len() < MAX_REFS_PER_NAME {
253+
entry.1.push(json!({
254+
"repositoryId": row.repository_id,
255+
"repositoryName": row.repository_name,
256+
"workflowId": row.workflow_id,
257+
"workflowPath": row.workflow_path,
258+
}));
259+
}
260+
}
261+
grouped
262+
.into_iter()
263+
.map(|(name, (count, references))| {
264+
json!({ "name": name, "referenceCount": count, "references": references })
265+
})
266+
.collect()
267+
}
268+
269+
/// GET /api/workspaces/{workspace_id}/secrets/requirements
270+
///
271+
/// Workflow-declared requirements detected at sync time: secret names the
272+
/// YAML references (`${{ secrets.X }}`) with no configured secret that could
273+
/// satisfy them, plus `${{ vars.X }}` references (informational — the
274+
/// platform doesn't manage plain variables). Names only, never values.
275+
pub async fn requirements(
276+
State(state): State<AppState>,
277+
CurrentUser(user): CurrentUser,
278+
Path(workspace_id): Path<Uuid>,
279+
) -> AppResult<Json<serde_json::Value>> {
280+
authz::require_permission(&state.pool, user.id, workspace_id, authz::SECRETS_READ).await?;
281+
282+
let missing = db::workflows::missing_secret_refs(&state.pool, workspace_id).await?;
283+
let vars = db::workflows::workspace_var_refs(&state.pool, workspace_id).await?;
284+
285+
Ok(Json(json!({
286+
// Only names that could actually become overup secrets surface as
287+
// missing — a lowercase or GITHUB_*-reserved ref is unconfigurable
288+
// here (GitHub folds case; we don't) and would be a dead-end button.
289+
"secrets": group_requirements(missing, |name| validate_name(name).is_ok()),
290+
"vars": group_requirements(vars, valid_ref_ident),
291+
})))
292+
}
293+
220294
#[derive(Debug, Deserialize)]
221295
#[serde(rename_all = "camelCase")]
222296
pub struct AuditQuery {

‎backend/src/routes/mod.rs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -324,6 +324,10 @@ pub fn build_router(state: AppState) -> anyhow::Result<Router> {
324324
"/workspaces/{workspace_id}/secrets/summary",
325325
get(secrets::summary),
326326
)
327+
.route(
328+
"/workspaces/{workspace_id}/secrets/requirements",
329+
get(secrets::requirements),
330+
)
327331
.route(
328332
"/workspaces/{workspace_id}/secrets/audit",
329333
get(secrets::audit),
@@ -344,6 +348,10 @@ pub fn build_router(state: AppState) -> anyhow::Result<Router> {
344348
"/workspaces/{workspace_id}/environments/summary",
345349
get(environments::summary),
346350
)
351+
.route(
352+
"/workspaces/{workspace_id}/environments/requirements",
353+
get(environments::requirements),
354+
)
347355
.route(
348356
"/workspaces/{workspace_id}/environments/audit",
349357
get(environments::audit),

0 commit comments

Comments
 (0)