This document defines what must live in platform core so that future packages (e.g., SRE Investigator, MCP adapters, repo tooling, etc.) can be installed and run without touching core code.
Design goal: core provides governance + runtime primitives. Packages provide domain logic (workers, connectors, workflows, UIs).
Protocol: CAP v2 is the canonical wire contract for bus and safety messages.
Core knows only:
- jobs, workflows, state, pointers, config, policy, audit
- scheduling, retries, timeouts, DLQ
- approvals, budgets, and constraints
Core must not know:
- Kubernetes, GitHub/GitLab, Datadog/Coralogix, Sentry, LLM providers
- “incident”, “PR”, “runbook”, “patch generation” semantics
- tool-specific topics or behavior
A package is an overlay on the platform:
- adds topics, workers, workflow templates, and config/policy overlays
- uses core APIs/contracts exactly as-is
- never requires core code changes for “new product logic”
(without these, packages will be hacks)
Status: Implemented using CAP v2 (github.com/cordum-io/cap/v2) with aliases in core/protocol/pb/v1.
Core must define and enforce:
- BusPacket (CAP envelope)
trace_id, sender_id, created_at, protocol_versionpayloadoneof includesJobRequest,JobResult,Heartbeat,JobProgress,JobCancel,SystemAlert
- Pointers
ContextPointer / ResultPointer / ArtifactPointer(store references, not big blobs)
- Control events
JobCancel + Heartbeat + JobProgress
- DLQ format
error_code + error_message + last_state + attempts(stored in DLQ entries)
Why packages need it: every worker and external integration becomes predictable, replayable, auditable.
Hard rule: version the envelope (protocol_version) so packages don’t break as the platform evolves.
- A package defines new
topics (e.g.,job.sre.collect.k8s,job.sre.patch.generate). - Workers subscribe to those topics and always speak
BusPacket{JobRequest/JobResult}. - Workers write outputs to
ResultPointerorArtifactPointer. - Core doesn’t need to “know” the meaning of the outputs.
Status: Implemented in core/workflow and cmd/cordum-workflow-engine (binary cordum-workflow-engine).
Core workflow engine must support vanilla steps that don’t require packages:
approval(human gate)delay(timer)condition(evaluated expression, boolean output)notify(emits a SystemAlert on the bus)worker(dispatches a job to a topic/pool)for_each(fan-out over array items; optionalmax_parallelthrottle)depends_onDAG dependencies (steps run when all deps succeed; independent steps run in parallel)
Required properties:
- durable run state (crash/restart safe)
- step retries with backoff + max attempts
- timeouts per step
- cancel propagation (run cancel stops running steps)
- full run timeline (inputs/outputs pointers, status transitions)
- schema validation for workflow input and step input/output
- rerun-from-step and dry-run mode
- dependency gating: failed/cancelled/timed-out deps block downstream steps (no implicit continue-on-error)
Why packages need it: packages are just workflows + workers. If the workflow engine isn’t bulletproof, the “Incident→PR” product will be unreliable.
- A package ships workflow templates that contain
workersteps pointing to the package’s topics. - Core executes the same state machine regardless of what the steps mean.
- If the package is uninstalled, those topics simply become unmapped and will DLQ.
Status: Implemented; routing comes from config/pools.yaml (topics + pool capabilities).
Core scheduler must do only:
- topic → pool mapping (from config)
- leasing/dispatch semantics
- timeouts, retries, DLQ
- pool backpressure (overload detection)
If no mapping exists: fail fast to DLQ with no_pool_mapping.
Why packages need it: installing a package becomes “add mapping + deploy workers”, not “change scheduler code”.
- Package installation adds config overlays:
pools.overlay.yaml(topic → pool)timeouts.overlay.yaml(step/job timeouts)
- Workers come online in that pool.
- Scheduler behavior remains unchanged.
- Handshake handling: The scheduler subscribes to
sys.handshakeand processesBusPacket{Handshake}messages. Worker-role handshakes update the in-memory worker registry with component capabilities, enabling capability-aware routing. - ErrorCode enum: Job failures now carry a structured
error_code_enum(ErrorCodeenum) alongside the deprecated stringerror_code. The scheduler auto-populateserror_code_enumfrom the string code when only the string is provided (e.g.,"timeout"maps toERROR_CODE_JOB_TIMEOUT). DLQ entries also carry the structured code. - Bus-layer validation: Incoming
JobRequestandJobResultpackets are validated using CAP SDK helpers (ValidateJobRequest/ValidateJobResult). Invalid packets are rejected, logged, and counted via thevalidation_rejections_totalmetric. - Enhanced SystemAlert: Alerts emitted by the workflow engine now include
severity(enum),source_component,details(map), andtrace_idalongside the deprecated string fields.
The scheduler and gateway gained submit-time and dispatch-time
boundary enforcement against unknown topics, ad-hoc payload shapes,
and unattested workers. All four mechanisms are mode-gated
(enforce / warn / off) so existing deployments degrade
gracefully:
- Topic Registry submit-time validation —
core/topics/registry.gois the canonical source of truth. Gateway rejects unknown topics with HTTP 400 (unknown_topicerror code) before publishing tosys.job.submit; scheduler does the same check before dispatch as a defense-in-depth boundary. Known topics with zero workers stay valid (degradedErrNoWorkersretry). - Schema enforcement —
SCHEMA_ENFORCEMENT=enforce|warn|off(defaultwarn). Job payloads validated against the topic's input schema via JSON Schema draft-07. Pack manifestinputSchemaandoutputSchemaregister at install time. - Worker attestation —
WORKER_ATTESTATION=enforce|warn|off(defaultoff). Scheduler verifiesHeartbeat.auth_tokenagainst the gateway's argon2id-hashed worker credential store. Cache is refresh-on-miss with merge-on-failure (prevents stale-cache starvation). - Worker readiness gating —
WORKER_READINESS_REQUIRED=true|false(defaultfalse). Scheduler dispatch picks only workers whoseready == trueand whoseHandshake.ready_topicsinclude the job's topic. Unknown workers (no handshake observed yet) are allowed to avoid starving net-new fleets — absence ≠ not ready. - Dispatch-time delegation re-verify —
core/auth/delegation.gore-verifies theJobRequest's delegation token signature, scope, expiry, and revocation status at dispatch time, NOT just at submit time. Closes the TOCTOU window between gateway-side accept and scheduler-side dispatch.
See docs/AGENT_PROTOCOL.md § "CAP v2.9.0 changes" for the
wire-level fields, docs/configuration-reference.md § "Gateway +
Scheduler — Boundary Hardening" for the full env-var matrix, and
docs/adr/009-control-plane-boundary-hardening.md for the design
rationale.
Status: Implemented; gRPC Check/Evaluate/Explain/Simulate with snapshotting.
Core kernel must evaluate a request and return:
ALLOWDENYREQUIRE_APPROVALALLOW_WITH_CONSTRAINTS
(rewrite budgets, sandbox, command allowlist, redaction level)- Optional remediations that suggest safer alternatives (topic/capability/label tweaks).
Policy management (P0 minimum):
- policy bundles loaded from file/URL + config service fragments (
cfg:system:policybundles) - config-service bundles may include metadata (
author,message, timestamps) and anenabledtoggle; admin overlays live under thesecops/prefix - signed + hot reload to new
PolicySnapshot(version, hash) - last-known-good fallback if verification fails
- decision audit record for every request:
{rule_id, version, decision, constraints, reason}
Safety kernel config-service source can be tuned via SAFETY_POLICY_CONFIG_SCOPE,
SAFETY_POLICY_CONFIG_ID, SAFETY_POLICY_CONFIG_KEY (or disabled with SAFETY_POLICY_CONFIG_DISABLE=1).
Why packages need it: without this, “safe autopatcher” is marketing, not reality.
- Packages do not implement security. They declare:
- topics/tools they need
- capability labels + risk tags
- Admins install policy overlays that:
- allow/deny specific capabilities
- require approvals for risky actions (prod writes, PR creation, shell exec)
- impose constraints (max diff size, deny-paths, network restrictions)
- Kernel gates every job/run/tool call before execution.
- When policy provides remediations, the gateway can apply them to create a new job without hand-editing inputs.
In addition to the input gate above, the safety kernel runs a two-phase output policy on every result:
- Phase 1 — Sync metadata fast-path on the scheduler hot path:
status, topic, worker identity, declared content-type. Cheap; runs
on every result; emits an early
ALLOWfor clearly-safe results so they can be released without waiting for content scan. - Phase 2 — Async content scan over the dereferenced result
payload (
res:<job_id>). Runs the configured scanner pipeline against the actual content; produces typed findings:secret_leak— credentials, tokens, API keys, private keyspii— names, emails, phone numbers, government IDsinjection— prompt injection, code injection, SQL fragments
Decisions returned by OutputPolicyService.CheckOutput (gRPC
contract in core/protocol/proto/v1/output_policy.proto):
ALLOW— release the result to the callerQUARANTINE— mark the jobOUTPUT_QUARANTINED(terminal state in the scheduler engine); operator must review before releaseREDACT— release a redacted copy with sensitive segments replaced by typed placeholders
Operator surfaces:
config/output_scanners.yaml— regex pattern definitions, per-scanner enable/disable, severity thresholds. Loaded by the safety kernel whenOUTPUT_POLICY_ENABLED=true.output_rulessection inconfig/safety.yaml— topic / capability / content-pattern matchers that trigger scanners.- Dashboard quarantine UX — quarantine badge on JobsPage list, remediation drawer on JobDetailPage, artifact panel for reviewing the redacted vs original payload.
See docs/output-policy.md for the full operator guide, the
finding-type reference, and the gRPC contract.
The Governance Timeline is a derivation view that joins safety decisions, output-policy scans, approval events, replay history, and operator overrides for a single job or workflow run into a single ordered narrative. It is materialized by the gateway from the underlying audit log and exposed via:
- REST:
GET /api/v1/governance/decisions(paginated; filterable by tenant, time window, decision class) - REST:
GET /api/v1/governance/health(rollup health indicator for the policy decision pipeline) - REST:
GET /api/v1/governance/approvals/analytics(approval-rate, time-to-approve, bottleneck breakdown) - Dashboard:
GovernanceTimelinecomponent on JobDetailPage's Governance tab, plus the Policy Studio overview / verification / replay tabs.
The timeline does NOT introduce a new event source — it composes
existing safety.decision, policy.decision, policy.scan,
policy.quarantine, policy.override, and policy.replay audit
events (see docs/audit.md § Event Types). Downstream SIEM
consumers should de-duplicate on job_id + event_type if they
want raw decisions only.
Status: Implemented; Redis-backed merge with version/hash snapshot.
Even before you “do packages”, core must support overlay config:
- base config (platform)
- optional fragments (future packages)
Must support:
- merged “effective config” snapshot with a version/hash
- live reload with rollback (scheduler reloads pools/timeouts)
- per-tenant overrides (later)
Why packages need it: package install becomes “drop overlay files”, not edit core config manually.
- A package ships overlays:
- routing (
pools) - policy fragments (stored under
cfg:system:policybundles) - budgets/timeouts
- optional schema registrations
- routing (
- Installer merges overlays into config service.
- Core reads the “effective config” snapshot and behaves accordingly.
Status: Implemented in sdk/runtime (wraps CAP runtime).
Core should ship a tiny Go library that defines:
- how a worker connects/subscribes to job topics
- how it loads context and writes results via pointers
- how it retries handlers with bounded attempts
- how it verifies/signs CAP envelopes (optional)
- how it exposes hooks for logging/observability
Use CAP worker helpers when you need heartbeats/progress/cancel handling.
Why packages need it: consistent worker behavior + fewer “mystery outages”.
- Package worker repos import the SDK.
- Upgrades become predictable (protocol_version + SDK versioning).
- Core doesn’t need to change for every new worker.
Status: Implemented in cmd/cordum-api-gateway (HTTP/WS + gRPC; binary cordum-api-gateway).
Package structure (core/controlplane/gateway/):
| Sub-package | Purpose |
|---|---|
gateway/ (root) |
HTTP/gRPC handlers, middleware chain, MCP bridge, server lifecycle (~20 source files) |
gateway/auth/ |
Auth providers (API key, basic, OIDC/JWT, composite), Redis-backed user and key stores |
gateway/packs/ |
Pack types, constants, manifest validation, marketplace utilities, tar extraction |
gateway/policybundles/ |
Policy bundle types, YAML rule parsing, policy merging, evaluation helpers, audit formatting |
Dependency graph: gateway → {auth, packs, policybundles}, policybundles → {auth, packs}. No circular imports.
At minimum:
- Workflows:
create/list/get/delete - Runs:
start/get/list/cancel/delete,rerun,timeline - Approvals: job approvals (including workflow gate approvals)
- Jobs:
submit/status/get result pointer,cancel,remediate - DLQ:
list/retry/delete - Policy:
evaluate/simulate/explain+ snapshot list - Config:
get/set/effective - Schemas: register/get/list/delete
- Locks: acquire/release/renew/get
- Artifacts: put/get
- Audit: decisions + run timeline
Why packages need it: packages use the same APIs for operations, UI, and integrations.
- A package registers workflow templates (optional).
- A package (or an external client) triggers runs via the gateway.
- Ops tooling uses the same APIs to debug failures and inspect evidence pointers.
(SRE Investigator + MCP adapter)
Status: Implemented with a Redis-backed store and retention classes.
You need a standard interface:
PutArtifact(content, metadata) -> artifact_ptrGetArtifact(ptr)
Support:
- size limits
- TTL/retention classes (e.g., 7d/30d)
- optional encryption at rest (later)
Why packages need it: logs, test outputs, diffs, evidence = artifacts. Don’t shove them into Redis ctx.
- SRE package stores “evidence bundle” as artifacts (log tails, kubectl output, CI logs).
- PR summaries link artifacts by pointer.
- Core remains unchanged; only the artifact storage backend may be swapped later.
Status: Partially implemented: secret:// detection + redaction helpers, policy enforcement via risk tags/labels.
Core must support:
- “secret refs”
e.g.,secret://vault/path#keyorsecret://k8s/ns/name - redaction utility for logs/evidence before LLM
- kernel rules can block flows if
secrets_presentdetected
Why packages need it: SRE investigator touches logs and env. This is where you get burned.
- Workers never read raw secrets unless policy allows and runner profile permits.
- Evidence is redacted before it becomes an artifact or LLM input.
- Kernel constraints enforce “no secret material to LLM”.
Status: Implemented via pool capability profiles (config/pools.yaml) and JobMetadata.requires.
Extend scheduler mapping to support constraints:
- pool requires:
docker,git,kubectl,network:egress,cpu,mem,gpu - job declares:
requires=[...],risk=[...]
Why packages need it: repo verify needs toolchain; LLM needs GPU; collectors need network.
- Package job submission includes
requires. - Scheduler chooses eligible pool without knowing anything about the domain.
Status: Partially implemented: safety constraints for max runtime/retries/artifact bytes/concurrency; gateway enforces max concurrent runs.
Per tenant / per actor:
- max concurrent runs
- max runtime
- max artifact bytes
- max retries
- max PR size (files/lines changed) via constraints
Why packages need it: “agent went wild” becomes bounded damage.
- SRE package PR creation step is constrained:
- max files, max lines, deny paths, require approval in prod
- Kernel returns
ALLOW_WITH_CONSTRAINTSthat the workflow engine/scheduler must honor.
Status: Implemented: rerun-from-step, dry-run, and run idempotency keys.
You need:
- rerun a run from step N
- rerun with same inputs (immutable pointers)
- “dry‑run” mode (no external side effects)
Why packages need it: debugging and safe iteration.
- “Incident→PR” can be re-run after policy updates or worker fixes.
- Dry-run supports “propose patch but don’t open PR” safely.
(don’t block MVP, but know what’s coming)
P2 core should evolve to:
- OIDC/JWT auth for humans
- service-to-service auth (mTLS or signed tokens)
- RBAC for control plane actions
- tenant isolation for data (ctx/res/artifacts)
- structured logs with
trace_id/run_id/job_id - Prometheus metrics across core services
- tracing propagation
- state store migrations (workflow schema evolution)
- protocol version negotiation
- “last-known-good” configs/policies
Status: Implemented. Licensing lives in core/licensing/ with Ed25519 signature
verification and three tiers (Community/Team/Enterprise). Entitlement enforcement
is applied across all services: gateway rate limits, scheduler concurrency caps,
workflow step limits, safety kernel policy bundle quotas, and audit retention
periods. Licenses degrade gracefully on expiry (services continue at Community
tier). Enterprise add-ons (license issuance, SSO/SAML, advanced RBAC, SIEM
export, support SLA) live in the enterprise and tools repos; this repo provides
the loading, validation, and tier enforcement layer.
Add these fields to the job metadata today:
tenant_id,actor_id,actor_typeidempotency_keypack_id(optional, empty now)capability(semantic action label, not just topic)risk_tags(prod/write/network/secrets/exec)requires(capabilities for routing)
Why this matters: it lets future packages plug into the same enforcement/routing/audit machinery without core changes.
Workflow steps support a meta block that maps to JobMetadata, so package templates can declare
capability, risk_tags, requires, and pack_id at the step level without touching core.
These are additions that pay off massively later without turning core into product soup.
POST /api/v1/policy/evaluate→ decision + matchedrule_id+ constraintsPOST /api/v1/policy/simulate→ same, but no side effects (for CI / PR reviews)GET /api/v1/policy/snapshots→ version/hash currently loadedGET /api/v1/policy/bundles→ list policy bundlesGET /api/v1/policy/bundles/{id}→ bundle detailPUT /api/v1/policy/bundles/{id}→ update bundle (requiresX-Principal-Role: admin)POST /api/v1/policy/bundles/{id}/simulate→ simulate against draft bundlePOST /api/v1/policy/publish→ publish bundles (requiresX-Principal-Role: admin)POST /api/v1/policy/rollback→ rollback bundles (requiresX-Principal-Role: admin)GET /api/v1/policy/audit→ policy publish/rollback audit
Why: makes policy changes reviewable and prevents “security theater”.
Status: Implemented in gateway and safety kernel.
Bundle IDs include / (e.g. secops/workflows). Replace / with ~ in the {id} path segment
or use the bundle_id query parameter.
- Package install pipelines can simulate policies before deployment.
- Admins can validate “will SRE Investigator be allowed to open PRs in prod?” before enabling.
Core should support:
- registering JSON Schemas (or accepting inline schemas with workflows/jobs)
- validating job inputs/outputs and step outputs
Why: packages become reliable and debuggable; you stop passing mystery blobs between steps.
Status: Implemented with Redis-backed schema registry and workflow input/step IO validation.
- SRE package enforces a schema for
IncidentContext,EvidenceBundle,PatchPlan. - Kernel can reject malformed or suspicious inputs early.
A tiny “lock service” inside core:
- lock by
{repo},{cluster/ns},{service/env},{incident_id} - modes: shared/exclusive, TTL, owner
Why: once you run autopatcher or MCP actions, two workflows racing will wreck you.
Status: Implemented with Redis-backed shared/exclusive locks and gateway APIs.
- SRE Investigator acquires exclusive lock on
{service/env}before patch generation/PR open. - Verify steps can hold shared locks; mutation steps require exclusive.
Even if core ships zero workers, define execution profiles packages can request:
sandbox=isolatednetwork=none|egress-allowlistfs=ro|rwtools=git,kubectl,go
Scheduler routes based on requires[].
Why: lets you enforce “this job can’t touch network” at the platform level.
Status: Partially implemented: scheduler routes by requires and constraints are passed via env; sandbox enforcement is up to workers/runners.
- Collectors request network egress; LLM steps request “no network”.
- Kernel enforces that risky steps can’t run in permissive profiles.
Standardize:
artifact_ptr- retention class (
short,standard,audit) - max size + chunking policy
Why: avoids shoving megabytes into Redis ctx and gives audit durability.
Status: Implemented (Redis-backed artifacts + retention classes).
- “evidence” is audit retention; “temp logs” are short retention.
Maintain an append-only timeline:
- state transitions, decisions, approvals, dispatches, result pointers
Why: audit, replay, postmortems, “why did it do that?”
Status: Implemented (run timeline stored in Redis and exposed via gateway).
- SRE Investigator PR body can link to a canonical run timeline.
- MCP calls can be fully reconstructed for compliance.
Budgets are safety:
- max runtime, max retries, max artifact bytes, max concurrent runs
- max diff size, max files touched, deny-path patterns (as constraints)
Why: keeps early packages safe and sellable.
Status: Partially implemented (policy constraints for runtime/retries/artifacts/concurrency).
- SRE patch generation constrained to
max_files_changed,max_lines_changed. - Kernel can auto-rewrite budgets per environment (prod stricter than dev).
Make it explicit:
idempotency_keyon submit/run- dedupe window + stable semantics
Why: webhook storms, retries, MCP clients will otherwise duplicate actions.
Status: Implemented for job submission and workflow run creation.
- Incident ingest uses incident_id as idempotency key.
- “Open PR” step uses
incident_id + repo + branchas dedupe key.
Ship cordumctl that can:
- create/run/delete workflows
- approve/reject
- inspect run timeline
- retry DLQ
Optional: a lightweight dashboard that talks to the gateway for run/status visibility.
Why: bring-up, debugging, demos without requiring a full UI stack.
Status: Implemented (cmd/cordumctl + smoke script, plus dashboard/; ships as cordumctl).
- Ops can run:
cordumctl pack install <path|url>/cordumctl pack uninstall <id>/cordumctl pack verify <id> - CLI still drives core workflows and approvals with no packs installed.
- Datadog/Coralogix/GitHub/K8s connectors (packages only)
- LLM providers / prompt logic (packages only)
- SRE Investigator logic (package)
- MCP proxy/controller logic (separate service —
cmd/cordum-mcp/provides the MCP server bridge)
Core should provide governance + runtime, not domain logic.
- Policy explain/simulate
- Resource locks
- Runner profiles + requires/constraints routing
These three are what make future packages safe and enterprise-real instead of toys.
When you install sre-investigator later, it should consist of:
- workers (containers) that subscribe to
job.sre.*topics - workflow templates that orchestrate those workers
- overlays:
pools.overlay.yamlmappingjob.sre.* → sre-investigator-pooltimeouts.overlay.yamlfor collector/verify stepssafety.overlay.yamladding:- allowlist for read-only collectors
- require approval for PR creation in prod
- constraints: deny-paths, max diff size, network rules
Core stays unchanged because:
- scheduler already routes by config
- workflow engine already supports job dispatch + approvals + retries
- kernel already evaluates capability/risk + applies constraints
- artifact pointers already store evidence
- audit log already records decisions and run timeline
Net effect: new product behavior appears by installing overlays + deploying workers, not editing core.