Status: Draft · SOP Layer: L3 · Date: 2026-07-01 SSOT: Yes — this document is the Single Source of Truth for all Capability Schema modules.
Capability is a First-class Object in the Capability Schema Specification. It has a unique ID, a model-independent semantic definition, standardized observation methods, and a reproducible scoring pipeline. A Capability is comparable across World Models, queryable by tools, and traceable to evidence.
A Capability's role in Capability Schema is equivalent to: an Experiment in TraceOS · a Table in SQL · a Metric Family in Prometheus.
capability:
id: "cap_temporal_stability"
name: "Temporal Stability"
version: "1.0"
definition:
description: >
The ability of a world model to maintain consistent predictions
over autoregressive rollout time.
invariants:
- prediction_consistency_degrades_monotonically
- delta_psnr_is_valid_proxy
observed_by:
metrics: # ← ground-truth required
- delta_psnr
predictors: # ← no ground-truth needed
- flow_instability
outputs:
score: { type: float, range: [0.0, 1.0] }
confidence: { type: float, range: [0.0, 1.0] }
evidence: { type: list, element: Evidence }
metadata:
domain: world_model
created: "2026-07-01"
status: experimental # experimental | stable | deprecated@dataclass(frozen=True)
class CapabilitySpec:
"""Immutable definition of a single Capability. Lives in capabilities/*.yaml."""
id: str # UUID7 "cap_..."
name: str # Human-readable, e.g. "Temporal Stability"
version: str # SemVer, e.g. "1.0"
description: str # One paragraph — what this Capability means
invariants: list[str] # Properties that must hold for any valid measurement
observed_by: ObservationSpec # Declarative binding — metric/predictor names only
outputs: OutputSpec # Shape of the Score this Capability produces
metadata: CapabilityMetadata # Domain, status, authorshipConstraint: The observed_by field contains names of Metrics and Predictors — never computation formulas, never threshold values, never normalization parameters. The same Metric name resolved by different implementations may use different normalization strategies (Recommended Profile), but the Capability definition does not prescribe which one.
@dataclass(frozen=True)
class ObservationSpec:
"""Declares what observes a Capability. Does not define how."""
metrics: list[str] # Metric names, e.g. ["delta_psnr"]
predictors: list[str] # Predictor names, e.g. ["flow_instability"]
@dataclass(frozen=True)
class MetricSpec:
"""A mathematical definition of one observation method."""
id: str # UUID7 "met_..."
capability_id: str # FK → CapabilitySpec.id
name: str # e.g. "delta_psnr"
description: str # What this metric measures
formula: str # Mathematical definition (LaTeX-compatible plain text)
unit: str # e.g. "dB"
range: tuple[float, float] # [min, max] in native units
higher_is_better: bool
requires_ground_truth: bool # Always True for Metrics
@dataclass(frozen=True)
class PredictorSpec:
"""An abstract interface for estimating a Capability without ground-truth."""
id: str # UUID7 "prd_..."
capability_id: str # FK → CapabilitySpec.id
name: str # e.g. "flow_instability"
description: str # What this predictor estimates
inputs: list[str] # Required inputs, e.g. ["world_model", "rollout_state"]
calibration_required: bool # Does it need a CalibrationBaseline?
requires_ground_truth: bool # Always False for PredictorsKey distinction: A Metric computes against ground-truth data (accurate, expensive, requires dataset). A Predictor estimates from model internals only (approximate, cheap, runs at inference time). Both produce values that feed into the same Capability Score.
@dataclass
class CapabilityQuery:
"""A request to evaluate a Capability. The unit of work."""
id: str # UUID7 "qry_..."
capability_id: str # Which Capability to evaluate
model: WorldModelABC # Any model implementing the ABC
dataset: DatasetHandle # Reference to evaluation data
observers: list[str] # Which metrics/predictors to run (subset of observed_by)
params: dict # Query-level params (e.g., rollout_steps=64)CREATED ──→ OBSERVING ──→ SCORING ──→ COMPLETED
│ │ │ │
└────────────┴──────────────┴────────────┘
│
FAILED
| Status | Meaning | Next |
|---|---|---|
| CREATED | Query created, not started | OBSERVING |
| OBSERVING | Metric(s)/Predictor(s) running | SCORING or FAILED |
| SCORING | Aggregating observations into Score | COMPLETED or FAILED |
| COMPLETED | Terminal success — Score produced | — |
| FAILED | Terminal failure — error payload | — (retry creates new Query) |
@dataclass(frozen=True)
class CapabilityScore:
"""The output of a single Capability evaluation."""
id: str # UUID7 "scr_..."
query_id: str # FK → CapabilityQuery.id
capability_id: str # FK → CapabilitySpec.id
model_name: str # From WorldModelABC.metadata.name
dataset_name: str # From DatasetHandle.name
value: float # 0.0 ~ 1.0, normalized
confidence: float # 0.0 ~ 1.0
native_values: dict[str, float] # Raw metric/predictor values before normalization
evidence: list[Evidence] # Traceable links to raw data
recommendation: str | None # Template-level actionable fix
timestamp: str # ISO 8601
spec_version: str # CapabilitySpec.version usedAll Capability Scores are normalized to [0.0, 1.0]. The normalization strategy is an implementation detail of the Reference Implementation (or any conformant implementation) and is NOT part of the Spec. However, the Conformance Test supplies a reference dataset + expected score range, and any implementation claiming conformance must produce Scores within that range.
@dataclass(frozen=True)
class Evidence:
"""A traceable link from a Score to its raw data source."""
id: str # UUID7 "evd_..."
source: str # "summary.json" | "rollout_005.mp4" | "metrics.jsonl"
type: str # "metric" | "video" | "log" | "trajectory" | "predictor_log"
location: str # Relative path or URI
description: str # Human-readable explanation of what this evidence shows
timestamp: str # ISO 8601@dataclass
class CapabilityReport:
"""The complete output document. Self-contained HTML by default."""
id: str # UUID7 "rpt_..."
query_id: str # FK → CapabilityQuery.id
capability_id: str # FK → CapabilitySpec.id
scores: list[CapabilityScore] # One per observer path
aggregate_score: CapabilityScore # Weighted or merged result
radar: RadarView # Capability structure visualization
timeline: TimelineView # Score-over-time or score-over-rollout visualization
recommendations: list[str] # Prioritized actionable items
generated_at: str # ISO 8601| View | What it shows | When present |
|---|---|---|
| Radar | Capability score on a gauge (single Capability) or radar chart (multi-Capability) | Always |
| Timeline | Score degradation over rollout steps | Temporal Capabilities |
| Failure Attribution | Which failure modes contributed to low scores | When score < 0.7 |
All entities have UUID7 (time-sortable) with short prefixes.
| Entity | Prefix | Example |
|---|---|---|
| CapabilitySpec | cap_ |
cap_3f7a8b2c1d4e |
| MetricSpec | met_ |
met_8c2d4f6a1b3e |
| PredictorSpec | prd_ |
prd_2e7f3a9c5d1b |
| CapabilityQuery | qry_ |
qry_5a1b9c3e7f2d |
| CapabilityScore | scr_ |
scr_6d1b4e8f2a3c |
| Evidence | evd_ |
evd_4a9c1e3f7b2d |
| CapabilityReport | rpt_ |
rpt_7b3d5f1a9c2e |
CapabilitySpec (cap_)
├── MetricSpec (met_)
│ └── compute(model, data) → raw values
├── PredictorSpec (prd_)
│ └── predict(model, state) → raw values
│
CapabilityQuery (qry_)
├── CapabilityScore (scr_) × N
│ └── Evidence (evd_) × M
└── CapabilityReport (rpt_)
└── [Radar | Timeline | Failure Attribution]
Every state change is an Event.
class EventType(Enum):
QUERY_CREATED = "capability.query.created"
OBSERVING_STARTED = "capability.observing.started"
METRIC_COMPUTED = "capability.metric.computed"
PREDICTOR_ESTIMATED = "capability.predictor.estimated"
SCORING_COMPLETED = "capability.scoring.completed"
EVIDENCE_LINKED = "capability.evidence.linked"
REPORT_GENERATED = "capability.report.generated"
QUERY_COMPLETED = "capability.query.completed"
QUERY_FAILED = "capability.query.failed"
@dataclass
class Event:
event_id: str # UUID7
event_type: EventType
query_id: str
capability_id: str
timestamp: str # ISO 8601
payload: dict
parent_event_id: str | NoneDesign rationale: Events are the audit trail. A complete Event log can replay a Query's full history. Consumers (TraceOS, CLI, future Dashboard) read the Event Stream, not Python objects.
The only interface that bridges the Spec and any concrete World Model.
class WorldModelABC(ABC):
"""Any generative world model. Exactly three methods."""
@abstractmethod
def encode(self, observation: Observation) -> Latent:
"""observation → internal representation"""
...
@abstractmethod
def decode(self, latent: Latent) -> Observation:
"""internal representation → observation"""
...
@abstractmethod
def step(self, latent: Latent, action: Action) -> Latent:
"""single-step forward dynamics — no rollout loop"""
...
@property
@abstractmethod
def metadata(self) -> dict:
"""{name, version, checkpoint_path, resolution, ...}"""
...Constitutional Rule: Any class implementing these three methods is a valid World Model for the Capability Schema. The Spec never references concrete model names. The Conformance Test passes a MockWorldModel through the pipeline to enforce this.
Capability is not hardcoded to world models. Domain knowledge lives in Schema YAML; the model layer is invariant.
Capability ← abstract concept (domain-agnostic)
↓
Observation ← measurement method (Metric or Predictor)
↓
Score ← normalized value (0.0~1.0, confidence, evidence)
↓
Evidence ← raw data reference (video, metric log, trajectory)
@dataclass(frozen=True)
class Capability:
"""Domain-agnostic. The semantic concept."""
id: str
name: str
domain: str # "world_model" | "llm" | "agent" | "audio"
@dataclass(frozen=True)
class Observation:
"""Domain-agnostic. A way to measure or estimate a Capability."""
capability_id: str
type: str # "metric" | "predictor"
name: str
native_unit: str
native_range: tuple[float, float]
@dataclass(frozen=True)
class Score:
"""Domain-agnostic. The normalized result."""
observation_name: str
value: float # 0.0 ~ 1.0
native_value: float
confidence: float
evidence_ids: list[str]
@dataclass(frozen=True)
class Evidence:
"""Domain-agnostic. Traceable data reference."""
source: str
type: str
location: str
description: strTo add a new domain (e.g., LLM evaluation), create a new domain schema YAML. The Engine code is unchanged. Only the Capability definitions change.
# temporal_stability.yaml — the first Capability definition
capability:
id: "cap_3f7a8b2c1d4e"
name: "Temporal Stability"
version: "1.0"
definition:
description: >
The ability of a world model to maintain consistent predictions
over autoregressive rollout time. A model with high Temporal
Stability produces rollouts that stay close to ground-truth
dynamics at least 32 steps into the future.
invariants:
- prediction_consistency_degrades_monotonically
- delta_psnr_is_valid_proxy
observed_by:
metrics:
- delta_psnr
predictors:
- flow_instability
outputs:
score:
type: float
range: [0.0, 1.0]
description: >
1.0 = rollout stays within 1 dB of ground-truth at 32 steps.
0.0 = rollout diverges immediately (ΔPSNR ≤ 0 at step 1).
confidence:
type: float
range: [0.0, 1.0]
description: >
Derived from the number of observation paths that agree.
evidence:
type: list
element: Evidence
description: >
Per-timestep ΔPSNR curve, rollout video frames, and
Predictor raw signal logs.
metadata:
domain: world_model
created: "2026-07-01"
status: experimental
author: Capability Schema Editorial BoardAny implementation claiming "Capability Schema Conformant" must:
- Parse
temporal_stability.yamlinto aCapabilitySpecobject. - Execute a
CapabilityQueryagainst a Reference World Model (supplied by the test suite) on the Reference Dataset (supplied by the test suite). - Produce a
CapabilityScorewithvalue ∈ [0.0, 1.0]andconfidence ∈ [0.0, 1.0]. - Produce a
CapabilityReportas a self-contained HTML file containing:- The Capability name and version
- The Score value with confidence
- At least one Evidence entry
- A radar or gauge visualization
- Pass the Reference Score range check: the produced Score must fall within the expected range defined by the Reference Implementation on the Reference Dataset.
The Conformance Test is owned by capability-schema-spec/conformance/. The Reference Implementation has no exemption — it must also pass.
Next: RFC-0002 — Metric Interface Specification References: CHARTER.md (Governance) · project-vision.md (Vision) · project-mission.md (Mission)