Mechanism taxonomy: nullable family, and stop grouping the pending

Tracks the registry's contract change. `family` is now the published mechanism
taxonomy and is nullable: a technique the registry has not classified is served
null with classification "pending". The producer's own clustering label moves
to `producer_family`.

The fix that matters is the grouping bug. `techniques_by_family()` and
`family_siblings` compared families directly, and `None == None` is true, so
every unclassified technique became a sibling of every other one - 323 of 420
against the live registry, presented to a caller as a real relationship. The
same shape was in the search haystack, which joined a None into a string and
raised. Grouping now requires a known mechanism, and the index simply does not
key on None.

A null family is also no longer a validation issue. Flagging it made the
registry's deliberate, published gap look like a data defect.

Both axes stay queryable and separate: families() is the mechanism axis (all
five, zero counts included), producer_families() the producer vocabulary, and
FamilyCount carries `axis` so a count is never read against the wrong one.
`memory_amp` means one thing on each and the counts differ.

STIX export verified byte-identical to the Rust backend on every x_nrdax
property including the pending technique's nulls.

142 tests, ruff and mypy clean. Checked against the live registry: 97
classified / 323 pending, family counts match, a pending technique has zero
siblings, and validate() reports no issues.

Author: Simon Morley

CHANGELOG.md

@@ -10,6 +10,44 @@ compatibility policy.
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-07-25
+
+### Changed
+
+- **Breaking (tracks the API):** `Technique.family` is now the published *mechanism*
+  taxonomy and is `str | None`. It is `None` while the registry has not classified a
+  technique, which on the live registry is 323 of 420. The producing pipeline's own
+  label moves to the new `producer_family` field. Code doing `t.family == "..."` or
+  grouping on `t.family` must handle `None`; `Technique.is_classified` is provided
+  for the intent.
+- `NRDAX.families()` now returns the five mechanism families (always all five,
+  zero counts included). The producer vocabulary moved to `NRDAX.producer_families()`.
+  `FamilyCount` gains `axis` (`"mechanism"` or `"producer-class"`), because a name
+  such as `memory_amp` occurs on both axes with different counts.
+- `by_family()` and `--family` filter the mechanism axis. `by_producer_family()` and
+  `--producer-family` filter the producer axis.
+- Search matches both axes; STIX export carries `x_nrdax_producer_family`,
+  `x_nrdax_surface`, `x_nrdax_bound_failure` and `x_nrdax_classification` alongside
+  `x_nrdax_family`, byte-identical to the backend emitter.
+
+### Fixed
+
+- **`techniques_by_family()` and `family_siblings` no longer group unclassified
+  techniques together.** Both compared families directly, and since `None == None`
+  every pending technique was a "sibling" of every other one - 323 of them against
+  the live registry. Grouping now requires a known mechanism.
+- A `null` family is no longer reported as a validation issue. A pending technique is
+  well-formed, and flagging it made the registry's own honest gap look like a data
+  defect.
+- The CLI renders `(pending classification)` rather than a bare `None`.
+
+### Added
+
+- `MECHANISM_FAMILIES`, `SURFACES`, `BOUND_FAILURES` and `CLASSIFICATION_STATES`
+  vocabularies; `Technique.surface`, `.bound_failure`, `.dual_with`, `.classification`
+  and `.is_classified`; `NRDAX.classified()`, `.unclassified()` and
+  `.techniques_by_producer_family()`.
+
 ## [0.3.0] - 2026-07-16
 
 ### Changed

src/nrdax/_version.py

@@ -7,4 +7,4 @@
 
 from __future__ import annotations
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"

src/nrdax/cli/formatting.py

@@ -57,7 +57,10 @@ def technique_detail(t: Technique) -> str:
         f"{t.id}  {t.display}",
         "=" * (len(t.id) + 2 + len(t.display)),
         f"slug (name)        {t.name}",
-        f"family             {t.family}",
+        f"family             {t.family or '(pending classification)'}",
+        f"producer family    {t.producer_family or '-'}",
+        f"surface            {t.surface or '-'}",
+        f"bound failure      {t.bound_failure or '-'}",
         f"status             {t.status}",
         f"reproduction       {t.reproduction_status}",
         f"first seen         {t.first_seen}",
@@ -111,7 +114,15 @@ def search_table(results: list[SearchResult], *, explain: bool = False) -> str:
     rows = []
     for r in results:
         t = r.technique
-        row = [f"{r.score:g}", t.id, truncate(t.display, 50), t.family, t.reproduction_status]
+        # `family` is None while a technique is pending classification; render the
+        # state rather than a bare "None" in the table cell.
+        row = [
+            f"{r.score:g}",
+            t.id,
+            truncate(t.display, 50),
+            t.family or "(pending)",
+            t.reproduction_status,
+        ]
         if explain:
             row.append(",".join(r.matched_fields))
         rows.append(row)

src/nrdax/cli/main.py

@@ -33,6 +33,7 @@
     DISCOVERY_ORIGINS,
     FAMILIES,
     FIDELITY_CLASSES,
+    MECHANISM_FAMILIES,
     NRDAX_SCHEMA_VERSION,
     REFERENCE_KINDS,
     REPRODUCTION_STATUSES,
@@ -104,6 +105,7 @@ def _predicate_kwargs(args: argparse.Namespace) -> dict[str, Any]:
     kw: dict[str, Any] = {}
     for name in (
         "family",
+        "producer_family",
         "chain",
         "status",
         "fidelity",
@@ -380,6 +382,11 @@ def cmd_cache(args: argparse.Namespace) -> int:
     "display_name",
     "mechanism",
     "family",
+    "producer_family",
+    "surface",
+    "bound_failure",
+    "dual_with",
+    "classification",
     "status",
     "first_seen",
     "instances",
@@ -456,7 +463,16 @@ def _add_source_arg(p: argparse.ArgumentParser) -> None:
 
 
 def _add_filter_args(p: argparse.ArgumentParser) -> None:
-    p.add_argument("--family", choices=FAMILIES, help="exact family")
+    p.add_argument(
+        "--family",
+        choices=MECHANISM_FAMILIES,
+        help="exact mechanism family (the published taxonomy)",
+    )
+    p.add_argument(
+        "--producer-family",
+        choices=FAMILIES,
+        help="exact producer family (the producing pipeline's own label)",
+    )
     p.add_argument("--chain", help="techniques with a reproduced instance on this chain")
     p.add_argument("--status", choices=STATUSES, help="lifecycle status")
     p.add_argument("--fidelity", choices=FIDELITY_CLASSES, help="any instance with this fidelity")

src/nrdax/exporters/stix_exporter.py

@@ -3,8 +3,10 @@
 Reproduces the backend's deterministic scheme (``stix/mod.rs``): ids are UUIDv5
 under a fixed namespace, timestamps come from ``first_seen`` (no wall-clock), the
 NRDAX id is anchored in ``external_references`` (``source_name: "nrdax"``), and the
-NRDAX-specific fields ride as custom properties ``x_nrdax_family`` /
-``x_nrdax_status`` / ``x_nrdax_chains``. Keys are alphabetically sorted and the
+NRDAX-specific fields ride as custom properties: ``x_nrdax_family`` (the published
+mechanism taxonomy, null while pending), ``x_nrdax_producer_family``,
+``x_nrdax_surface``, ``x_nrdax_bound_failure``, ``x_nrdax_classification``,
+``x_nrdax_status`` and ``x_nrdax_chains``. Keys are alphabetically sorted and the
 document is 2-space-indented with a trailing newline, matching the feed's
 ``stix.json`` exactly — so a bundle exported here is identical to the one served.
 """
@@ -61,6 +63,10 @@ def attack_pattern(technique: Technique) -> dict[str, Any]:
         "description": technique.mechanism,
         "external_references": refs,
         "x_nrdax_family": technique.family,
+        "x_nrdax_producer_family": technique.producer_family,
+        "x_nrdax_surface": technique.surface,
+        "x_nrdax_bound_failure": technique.bound_failure,
+        "x_nrdax_classification": technique.classification,
         "x_nrdax_status": technique.status,
         "x_nrdax_chains": chains,
     }

src/nrdax/models.py

@@ -23,12 +23,15 @@
 
 from .errors import IssueCollector
 from .vocab import (
+    BOUND_FAILURES,
     DISCOVERY_ORIGINS,
     FAMILIES,
     FIDELITY_CLASSES,
+    MECHANISM_FAMILIES,
     NRDAX_SITE,
     REFERENCE_KINDS,
     STATUSES,
+    SURFACES,
     TECHNIQUE_ID_PATTERN,
     fidelity_strength,
 )
@@ -172,26 +175,56 @@ def fidelity_strength(self) -> int:
 class Technique:
     """The citable unit. Opaque stable ``id``; ``family`` is an attribute (a
     technique can be reclassified without its id changing); zero or more instances
-    and external references."""
+    and external references.
+
+    ``family`` is the published MECHANISM taxonomy and is ``None`` while the registry
+    has not classified the technique - it is never inferred from ``producer_family``,
+    because the producer's surface-defined labels have no honest mechanical target.
+    Anything grouping by family must treat ``None`` as unknown rather than as a shared
+    value; see :meth:`is_classified`.
+    """
 
     id: str
     name: str
     mechanism: str
-    family: str
+    #: Published mechanism family, or None while pending classification.
+    family: str | None
     status: str
     first_seen: str
     display_name: str | None = None
+    #: The producing pipeline's own clustering label. Provenance, not the taxonomy.
+    producer_family: str | None = None
+    #: Where the attacker's input enters. Present only when classified.
+    surface: str | None = None
+    #: Why the node's bound failed to apply. Present only when classified.
+    bound_failure: str | None = None
+    #: Secondary family where a technique is genuinely dual and no reproduction
+    #: measured which resource binds first.
+    dual_with: str | None = None
+    #: ``curated`` or ``pending``; always present on a current registry response.
+    classification: str = "pending"
     instances: list[Instance] = field(default_factory=list)
     external_references: list[ExternalReference] = field(default_factory=list)
     provenance_note: str | None = None
     extra: dict[str, Any] = field(default_factory=dict)
 
+    @property
+    def is_classified(self) -> bool:
+        """Whether this technique carries a mechanism family. Prefer this over
+        truth-testing ``family``: it states the intent at the call site."""
+        return self.family is not None
+
     _KNOWN: ClassVar[set[str]] = {
         "id",
         "name",
         "display_name",
         "mechanism",
         "family",
+        "producer_family",
+        "surface",
+        "bound_failure",
+        "dual_with",
+        "classification",
         "status",
         "first_seen",
         "instances",
@@ -211,9 +244,34 @@ def from_dict(
             issues.add(f"{locator}.id", f"id does not match {TECHNIQUE_ID_PATTERN}")
         name = _require_str(data, "name", issues, locator)
         mechanism = _require_str(data, "mechanism", issues, locator)
-        family = _require_str(data, "family", issues, locator)
-        if issues is not None and family and family not in FAMILIES:
-            issues.add(f"{locator}.family", f"unknown family {family!r}", "warning")
+        # `family` is nullable by contract: a technique the registry has not
+        # classified is served null with classification "pending". That is a
+        # well-formed record, not a malformed one, so an absent family is never
+        # reported as an issue - only a present-but-unknown one.
+        family = _optional_str(data, "family", issues, locator)
+        if issues is not None and family and family not in MECHANISM_FAMILIES:
+            issues.add(f"{locator}.family", f"unknown mechanism family {family!r}", "warning")
+        producer_family = _optional_str(data, "producer_family", issues, locator)
+        if issues is not None and producer_family and producer_family not in FAMILIES:
+            issues.add(
+                f"{locator}.producer_family",
+                f"unknown producer family {producer_family!r}",
+                "warning",
+            )
+        surface = _optional_str(data, "surface", issues, locator)
+        if issues is not None and surface and surface not in SURFACES:
+            issues.add(f"{locator}.surface", f"unknown surface {surface!r}", "warning")
+        bound_failure = _optional_str(data, "bound_failure", issues, locator)
+        if issues is not None and bound_failure and bound_failure not in BOUND_FAILURES:
+            issues.add(
+                f"{locator}.bound_failure",
+                f"unknown bound failure {bound_failure!r}",
+                "warning",
+            )
+        dual_with = _optional_str(data, "dual_with", issues, locator)
+        classification = _optional_str(data, "classification", issues, locator) or (
+            "curated" if family else "pending"
+        )
         status = _require_str(data, "status", issues, locator)
         if issues is not None and status and status not in STATUSES:
             issues.add(f"{locator}.status", f"unknown status {status!r}", "warning")
@@ -244,6 +302,11 @@ def from_dict(
             status=status,
             first_seen=first_seen,
             display_name=display_name,
+            producer_family=producer_family,
+            surface=surface,
+            bound_failure=bound_failure,
+            dual_with=dual_with,
+            classification=classification,
             instances=instances,
             external_references=refs,
             provenance_note=provenance_note,
@@ -257,6 +320,15 @@ def to_dict(self) -> dict[str, Any]:
             out["display_name"] = self.display_name
         out["mechanism"] = self.mechanism
         out["family"] = self.family
+        if self.producer_family is not None:
+            out["producer_family"] = self.producer_family
+        if self.surface is not None:
+            out["surface"] = self.surface
+        if self.bound_failure is not None:
+            out["bound_failure"] = self.bound_failure
+        if self.dual_with is not None:
+            out["dual_with"] = self.dual_with
+        out["classification"] = self.classification
         out["status"] = self.status
         out["first_seen"] = self.first_seen
         out["instances"] = [i.to_dict() for i in self.instances]
@@ -410,7 +482,13 @@ def to_dict(self) -> dict[str, Any]:
 
 @dataclass(frozen=True)
 class FamilyCount:
-    """A family and how many techniques currently carry it."""
+    """A family and how many techniques currently carry it.
+
+    ``axis`` says which taxonomy the name belongs to: ``mechanism`` for the published
+    families, ``producer-class`` for the producing pipeline's own labels. A name such
+    as ``memory_amp`` occurs on both axes with different counts, so a count read
+    without its axis is meaningless."""
 
     name: str
     technique_count: int
+    axis: str = "mechanism"

src/nrdax/queries/filters.py

@@ -35,7 +35,21 @@ def _validate(value: str, allowed: tuple[str, ...], what: str) -> str:
 
 
 def by_family(name: str) -> Predicate:
-    return lambda t: t.family == name
+    """Match on the published MECHANISM family. A technique pending classification
+    has ``family is None`` and matches nothing here - never every other pending
+    technique, which a bare ``==`` comparison would give you."""
+    return lambda t: t.family is not None and t.family == name
+
+
+def by_producer_family(name: str) -> Predicate:
+    """Match on the producing pipeline's own label, a different axis from
+    :func:`by_family`."""
+    return lambda t: t.producer_family is not None and t.producer_family == name
+
+
+def unclassified() -> Predicate:
+    """Match techniques with no mechanism family yet."""
+    return lambda t: t.family is None
 
 
 def by_status(status: str) -> Predicate:
@@ -111,6 +125,7 @@ def all_of(*predicates: Predicate) -> Predicate:
 def build_predicate(
     *,
     family: str | None = None,
+    producer_family: str | None = None,
     status: str | None = None,
     chain: str | None = None,
     fidelity: str | None = None,
@@ -126,6 +141,8 @@ def build_predicate(
     preds: list[Predicate] = []
     if family is not None:
         preds.append(by_family(family))
+    if producer_family is not None:
+        preds.append(by_producer_family(producer_family))
     if status is not None:
         preds.append(by_status(status))
     if chain is not None:

src/nrdax/queries/search.py

@@ -57,7 +57,9 @@ def _field_texts(t: Technique) -> dict[str, list[str]]:
         "id": [t.id],
         "name": [t.name],
         "display_name": [t.display_name] if t.display_name else [],
-        "family": [t.family],
+        # Both taxonomy axes are searchable. `family` is None while a technique is
+        # pending classification, so it is filtered rather than joined blindly.
+        "family": [x for x in (t.family, t.producer_family) if x],
         "chain": [i.chain for i in t.instances],
         "primitive_id": [i.primitive_id for i in t.instances],
         "reference": [ref.id for _, ref in t.iter_references()]