feat(descriptions): carry each series' publisher codes

python/descriptions/CHANGELOG.md

@@ -149,5 +149,17 @@ and this package adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   members of one side with no partner on the other. The `sum` and `compare`
   descriptions cover the paired form: members of `data` and `other` pair by
   role through `on`, and a call to `compare` returns `data` minus `other`.
+- `about.source_codes` on a card and a dataset version, and `source_codes` on
+  a card's drawn value: the publishers' own series codes for the variables
+  covered, as `{provenance, code, exact?}` in the shape and order of Search's
+  `stat_vars.source_codes`, 1 to 50 of them, absent when there are none or
+  more. `exact: true` marks a code that names the whole of what lists it. A
+  computed version and a source card refuse `about.source_codes`, and a card
+  drawn from a computed version refuses `exact`. `check_description` raises
+  `source_codes_not_ascending` when a list, under `about` or on a drawn value,
+  is not strictly ascending by provenance and then code, which catches a
+  list out of order and a pair listed twice with and without `exact`.
+- Vectors rejecting a drawn value's `source_codes` that list one pair twice,
+  once exact, and an `about.source_codes` list in descending order.
 
 [unreleased]: https://github.com/DevelopmentFinanceObservatory/spec-kit/commits/main/python

python/descriptions/README.md

@@ -95,7 +95,7 @@ A `pooled_entities` count names its graph role in `role`, such as `financingProv
 
 ### Structure rules
 
-`check_steps` checks what the schema cannot, on a list that already validates against `steps.schema.json`, and raises on the first rule broken. `check_description` runs the count bound, the I-JSON check and the id, input and `sides` rules on a description's `steps` when they are a list, then checks the description's caveats and gaps and its drawn dimensions' `values`.
+`check_steps` checks what the schema cannot, on a list that already validates against `steps.schema.json`, and raises on the first rule broken. `check_description` runs the count bound, the I-JSON check and the id, input and `sides` rules on a description's `steps` when they are a list, then checks the description's caveats and gaps, its drawn dimensions' `values` and its `source_codes` lists.
 
 | Reason                        | Meaning                                                                                                            |
 | ----------------------------- | ------------------------------------------------------------------------------------------------------------------ |
@@ -111,6 +111,7 @@ A `pooled_entities` count names its graph role in `role`, such as `financingProv
 | `unknown_gap_step`            | A description's gap names a step its `steps` do not hold                                                           |
 | `drawn_values_count_mismatch` | A drawn dimension lists `values` whose number differs from its `count`                                             |
 | `duplicate_drawn_value`       | A drawn dimension's `values` repeat a `ref`                                                                        |
+| `source_codes_not_ascending`  | A `source_codes` list is out of order by provenance and then code, or lists a pair twice                           |
 
 The size bound is on the stored list and `check_steps` checks it where the list is written, counting the list serialised by JCS (RFC 8785). `check_description` leaves it out, since a served step's labels come from the vocabulary and its `text` is added, so the served bytes differ from the stored list's. JCS spells numbers as ECMAScript does, so a TypeScript consumer counts the same bytes with `JSON.stringify` on the same members, once it refuses what I-JSON cannot carry (`String.prototype.isWellFormed`, `Number.isSafeInteger`, `Number.isFinite`). The bounds are checked first, on the whole list. Their errors, and those for a caveat or a gap, have no `step`. An input may name only an earlier step, so a list that passes cannot hold a cycle. Reasons are permanent once published. A changed meaning gets a new reason.
 
@@ -151,6 +152,7 @@ A description carries `describes`, `title`, the five blocks `about`, `sources`,
 - An open dimension with a known `count` carries `top`, up to two values by variable count, `more`, the count of the rest, and `values`, the route that pages them, provider included. A dimension whose values are unknown has `count: null` with a gap and none of the three.
 - A source that is not an input carries `publisher`, `licence` and `series`, each `null` with a gap when unknown. A dataset version has no inputs, so its sources carry no `content_version`.
 - `about.measure` carries a `definition`, `null` with gap `about.measure.definition` when nothing defines the measure. Each `about.fixed[].value`, on a version or a card, may carry a `definition`, the graph's description of the value, 1 to 4096 characters, absent when the graph holds none. A card's `about.drawn.dimensions[]` may list the `values` it draws on that dimension, each with its `definition`, in the order the card first draws them. The list holds 1 to 50 values, and its presence means it names every value the card draws there. A list longer or shorter than the dimension's `count`, or one that repeats a `ref`, breaks the structure rules, and a card that draws none or more than 50 leaves `values` out.
+- `about.source_codes` lists the codes publishers file the covered variables under, from each variable's `sourceIndicatorCode` arc, as `{provenance, code, exact?}`, sorted by provenance and then code, each once, as Search serves `stat_vars.source_codes`. The schema cannot hold the order, or tell a repeated pair that differs only in `exact`, so `check_description` raises `source_codes_not_ascending` on a list that is not strictly ascending by `(provenance, code)`. A card lists those of every series it draws, and a series of a computed version takes the codes of the variables it was computed from. A dataset version lists those of the one variable its definition resolved to. A card's drawn value on a dimension the variables declare carries the codes of the drawn variables holding it. A list holds 1 to 50 codes and is absent when there are none or more. `exact: true` marks a code that names the whole of what lists it: every series there is read from that provenance alone and is one the code names, and the provenance gives none of them another code. A code without it names variables the series are built from or hold beside others. A computed version and a source card carry no `about.source_codes`, and a card drawn from a computed version carries no exact code.
 - Every reference is `{ref, label}`, and every unit, variable side and input version outside `steps` carries its `label` too. A label, source name, publisher or licence taken from the evidence stored at mint, in place of today's vocabulary, sits in an object carrying `at_mint: true`.
 - A computed version's inputs, in `sources`, `freshness.inputs` and `cite.inputs`, name each version it was computed from by `id` and `content_version`, with a `label`. An input that no step retrieves, as on a version published without steps, has an unknown id: it is served with `id: null`, or in `sources` with `ref: null` and no `description` link, and a gap naming that member. Its label is the one stored with it, else its content version with a gap. Only an input, which carries `content_version`, may leave its `ref` null.
 - `steps` is a step list whose every step adds `text`, the composer's sentence for that step, or `null` when the record holds none, with a gap. Both schemas build a step from `step_members` in `steps.schema.json`: a stored step closes it with `unevaluatedProperties: false`, so a stored list refuses `text`, and a served step requires `text` and closes it the same way.

python/descriptions/src/dfo_descriptions/schemas/description.schema.json

@@ -124,7 +124,8 @@
               "fixed": false,
               "varies": false,
               "variables": false,
-              "drawn": false
+              "drawn": false,
+              "source_codes": false
             }
           },
           "freshness": {
@@ -218,7 +219,8 @@
                 "measure": false,
                 "fixed": false,
                 "drawn": false,
-                "unit": false
+                "unit": false,
+                "source_codes": false
               }
             },
             "freshness": {
@@ -253,7 +255,29 @@
               "about": {
                 "properties": {
                   "measure": false,
-                  "fixed": false
+                  "fixed": false,
+                  "source_codes": {
+                    "$ref": "#/$defs/built_from_codes"
+                  },
+                  "drawn": {
+                    "properties": {
+                      "dimensions": {
+                        "items": {
+                          "properties": {
+                            "values": {
+                              "items": {
+                                "properties": {
+                                  "source_codes": {
+                                    "$ref": "#/$defs/built_from_codes"
+                                  }
+                                }
+                              }
+                            }
+                          }
+                        }
+                      }
+                    }
+                  }
                 }
               }
             }
@@ -412,6 +436,67 @@
       "minLength": 1,
       "maxLength": 4096
     },
+    "drawn_value": {
+      "description": "A value a card draws on a dimension: `ref`, `label`, `at_mint` and `definition` as in `defined_ref`, and `source_codes`, the codes of the drawn variables that hold the value, present only on a dimension the variables themselves declare, absent when none of them has a code.",
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["ref", "label"],
+      "properties": {
+        "ref": {
+          "$ref": "steps.schema.json#/$defs/text"
+        },
+        "label": {
+          "$ref": "steps.schema.json#/$defs/text"
+        },
+        "at_mint": {
+          "const": true
+        },
+        "definition": {
+          "$ref": "#/$defs/definition"
+        },
+        "source_codes": {
+          "$ref": "#/$defs/source_codes"
+        }
+      }
+    },
+    "source_codes": {
+      "description": "Publisher series codes, each once, sorted by provenance and then code, as Search serves `stat_vars.source_codes`.",
+      "type": "array",
+      "minItems": 1,
+      "maxItems": 50,
+      "uniqueItems": true,
+      "items": {
+        "$ref": "#/$defs/source_code"
+      }
+    },
+    "built_from_codes": {
+      "description": "Codes none of which is exact, as a card drawn from a computed version lists them: its series are built from the variables the codes name.",
+      "items": {
+        "properties": {
+          "exact": false
+        }
+      }
+    },
+    "source_code": {
+      "description": "A code a publisher files a series under, such as World Bank `DT.TDS.DEGG.CD`, from the variable's `sourceIndicatorCode` arc: `provenance`, the dcid of the provenance issuing it, and `code`, upper-cased. `exact: true` when the code names the whole of what lists it: every series there is read from `provenance` alone and is one the code names, `provenance` gives none of them another code, and the version is a dataset version. Absent, the series are built from the codes listed, or hold others beside them.",
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["provenance", "code"],
+      "properties": {
+        "provenance": {
+          "$ref": "steps.schema.json#/$defs/text"
+        },
+        "code": {
+          "type": "string",
+          "minLength": 1,
+          "maxLength": 512,
+          "$ref": "steps.schema.json#/$defs/one_line"
+        },
+        "exact": {
+          "const": true
+        }
+      }
+    },
     "input_ref": {
       "description": "A computed version's input as served: its id, content version and label. `id` is `null` when the version holds no step that retrieves the input, with a gap naming it; the label is then the one stored with the input, else its content version with a gap.",
       "type": "object",
@@ -715,6 +800,10 @@
         "unit": {
           "$ref": "#/$defs/unit"
         },
+        "source_codes": {
+          "description": "The publishers' own codes for the variables the description covers: on a card, those of every series it draws, a series of a computed version taking the codes of the variables it was computed from; on a dataset version, those of the one variable its definition resolved to. Absent when none of them has a code, and when they hold more than 50.",
+          "$ref": "#/$defs/source_codes"
+        },
         "drawn": {
           "description": "A card's selection against its version: per dimension, how many values it draws of how many the version holds; and its window against the version's period.",
           "type": "object",
@@ -743,7 +832,7 @@
                     "minItems": 1,
                     "maxItems": 50,
                     "items": {
-                      "$ref": "#/$defs/defined_ref"
+                      "$ref": "#/$defs/drawn_value"
                     }
                   }
                 }

python/descriptions/src/dfo_descriptions/structure.py

@@ -5,6 +5,7 @@
 
 from collections.abc import Mapping, Sequence
 from enum import StrEnum
+from itertools import pairwise
 from typing import Any
 
 import rfc8785
@@ -73,6 +74,11 @@ class StepsErrorReason(StrEnum):
     DUPLICATE_DRAWN_VALUE = "duplicate_drawn_value"
     """A drawn dimension's `values` repeat a `ref`."""
 
+    SOURCE_CODES_NOT_ASCENDING = "source_codes_not_ascending"
+    """A `source_codes` list, under `about` or on a drawn value, is not
+    strictly ascending by provenance and then code: it is out of order or
+    lists a `(provenance, code)` pair twice, with or without `exact`."""
+
 
 class StepsError(ValueError):
     """A step list or description breaks the structure rules.
@@ -156,7 +162,9 @@ def check_description(description: Mapping[str, Any]) -> None:
     step in `steps`. A gap names a step when its path has at least three
     segments and starts `steps.`, as in `steps.s3.op`.
     A drawn dimension that lists `values` must list `count` of them, each
-    `ref` once.
+    `ref` once. Every `source_codes` list, under `about` and on each drawn
+    value, must be strictly ascending by provenance and then code, which
+    sorts it and holds each pair once.
 
     Args:
         description: A description that already validates against
@@ -188,6 +196,7 @@ def check_description(description: Mapping[str, Any]) -> None:
                 detail=f"gap {gap!r} names step {segments[1]!r}",
             )
     _check_drawn_values(description)
+    _check_source_codes(description)
 
 
 def _check_count(steps: Sequence[Mapping[str, Any]]) -> None:
@@ -276,3 +285,23 @@ def _check_drawn_values(description: Mapping[str, Any]) -> None:
                 step=None,
                 detail=f"drawn dimension {position} lists a ref twice",
             )
+
+
+def _check_source_codes(description: Mapping[str, Any]) -> None:
+    """Each `source_codes` list is strictly ascending by `(provenance, code)`."""
+    about = description["about"]
+    lists = [("about.source_codes", about.get("source_codes"))]
+    drawn = about.get("drawn") or {}
+    for position, dimension in enumerate(drawn.get("dimensions", [])):
+        for index, value in enumerate(dimension.get("values") or []):
+            where = f"drawn dimension {position} value {index}"
+            lists.append((where, value.get("source_codes")))
+    for where, codes in lists:
+        keys = [(code["provenance"], code["code"]) for code in codes or []]
+        for earlier, later in pairwise(keys):
+            if earlier >= later:
+                raise StepsError(
+                    reason=StepsErrorReason.SOURCE_CODES_NOT_ASCENDING,
+                    step=None,
+                    detail=f"{where} lists {later} after {earlier}",
+                )