Add GARI and generic detector layout support to decoder CLIs

Rebase the final PR quantumlib#277 feature onto detector traversal semantics from PR quantumlib#306. Remove the obsolete GARI order inversion so generated physical detector prefixes use detector_at_position directly.

Co-authored-by: Aria Shahingohar <ariash@google.com>

Author: 2 people

README.md

@@ -144,9 +144,78 @@ Using a Detection Event File and Observable Flips:
 ./tesseract --in events.01 --in-format 01 --obs_in obs.01 --obs-in-format 01 --dem surface_code.dem --out decoded.txt
 ```
 
+Tesseract also accepts explicit detector traversal orders in a JSON file. The
+file has the same list-of-lists form as Python's `TesseractConfig.det_orders`,
+and every inner list must be a complete permutation of the DEM detector IDs:
+
+```json
+[[0, 2, 1, 3], [3, 1, 2, 0]]
+```
+
+Pass it with `--detector-orders orders.json`. This replaces the orders normally
+generated by `--num-det-orders`, `--det-order-seed`, and the `--det-order-*`
+method flags.
+
 Tesseract supports reading and writing from all of Stim's standard [output
 formats](https://github.com/quantumlib/Stim/blob/main/doc/result_formats.md).
 
+### Decoding with GARI
+
+Generate a GARI matrix DEM from a source circuit:
+
+```bash
+python src/py/_tesseract_py_util/gari.py \
+    --circuit circuit_file.stim \
+    --prior xor \
+    --out-dir gari_output
+```
+
+This writes `gari_output/circuit_file_gari_xor.dem`. Its physical detector rows preserve the
+source detector IDs, and its added virtual rows form a suffix. GARI-aware detector orders can be
+written in the generic CLI format when reordering is wanted:
+
+```python
+import json
+import stim
+from tesseract_decoder import demutil
+
+circuit = stim.Circuit.from_file("circuit_file.stim")
+gari_dem = stim.DetectorErrorModel.from_file("gari_output/circuit_file_gari_xor.dem")
+orders = demutil.gari.build_detector_orders(circuit, gari_dem, num_det_orders=5)
+with open("gari_orders.json", "w") as f:
+    json.dump(orders, f)
+```
+
+Sample from the source circuit and decode with the generated DEM:
+
+```bash
+./bazel-bin/src/tesseract \
+    --circuit circuit_file.stim \
+    --dem gari_output/circuit_file_gari_xor.dem \
+    --detector-orders gari_orders.json \
+    --sample-num-shots 100 \
+    --sample-seed 1234 \
+    --threads 1 \
+    --pqlimit 1000000 \
+    --beam 5 \
+    --beam-climbing \
+    --no-revisit-dets \
+    --print-stats \
+    --stats-out gari-stats.json
+```
+
+The GARI matrix DEM is a decoding representation and must not be sampled. When a circuit and a
+larger DEM are supplied together, both command-line decoders read or sample the circuit's detector
+prefix and leave the DEM's virtual suffix zero. Their observable counts must agree. When reading a
+source-width event file, pass both `--circuit` and `--dem`; with `--dem` alone the input width is the
+full DEM width.
+
+For matrix analysis, `gari.py --row-order block` writes a `_block.dem` file in the internal
+physical-X, physical-Z, virtual-Z, virtual-X row order. This research form does not accept source
+syndromes as a direct prefix. See
+[GARI transformed matrices](src/py/README.md#gari-transformed-matrices) for supported circuit
+conventions and Python API details.
+
 ### Performance Optimization
 
 Here are some tips for improving performance:

docs/tutorial.ipynb

@@ -935,7 +935,7 @@
       "source": [
         "gari = tesseract_decoder.demutil.gari\n",
         "\n",
-        "gari_dem, gari_layout = gari.circuit_to_gari(\n",
+        "gari_dem = gari.circuit_to_gari(\n",
         "    circuit,\n",
         "    prior_function=gari.tesseract_xor_prior_probabilities,\n",
         ")"
@@ -950,7 +950,8 @@
       "source": [
         "Sample detection events only from the original circuit. The GARI matrix DEM\n",
         "stores the transformed matrices for decoding and is not sampled. Copy the\n",
-        "source syndrome into its physical rows; the added virtual entries stay zero."
+        "source syndrome into the detector prefix; the added virtual entries stay\n",
+        "zero."
       ]
     },
     {
@@ -964,34 +965,26 @@
       "source": [
         "num_shots = 10\n",
         "gari_dets = np.zeros((num_shots, gari_dem.num_detectors), dtype=bool)\n",
-        "gari_dets[:, gari_layout[\"source_to_gari\"]] = dets[:num_shots]"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "abc39564",
-      "metadata": {
-        "id": "gari-detector-order"
-      },
-      "source": [
-        "The layout is physical-then-virtual. Setting `num_det_orders=0` selects one\n",
-        "ascending detector order, so Tesseract processes the rows in that order."
+        "gari_dets[:, :dets.shape[1]] = dets[:num_shots]"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
-      "id": "bc0c802c",
+      "id": "3e397ee8",
       "metadata": {
         "id": "gari-decode-example"
       },
       "outputs": [],
       "source": [
-        "short_beam = tesseract_decoder.make_tesseract_sinter_decoders_dict()[\n",
-        "    \"tesseract-short-beam\"\n",
-        "]\n",
-        "short_beam.num_det_orders = 0\n",
-        "gari_decoder = short_beam.compile_decoder_for_dem(dem=gari_dem).decoder\n",
+        "gari_config = tesseract.TesseractConfig(\n",
+        "    dem=gari_dem,\n",
+        "    det_beam=15,\n",
+        "    beam_climbing=True,\n",
+        "    no_revisit_dets=True,\n",
+        "    pqlimit=200_000,\n",
+        ")\n",
+        "gari_decoder = gari_config.compile_decoder()\n",
         "predicted_observables = gari_decoder.decode_batch(gari_dets)\n",
         "logical_failures = np.count_nonzero(\n",
         "    np.any(predicted_observables != obs[:num_shots], axis=1)\n",

docs/tutorial.py

@@ -358,31 +358,31 @@ def run_tesseract_decoder(decoder, dets, obs):
 # %% id="gari-transform-example"
 gari = tesseract_decoder.demutil.gari
 
-gari_dem, gari_layout = gari.circuit_to_gari(
+gari_dem = gari.circuit_to_gari(
     circuit,
     prior_function=gari.tesseract_xor_prior_probabilities,
 )
 
 # %% [markdown] id="gari-syndrome-layout"
 # Sample detection events only from the original circuit. The GARI matrix DEM
 # stores the transformed matrices for decoding and is not sampled. Copy the
-# source syndrome into its physical rows; the added virtual entries stay zero.
+# source syndrome into the detector prefix; the added virtual entries stay
+# zero.
 
 # %% id="gari-sample-example"
 num_shots = 10
 gari_dets = np.zeros((num_shots, gari_dem.num_detectors), dtype=bool)
-gari_dets[:, gari_layout["source_to_gari"]] = dets[:num_shots]
-
-# %% [markdown] id="gari-detector-order"
-# The layout is physical-then-virtual. Setting `num_det_orders=0` selects one
-# ascending detector order, so Tesseract processes the rows in that order.
+gari_dets[:, :dets.shape[1]] = dets[:num_shots]
 
 # %% id="gari-decode-example"
-short_beam = tesseract_decoder.make_tesseract_sinter_decoders_dict()[
-    "tesseract-short-beam"
-]
-short_beam.num_det_orders = 0
-gari_decoder = short_beam.compile_decoder_for_dem(dem=gari_dem).decoder
+gari_config = tesseract.TesseractConfig(
+    dem=gari_dem,
+    det_beam=15,
+    beam_climbing=True,
+    no_revisit_dets=True,
+    pqlimit=200_000,
+)
+gari_decoder = gari_config.compile_decoder()
 predicted_observables = gari_decoder.decode_batch(gari_dets)
 logical_failures = np.count_nonzero(
     np.any(predicted_observables != obs[:num_shots], axis=1)

src/py/README.md

@@ -707,29 +707,34 @@ nice_calibrated_dem = demutil.regeneralize_spatial_dem(
 #### GARI transformed matrices
 
 `demutil.gari.circuit_to_gari` converts a supported correlated CSS Stim
-circuit into a GARI matrix DEM and companion layout for Tesseract. It
-generates a flattened source DEM with `decompose_errors=False`. Detectors must
-follow the repository's fourth-coordinate convention: values `0`–`2` identify
-X detectors and `3`–`5` identify Z detectors.
+circuit into a GARI matrix DEM. It generates a flattened source DEM with
+`decompose_errors=False`. Detectors must follow the repository's
+fourth-coordinate convention: values `0`–`2` identify X detectors and `3`–`5`
+identify Z detectors.
 
 ```python
 import stim
 from tesseract_decoder import demutil
 
 circuit = stim.Circuit.from_file("circuitFile.stim")
-gari_dem, gari_layout = demutil.gari.circuit_to_gari(
+gari_dem = demutil.gari.circuit_to_gari(
     circuit,
     prior_function=demutil.gari.tesseract_xor_prior_probabilities,
 )
 ```
 
-`circuit_to_gari` returns:
+The returned DEM preserves the source detector IDs as a prefix and appends the
+virtual detector rows. For matrix analysis,
+`circuit_to_gari(..., row_order="block")` instead emits the internal
+`[physical X, physical Z, virtual Z, virtual X]` row order. This research form
+does not accept source syndromes as a direct prefix.
 
-* `gari_dem`: the augmented detector and logical matrices stored using Stim
-  DEM syntax.
-* `gari_layout`: a `tesseract.gari_layout.v1` dictionary containing the source
-  and GARI detector counts, the `source_to_gari` detector mapping, and the
-  `physical_then_virtual` detector order.
+`demutil.gari.build_detector_orders(circuit, gari_dem, num_det_orders, ...)`
+uses the source circuit to build BFS, coordinate, or index orders and then
+appends the virtual detector IDs. The resulting list has the same format as
+`TesseractConfig.det_orders` and the Tesseract CLI's `--detector-orders` JSON
+file. It applies to the default source-aligned GARI DEM, not the research-only
+block form.
 
 Related public APIs:
 
@@ -738,13 +743,14 @@ Related public APIs:
 * `demutil.gari.GariTransform` is passed to prior-policy callbacks. It exposes
   the transformed detector and logical matrices, the `U` and `V` projection
   matrices, the source `e_Z`, `e_X`, and `e_Y` column indices, and the source
-  detector mapping.
+  detector mapping into the internal block rows.
 * `paper_prior_probabilities`, `tesseract_xor_prior_probabilities`, and
   `tesseract_lp_max_barred_cost_prior_probabilities` return one probability for
   each transformed GARI column. A user-defined prior can follow the same
   callable interface.
 
 The returned GARI matrix DEM stores transformed matrices for decoding and must
-not be sampled. Sample from the original circuit and use the companion layout
-to place its physical syndrome. See the
+not be sampled. Sample from the original circuit, copy its syndrome into the
+beginning of a zero-filled GARI syndrome, and leave the virtual suffix zero.
+See the
 [GARI tutorial](../../docs/tutorial.ipynb) for a complete decoding example.