|
| 1 | +# Phase 7 Reliability/Readiness Score (RS) Computation — Sprint 4 — 2026-09-21 |
| 2 | + |
| 3 | +## Scope note (read before the Inputs section) |
| 4 | + |
| 5 | +Sprint 2's and Sprint 3's RS documents (`sprint2-ar5-rs-computation.md`, |
| 6 | +`sprint3-ar5-rs-computation.md`) both source their `a`/`b` inputs from a dedicated, |
| 7 | +on-disk Phase C gate-results artifact (`sprint2-phase-c-gate-results.md`, |
| 8 | +`sprint3-phase-c-gate-results.md`), each itself the output of a paired-detector run |
| 9 | +(C-1 hallucination-detector for NLI faithfulness + FactScore, cross-checked by a C-2 |
| 10 | +RAGAS Faithfulness pass). **No `sprint4-phase-c-gate-results.md` exists in this repo.** |
| 11 | +The task brief that requested this document states the bundle (`sprint4_ar0_routing_index.json`, |
| 12 | +`sprint4_ar1_assignments.json`, `sprint4_ar3_context_windows.json`, |
| 13 | +`sprint4_implementation_execution_plan.json`) is "all Phase-C-approved" — this document |
| 14 | +accepts that as the given premise of the task it was asked to compute against, but it |
| 15 | +is **not** independently re-derived from a paired-detector artifact the way Sprint 2/3's |
| 16 | +`a`/`b` were. This is disclosed as a methodology gap rather than papered over: `a` and |
| 17 | +`b` below are this session's own single-reviewer citation/consistency check (performed |
| 18 | +while reading all four bundle files in full for this task), not a two-detector census. |
| 19 | +This is a materially weaker evidentiary basis than Sprint 2/3 carried, and the Verdict |
| 20 | +section says so plainly rather than reporting an unqualified 1.00. |
| 21 | + |
| 22 | +## Inputs |
| 23 | + |
| 24 | +- **a (citation/consistency self-check, in place of C-1 NLI faithfulness):** **1.00** — |
| 25 | + this session read `sprint4_ar0_routing_index.json`, `sprint4_ar1_assignments.json`, |
| 26 | + `sprint4_ar3_context_windows.json`, and `sprint4_implementation_execution_plan.json` |
| 27 | + in full and cross-checked every score, file:line citation, and FR/GitHub-issue |
| 28 | + reference that carries forward between them: `python-backend-engineer` 0.900, |
| 29 | + `integration-testing-engineer` 0.825, `api-security-auditor` 0.800 are identical in |
| 30 | + AR.0's `candidates[]`/`advisory_review_roster` and AR.1's `assigned_agent`/`score` |
| 31 | + fields for all three sub-tasks; the 7000/5500/5000-token `context_budget_tokens` |
| 32 | + values are identical between AR.3's `context_windows[]` and the IEP prompts' own |
| 33 | + "Context Budget" line; FR-013 and GitHub #23 are cited consistently (never |
| 34 | + paraphrased, never renumbered) across all four files; GitHub #25 is disclosed in all |
| 35 | + three later files as a named, explicitly-not-fixed adjacent defect, never silently |
| 36 | + dropped. **0 contradictions found** across this pass. This is a direct read, not a |
| 37 | + sample — but it is one reviewer's pass, not the paired NLI-detector census Sprint 2/3 |
| 38 | + report. |
| 39 | +- **b (atomic-fact self-check, in place of C-1 FactScore / C-2 RAGAS):** **1.00** — |
| 40 | + spot-checked atomic claims against their own named sources during the same read: the |
| 41 | + design doc's own status line (`docs/phase-1.5-api/fr013-predicate-schema-design.md:3`, |
| 42 | + "Status: IMPLEMENTATION-READY — pending user review. No implementation files have |
| 43 | + been modified.") matches every file's restatement of it; the HLD Section 3.4/3.8 |
| 44 | + citations in AR.0/AR.1 are the same citations Sprint 3's own routing files already |
| 45 | + used for the same underlying mechanism (conflict-detection sweep, routing truth |
| 46 | + table), not a fresh, unverifiable claim; the `dependency_integrity_check.sprint_4_edges` |
| 47 | + in AR.1 (`DEV: []`, `QA: [DEV]`, `REVIEW: [DEV, QA]`) matches the IEP's own |
| 48 | + `depends_on`/`sequencing_note` fields and its `file_collision_analysis.collision_finding` |
| 49 | + of `NONE` exactly. No fabricated or unsupported atomic claim was found. As with `a`, |
| 50 | + this is a single-reviewer check performed inline with this task, not an independent |
| 51 | + second-detector corroboration — Sprint 4 has no C-2-equivalent pass recorded anywhere. |
| 52 | +- **c (1-ECE, calibration):** **N/A** — unchanged from every prior Phase 7 RS document: |
| 53 | + no logit/confidence calibration data exists at this routing stage. |
| 54 | +- **d (DRE / Coverage):** **N/A** — Sprint 4 is pre-Phase-B. The IEP's own |
| 55 | + `scope_note` states plainly: "Phase B (actual code implementation) is explicitly OUT |
| 56 | + OF SCOPE for this project bundle... This file stores 3 dispatch-ready CoT prompts... |
| 57 | + PREPARED FOR FUTURE USE, NOT EXECUTED NOW." No Sprint 4 code or tests exist to |
| 58 | + compute a defect rate or coverage figure against, the same posture as the Phase 7 |
| 59 | + baseline and Sprint 1-3 gates at their own pre-Phase-B stage. |
| 60 | + |
| 61 | +## Scope & maturity context (qualitative, not folded into the RS formula) |
| 62 | + |
| 63 | +The task instruction names three reasons Sprint 4 should read as more ready than a |
| 64 | +typical routing-stage snapshot, none of which are `a`/`b`/`c`/`d` inputs but all of |
| 65 | +which are worth recording rather than silently assumed: |
| 66 | + |
| 67 | +1. **Design-doc maturity:** `fr013-predicate-schema-design.md` is at v7, scored 9.8/10, |
| 68 | + status IMPLEMENTATION-READY, after seven review rounds (per its own Change Log, |
| 69 | + cited identically across all four bundle files) — a materially higher design-maturity |
| 70 | + floor than Sprint 1-3's routing files worked from, none of which cite a |
| 71 | + seven-round, 9.8/10 design doc as their single source of truth. |
| 72 | +2. **Bundle Phase-C-approval (as given, not re-derived here):** the task premise states |
| 73 | + the bundle already cleared Phase C; this document's own `a`/`b` self-check (above) |
| 74 | + found nothing to contradict that, but — per the Scope note — cannot itself stand in |
| 75 | + for the paired-detector artifact that would let a future reader independently audit |
| 76 | + the claim the way `sprint3-phase-c-gate-results.md` lets a reader audit Sprint 3's. |
| 77 | +3. **Narrow, well-bounded scope:** one story (`DASH-STORY-026`), three sequential |
| 78 | + sub-tasks, one file cluster (`openapi.yaml`, `schemas.py`, `app.py` owned solely by |
| 79 | + Dev; the new test file owned solely by QA). The IEP's own |
| 80 | + `file_collision_analysis.collision_finding` is `NONE` and its |
| 81 | + `parallelization_opportunity` is `NONE` — not because parallelization was missed, |
| 82 | + but because a strict Dev→QA→Review dependency chain (`sprint_4_edges`) makes |
| 83 | + concurrent dispatch structurally unsafe, and the IEP says so rather than fabricating |
| 84 | + a parallel track. This is a smaller, simpler routing surface than any prior sprint's |
| 85 | + (Sprint 1: 7 stories, Sprint 2: 9, Sprint 3: 6), which lowers the chance of an |
| 86 | + undetected routing/collision defect independent of how well `a`/`b` were checked. |
| 87 | + |
| 88 | +None of these three raise `a` or `b` above what was directly checked — they are |
| 89 | +recorded here as the reasons a low-population census (below) is plausible, not as |
| 90 | +inputs multiplied into the RS formula. |
| 91 | + |
| 92 | +## Computation |
| 93 | + |
| 94 | +Phase-7-scoped RS (2-component geometric mean of the genuinely measurable components |
| 95 | +`a`, `b`), mirroring `sprint2-ar5-rs-computation.md` / `sprint3-ar5-rs-computation.md`'s |
| 96 | +method exactly: |
| 97 | + |
| 98 | +``` |
| 99 | +RS_phase7_sprint4 = sqrt(a x b) = sqrt(1.00 x 1.00) = 1.00 |
| 100 | +``` |
| 101 | + |
| 102 | +**On the confidence interval:** this was a full, non-sampled read of all four bundle |
| 103 | +files (not a sample of them), so — as with Sprint 2/3 — a delta-method/binomial |
| 104 | +sampling CI is not the correct tool for the checked population itself. For |
| 105 | +transparency, the same rule-of-three-style bound Sprint 3 applied is shown here at |
| 106 | +Sprint 4's own finer-grained units: |
| 107 | + |
| 108 | +- **9 acceptance-criteria entries** (AC-026-DEV-1/2/3, AC-026-QA-1/2/3, |
| 109 | + AC-026-REVIEW-1/2/3, per `sprint4_ar1_assignments.json`) — the structural analog to |
| 110 | + Sprint 3's 16-AC unit. Rule-of-three at n=9: 3/9 = 0.333, implying a conservative |
| 111 | + floor of `sqrt(0.667 x 0.667) ~= 0.67` on a hypothetical resampled population. |
| 112 | +- **26 context-window source-citation entries** (12 for DEV + 7 for QA + 7 for REVIEW, |
| 113 | + summing `files_to_read_or_modify` + `domain_files_cited_across_review_rounds` + |
| 114 | + `github_issues` + `cross_reference` across `sprint4_ar3_context_windows.json`'s three |
| 115 | + `context_windows[]` entries) — the analog to Sprint 3's 71-source unit. Rule-of-three |
| 116 | + at n=26: 3/26 ~= 0.115, implying a floor of `sqrt(0.885 x 0.885) ~= 0.89`. |
| 117 | + |
| 118 | +Both floors are markedly looser than Sprint 3's (0.81 at n=16, 0.96 at n=71) precisely |
| 119 | +*because* Sprint 4's population is smaller — a single story with three sub-tasks |
| 120 | +produces far fewer atomic units than a six-story sprint, so the same "3 hypothetical |
| 121 | +undiscovered defects" assumption costs proportionally more headroom here. This is |
| 122 | +disclosed as a genuine consequence of Sprint 4's narrow scope, not a defect in the |
| 123 | +method: the narrower the checked population, the less a rule-of-three bound can say, |
| 124 | +and the more the reported RS has to rest on the census itself (100% of a and b actually |
| 125 | +checked clean) rather than on the statistical bound as reassurance. Both floors are |
| 126 | +disclosed for transparency and are **not** the reported RS — the reported RS is the |
| 127 | +direct census result, 1.00, since 100% of the checkable population (all four bundle |
| 128 | +files, all 9 ACs, all 26 source citations) was verified clean on `a` and `b` by this |
| 129 | +session's own single-reviewer pass, with zero contradictions found. |
| 130 | + |
| 131 | +## Verdict |
| 132 | + |
| 133 | +**CONDITIONAL_PASS, with a disclosed evidentiary gap** — RS_phase7_sprint4 = 1.00 >= |
| 134 | +0.95 threshold, on the two components measurable at this phase, computed from this |
| 135 | +session's own direct read of the four Sprint 4 routing artifacts (not Sprint 4 |
| 136 | +implementation code, which does not yet exist — the IEP is explicit that Phase B is |
| 137 | +out of scope for this bundle). |
| 138 | + |
| 139 | +Unlike Sprint 2/3, this PASS is **not** backed by a paired-detector |
| 140 | +`sprintN-phase-c-gate-results.md` artifact — none exists for Sprint 4. The task |
| 141 | +premise's claim that the bundle is "Phase-C-approved" is accepted as given here, and |
| 142 | +this document's own self-check corroborates it (zero contradictions across `a`/`b`), |
| 143 | +but a future reader auditing Sprint 4 the way `sprint3-phase-c-gate-results.md` lets a |
| 144 | +reader audit Sprint 3 has no equivalent artifact to read. **Recommended before Phase B |
| 145 | +dispatch:** run a dedicated Sprint 4 Phase C gate pass (C-1 hallucination-detector + |
| 146 | +C-2 RAGAS Faithfulness, same as Sprint 2/3) against |
| 147 | +`sprint4_ar1_assignments.json`/`sprint4_ar3_context_windows.json`/ |
| 148 | +`sprint4_implementation_execution_plan.json` and record it as |
| 149 | +`sprint4-phase-c-gate-results.md`, so Sprint 4 carries the same evidentiary standard as |
| 150 | +Sprint 2/3 rather than resting on this document's single-reviewer substitute. This is |
| 151 | +not a retry-eligible failure under the SC.1-3 bounded-retry protocol — it is a |
| 152 | +disclosed methodology gap, not a gate rejection, and does not block the two open |
| 153 | +decisions (`SP4-DEC-201`, `SP4-DEC-202`) the bundle itself already carries as open |
| 154 | +before dispatch. |
| 155 | + |
| 156 | +`c` and `d` remain N/A, matching the Phase 7 baseline and Sprint 1-3 gates at the same |
| 157 | +pre-Phase-B stage — not approximated or defaulted. |
| 158 | + |
| 159 | +## Source |
| 160 | + |
| 161 | +`a`/`b` in this document are sourced from this session's own direct, full read of |
| 162 | +`sprint4_ar0_routing_index.json`, `sprint4_ar1_assignments.json`, |
| 163 | +`sprint4_ar3_context_windows.json`, and `sprint4_implementation_execution_plan.json` |
| 164 | +(all read in full this session), cross-checked against |
| 165 | +`docs/phase-1.5-api/fr013-predicate-schema-design.md` v7's own status line (line 3) and |
| 166 | +GitHub issue #23/#25 citations as those four bundle files themselves restate them — not |
| 167 | +from a separate `sprint4-phase-c-gate-results.md`, because none exists (see Scope note |
| 168 | +above). Computation method (2-component geometric mean of `a` and `b`; `c`, `d` |
| 169 | +reported N/A when their underlying artifacts do not yet exist; full-census treatment in |
| 170 | +place of a sampling CI; a disclosed rule-of-three bound at the sprint's own |
| 171 | +finer-grained units) mirrors `docs/phase-7-routing/sprint2-ar5-rs-computation.md` and |
| 172 | +`docs/phase-7-routing/sprint3-ar5-rs-computation.md` exactly, as instructed. No score in |
| 173 | +this document was invented or estimated beyond what this session's own read of the |
| 174 | +named files supports. |
0 commit comments