|
| 1 | +# Exhaustive GT/AUTO human-review gate |
| 2 | + |
| 3 | +[English](HUMAN_REVIEW.md) | [简体中文](HUMAN_REVIEW.zh-CN.md) |
| 4 | + |
| 5 | +## Motivation |
| 6 | + |
| 7 | +A Teacher prediction is evidence, not ground truth. Confidence alone cannot safely decide whether a box is a missing object, a duplicate of an existing annotation, a differently sized box for the same target, or a conflict with another class. The review gate makes those cases explicit before any derived label dataset is created. |
| 8 | + |
| 9 | +## Complete image/class state space |
| 10 | + |
| 11 | +Every image and class belongs to exactly one state: |
| 12 | + |
| 13 | +| State | Existing GT | Teacher evidence | Default handling | |
| 14 | +|---|---:|---:|---| |
| 15 | +| `GT0_AUTO0` | No | No | No action; retained for coverage accounting | |
| 16 | +| `GT1_AUTO0` | Yes | No | Keep the existing GT | |
| 17 | +| `GT0_AUTO1` | No | Yes | Evaluate confidence, geometry and split policy | |
| 18 | +| `GT1_AUTO1` | Yes | Yes | Resolve already-labelled, same-target ambiguity or distinct targets | |
| 19 | + |
| 20 | +The matrix is generated for every image/class pair, not only for images containing candidates. This prevents a report from silently ignoring absence cases. |
| 21 | + |
| 22 | +## Complementary geometry |
| 23 | + |
| 24 | +For candidate box $A$ and target box $B$, intersection-over-union is: |
| 25 | + |
| 26 | +$$ |
| 27 | +\operatorname{IoU}(A,B)=\frac{|A\cap B|}{|A\cup B|} |
| 28 | +$$ |
| 29 | + |
| 30 | +IoU becomes small when one valid box is contained in a much larger box. Intersection-over-smaller-area therefore provides a second signal: |
| 31 | + |
| 32 | +$$ |
| 33 | +\operatorname{IoS}(A,B)=\frac{|A\cap B|}{\min(|A|,|B|)} |
| 34 | +$$ |
| 35 | + |
| 36 | +The normalized center distance is: |
| 37 | + |
| 38 | +$$ |
| 39 | +d_c(A,B)=\frac{\sqrt{(c_x^A-c_x^B)^2+(c_y^A-c_y^B)^2}}{\sqrt{\min(|A|,|B|)}} |
| 40 | +$$ |
| 41 | + |
| 42 | +The area ratio is: |
| 43 | + |
| 44 | +$$ |
| 45 | +r_a(A,B)=\frac{\max(|A|,|B|)}{\min(|A|,|B|)} |
| 46 | +$$ |
| 47 | + |
| 48 | +The default policy uses these signals as follows: |
| 49 | + |
| 50 | +- Already labelled: `IoU >= 0.60`, or `IoS >= 0.90` with normalized center distance `<= 0.35`. |
| 51 | +- Same target, ambiguous extent: `IoU >= 0.20`, or `IoS >= 0.55` with normalized center distance `<= 0.80`. |
| 52 | +- Near-duplicate Teacher candidates: same-class candidate IoU `>= 0.90`; only the higher-confidence candidate survives. |
| 53 | +- Cross-class conflict: cross-class IoU `>= 0.65` and area ratio `<= 2.0`; route to a human instead of deleting a potentially valid nested small object. |
| 54 | + |
| 55 | +## Candidate terminal states |
| 56 | + |
| 57 | +Each Teacher candidate receives one terminal state: |
| 58 | + |
| 59 | +| Terminal state | Human review | Allowed result | |
| 60 | +|---|---:|---| |
| 61 | +| `INVALID` | No | Reject | |
| 62 | +| `MODEL_DUPLICATE` | No | Reject lower-confidence duplicate | |
| 63 | +| `GT_ALREADY_LABELED` | No | Keep GT, do not add another box | |
| 64 | +| `GT_SAME_AMBIGUOUS` | Yes | Explicitly replace the matched GT or reject | |
| 65 | +| `GT_CROSS_CLASS_CONFLICT` | Yes | Add as a distinct class or reject | |
| 66 | +| `TRAIN_MISSING_HIGH/MEDIUM` | Yes | Add or reject | |
| 67 | +| `EVAL_MISSING_HIGH/MEDIUM` | Yes | Accept as gold-evaluation evidence or reject | |
| 68 | +| `BELOW_REVIEW_THRESHOLD` | No by default | Preserve in the audit table; optionally include in review | |
| 69 | +| `SPLIT_DISABLED` | No | Reject | |
| 70 | + |
| 71 | +High confidence does not bypass the human gate. The policy separates evidence strength from permission to change labels. |
| 72 | + |
| 73 | +## Offline review bundle |
| 74 | + |
| 75 | +```powershell |
| 76 | +yolo-label-recovery review-build D:\data\dataset D:\runs\candidates_all.csv ` |
| 77 | + --policy configs\review_policy.example.yaml ` |
| 78 | + --output-dir D:\runs\company-review ` |
| 79 | + --render ` |
| 80 | + --redact-paths |
| 81 | +``` |
| 82 | + |
| 83 | +The output contains the complete audit table, review queue, decision template, rendered candidates, matrix coverage, an HTML summary and an offline Tk reviewer. The reviewer autosaves `company_decisions.csv` and resumes from the first unfinished row. |
| 84 | + |
| 85 | +Review keys: |
| 86 | + |
| 87 | +- `A`: accept a distinct missing box. |
| 88 | +- `P`: replace the highlighted same-class GT. |
| 89 | +- `E`: accept a val/test gold-label candidate. |
| 90 | +- `D`: reject. |
| 91 | +- `U`: temporarily uncertain; must be resolved before apply. |
| 92 | + |
| 93 | +## Safe apply |
| 94 | + |
| 95 | +```powershell |
| 96 | +yolo-label-recovery review-apply D:\data\dataset D:\runs\company-review\company_decisions.csv ` |
| 97 | + --policy configs\review_policy.example.yaml ` |
| 98 | + --output-root D:\data\dataset-reviewed |
| 99 | +``` |
| 100 | + |
| 101 | +The apply stage enforces the following invariants: |
| 102 | + |
| 103 | +- Blank and uncertain decisions block the entire operation. |
| 104 | +- The output cannot equal, contain or be a parent of the source dataset. |
| 105 | +- Source images and labels are never modified. |
| 106 | +- Replace is valid only for `GT_SAME_AMBIGUOUS` and only if the referenced GT has not changed since review. |
| 107 | +- Accepted additions are checked again against the latest derived labels before writing. |
| 108 | +- Val/test decisions are held by default to avoid silently changing the evaluation contract. |
| 109 | +- Every applied, replaced, held and rejected row remains auditable. |
| 110 | + |
| 111 | +## Public demo |
| 112 | + |
| 113 | +```powershell |
| 114 | +python examples\create_review_fixture.py --output-dir .demo-review-fixture |
| 115 | +yolo-label-recovery review-build .demo-review-fixture\dataset .demo-review-fixture\candidates.csv ` |
| 116 | + --output-dir .demo-review-result --render --redact-paths |
| 117 | +``` |
| 118 | + |
| 119 | +The fixture intentionally covers all four image/class states and the main candidate terminal decisions without requiring a GPU, private image or model weight. |
0 commit comments