REV3 ring-buffer detection (#18 / 796cfdc) misroutes most legacy MBB files

[cecil-t]

Flagging a side effect of the REV3 detection added in #18 (commit 796cfdc, by ilja-radusch).

The condition len(raw())==0x40000 and index_of_sequence(b'\xa1\xa1\xa1\xa1') is meant to catch the new ring-buffer format, but classic REV0/REV1 files also carry a \xa1\xa1\xa1\xa1 marker near their own header, for an unrelated field, and they're almost always exactly 262144 bytes too. index_of_sequence returns the first match in the file, so it finds this early legacy marker just as easily as a genuine ring-buffer one, and the condition can't tell them apart.

Checked this against a large set of legacy MBB files by independently verifying where a real, check-digit-valid VIN actually sits in each file (0x240 for REV0, 0x252 for REV1), bypassing the parser's own branching entirely. 2,987 of 3,635 legacy files, 82%, are being routed into REV3 by this condition even though they're genuinely REV0 or REV1. Once routed there, Serial number, Firmware rev., Board rev. and Model all come back as the literal string "Unknown," and the VIN field only looks correct because that branch falls back to copying it from the filename rather than decoding it from the file.

I confirmed the FST platform files this was written for are correctly routed, so this isn't wrong in general, it's specifically the 262144-byte-plus-marker check overlapping with the legacy format's own layout.

I've got a fix I'm testing that requires the marker to actually indicate ring-buffer structure rather than just being present somewhere in a 262144-byte file. Will follow up with numbers once I've validated it against the full data set.

[cecil-t]

Update, since writing this up: this is now deployed and running against live uploads on my online parser, and validated against my full data set offline. Ran a handful of real files through it since deploying, parsing clean.

Follow-up with the numbers:

Root cause: get_version_and_header()'s REV3 (ring-buffer) detection, added in 796cfdc, routes any file that is exactly 262,144 bytes and contains \xa1\xa1\xa1\xa1 anywhere into the ring-buffer branch. Classic REV0/REV1 files are almost always exactly that size too, and carry their own unrelated \xa1\xa1\xa1\xa1 marker near their own header. Result: 2,987 of 3,635 legacy MBB files in my data set (82.2%) were being routed into REV3, which hardcodes Serial number, Firmware rev., Board rev., and Model to 'Unknown', the same four fields from the original 2017 report, broken again by different, newer code.

The fix: try the classic VIN-offset detection first; only fall back to the ring-buffer fix if none of the three legacy offsets produce a valid VIN. The raw()[0] == 0xb2 check that correctly identifies real ring-buffer files is untouched.

Validated on the full data set (3,635 legacy MBB files, content-hash deduplicated, cross-checked against an independent VIN-offset):

2,987 files move from misrouted to correctly routed. Zero regressions. Every file that was correctly routed before still is.
FST-platform files (418, the population this detection targets): 417/418 still correctly route to REV3; the one exception was already broken before this fix for an unrelated reason (a 644-byte malformed capture) and is now more accurately labeled, not newly broken.
Serial number plausibility on 2017+ legacy files: 2.2% → 94.0%. Firmware rev.: 1.3% → 94.0%. Model: 1.4% → 94.1%. Board rev. present at all: 1.3% → 94.0%.
The existing # TODO confirm Board rev. candidate offset for REV1 (0x268) now runs on 1,958 files instead of 28: same 8 distinct values, 93.2% stable per-VIN, holds up at the larger sample.

One more thing caught on self-review before calling this done: the 79 files that fall through to neither a ring-buffer nor a legacy VIN match were missing three header keys entirely (Serial number/Firmware rev./Board rev. simply absent from the dict, rather than 'Unknown' like every other unidentifiable-field case in this function). Audited every downstream consumer (text/CSV/JSON emitters, the +/+= merge operators), there is no crash risk, but those three fields now explicitly read 'Unknown' for this bucket too, matching REV3's own pattern. Re-ran the full validation after this change: identical numbers across the board, confirming it only touched those 79 files' dict completeness.

On PR #19 (merged just before this): no conflict, different function entirely. The data set has 117 files matching that same description. Checked whether the two bugs were compounding (the "Board Status / No additional data" ladder PR #19 describes): they weren't. PR #19's fix lives in the shared entry loop used by REV0/REV1/REV3 alike, so it already covered misrouted files the same as correctly-routed ones.