fix: retain failed BLE evidence snapshots

Author: DevanMetz

.github/workflows/release.yml

@@ -37,6 +37,8 @@ jobs:
         run: python3 tools/test_ride_history.py --compiler cc
       - name: Test ride recording
         run: python3 tools/test_ride_recording.py --compiler cc
+      - name: Test BLE evidence storage
+        run: python3 tools/test_ble_evidence.py --compiler cc
       - name: Test Ring history storage
         run: python3 tools/test_ring_history.py --compiler cc
       - name: Test ebook pages
@@ -178,6 +180,22 @@ jobs:
           if-no-files-found: warn
           retention-days: 14
 
+      - name: Retain BLE evidence storage checks
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: ble-evidence-logs
+          path: |
+            build/ble-evidence/compile.log
+            build/ble-evidence/result.log
+            build/ble-evidence/source.json
+            build/ble-evidence/ble_evidence.inc
+            build/ble-evidence/ble_evidence_types.inc
+            build/ble-evidence/ble_evidence_config.inc
+            build/ble-evidence/storage_source.inc
+          if-no-files-found: warn
+          retention-days: 14
+
       - name: Retain Ring history test evidence
         uses: actions/upload-artifact@v4
         if: always()

ROADMAP.md

@@ -164,6 +164,7 @@ Phase 3 software progress has begun:
 - [x] MQTT PASTE BYTES prepares a separate 0-128-byte publish draft from Byte Lab, including empty payloads. Explicit Text/Bytes selection preserves the current text field; queued requests retain captured bytes/topic/QoS/retain through later edits and Home. Paste/mode changes never send or log, cleanup rejects stale taps, and profiles/metadata exclude the body. Ten native scenarios cover every length, pending/active ownership, errors and 25 stable cycles. Four actual Byte Lab/SDK/broker round trips cover QoS 0/1/2 and empty data; three private mutations are rejected. The built firmware remains unflashed; physical validation is open.
 - [ ] Validate authenticated TLS and WSS brokers, saved-profile reboot/overwrite/delete/corruption recovery, cleartext confirmation/expiry, QoS 0/2, wildcard subscriptions, fragmentation and receive-ring overflow, failed auth/DNS/TLS/timeout paths, SD full/removal, and the second panel family. NVS encryption remains part of the opt-in hardened-device profile.
 - [x] BLE GATT Explorer reuses the shared NimBLE scanner but never starts it or connects automatically. It caps results at eight devices, 16 services, and 32 characteristics; requires product-specific BLE profiles to be off; exposes UUIDs, handles, and properties; supports explicit reads and one notification/indication subscription; gates 1-64-byte raw writes behind an unchanged five-second confirmation; and atomically exports the bounded evidence snapshot to SD only after an explicit tap.
+- [x] BLE evidence never publishes after a rejected header/row write, retains the named TMP and preserves the first cause through close/error reporting. Creation stops on noncollision directory/probe/open/stream errors and reports `EEXIST` after all ten names are occupied. All 50 native callback/formatter/storage cases pass; the previous callback fails 28 controls, including eight rejected writes incorrectly published. Exact bytes, cache isolation/capacity, admission gates and 25 failure/restart pairs are checked with closed streams/descriptors and balanced modeled locks. Linux CI retains source hashes and logs; physical SD, live BLE, panel and power cuts remain open.
 - [x] BLE COPY LATEST VALUE copies a complete 0-64-byte received payload and disarms write confirmation without BLE traffic. It uses the original value handle, rejects truncated or failed buffer copies, and preserves the shared clipboard through disconnect. A read returns one ATT response, not an assembled long attribute. On-device startup tests cover all lengths 0-64, a 65-byte rejection, chained/short buffers, read errors, notify/indicate, stale connections and disconnect. Live GATT copy and evidence-export checks remain open.
 - [x] On ST7121 hardware, an explicit scan filled the eight-device cap and found the owned `KICKR CORE 4816`. Read-only discovery reported 12 services and the disclosed 32-characteristic cap; Manufacturer Name returned `WahooFitness`; Indoor Bike Data `0x2AD2` subscribed, delivered an 8-byte notification, and unsubscribed. The published CSV contained eight advertisements, 32 characteristic rows, and the correctly labeled notification row. Home released the live connection, immediate re-entry was idle, System reported `Reset: USB`, and internal heap remained 62 KB free/60 KB minimum.
 - [ ] Validate no-target and overflowing scans, connection cancel/failure, pairing/authentication errors, indications and fragmented/over-64-byte values, the raw-write confirmation/expiry path on a disposable target, SD full/removal and retained-temp behavior, interaction with each optional product profile, and the second panel family.

docs/architecture.md

@@ -91,6 +91,8 @@ Ride history shows totals only after recovery, reading and closing the summary s
 
 Ride recording checks both directories and advances through its 100 same-second names only on CSV/TMP collisions. Other creation errors stop Start; a failed stream conversion closes its descriptor and attempts to remove the newly created empty TMP. Rejected headers/rows and periodic sync failures use one abort path that closes recording ownership, retains the TMP and reports the captured first cause. Tick and Stop also abort on known SD loss before another row or publication. Healthy Stop commits the CSV before rebuilding history, so an index failure leaves the published ride authoritative. Native tests compile the actual lifecycle and history helpers together with storage; UI, BLE data, time and SD boundaries are controlled.
 
+BLE evidence copies its bounded shared cache under the existing lock, then writes that snapshot synchronously. Each directory, final-name probe and file-creation boundary uses a fresh error; only actual CSV/TMP collisions advance the ten-name search. Header and row failures capture the first cause and close the stream without invoking publication. The named TMP remains available for inspection, and cleanup/error reporting cannot replace that cause. Healthy saves use the unchanged storage helper and CSV schema. Native checks compile the callback, its format/status helpers and cache typedefs with modeled BLE/UUID/lock boundaries and real files; no worker or LVGL pool change is introduced.
+
 Ring's saved heart-rate timestamp becomes loaded only after incomplete-tail repair, seek/size checks, the expected bounded tail read and close succeed. Read or close errors leave the checkpoint unchanged and retryable. Its existing health timer retains queued samples and defers history requests until that load succeeds, then filters timestamps against the recovered checkpoint. Known SD unavailability blocks queue processing even when a checkpoint is cached; timeout state still advances before the storage gate. Appends preserve the first error through close and tail repair. Native files and controlled queue/timer/SD boundaries verify this behavior; physical SD, BLE and scheduling remain open.
 
 Ebooks reads up to 8 KiB into its existing cached PSRAM buffer. Page text, current/next byte positions, navigation state and scroll position change only after successful seek, read and close. Failure preserves the confirmed page and positions, reports the first error and allows retry when storage is available. Reader initialization clears both offsets and disables navigation until loading succeeds. Callbacks recheck disabled state and screen pointers; negative positions and additions beyond the platform's `long` range are rejected. Native files and controlled UI/heap/SD boundaries verify the transitions; physical SD and panel behavior remain open.

docs/data-formats.md

@@ -151,4 +151,6 @@ unix_time,event,address,name,rssi,service_uuid,characteristic_uuid,handle,proper
 
 `event` is `advertisement`, `characteristic`, `read`, `notification`, or `indication`. Properties use `R` for read, `W` for write with response, `w` for write without response, `N` for notify, and `I` for indicate. The snapshot is deliberately bounded to eight advertisers, 16 services, 32 characteristics, and 64 displayed value bytes. A trailing ` ...` marks a truncated value; `[unavailable]` means receive-buffer copying failed and no bytes are exported. These markers are not hex bytes. Earlier firmware labeled indications as `notification`. It includes nearby device names and addresses, so saving is explicit and the UI discloses this privacy tradeoff.
 
+BLE evidence creation stops on directory/probe/open/stream errors and retries only occupied CSV/TMP names; exhausting the ten same-second names reports `EEXIST`. A failed stream conversion attempts to remove its newly created empty TMP. Each CSV write must succeed before the publication helper runs. A rejected header or row closes the stream, retains the named TMP and reports its first cause without retrying publication; missing errno falls back to `EIO`. Healthy publication still requires flush, sync and close. Native byte/fault checks cover retained generations and same-second restart; physical SD and power-loss recovery remain hardware checks.
+
 These formats are still pre-1.0. Add fields at the end when possible, never silently change a unit, and document any incompatible migration in release notes.

docs/testing.md

@@ -24,6 +24,7 @@ Python 3 and Node.js are needed for the quick checks. The configuration checks a
 | Offline apps, SPI or I2C editor/result flows | [Offline and peripheral UI](#offline-and-peripheral-ui). |
 | Serial, saved serial logs or RTU Frames | [Serial and RTU UI](#serial-and-rtu-ui). |
 | UART or RS-485 SD log writes | [Serial log writers](#serial-log-writers) and a firmware build. |
+| BLE evidence export | [BLE evidence storage](#ble-evidence-storage) and a firmware build. |
 | Modbus TCP, NTP, Wake-on-LAN, UDP or HTTP fixtures | [Network fixtures and UI](#network-fixtures-and-ui). |
 | MQTT app, packet codec or client protocol | [Lifecycle/storage](#mqtt-lifecycle-and-storage) and [client/broker fixture](#mqtt-client-and-broker-fixture). |
 | OTA manifest parsing and update eligibility | [OTA manifests](#ota-manifests) plus a firmware build. |
@@ -211,6 +212,20 @@ UI status/control callbacks, clocks/ticks, SD admission and I/O failures are con
 
 Full serial/function/storage/header/test hashes, generated function/config/storage source and compile/result logs stay under `build/serial-log-write/`. Linux CI retains `serial-log-write-logs`, including available failure diagnostics. These 82 cases are separate from the Windows native total.
 
+## BLE evidence storage
+
+```powershell
+python tools/test_ble_evidence.py --compiler 'C:/path/to/clang.exe'
+```
+
+This portable fixture compiles the actual BLE save callback, its status/address/property/name helpers, its three cache typedefs and the complete `storage_io.c`. Its 50 cases check exact native CSV/TMP bytes, header and each row failure, directory/probe/open/stream errors, flush/sync/close/publication failures, mixed collisions, all ten occupied names and 25 rejected-header/restart pairs. Every case checks closed streams/descriptors and balanced modeled locks. The previous callback fails 28 controls, including eight rejected writes incorrectly published as CSVs; other controls concern creation/error handling and the retained-file notice.
+
+Healthy cases cover read/notification/indication labels, empty/truncated/unavailable values, CSV name sanitization, missing peers, invalid service indexes and all eight device/16 service/32 characteristic/64 value-byte cache limits. Mutating the shared cache after the snapshot lock exits preserves the originally captured output. Maximum-cache entries repeat controlled identities to exercise capacity; this fixture does not validate live discovery identities.
+
+Cache contents, NimBLE type boundaries and 16-bit UUID conversion, locks, SD admission/error reporting, clocks and I/O faults are controlled. The snapshot is taken through the actual callback; no NimBLE, BLE traffic, real LVGL, SDK mount/error filtering, 32/128-bit UUID conversion or physical FAT/SD/panel/power behavior is executed. UTC fixes filenames, Windows descriptors use binary mode for SDK LF bytes, and cleanup removes only owned files and directories. Failed writes close without a publication retry and retain a named TMP; those bytes can be partial or complete-looking without confirmed physical durability.
+
+Full BLE/function/cache-type/storage/header/test hashes, generated function/type/config/storage source and compile/result logs stay under `build/ble-evidence/`. Linux CI retains `ble-evidence-logs`, including available failure diagnostics. These 50 cases are separate from the Windows native total and live GATT tests.
+
 ## CSV viewer storage and UI
 
 ```powershell

main/ble_tool.c

@@ -963,32 +963,46 @@ static void save_evidence_clicked(lv_event_t *event)
     strftime(day, sizeof(day), "%y%m%d", &local);
     char directory[32];
     snprintf(directory, sizeof(directory), "/sdcard/BLE/%s", day);
-    if ((mkdir("/sdcard/BLE", 0775) != 0 && errno != EEXIST) ||
-        (mkdir(directory, 0775) != 0 && errno != EEXIST)) {
-        int error = errno ? errno : EIO;
-        if (report_storage_error) report_storage_error(error);
-        set_status("Evidence directory failed: %s", strerror(error));
-        return;
+    const char *folders[] = {"/sdcard/BLE", directory};
+    for (unsigned i = 0; i < 2; i++) {
+        errno = 0;
+        if (mkdir(folders[i], 0775) != 0 && errno != EEXIST) {
+            int error = errno ? errno : EIO;
+            if (report_storage_error) report_storage_error(error);
+            set_status("Evidence directory failed: %s", strerror(error));
+            return;
+        }
     }
 
     char temporary_path[48] = "";
     char final_path[48] = "";
     int descriptor = -1;
+    int create_error = EEXIST;
     for (int collision = 0; collision < 10; collision++) {
         snprintf(temporary_path, sizeof(temporary_path), "%s/B%02d%02d%02d%d.TMP",
                  directory, local.tm_hour, local.tm_min, local.tm_sec, collision);
         snprintf(final_path, sizeof(final_path), "%s/B%02d%02d%02d%d.CSV",
                  directory, local.tm_hour, local.tm_min, local.tm_sec, collision);
+        errno = 0;
         if (access(final_path, F_OK) == 0) continue;
+        if (errno != ENOENT) {
+            create_error = errno ? errno : EIO;
+            break;
+        }
+        errno = 0;
         descriptor = open(temporary_path, O_WRONLY | O_CREAT | O_EXCL, 0664);
-        if (descriptor >= 0 || errno != EEXIST) break;
+        if (descriptor >= 0) break;
+        if (errno != EEXIST) {
+            create_error = errno ? errno : EIO;
+            break;
+        }
     }
     if (descriptor < 0) {
-        int error = errno ? errno : EEXIST;
-        if (report_storage_error) report_storage_error(error);
-        set_status("Evidence file could not be created: %s", strerror(error));
+        if (report_storage_error) report_storage_error(create_error);
+        set_status("Evidence file could not be created: %s", strerror(create_error));
         return;
     }
+    errno = 0;
     FILE *file = fdopen(descriptor, "wb");
     if (!file) {
         int error = errno ? errno : EIO;
@@ -1005,17 +1019,20 @@ static void save_evidence_clicked(lv_event_t *event)
         address_text(&device_snapshot[peer_index].address, peer_address);
         csv_text(device_snapshot[peer_index].name, peer_name, sizeof(peer_name));
     }
-    bool ok = fprintf(file, "unix_time,event,address,name,rssi,service_uuid,"
-                      "characteristic_uuid,handle,properties,value_hex\n") >= 0;
-    for (size_t i = 0; ok && i < devices_found; i++) {
+    errno = 0;
+    int error = fprintf(file, "unix_time,event,address,name,rssi,service_uuid,"
+                        "characteristic_uuid,handle,properties,value_hex\n") < 0 ? (errno ? errno : EIO) : 0;
+    for (size_t i = 0; !error && i < devices_found; i++) {
         char address[18];
         char name[33];
         address_text(&device_snapshot[i].address, address);
         csv_text(device_snapshot[i].name, name, sizeof(name));
-        ok = fprintf(file, "%lld,advertisement,%s,%s,%d,,,,,\n",
-                     (long long)now, address, name, device_snapshot[i].rssi) >= 0;
+        errno = 0;
+        if (fprintf(file, "%lld,advertisement,%s,%s,%d,,,,,\n",
+                    (long long)now, address, name, device_snapshot[i].rssi) < 0)
+            error = errno ? errno : EIO;
     }
-    for (size_t i = 0; ok && i < characteristics_found; i++) {
+    for (size_t i = 0; !error && i < characteristics_found; i++) {
         if (characteristic_snapshot[i].service_index >= services_found) continue;
         char service_uuid[BLE_UUID_STR_LEN];
         char characteristic_uuid[BLE_UUID_STR_LEN];
@@ -1024,27 +1041,35 @@ static void save_evidence_clicked(lv_event_t *event)
                         service_uuid);
         ble_uuid_to_str(&characteristic_snapshot[i].uuid.u, characteristic_uuid);
         properties_text(characteristic_snapshot[i].properties, properties);
-        ok = fprintf(file, "%lld,characteristic,%s,%s,,%s,%s,0x%04X,%s,\n",
-                     (long long)now, peer_address, peer_name, service_uuid,
-                     characteristic_uuid, characteristic_snapshot[i].val_handle,
-                     properties) >= 0;
+        errno = 0;
+        if (fprintf(file, "%lld,characteristic,%s,%s,,%s,%s,0x%04X,%s,\n",
+                    (long long)now, peer_address, peer_name, service_uuid,
+                    characteristic_uuid, characteristic_snapshot[i].val_handle,
+                    properties) < 0)
+            error = errno ? errno : EIO;
     }
-    if (ok && value_seen) {
+    if (!error && value_seen) {
         char value_hex[BLE_TOOL_VALUE_MAX * 3 + 1] = "";
         size_t used = 0;
         for (size_t i = 0; i < value_length && used < sizeof(value_hex); i++)
             used += (size_t)snprintf(value_hex + used, sizeof(value_hex) - used,
                                      "%s%02X", i ? " " : "", value_snapshot[i]);
-        ok = fprintf(file, "%lld,%s,%s,%s,,,,0x%04X,,%s%s\n", (long long)now,
-                     value_was_indication ? "indication" : value_was_notification ? "notification" : "read",
-                     peer_address, peer_name, value_handle, value_hex,
-                     value_copy_failed ? "[unavailable]" : value_total > value_length ? " ..." : "") >= 0;
+        errno = 0;
+        if (fprintf(file, "%lld,%s,%s,%s,,,,0x%04X,,%s%s\n", (long long)now,
+                    value_was_indication ? "indication" : value_was_notification ? "notification" : "read",
+                    peer_address, peer_name, value_handle, value_hex,
+                    value_copy_failed ? "[unavailable]" : value_total > value_length ? " ..." : "") < 0)
+            error = errno ? errno : EIO;
     }
-    if (!ok && !errno) errno = EIO;
-    if (storage_commit_new_file(&file, temporary_path, final_path) != 0) {
-        int error = errno ? errno : EIO;
+    if (error) fclose(file);
+    else {
+        errno = 0;
+        if (storage_commit_new_file(&file, temporary_path, final_path) != 0) error = errno ? errno : EIO;
+    }
+    if (error) {
         if (report_storage_error) report_storage_error(error);
-        set_status("Evidence save failed: %s", strerror(error));
+        const char *name = strrchr(temporary_path, '/');
+        set_status("Evidence not published; %s retained (%s)", name ? name + 1 : "TMP", strerror(error));
         return;
     }
     set_status("Evidence saved to %s", final_path);