SPECIFICATION.txt

===============================================================================
 TSM - TEMPORAL SIGNAL MEDIUM
 Format specification v5.2 "Compact Escape"
===============================================================================

 Copyright (c) 2026 Francesco De Simone.

 This document is licensed under the Creative Commons Attribution 4.0
 International licence (CC BY 4.0):

     https://creativecommons.org/licenses/by/4.0/

 Anyone may implement this format, in any language, for any purpose, commercial
 or not, without asking and without using a line of the reference code. What
 attribution asks is that the format be named as TSM, by Francesco De Simone.

 The reference implementation beside this document is separately licensed under
 the Apache License 2.0 - see LICENSE and NOTICE.


-------------------------------------------------------------------------------
 1. WHAT TSM IS
-------------------------------------------------------------------------------

 TSM = Temporal Signal Medium.

 It is a generic, source-agnostic format for storing a two-state temporal signal
 (HIGH/LOW) by describing its edge/toggle transitions over time, instead of the
 audio samples. It is not tied to any machine, protocol or historical source: it
 represents "when" the signal switches, with tick resolution (a configurable time
 unit, default 4000 ns = 4 us).

 Current version: TSM v5.2 "Compact Escape" (magic "TSM5", version 5).
 Reference toolchain license: MIT (see LICENSE).

 Conceptual model (3 region types):

   ZERO_SILENCE   (4)  true zero-amplitude gap (real silence)
   SILENCE/HOLD   (0)  the signal stays stable HIGH or LOW, no edge
   INDEXED_DELTA  (3)  active edge/toggle stream

 A TSM tape is therefore a sequence of regions: long silences/holds compressed
 into a few bytes, and active blocks encoded as a compact stream of time deltas.


-------------------------------------------------------------------------------
 2. FORMAT SPECIFICATION (v5.2)
-------------------------------------------------------------------------------

 References: TSM_v5_2_FORMAT_SPEC.md, README_TSM_V5_2_COMPACT_ESCAPE.md,
 structs in src_v5/tsm_v5.h.

 2.1 HEADER - TSM5Header, 80 bytes
 ---------------------------------
   magic[4]               char   "TSM5"
   version                u16    5
   header_size            u16    80
   total_duration_ticks   u64    total duration in ticks
   base_time_unit_ns      u64    base time unit (ns/tick), default 4000
   region_count           u32    number of regions
   region_entry_size      u32    104
   region_table_offset    u64    offset of the region table
   data_offset            u64    offset of the payload data
   flags, reserved32      u32    reserved
   reserved0              u64    initial signal state: 0 = LOW, 1 = HIGH
   reserved1              u64    original source sample count
   reserved2              u64    original source sample rate

 2.2 REGION - TSM5Region, 104 bytes
 ----------------------------------
   Main fields: start_ticks, duration_ticks, region_mode (0/3/4), encoding,
   time_unit_ns (per-region), table_count / table_entry_size (delta table),
   data_offset / data_size (payload), aux_*, flags, reserved0 (state
   before/through the region: 0 LOW / 1 HIGH).

   Compact Escape key point: for INDEXED_DELTA regions,
   region.flags = raw_delta_bytes (1, 2, 3 or 4) - the byte width of the raw
   delta that follows an escape.

 2.3 COMPACT ESCAPE ENCODING (encoding = 7, NIBBLE_ESCAPE_INDEX)
 --------------------------------------------------------------
   The active payload is a nibble stream (4 bits):

     nibble 0..14 = index into the region's delta table
     nibble 15    = ESCAPE

   After an escape: align to the next byte boundary, then read a raw
   little-endian delta region.flags bytes wide. Each region has its own delta
   table (max 15 entries, because nibble 15 is reserved for the escape ->
   I4/I8/I16 profiles).

   Rationale: common deltas (the fixed-duration bits of a loader) fit in 4 bits;
   rare/long deltas use the escape. This recovers compactness while preserving
   timing accuracy (the old v5.1 per-byte escape was accurate but often ~2x the
   source).

 2.4 PLAYBACK PSEUDOCODE
 -----------------------
   for each region:
       if ZERO_SILENCE:   output zero for region duration
       else if SILENCE:   output current state for region duration
       else if INDEXED_DELTA:
           restore pre-edge state
           toggle at region start
           while region active:
               code = read_nibble();
               if (code < 15) delta = table[code];
               else { align_to_byte(); delta = read_raw_delta(region.flags); }
               wait(delta);
               toggle();


 2.5 LABEL SECTION - optional, appended after the payloads
 ---------------------------------------------------------
   A TSM is a stream of edges and nothing else, which is the point: it does not
   know what the tape carries. That leaves nowhere for what a person worked out
   about the recording - what the programs are called, which system wrote each
   region, where the tape came from, what has to be typed to load it - and that
   knowledge was being lost with every file written.

   The label section holds it. It is optional, it is appended AFTER the last
   region payload, and it is found from the END of the file:

     [ ... region payloads ... ][record][record] ... [u32 count][u32 bytes]["TSML"]

   Nothing that comes before it moves. A reader that knows nothing of this
   never looks at the end of the file and never notices it is there; a reader
   that wants the labels seeks twelve bytes back from the end, checks the
   magic, and walks the records from `end - 12 - bytes`.

   There is deliberately NO flag in the header announcing the section. A bit
   would save a twelve-byte seek and cost the guarantee worth having: a TSM
   carrying labels is a TSM without them plus bytes at the end, not one byte
   otherwise different. It would also be a second thing to keep true, and a
   tool that rewrote the payloads and dropped the section would leave it lying,
   where the footer can only lie about itself and is checked against its own
   count and length.

   Footer, the last 12 bytes of the file:

     u32  count           how many records were written
     u32  bytes           the records' total size, not counting this footer
     u8   magic[4]        "TSML"

   One record, repeated `count` times:

     u32  record_bytes    including this field, padded to a multiple of 4
     u32  kind            see the table below
     u32  region          a region index, or 0xFFFFFFFF for the whole tape
     u64  from_ticks      the counter the label runs between, in the file's
     u64  to_ticks        own base_time_unit_ns
     u32  text_bytes
     u8   text[]          UTF-8, not NUL-terminated

   Because every record carries its own length, a reader steps over a kind it
   does not know. The list can therefore grow without any existing reader
   having to change.

     kind  applies to    the text is
     ----  ----------    -----------
       1   a region      what the program recorded there is called
       2   a region      which system wrote it, as a short token ("kcs1200")
       3   the tape      where the recording came from, and when
       4   either        what has to be done to load it

   A reader must treat the section as absent - not as an error - when the magic
   is missing, when `bytes` is larger than the file, or when a record runs past
   the end of the section. Saying nothing about itself is what every TSM
   written before this section existed does, and they are all still valid.

   Counters are in ticks so that a label means the same thing whatever rate the
   file is rendered at. Turning one into seconds is
   `ticks * base_time_unit_ns / 1e9`.

   Reference reader: `tsm5_read_labels()` in src/tsm_v5.c, which takes the file
   in memory because the section is found from the end.


-------------------------------------------------------------------------------
 3. C SOURCES
-------------------------------------------------------------------------------

 3.1 REFERENCE TOOLCHAIN - src/
 --------------------------------------------
   Build: build_v5_2_compact.bat (requires gcc in PATH, -std=c99 -O2 -Wall).
   Output in bin/. 8 compilation steps.

   Core modules (library):
     tsm_v5.c / tsm_v5.h    Format I/O: read/write of header and regions,
                            u16/u32/u64 LE helpers, samples <-> ticks
                            conversions, u64 -> dec. Defines all the constants
                            (TSM5_MODE_*, TSM5_ENCODING_*, TSM5Header,
                            TSM5Region).
     wav_io_simple.c / .h   Minimal WAV loader: WavData { sample_rate, channels,
                            sample_count, double* samples },
                            wav_load_pcm16_mono_simple().
     indexed_common.c / .h  Shared structures/algorithms for segmentation and
                            building the indexed delta tables.

   CLI tools (executables):
     wav_tsm_advisor_v5_indexed.c -> wav_tsm_advisor_v5_indexed.exe
        Analyzes a WAV, reports common deltas and "zero-like" gaps, and generates
        the conversion/render batch files using the source WAV sample rate.
     wav2tsm_indexed_v5.c -> wav2tsm_indexed_v5.exe
        WAV -> TSM v5.2 compact (nibble+escape). Default raw_delta_bytes = 4;
        I4/I8/I16 profiles (table size).
     tsm2wav_v5.c -> tsm2wav_v5.exe
        TSM -> WAV: reference renderer. Handles HOLD/SILENCE, ZERO_SILENCE,
        INDEXED_DELTA (+ legacy nibble-packed for inspection).
     wav_gap_scan.c -> wav_gap_scan.exe
        Detects ~zero-amplitude gaps in a WAV. Use: compare original vs
        re-rendered WAV.
     tsm_v5_indexed_audit.c -> tsm_v5_indexed_audit.exe
        Structural diagnostics: dump of header, regions, delta table of a .tsm.

 3.2 OTHER IMPLEMENTATIONS
 -------------------------
   The tools in src/ are the reference, not the only way. TSM is deliberately
   small enough that a player is a few hundred lines: a reader needs the header,
   the region table, and the nibble stream of section 2.3.

   The author's own emulator carries a second, independent implementation - a
   runtime decoder that renders the signal to a virtual datacorder in step with
   the CPU rather than writing a WAV. It is not part of this repository, and
   nothing here depends on it. It is worth mentioning only because the two were
   written from this document separately, which is the test a specification has
   to pass: two implementations that agree, neither of them consulting the
   other's code.

   What a playback implementation must get right, and nothing else:

     - Enter each region at the level in its reserved0 field.
     - ZERO_SILENCE emits no signal for its duration; SILENCE holds the level.
     - In INDEXED_DELTA, read nibbles low half first; nibble 15 is the escape
       and is followed by a 32-bit little-endian raw delta.
     - Each delta is a number of ticks of that region's own time_unit_ns, and
       every delta is a toggle.
     - A region's duration is authoritative: a stream that runs out early holds
       its last level, and one that overruns is truncated.


-------------------------------------------------------------------------------
 4. UTILITIES AND WORKFLOW
-------------------------------------------------------------------------------

 4.1 DRAG-AND-DROP BATCH FILES (batch/, Windows)
 -------------------------------------------
   analyze_wav_generate_v5_2_batches.bat
      Drop a WAV -> advisor -> _v5_2_advisor.txt report + batch
      TSM_v5_2_I16_<file>.bat.
   tsm_v5_indexed_audit_dragdrop.bat
      Drop a .tsm -> audit report.
   wav_gap_scan_dragdrop.bat
      Drop a WAV -> gap report (threshold 0.03, min 20 ms).
   build_v5_2_compact.bat
      Builds the whole toolchain.

   On anything else, and on Windows with make installed, the Makefile beside
   this document builds the same five tools:  make  and  make check .

 4.2 DEFAULT PARAMETERS (advisor)
 --------------------------------
   time_unit_ns    = 4000      edge_threshold  = 0.35
   min_delta_ticks = 1         gain            = 8.0
   clip_level      = 1.0       zero_threshold  = 0.03
   min_zero_ms     = 20

 4.3 TYPICAL WORKFLOW
 --------------------
   1. Start from a WAV. If the recording is in some other container, convert
      it first with whatever tool reads that container faithfully.
   2. Drop the WAV onto analyze_wav_generate_v5_2_batches.bat.
   3. Run the generated TSM_v5_2_I16_<file>.bat first (I16 baseline).
   4. Test the re-rendered WAV (tsm2wav) and compare gaps (wav_gap_scan).
   5. Try I8 / I4 for more compactness.

 4.4 VALIDATION STATUS
 ---------------------
   ZX Spectrum : OK with reconstructed WAVs, I16 baseline.
   MSX cassette: OK with reconstructed WAVs.
   Sega SC-3000: OK on original recordings; a WAV encoded to TSM and rendered
                 back decodes to the same bytes, front for front.


-------------------------------------------------------------------------------
 5. POSSIBLE APPLICATIONS
-------------------------------------------------------------------------------

 5.1 RETROCOMPUTING (primary application)
 ----------------------------------------
   TSM was born to faithfully preserve the cassette tapes of 8-bit home computers
   and consoles. Its strength is that it stores the signal at the edge level, not
   its logical interpretation:

     - Encoding independence. Unlike "logical" formats (.cas, .tap, .tzx to some
       extent), TSM does not need to "understand" the loader. It captures the
       timing of the transitions, so it represents without trouble:
         * standard KCS (SC-3000/SG-1000, MSX),
         * the ZX Spectrum ROM loader,
         * turbo loaders and custom loaders,
         * multi-format / mixed tapes - a real one exists: a Sega SC-3000
           BASIC loader followed by blocks written in the ZX Spectrum's pulse
           protocol, on the same cassette, read and re-encoded as one TSM,
         * timing-based protection schemes.
     - Hardware-level playback. An emulator can feed a TSM to its virtual
       datacorder and let the machine's own loader demodulate it, exactly as the
       real one would, instead of injecting already-decoded bytes. That is what
       keeps the original loaders - including the ones nobody has documented -
       working unchanged.
     - Compactness with fidelity. Hold/zero-silence compress the long pilot tones
       and the silences; the nibble+escape stream keeps the data blocks small
       while preserving tick-level accuracy (4 us by default).
     - Friendly distribution. A TSM tape can be wrapped in a steganographic PNG
       (footer "TSM5") complete with a J-card: it is distributed and loaded like
       a regular image.

   In short: TSM is the "high-level signal abstraction" equivalent of a WAV
   sampling of the tape, but orders of magnitude smaller and suited to real-time
   loading in emulation.

 5.2 OTHER APPLICATIONS (source-agnostic format)
 -----------------------------------------------
   Being a generic Temporal Signal Medium, TSM can archive/compress any sampled
   two-state temporal signal:

     - captured serial protocols and digital buses (UART/RTTY, telegraphy/CW,
       line signals), for analysis or replay;
     - 1-bit instrumentation / logic-analyzer traces with long stable periods;
     - any pipeline where it is convenient to keep the transitions instead of the
       samples, exploiting the hold/zero-silence segmentation for compression.

   For these uses the same tools apply: wav2tsm to import from WAV, tsm2wav to
   re-render, audit/gap_scan for diagnostics.


-------------------------------------------------------------------------------
 6. QUICK REFERENCES
-------------------------------------------------------------------------------

   This document         SPECIFICATION.txt  (CC BY 4.0)
   C header and structs  src/tsm_v5.h
   Reference encoder     src/wav2tsm_indexed_v5.c
   Reference renderer    src/tsm2wav_v5.c
   Structural audit      src/tsm_v5_indexed_audit.c
   Build, any platform   Makefile          (make, make check)
   Build, Windows only   batch/build_v5_2_compact.bat
   Licence, code         LICENSE, NOTICE   (Apache-2.0)

===============================================================================