Skip to content

Commit 34c5e73

Browse files
LC-LinkousLC-Linkous
committed
tests, docs, and scratch work dump
1 parent f7ceb45 commit 34c5e73

22 files changed

Lines changed: 2974 additions & 930 deletions

‎.github/workflows/tests.yml‎

Lines changed: 61 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -50,11 +50,65 @@ jobs:
5050
enable-cache: true
5151
python-version: ${{ matrix.python-version }}
5252

53+
- name: Lockfile is in sync with pyproject
54+
# A stale uv.lock means CI tests a different dependency set than a
55+
# fresh `uv sync` would install. Fail early instead.
56+
run: uv lock --check
57+
5358
- name: Sync environment (numpy + scipy + dev group, editable install)
5459
run: uv sync
5560

5661
- name: Run test suite (hardware tests deselected)
57-
run: uv run pytest -m "not hardware" --cov=sdr_dsp --cov-report=xml --cov-report=term-missing
62+
# 85% floor, same as hackrfpy. The tree is at ~93%; the gate is there
63+
# so a large untested module cannot land unnoticed.
64+
run: uv run pytest -m "not hardware" --cov=sdr_dsp --cov-report=xml --cov-report=term-missing --cov-fail-under=85
65+
66+
oldest-deps:
67+
name: pytest (ubuntu, py3.11, oldest declared numpy/scipy)
68+
# pyproject promises numpy>=1.26 and scipy>=1.11; the matrix above only
69+
# ever tests whatever the lockfile resolved to (the newest). This leg
70+
# installs the OLDEST versions the project claims to support, so the
71+
# claim is tested rather than assumed.
72+
runs-on: ubuntu-latest
73+
steps:
74+
- uses: actions/checkout@v4
75+
- uses: astral-sh/setup-uv@v5
76+
with:
77+
python-version: "3.11"
78+
- name: Sync with lowest allowed direct dependencies
79+
run: uv sync --resolution lowest-direct
80+
- name: Show what resolved
81+
run: uv run --no-sync python -c "import numpy, scipy; print('numpy', numpy.__version__, 'scipy', scipy.__version__)"
82+
- name: Run test suite
83+
# --no-sync: a plain `uv run` re-syncs to the lockfile (newest) first,
84+
# which would silently undo the step above.
85+
run: uv run --no-sync pytest -m "not hardware" -q
86+
87+
package:
88+
name: build wheel + import from a clean env
89+
# The package __init__ lazy-loads submodules (PEP 562) and ships a
90+
# py.typed marker; both are easy to break in a way the editable install
91+
# used by the tests never notices. Build the wheel, install it alone
92+
# into an empty environment, and import the public surface from there.
93+
runs-on: ubuntu-latest
94+
steps:
95+
- uses: actions/checkout@v4
96+
- uses: astral-sh/setup-uv@v5
97+
with:
98+
python-version: "3.12"
99+
- name: Build sdist + wheel
100+
run: uv build
101+
- name: Import the wheel in an isolated environment
102+
run: |
103+
uv run --isolated --no-project --with dist/*.whl python - <<'PY'
104+
import importlib.resources as r
105+
import sdr_dsp, sdr_dsp.core, sdr_dsp.io, sdr_dsp.sinks, sdr_dsp.sources
106+
import sdr_dsp.stream, sdr_dsp.link
107+
from sdr_dsp import Pipeline, fm_demod, design_lowpass
108+
from sdr_dsp.stream import StreamingFIR, StreamingResampler
109+
assert (r.files("sdr_dsp") / "py.typed").is_file(), "py.typed missing from wheel"
110+
print("wheel imports OK:", sdr_dsp.__name__)
111+
PY
58112
59113
lint:
60114
name: ruff + mypy
@@ -75,9 +129,10 @@ jobs:
75129
run: uv run ruff check .
76130

77131
- name: Mypy
78-
# Blocking. Not the py.typed-strict gate hackrfpy runs (sdr_dsp is not
79-
# fully annotated yet and ships no py.typed marker), but mypy's default
80-
# checks -- real type errors, incompatible assignments, wrong argument
81-
# types. The tree is clean at that level; a regression fails the build.
82-
# The path to strict + py.typed is tracked in TODO.
132+
# Blocking. Not the strict gate hackrfpy runs: sdr_dsp DOES ship a
133+
# py.typed marker but most signatures are not annotated yet, so this
134+
# is mypy's default checks plus check_untyped_defs -- real type errors,
135+
# incompatible assignments, wrong argument types. The tree is clean at
136+
# that level; a regression fails the build. The path to strict is in
137+
# docs/TODO.md.
83138
run: uv run mypy

‎TODO.md‎

Lines changed: 291 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/LOG.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,146 @@ that have hooks, but have not been implemented yet.
99

1010

1111

12+
## 2026-09-25 — Gain search made device-agnostic: GainModel (patch 0022)
13+
14+
Lifted the HackRF-specific gain logic into a `GainModel` abstraction so auto-gain
15+
works on any SDR, not just the HackRF One. `search_gain` is now a thin wrapper;
16+
the engine underneath drives an arbitrary radio. Suite: 797 → 808 passed.
17+
18+
- **`GainStage` + `GainModel`** describe a radio's gain as named stages, each
19+
"frontend" (improves SNR: RF amp, LNA, a tuner's single gain) or "level"
20+
(sets ADC level after SNR: the HackRF VGA). Factories: `hackrf_gain_model()`
21+
(LNA + optional +14 dB amp + VGA) and `rtlsdr_gain_model()` (one gain knob,
22+
no separate level stage).
23+
- **`search_gain_model(probe, model, quality=…)`** is the general engine: the
24+
same two-stage search (front end for SNR at minimum level, then walk the level
25+
stage for the ADC window) over any model. When a radio has no separate level
26+
stage, it walks the SNR-chosen knob itself rather than pretending a second knob
27+
exists -- the honest single-knob behavior.
28+
- **`search_gain` is now a wrapper** over the engine with the HackRF model,
29+
keeping the `probe(lna, vga, amp)` signature and `{lna,vga,amp,…}` result
30+
unchanged; the 13 existing `test_gain_and_health` tests pass untouched, which
31+
is the fidelity proof.
32+
- **The SNR-first guarantee now holds on a second radio.** New
33+
`tests/test_gain_model.py` (11 tests) drives a single-knob (RTL-SDR-like) mock
34+
tuner: it converges in-window on a decent antenna, reports `too_weak` on a
35+
DEAD antenna (does not bless amplified noise -- the whole point of the
36+
SNR-first design), reports `clipping` on overload, and stays on the gain grid.
37+
The same engine still enables the amp for a weak HackRF antenna.
38+
39+
Caveat: the single-knob path is validated against a mock tuner model, not a real
40+
RTL-SDR (none on hand). A coarse gain grid can leave the level just shy of the
41+
window near a boundary; the search reports that honestly rather than faking
42+
success. This closes the one device-coupling the modularity docs called out --
43+
RX, TX, and gain search are all radio-agnostic now.
44+
45+
## 2026-09-25 (earlier) — Modularity kit: prove the library isn't HackRF-locked (patch 0021)
46+
47+
Made device-independence concrete AND enforced, so "any SDR works" is a test
48+
result rather than a claim. Suite: 785 → 797 passed.
49+
50+
- **`tests/test_modularity.py` (7 tests)** -- the guard with teeth. It AST-scans
51+
`src/sdr_dsp/` and FAILS if any core module imports a device SDK (hackrfpy,
52+
pyrtlsdr, SoapySDR, uhd, …); a deliberately smuggled `import hackrfpy` was
53+
confirmed to trip it. It also drives the full pipeline (PSD, pilot check) from
54+
a NON-HackRF source, so the structural guarantee is checked both ways.
55+
- **`examples/sdr_adapter_template.py`** -- a copy-and-fill adapter (IQSource +
56+
guarded TXSink) with the four device-specific spots marked, sample-format
57+
normalization (cu8, the RTL-SDR wire format) as the worked example, and a
58+
built-in fake device so it self-demonstrates end to end with no hardware.
59+
- **`docs/ADDING_AN_SDR.md`** -- the porting process and record: the steps, the
60+
normalization table, the "free vs. needs-work" split (RX/TX free; auto-gain
61+
WAS the one HackRF-shaped piece, since fixed in 0022), and a per-SDR
62+
porting-record table (HackRF + template filled, RTL-SDR stubbed to copy).
63+
- **`docs/MODULARITY.md`** (written just before the kit) -- the HAL layer model
64+
(core / `IQSource`+`TXSink` Protocols / device adapter / hardware) with a
65+
diagram and the sample-format normalization table.
66+
67+
## 2026-09-24 (later) — Typing gate tightened; py.typed made honest (patches 0019–0020)
68+
69+
The package ships `src/sdr_dsp/py.typed`, but the pyproject comment still claimed
70+
it did not and kept the strict flags off. Corrected the record and took a real
71+
step toward typed. Suite: 785 (unchanged); mypy now stricter.
72+
73+
- **`check_untyped_defs` enabled** in `[tool.mypy]` -- mypy now type-checks the
74+
BODIES of unannotated functions (catches real bugs without demanding
75+
signatures everywhere). Fixed the 10 issues it surfaced: empty-container
76+
annotations (`bits: list[int] = []` in psk/qam, deques in arq, dicts in
77+
drivers/pipeline), a `results` refactor in the pipeline (drop the `list|None`
78+
union that the append site couldn't prove safe), and int-casting the edge
79+
indices in `find_bursts` so the spans list is uniformly typed.
80+
- **pyproject comment corrected**: py.typed IS shipped; the full strict
81+
disallow-untyped flags remain the per-module annotation campaign (~350 errors
82+
under `--strict` today), turned on as each module is annotated.
83+
- **`[project.urls]`** (patch 0019): Homepage / Repository / Issues, now that
84+
the repo is public.
85+
86+
## 2026-09-24 — TX examples across the sink seam (patch 0018)
87+
88+
`examples/transmit_examples.py`: modulate the library's own schemes as framed
89+
packets and key them through a `TXSink`, proving `demod(modulate(x)) == x` across
90+
the device boundary in software. The last big no-hardware chunk before the
91+
two-SDR bench test. Suite: 756 → 785 passed.
92+
93+
- **OOK / FSK / BPSK / QPSK, framed** (`build_frame` → modulate → sink → read
94+
back → demod → `find_frames`), all recovered with a valid CRC. Offline by
95+
default via `LoopbackSink`; the example self-checks (nonzero exit if any scheme
96+
fails to round-trip). `--sink hackrf` drives the guarded `HackRFSink` to the
97+
device seam and stops there -- the actual radiation is the bench step.
98+
- **RX chains are the hardware-robust ones** from the demod docstrings and
99+
`two_station_link`: smoothed auto-threshold FSK, transition-phase symbol
100+
decimation via `sample_symbols`, guard-padded bursts. QPSK decimates at known
101+
symbol centers on the clean loopback (unknown-delay timing recovery is the
102+
Phase E two-SDR concern, noted in the code).
103+
- **24 offline tests** (`test_transmit_examples.py`): every scheme × four
104+
payloads, `key_out` genuinely drives the sink protocol, a non-buffering
105+
device-like sink is handled without crashing, and the `HackRFSink` seam is
106+
confirmed guarded (`armed=False`, refuses to transmit).
107+
108+
## 2026-09-24 (earlier) — FM carrier-offset estimate & correct (patch 0017)
109+
110+
New core capability: `estimate_fm_cfo` / `correct_fm_cfo`, validated on the real
111+
103.7 MHz capture's genuine crystal offset. Suite: 746 → 756 passed.
112+
113+
- **Method: the discriminator DC term.** Under a constant carrier offset Δf the
114+
differential discriminator picks up that same Δf at every sample, while the
115+
program audio averages to zero -- so the MEAN of the demodulated instantaneous
116+
frequency IS the offset. Channelized first (like `fm_pilot_excess_db`) so
117+
out-of-channel energy doesn't corrupt the mean; impulsive discriminator clicks
118+
are trimmed before averaging.
119+
- **NOT the pilot.** With a differential discriminator a carrier offset adds a DC
120+
term but does NOT move the 19 kHz pilot (differentiation is invariant to a
121+
constant frequency shift), so the pilot carries no CFO information -- the trap
122+
the TODO's "or the pilot frequency" wording invited. A test pins this so nobody
123+
"improves" the estimator into it; the pilot's real role is a presence gate.
124+
- **`correct_fm_cfo`** reuses `frequency_shift(-cfo)` and returns
125+
`(corrected_iq, cfo_hz)` so the applied correction is visible; estimation never
126+
mutates data (separate call), matching `estimate_cfo`'s contract.
127+
- **Validated:** synthetic offsets (±500 Hz to ±12 kHz) recovered to <10 Hz with
128+
correct sign; the real capture carries −315 Hz (~3 ppm) corrected to −2 Hz,
129+
stable across the two halves (a real offset, not noise). 10 tests
130+
(`test_cfo.py`).
131+
132+
## 2026-09-23 (later) — Sub-8 Msps warning dedup; lint gate re-green (patches 0015–0016)
133+
134+
Two hygiene fixes after the CI landing. Suite: 744 → 746 passed.
135+
136+
- **The sub-8 Msps warning now dedupes for real (0016).** `dedup_warnings` used
137+
`redirect_stderr`, but hackrfpy emits the notice through
138+
`logging.getLogger("hackrfpy").warning()`; once anything installs a
139+
`StreamHandler` bound to the real stderr (a plain `logging.basicConfig` does
140+
it), `redirect_stderr` never sees the record and the same line printed once per
141+
probe (~16× per station in preflight). Fixed at the logging layer: a
142+
`logging.Filter` on the hackrfpy logger drops repeat WARNING+ records before
143+
any handler, bound or not; INFO progress chatter is left alone. Two regression
144+
tests cover the bound-handler path the old code missed.
145+
- **Lint gate re-greened (0015).** Three ruff errors had slipped into two
146+
untracked diagnostic scripts that never passed the gate --
147+
`tools/diagnose_rate.py` (unused numpy import; a lambda assignment) and
148+
`tools/diagnose_path.py` (missing trailing newline). Fixed in-style (fix, not
149+
a blanket per-file-ignore, matching `preflight_collection`), so CI's ruff step
150+
is green again.
151+
12152
## 2026-09-23 — CI pipeline, typing gate, RNG determinism (+ the real flake)
13153

14154
Made the pipeline trustworthy: a CI workflow, a green mypy gate, deterministic

‎docs/README.md‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1-
Note that the documentation is being AI summarized to fix spelling errors and make the developement easier to follow.
1+
# docs/
22

3-
All mistakes are human, and will likely take a revision or two to fix experimentally.
3+
| File | What it is |
4+
|---|---|
5+
| `LOG.md` | Running development log, newest first. Start here when coming back to the project. |
6+
| `TODO.md` | Open work, in priority order. The LOG's deferrals live here, not in prose. |
7+
| `sdr_dsp_REFERENCE.md` | The deep technical reference for the library. |
8+
| `EXAMPLES.md` | Catalog of every example script. |
9+
| `MODULARITY.md` / `ADDING_AN_SDR.md` | The IQSource/TXSink seam and how to drive the library from another radio. |
10+
| `HARDWARE.md` | HackRF One characteristics that set expectations for the demods. |
11+
| `MODULATIONS.md` | The modulation families covered. |
12+
| `DC_SPIKE.md` | Where the DC spike comes from and the offset-tuning recipe. |
13+
| `PCAP_CONVERTER_NOTE.md` | Note on packet-capture conversion. |
14+
| `dev_handbook/` | Generated API handbook plus the scripts that produce it. |
15+
16+
The package README (`../sdr_dsp/README.md`) is the user-facing one.
17+
18+
*Documentation is AI-assisted for spelling and structure; all mistakes are
19+
human and will take a revision or two to fix experimentally.*

‎docs/README2.md‎

Lines changed: 0 additions & 57 deletions
This file was deleted.

‎docs/TODO.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# TODO
2+
3+
Open work, roughly in priority order. Items the LOG defers "to TODO" land
4+
here. Strike through or delete when done; the LOG records what happened.
5+
6+
## Hardware (the gate everything else waits on)
7+
8+
- [ ] **Wired one-way TX bench test** of the FSK packet chain
9+
(`examples/two_station_link.py`). Everything digital, TX and protocol
10+
is software-only until this runs (REFERENCE §12).
11+
- [ ] **Windows stdout-pipe corruption at 8 Msps** (`capture_array` /
12+
`open_receiver`; LOG 2026-09-20). Unresolved upstream in hackrfpy 1.1.0.
13+
Suspect throughput rather than framing: watch `hackrf_transfer`'s
14+
dropped-sample stats on stderr, try a large `read_samples`, and A/B
15+
against the file path again. Until it's settled, every measurement
16+
probe stays on `sources/probe.py`; `HackRFCapture` (live adapter)
17+
still streams through the pipe.
18+
- [ ] Re-verify `HackRFCapture.blocks()` live once hackrfpy ships the
19+
`PersistentReceiver.blocks()` odd-byte fix (I/Q swap after an odd
20+
chunk; found 2026-10-24).
21+
- [ ] CFO-ppm capture experiment: the cheap go/no-go for whether real
22+
same-model devices separate on the fingerprint features (LOG 07-11).
23+
24+
## Library
25+
26+
- [ ] Lift the HackRF three-knob gain model out of `core/gain_search.py`
27+
into a `GainModel` protocol so `search_gain` can drive another radio
28+
(MODULARITY §2).
29+
- [ ] Give `live_fm_listen.py`'s `FMStream` over to the `Streaming*` stages
30+
(`stream/stages.py`) now that they exist; it should become a thin
31+
composition, and the stereo path can then go live too.
32+
- [ ] F3 transient features (fingerprint plan) -- deferred to the hardware era.
33+
34+
## Tooling / typing
35+
36+
- [ ] mypy strict + `py.typed` honesty: the marker ships but most signatures
37+
are unannotated (~350 errors under `disallow_untyped_defs`). Enable the
38+
strict flags per module as each is annotated; until then the marker
39+
overpromises, exactly the hackrfpy pre-1.1.0 situation.
40+
- [ ] Line-ending normalization: `.gitattributes` is committed, the
41+
`git add --renormalize .` commit has not been made. Do it alone.
42+
43+
## Repo hygiene
44+
45+
- [ ] `sample_data/hackrf_one/*.iq` is 160 MB of raw IQ committed directly,
46+
byte-identical (sha256) to `hackrfpy/tests/fm_reference/`. Move to Git
47+
LFS or a release asset and fetch on demand; each file is 80 MB, near
48+
GitHub's 100 MB hard limit, and `.git` is already 121 MB.
49+
- [ ] LOG entries dated 2026-09-21..23 describe work that is in commits
50+
stamped 2026-09-20 (the LOG was written ahead of the commits). Note
51+
it in the LOG header or accept the drift.

0 commit comments

Comments
 (0)