Spec v4 is frozen (plan.md §21). Deviations and engineering choices made during runs are recorded here with the date and the run they first applied to, not as spec revisions.
Informal expectations (e.g. "a plain tanh RNN lands around 1.7–2.0") may live here. They never appear in the scoreboard or in public claims (§2).
| Date | Decision | Applies from | Why |
|---|---|---|---|
| 2026-09-13 | Path gate thresholds: reachable ≥ 0.95, p90 ≤ microsteps × 3 chars | first build_graph | §6 leaves the numbers to engineering; logged in gate.json |
| 2026-09-13 | Degree-preserving plateau: min 10×E swaps, then chunks of E until survival drops < 1e-3/chunk; two seeds must agree within 0.01 | first build_graph | §4 |
| 2026-09-13 | Reference losses use add-1 smoothing with counts from the train split | prepare_data | §2 |
| 2026-09-13 | Trim step (§3.3 D): when the post-trim SCC is short of target, re-trim to target + deficit rather than lowering k (k is already the largest core ≥ target, so lowering it cannot help). Final size may exceed target by a few nodes; logged as final_neurons | first build_graph | spec D as written loops to k=0 on a one-node shortfall |
| 2026-09-13 | MaleCNS is read from the official bulk release (gs://flyem-male-cns/v1.0/connectome-data/flat-connectome/, CC-BY 4.0, md5-verified against the bucket listing), not neuPrint |
data/fly/fetch_malecns.py | neuprint.janelia.org is unreachable from this node (TCP timeout, sandbox or not); the bucket is the primary release and needs no token. The verified neuPrint dataset name is male-cns:v1.0 (was guessed as male-cns); kept as the --neuprint fallback |
| 2026-09-13 | Region column is superclass; region = central_brain iff superclass starts with cb_ (cb_intrinsic, cb_sensory, cb_motor, cb_endocrine, cb_efferent, cb_sensory_tbc = 37,229 bodies). vnc_* -> vnc, ol_* -> optic_lobe; ascending/descending, visual_projection/centrifugal, ENS, sensory_ascending -> other (span compartments, excluded); NaN -> unannotated |
data/fly/build_edges.py | §3.1: only unambiguous central-brain annotations enter the pool; the full crosstab is printed by build_edges.py |
| 2026-09-13 | Edge table comes from the full connectome-weights-...-minconf-0.5.feather (151.9M segment pairs), filtered to annotated bodies by the extractor. On the cb_ pool it differs from the traced-only file by 988 of 7.29M edges (0.014%), all touching cb_-annotated bodies whose status is orphan/leaf/anchor rather than traced; these are low-degree and are removed by the dense-core step |
first build_graph | User asked for the full graph; the difference is logged rather than silently chosen |
| 2026-09-13 | Reservoir gate (§11) passed on cb1k: FrozenFly best val 2.253 / final 2.346 nats vs bigram 2.482 (full split) and 2.720 (dev excerpt's own split). Dev val is the excerpt's last 500 chars, so the number is a gate, not a scoreboard entry | dev_1k frozen_seed1 | §15 |
| 2026-09-13 | cb5k benchmark (§8.3), one GPU, launch config (B=32, T=64, microsteps=2, 524,324 edges): 296 ms/step, 6.9k chars/s, 2.16 GB peak. 20k steps ≈ 100 min. Six paired runs fit on six GPUs concurrently | launch | The orientation note's ~145k-edge estimate was for a different core; the real cb5k core at min_synapses=3 has mean degree 105 |
| 2026-09-13 | Prior art, verified on the Hub: ngxson/fly-llm-hf (CC-BY 4.0, 2026-09-11) is a MaleCNS v1.0 frozen reservoir LM: 49,393 neurons (cb_sensory + visual_projection + cb_intrinsic + ascending + descending), 9.05M edges as CSR buffers in safetensors, signed synapse counts scaled to spectral radius 0.99, only input projection / per-neuron gains / readout trained, TinyStories, BPE-1024; one unseeded permuted-wiring control preserving in-degree only. eob/gpt-fly is a GPT-2 with FlyWire-derived MLP masks. Neither trains the connectome's edge values |
publication | FlyGPT's distinction is that the fly's edge values are trained, with a stricter cb_-only pool, in+out-degree-preserving control, and the 5-paired-seed rule. Model cards must credit fly-llm-hf as prior art in the first paragraph |
| 2026-09-13 | Hugging Face packaging plan (gated on a trained 5k model, §16 step 11): custom trust_remote_code model, model.safetensors with int32 graph.edge_index/graph.synapse_count, int64 graph.node_id (MaleCNS body ids), bf16 recurrent.edge_values/raw_leak/bias and adapters; sparse matmul reconstructed in fp32 at load (§8.2). Publish our own extraction as -init, -shakespeare, -degree-scrambled checkpoints; borrow the cache/generate plumbing pattern from fly-llm-hf with attribution, never its graph or weights |
export_hf.py | user direction 2026-09-13 |
| 2026-09-13 | ~/malecns-hf: the full traced MaleCNS v1.0 graph (164,587 neurons, 25,563,197 edges, 101 autapses kept, min synapses 1) as a lossless HF repo: int32 edge_index + exact int32 synapse counts canonical, bf16 copy derived only, NT/region/superclass as separate int8 tensors, named subset masks; build script verifies the saved edges against the release table. No sign convention, scaling, or dynamics imposed |
data/fly/export_malecns_hf.py | user direction 2026-09-13: a packaging layer, distinct from fly-llm-hf's derived reservoir |
| 2026-09-13 | 1k overfit gate (§15) initially failed at the scaffold defaults (init_scale 0.1, 3000 steps: train loss 1.26) while a parameter-matched tanh RNN memorized (0.13). Spectral radius at init ≈ init_scale (0.1 → nearly dead recurrence). Diagnostics on cb1k, same seed/data: init 0.5 → 0.94, init 1.0 → 0.63 at 3000 steps; at 10k steps init 1.0 → 0.20 (verbatim excerpt reproduction), init 0.1 → 0.28, init 1.0 + rec-LR 3e-3 → 0.15. Decision: init_scale: 1.0 in launch.yaml and dev_1k.yaml (radius 1.01–1.08 real, 1.01–1.02 scrambled, cb5k), dev gate runs 10k steps. Recurrent LR stays at the spec's 3e-4 for launch |
launch, dev_1k | §11 scales the reservoir to radius ~0.9–1.0; the trainable model now starts in the same regime. Not a tuning of microsteps. The gate run is runs/flygpt-diag-long-init10/cb1k/real_seed1 (config-identical to the updated dev_1k.yaml apart from the project name) |
| 2026-09-13 | QuixiAI/MaleCNS (built at ~/malecns-hf) redesigned as a literal mirror: canonical source is the FULL segment-to-segment table (88,384,522 bodies, 151,856,684 edges) in file row order, tensors neuron_id int64, edge_src/edge_dst/synapse_count int32; annotations as neuron_status/neuron_superclass int8 with release vocabularies; NT table as its own group (nt_neuron_index, nt_class, nt_confidence float64); verbatim parquet sidecars. No bf16 copy, no derived region tensor, no traced subfolder: status == Traced reproduces the release's traced-only table exactly (checked in the build). Counts are called "anatomical connection weights / synapse counts", never neural weights |
export_malecns_hf.py | user direction 2026-09-13: a lossless packaging layer must not make modeling choices (rounding, mappings, subsets) for downstream users |
| 2026-09-13 | FlyGPT consumes QuixiAI/MaleCNS (pinned revision 6330780a) instead of the raw release files; build_edges.py --source raw remains only to rebuild that repo. Edges from both routes are identical (151,856,561 directed edges after dropping self-loops). Trained checkpoints will be published to QuixiAI/FlyGPT with base_model: QuixiAI/MaleCNS in the card |
build_edges, export_hf | user direction 2026-09-13: the connectome repo is the foundation for everything downstream |
| 2026-09-14 | Fused CUDA kernels for the recurrence (flygpt/csrc/fly_recurrence.cu, flygpt/kernels.py; toolkit CUDA 13.2 + torch 2.14+cu132 installed for this). Profile of the COO path: ~14k kernel launches per step, CPU-launch-bound, and autograd accumulating 128 sparse-sparse adds per step. New path: one autograd Function per 64-char window; state kept [N,B] so a neuron's batch row is one 128-B line; 16 warps per row for the CSR product and its transpose, warp-per-edge dot for the edge gradient. cb5k, B=32, T=64, M=2, one B200: COO 220 ms/step → CSR 126 → fused 49 (1 warp/row) → 24.6 (4) → 17.8 (8) → 16.5 (16); peak memory 2.17 → 0.36 GB. Logits and every gradient match the COO path to ≤2e-9 (tests/test_kernels.py, B=4/32/40); train.py val losses at steps 50–200 identical to the COO run. Default model.backend: cuda, falls back to COO with a warning when nvcc is absent; generation still uses the COO step |
launch (relaunched from step 0 with the fused path) | user asked for custom kernels instead of a 2.5 h launch; spec §8.1 anticipated the CSR benchmark |
| 2026-09-14 | cb5k launch result (§13, 5 paired seeds, 20k steps, fused kernels, 6 min/run). Final val loss (nats/char): real 1.737 (1.728–1.752), degree_preserving 1.748 (1.738–1.759); bigram 2.482. Paired Δ = scrambled − real: +0.0175, +0.0125, +0.0104, +0.0076, +0.0073; mean +0.0111. §12 rule: 5/5 same sign passes, but mean Δ < 0.05 → "no detectable difference at this scale" (same verdict on best_val: mean +0.0127). Real is ahead by ~0.011–0.017 at every checkpoint from step 1000 on; the gap shrinks slightly late in training. Stability: frac_saturated 0.010, mean leak 0.60, no divergence. Train loss 1.56 vs val 1.74 at the end: mild overfitting, run length is not the limiter | results/ | The fly learns Shakespeare (far below bigram, structured speech in generations). Wiring effect at 5k is consistent in sign but below the pre-registered minimum effect; report exactly that (§19). Follow-ups per §14: neuron-count ladder, min_synapses {1,5}, microstep sweep |
| 2026-09-14 | Full training: configs/launch_100k.yaml = launch.yaml with steps 100k (eval every 500), project flygpt-v0-100k. Same graph, seeds, LRs, batch, context. Rationale: at 20k steps val was still dropping ~0.05 nats per 6k steps (41 epochs seen); best-val checkpoint is kept so late overfitting cannot hurt the published model. The 20k results are archived under results/cb5k_20k/ |
100k launch | user: "fully train it"; §13 does not fix the step count |
| 2026-09-14 | Whole nervous system (§10, §16 step 14): configs/full_cns.yaml, graph cns_full. Pool = every body with status == Traced (new status_filter in GraphConfig; default None so cb5k is unchanged), region_filter whole_cns, min_synapses 3. target_neurons 200,000 exceeds the largest SCC, so the extractor keeps the whole SCC with no core pruning or trim: the graph is the strongly connected traced nervous system (sizing run: 160,514 neurons, 10.4M edges; region mix 57% optic lobe, 23% central brain, 12% VNC, 8% spanning). I/O = top 2048 by out-degree / top 4096 by in-degree (8× the 5k interface, adapters still small vs 10.4M edges). Same model, LRs, gates. First build with one control seed for a dev look; 5 paired seeds only if the wiring question is pursued at this scale |
cns_full | user: "train the whole nervous system" |
| 2026-09-14 | cns_full pre-flight (real graph): 160,514 neurons, 10,402,842 edges, 11.06M params (adapters 3%), reciprocal pairs 1.0M. init_scale 1.0 gave spectral radius 1.60 (heavier-tailed degrees than cb5k); set init_scale: 0.65 → radius 0.85–1.04 across seeds, the same "≈1 at init" principle as cb5k. Step ≈ 770 ms with the GPU shared, peak 8.8 GB; steps set to 50k (best-val checkpoint kept), extend if val is still falling |
cns_full | pre-flight before spending GPU-hours |
| 2026-09-14 | Data parallelism: train.py under torchrun (DDP). Each rank draws its own batch from a rank-specific data-order stream (seed×1000+rank, identical rule across conditions so pairing holds), gradients all-reduced, global batch = world × batch_size; rank 0 logs, evaluates, generates, checkpoints (state dict without the DDP prefix). Not ZeRO/FSDP: the largest model is 11M params / 8.8 GB, there is nothing to shard, and FSDP would split the edge_values parameter the fused kernel needs whole. Whole-CNS runs use all 8 GPUs (global batch 256), 12,500 steps ≈ 200 epochs, real then scrambled via scripts/launch_cns_ddp.sh so both see the same global batch |
cns_full | user: use all 8 GPUs on one model |
| 2026-09-14 | 100k relaunch of seed 5 (the first attempt crashed at startup on a set_device("cuda") bug introduced with DDP; fixed, single-process path only sets the device when it has an index). GPU split while seed 5 reruns: seed-5 pair on GPUs 0–1, whole-CNS real on GPUs 2–7 via torchrun (global batch 192, steps 16,700 = the same 205M characters as the 8-GPU plan), scrambled control follows on the same six. Measured: 0.54 s/step, 13 GB/GPU, 100% util |
cns_full, 100k seed 5 | keep all 8 GPUs busy without changing what the paired CNS runs see |
| 2026-09-14 | cb5k at 100k steps (5 paired seeds). Final val: real 1.609 (1.603–1.619), scrambled 1.613 (1.606–1.630); best-val: real 1.5778–1.5992, scrambled 1.5816–1.6056. Paired Δ (final): +0.008, +0.011, +0.003, −0.002, −0.004, mean +0.003, mixed signs; Δ (best): all positive, mean +0.004. Verdict: no detectable difference at this scale. Compared with 20k (mean Δ +0.011, 5/5 positive): the real graph's advantage is an early-training speed effect that vanishes at convergence. Best val typically at 76k–94k of 100k; train ≈ 1.45 → mild overfitting, run length no longer the limiter. Public wording (§19): the fly connectome learns Shakespeare (1.58 nats vs bigram 2.48); at 5,000 neurons its specific wiring does not measurably beat a degree-matched scramble | results/ (100k) | §12, §14: failure is content |
| 2026-09-14 | Parameter-matched baselines (§11), 100k steps, 3 seeds, same data/split/context. Best val: GRU 1.522, transformer 1.548, dense tanh RNN 1.581, FlyGPT real 1.587, scrambled 1.591. Final val: GRU 2.048, RNN 2.013, transformer 1.724, FlyGPT 1.609. Every dense baseline overfits hard after its best step (best reached at 7k–27k steps); the fly graphs do not (best at 76k–94k, final within 0.02 of best). Reading: at ~578k parameters the connectome buys no accuracy over a dense recurrent net of the same size, but its fixed sparse structure acts as a strong regularizer. Scoreboard reports both columns for this reason | results/scoreboard.md | §11 |
| 2026-09-14 | FrozenFly reservoir at 100k steps (§11), 2 of 3 seeds done. Best val 1.854 / 1.862, final 1.869 / 1.889, with only 43,873 trainable parameters (adapters; the 524,324 edge values are frozen random-sign, scaled to spectral radius 0.95). Trainable-edge FlyGPT on the same graph: best 1.578. So training the fly's synapses is worth ~0.28 nats over using the same wiring as a fixed reservoir, which is exactly the axis separating this project from ngxson/fly-llm-hf. Gate §15 (beat bigram 2.482) passes by a wide margin | results/scoreboard.md | §11 |
| 2026-09-14 | Whole nervous system trained (§16 step 14). 160,514 neurons, 10,402,842 trainable synapses, 11.06M params, 16,700 steps at global batch 192 on 6 GPUs (132 min, 205M chars ≈ 200 epochs). Best val 1.5957 @ step 15,375, final 1.6291, train 1.333, frac_saturated 0.0013, mean leak 0.554. Scale did not help: cb5k best was 1.578 with 1/18 the neurons and 1/20 the synapses, so the whole CNS is 0.018 worse on val. Consistent with cb10k (best 1.610 so far). Reading: at context 64 the binding constraint is the context window and the 1.1M-char corpus, not model capacity; the larger graphs overfit sooner (train 1.333 vs val 1.596). Generation quality nonetheless reads better by eye: correct rare character names (AUTOLYCUS, COMINIUS, MENENIUS, HERMIONE, GREMIO, QUEEN MARGARET) and more grammatical lines. Do not claim scale improves the model | cns_full | §14 scaling ladder |
| 2026-09-14 | Stopped all runs at user request, 04:05 UTC. Killed mid-flight: whole-CNS degree_preserving seed 1 (step ~1,300/16,700, no usable result) and cb10k real seed 1 (step 42,876/100,000, best 1.6345 @ 38,000, partial). Correction to the §5 scaling claim: cb10k degree_preserving seed 1 completed at best 1.5613 @ 94,000 (final 1.5918) vs cb5k degree_preserving seed 1 at best 1.5816. So 10k is 0.020 BETTER than 5k on the matched seed and condition, while the whole CNS (1.5957) is worse than cb5k's 5-seed mean (1.587). The pattern is non-monotonic, not "scale does not help": 5k → 10k improves, 160k regresses. The cb10k real half was killed before finishing, so no paired Δ exists at 10k. conclusions.md §5 corrected accordingly | scaling ladder | accuracy of a published conclusion |