work on sigma_store=canonical and bootstrap_rec3D

Author: hael

doc/implementation_notes/pcg_priors.md

@@ -3473,6 +3473,32 @@ the acceptable-looking outputs do not validate the prior.
      nparts=... nthr=... rec_backend=pcg` (UI now exposes rec_backend,
      maxits_pcg, rtol).
 
+11. **Canonical sigma2 store review, streptavidin 2/10 failures (2026-09-07).**
+   - Proven non-equivalences with the legacy STAR store: (a) freshness,
+     canonical commits residuals(N) right after matcher pass N and its
+     stage-boundary and final reconstructions consume them, legacy's STAR N
+     holds residuals(N-1); (b) the final reconstruction: legacy refreshes
+     sigmas at native sampling whenever the registration box differs from
+     the native box, canonical reused any structurally consumable state and
+     skipped the refresh. Precision, averaging and grouping are the same.
+   - Failure surfaces only canonical had: hard abort on any active record
+     with a non-finite or non-positive shell; generation/digest/checksum/
+     coverage aborts on merge; silent re-seed from image power on any
+     validation failure. Design hazard: the state was registered as an
+     absolute path that survives every project copy, so a copied execution
+     project consumed and mutated the originating run's sigma2_state.bin;
+     candidate and range names carried no transaction identity.
+   - Fixes: registration by name resolved against the project file's
+     directory (absolute only for explicit cross-directory states);
+     abinitio3D drops an inherited registration at start; candidate and
+     range names carry the generation they commit
+     (`<stem>.g<N>.next`, `<stem>.g<N>.part<NN>.range`) and a worker checks
+     the candidate's generation; invalid records are skipped with a warning
+     in reduction and commit validation; the canonical final reconstruction
+     applies the legacy registration-box rule. Test
+     `simple_test_exec prg=sigma2_state` extended (invalid-record skip,
+     scoped names). Uncompiled.
+
 ## 11. The NU machinery as the prior infrastructure
 
 The nonuniform-regularization machinery

doc/policies/abinitio3D_policy.md

@@ -144,6 +144,18 @@ program takes any project with 3D orientations, it is also the standalone
 test for this stage (`simple_exec prg=bootstrap_rec3D projfile=... pgrp=...
 mskdiam=... nparts=... nthr=... rec_backend=...`), so failures in the final
 reconstruction can be reproduced in minutes rather than after a full run.
+Its bootstrap map is a gridding assembly that carries the last stage's
+`filt_mode`, `nu_refine` and `automsk` (the residual sigmas depend on the
+regularization of the reference they are scored against, so that reference is
+regularized like the last stage's matching references); the shipped map is
+classical and runs on the workflow's backend with the PCG cold-solve budget
+applied inside `bootstrap_rec3D` (2026-09-07).
+Whether the final reconstruction refreshes its sigmas at native sampling is
+decided by one rule for both sigma stores: a registration box different from
+the native box (2026-09-07; the canonical store used to reuse its committed
+state regardless). With `sigma_store=canonical`, abinitio3D drops any state
+registration inherited with its input project before the first stage, so
+every run seeds and owns its sigmas in its own directory.
 
 ## 4. Low-Pass and Cropping
 

doc/policies/refine3D_policy.md

@@ -189,7 +189,26 @@ an empty directory, and every final reconstruction at a new sampling. Since
   sequence: seed, bootstrap map, residual pass, consolidation, final map;
   abinitio3D's `calc_final_rec` and refine3D_auto call it and carry no copy
   of the sequence. It runs standalone on any project with 3D orientations
-  and is the test entry point for the final-reconstruction stage.
+  and is the test entry point for the final-reconstruction stage. The
+  bootstrap map only serves as the residual reference, so it is always a
+  gridding assembly with ML regularization (one particle pass, no
+  postprocessing) that keeps the workflow's `filt_mode`, `nu_refine` and
+  `automsk`: the residual sigmas depend on the regularization of the
+  reference they are scored against, so that reference is regularized
+  exactly as the refinement's matching references were; the shipped map keeps the
+  caller's backend and, on PCG, starts from nothing at the native box and
+  therefore gets the cold-solve budget of at least `FINAL_PCG_MAXITS_FLOOR`
+  (5) iterations whoever the caller is (2026-09-07). The base plus ML solve
+  pair of that final PCG reconstruction is inherent to ML regularization:
+  the prior is built from the base pair's independent-half FSC, which is
+  also the reported FSC and the unfiltered pair postprocessing uses.
+- The registration-box rule is the same for both stores (2026-09-07): a
+  final reconstruction whose registration (crop) box differs from the native
+  box refreshes the sigmas at native sampling (image-power seed, bootstrap
+  map, residual pass) before the shipped map. Before this the canonical store
+  reused its committed state whenever it was structurally consumable, which
+  is always true at the native box, and so skipped the refresh legacy
+  performs; the two stores could not produce the same final map.
 
 ## 6. Reference Preparation
 

doc/policies/sigma_calculation_policy.md

@@ -148,17 +148,38 @@ normalization by `2*pftsz`.
 
 Each canonical iteration is transactional:
 
-1. the master creates `sigma2_state.next`, copying the committed generation
-   when unchanged rows must survive;
-2. each worker writes one exclusive temporary range for its assigned global
-   particle rows;
+1. the master creates the candidate `<state stem>.g<N>.next`, where N is the
+   generation this update commits (committed generation + 1), copying the
+   committed generation when unchanged rows must survive;
+2. each worker writes one exclusive range file
+   `<state stem>.g<N>.part<NN>.range` for its assigned global particle rows,
+   after checking that the candidate it found carries generation N;
 3. the master verifies generation/layout identity and exact non-overlapping
    scheduled coverage, merges the ranges, and reduces active records into the
    grouped model;
 4. commit-time scientific and integrity validation runs;
 5. the candidate is synced and atomically published over the committed file,
    followed by directory sync.
 
+Candidate and range names carry the generation they belong to (2026-09-07),
+so a file left by another transaction, a crashed run or a second run in the
+same directory, can never be merged into this one; the mismatch fails loudly.
+
+Execution-local state (2026-09-07): the project registers its state by name
+only when the state lives in the project file's own directory, and the name
+is resolved against that directory on lookup. A project copied into a new
+execution directory (`mkdir=yes`) therefore no longer points back into the
+originating run's `sigma2_state.bin`; the copy seeds its own state. An
+absolute registration is kept only for an explicit cross-directory state
+(the streaming pool, project merges). abinitio3D drops any registration
+inherited with its input project at start, so a fresh workflow always seeds.
+
+Invalid records (2026-09-07): an active particle record with a non-finite or
+non-positive shell is skipped by the reduction and by commit-time validation,
+with a warning giving the count, instead of aborting the run. The legacy
+store averages whatever its part files hold; the canonical store now degrades
+the same way rather than failing where legacy would not.
+
 A failed or incomplete candidate never replaces the previous committed state.
 The committed format is independent of partition count and number padding.
 

doc/refactoring_notes/canonical_sigma2_state_refactoring.md

@@ -155,8 +155,10 @@ the complete generation.
 
 ## 5. Distributed Write and Recovery
 
-The committed `sigma2_state.bin` is immutable. Every update targets
-`sigma2_state.next`; only the master may create, size, validate, publish, or
+The committed `sigma2_state.bin` is immutable. Every update targets the
+generation-scoped candidate `sigma2_state.g<N>.next` (N = the generation the
+update commits; worker ranges are `sigma2_state.g<N>.part<NN>.range`, both
+since 2026-09-07); only the master may create, size, validate, publish, or
 discard it.
 
 Protocol:

production/tests/simple_test_sigma2_state.f90

@@ -5,18 +5,19 @@ program simple_test_sigma2_state
 use simple_syslib,            only: del_file, file_exists
 use simple_sigma2_state_file, only: sigma2_state_header, sigma2_state_init_header, &
     &sigma2_state_create_candidate, sigma2_state_write_local_range, sigma2_state_read_header, &
-    &sigma2_state_validate_file, SIGMA2_GROUP_GLOBAL, SIGMA2_GROUP_STACK, SIGMA2_PROV_PSPEC, &
-    &SIGMA2_PROV_RESIDUAL, SIGMA2_STATE_COMMITTED
+    &sigma2_state_read_groups, sigma2_state_validate_file, SIGMA2_GROUP_GLOBAL, SIGMA2_GROUP_STACK, &
+    &SIGMA2_PROV_PSPEC, SIGMA2_PROV_RESIDUAL, SIGMA2_STATE_COMMITTED
 use simple_sigma2_state,      only: sigma2_state_layout_digest, sigma2_state_merge_local_ranges, &
     &sigma2_state_reduce_groups, sigma2_state_validate_identity, sigma2_state_validate_science, &
     &sigma2_state_candidate_path, sigma2_state_commit, sigma2_state_prepare_update, &
-    &sigma2_state_range_path
+    &sigma2_state_range_path, sigma2_state_next_generation
 implicit none
 
 call exercise_policy('global', SIGMA2_GROUP_GLOBAL, 1, 4)
 call exercise_policy('group',  SIGMA2_GROUP_STACK,  2, 8)
 call exercise_recovery_guards()
 call exercise_update_preparation()
+call exercise_invalid_record_skip()
 write(*,'(A)') 'SIMPLE_TEST_SIGMA2_STATE NORMAL STOP'
 
 contains
@@ -171,13 +172,15 @@ subroutine exercise_update_preparation()
         logical :: active(4)
         integer :: eo(4), groups(4), status
         integer(int64), parameter :: DIGEST = 991_int64
+        integer(int64) :: next_gen
         character(len=128) :: message
         character(len=*), parameter :: COMMITTED = 'sigma2_state.bin'
-        candidate_path = sigma2_state_candidate_path(COMMITTED)
-        range_path = sigma2_state_range_path(COMMITTED, 2, 3)
+        candidate_path = sigma2_state_candidate_path(COMMITTED, 1_int64)
+        range_path = sigma2_state_range_path(COMMITTED, 1_int64, 2, 3)
         call del_file(COMMITTED)
         call del_file(candidate_path)
         call del_file(range_path)
+        call del_file(sigma2_state_candidate_path(COMMITTED, 2_int64))
         spectra(:,1) = [1.0,2.0,3.0]
         spectra(:,2) = [2.0,3.0,4.0]
         spectra(:,3) = [3.0,4.0,5.0]
@@ -197,19 +200,80 @@ subroutine exercise_update_preparation()
         call require_ok(status, message)
         call sigma2_state_commit(candidate_path%to_char(), COMMITTED, active, eo, groups, status, message)
         call require_ok(status, message)
+        call require(index(candidate_path%to_char(), 'sigma2_state.g1.next') > 0, &
+            &'candidate path is scoped to the generation it commits')
+        call require(index(range_path%to_char(), 'sigma2_state.g1.part002.range') > 0, &
+            &'canonical range path is scoped to the generation and includes the padded partition')
+        ! the next transaction is named for the generation it will commit
+        call sigma2_state_next_generation(COMMITTED, next_gen, status, message)
+        call require_ok(status, message)
+        call require(next_gen == 2_int64, 'next generation follows the committed one')
+        candidate_path = sigma2_state_candidate_path(COMMITTED, next_gen)
+        call require(index(candidate_path%to_char(), 'sigma2_state.g2.next') > 0, &
+            &'next candidate path is scoped to the next generation')
         call sigma2_state_prepare_update(COMMITTED, candidate_path%to_char(), status, message)
         call require_ok(status, message)
         call sigma2_state_read_header(candidate_path%to_char(), header, status, message)
         call require_ok(status, message)
         call require(header%generation == 2_int64, 'prepared update advances the generation')
         call require(header%provenance == SIGMA2_PROV_RESIDUAL, 'prepared update records residual provenance')
-        call require(index(range_path%to_char(), 'sigma2_state_range_part002.bin') > 0, &
-            &'canonical range path includes the padded partition')
         call del_file(COMMITTED)
         call del_file(candidate_path)
         call del_file(range_path)
     end subroutine exercise_update_preparation
 
+    !> an active record with a non-positive shell is skipped by the
+    !! reduction (with a warning) instead of aborting; the grouped model
+    !! is the mean of the valid records and commit-time validation agrees
+    subroutine exercise_invalid_record_skip()
+        type(sigma2_state_header) :: header
+        type(string) :: candidate_path, range_path
+        real(real32), allocatable :: stored(:,:,:)
+        real(real32) :: spectra(3,4), expected_even(3), expected_odd(3)
+        logical :: active(4)
+        integer :: eo(4), groups(4), status
+        integer(int64), parameter :: DIGEST = 1231_int64
+        character(len=128) :: message
+        character(len=*), parameter :: COMMITTED = 'skip_sigma2_state.bin'
+        candidate_path = sigma2_state_candidate_path(COMMITTED, 1_int64)
+        range_path     = sigma2_state_range_path(COMMITTED, 1_int64, 1, 1)
+        call del_file(COMMITTED)
+        call del_file(candidate_path)
+        call del_file(range_path)
+        spectra(:,1) = [1.0,2.0,3.0]
+        spectra(:,2) = [2.0,3.0,4.0]
+        spectra(:,3) = [3.0,0.0,5.0] ! invalid: a non-positive shell
+        spectra(:,4) = [4.0,5.0,6.0]
+        active = .true.; eo = [0,1,0,1]; groups = 1
+        expected_even = spectra(:,1)
+        expected_odd  = 0.5*(spectra(:,2)+spectra(:,4))
+        call sigma2_state_init_header(header, 1, 3, 4, 8, 1.5, 1, SIGMA2_GROUP_GLOBAL, &
+            &1_int64, DIGEST, SIGMA2_PROV_PSPEC)
+        call sigma2_state_create_candidate(candidate_path%to_char(), header, status, message)
+        call require_ok(status, message)
+        call sigma2_state_write_local_range(range_path%to_char(), 1_int64, DIGEST, 1, spectra, &
+            &1, 3, status, message)
+        call require_ok(status, message)
+        call sigma2_state_merge_local_ranges(candidate_path%to_char(), [range_path], &
+            &[.true.,.true.,.true.,.true.], status, message)
+        call require_ok(status, message)
+        call sigma2_state_reduce_groups(candidate_path%to_char(), active, eo, groups, status, message)
+        call require(status == 0, 'invalid record is skipped, not fatal')
+        call sigma2_state_read_groups(candidate_path%to_char(), stored, status, message)
+        call require_ok(status, message)
+        call require(all(abs(stored(:,1,1)-expected_even) <= 1.e-6*expected_even), &
+            &'even group mean excludes the invalid record')
+        call require(all(abs(stored(:,2,1)-expected_odd) <= 1.e-6*expected_odd), &
+            &'odd group mean is the mean of the valid records')
+        call sigma2_state_validate_science(candidate_path%to_char(), active, eo, groups, status, message)
+        call require(status == 0, 'commit-time validation applies the same skip rule')
+        call sigma2_state_commit(candidate_path%to_char(), COMMITTED, active, eo, groups, status, message)
+        call require_ok(status, message)
+        call del_file(COMMITTED)
+        call del_file(candidate_path)
+        call del_file(range_path)
+    end subroutine exercise_invalid_record_skip
+
     subroutine assert_committed_generation(path, expected)
         character(len=*), intent(in) :: path
         integer(int64),   intent(in) :: expected

src/fileio/simple_projfile_utils.f90

@@ -328,7 +328,7 @@ subroutine concatenate_canonical_states
             layout_digest = sigma2_state_project_layout_digest(merged_proj, merged_proj%os_ptcl2D)
             if( layout_digest == 0_int64 ) THROW_HARD('cannot derive merged canonical sigma2 layout identity')
             target_path = dir//'sigma2_state.bin'
-            candidate_path = sigma2_state_candidate_path(target_path%to_char())
+            candidate_path = sigma2_state_candidate_path(target_path%to_char(), generation)
             call sigma2_state_init_header(target_header, int(target_header%kfrom), int(target_header%kto), &
                 &nallptcls, int(target_header%box), real(target_header%smpd), ngroups, &
                 &target_header%grouping, generation, layout_digest, SIGMA2_PROV_RESIDUAL)
@@ -697,7 +697,7 @@ subroutine merge_selected_canonical_field( target_particles )
                 if( layout_digest == 0_int64 ) THROW_HARD('cannot derive merged canonical sigma2 layout identity')
                 target_dir = get_fpath(projfile_out)
                 target_path = target_dir//'sigma2_state.bin'
-                candidate_path = sigma2_state_candidate_path(target_path%to_char())
+                candidate_path = sigma2_state_candidate_path(target_path%to_char(), generation)
                 call sigma2_state_init_header(target_header, int(target_header%kfrom), int(target_header%kto), &
                     &nrows, int(target_header%box), real(target_header%smpd), ngroups, &
                     &target_header%grouping, generation, layout_digest, SIGMA2_PROV_RESIDUAL)

src/main/commanders/simple/simple_commanders_abinitio.f90

@@ -642,6 +642,16 @@ subroutine exec_abinitio3D( self, cline )
         call nice_comm%cycle()
         ! read project
         call spproj%read(params%projfile)
+        ! A fresh abinitio3D never continues another run's sigma2 estimate: a
+        ! canonical registration inherited with the project is dropped so the
+        ! first euclid stage seeds in this run's own directory (2026-09-07)
+        if( spproj%projinfo%get_noris() == 1 )then
+            if( spproj%projinfo%isthere(1, 'sigma2_state') )then
+                call spproj%projinfo%delete_entry('sigma2_state')
+                call spproj%write_segment_inside('projinfo', params%projfile)
+                write(logfhandle,'(A)') '>>> ABINITIO3D: dropped an inherited canonical sigma2 registration; sigmas are seeded here'
+            endif
+        endif
         ! provide initialization of 3D alignment using class averages?
         start_stage = 1
         l_ini3D     = .false.