Follow-on to #96, #97, #98. Full design doc lands with slice 1 at docs/llmsr-objective-formulation.md.
Problem
Symbolic regression looks for a functional FORM. The constants are usually a nuisance. The fit path conflates them:
score = m( f(X; theta*), y ) theta* = argmin_theta l( f(X;theta), y )
One pooled dataset, one theta, one scalar. If the same law governs 40 cells with 40 different constant sets, this scores the form against a single compromise theta and charges the FORM for a mismatch that belongs to the PARAMETERS. A correct law loses to a wrong-but-flexible one.
Wanted:
score = < m( f(X_i; theta*_i), y_i ) >_i theta*_i = argmin_theta l( f(X_i;theta), y_i )
Each group (cell, trial, subject, condition) refits its own constants under the same form. This is profile likelihood: per-group constants are nuisance parameters profiled out, so what remains measures only the form.
What #96 to #98 shipped, and the gap
Shipped: an arbitrary callable (y_pred, y_true) -> float registered from the scientist's own module, data_shape dispatch, and guard for hard accept/reject.
Not expressible: per-group refit, a score vector, any scorer that must refit or rescale as part of computing the error, per-point weights, or parameter-dependent penalties. The scorer sees neither X nor theta, although Constraint.check does see theta, so chi-squared with per-point sigma cannot be written at all. The only workaround is closing over module-level data, which welds a metric to one dataset.
Formulation
Five plug-in points, defaults reproducing current behavior:
partition: Data -> [Group] # who gets their own theta (default: one group)
bind: (equation, Group) -> CallPlan # today a closed two-value enum
score: (form, group, fit) -> float # per-group, MAY refit or rescale internally
constrain: (pred, ctx) -> bool # hard gates, per group
optimize: (loss_fn, ctx) -> theta # today: hardcoded BFGS
The load-bearing change is the control inversion. Today the pipeline fits and then hands the metric a finished prediction, so a metric can never refit and therefore can never profile out a nuisance parameter. The scorer should instead receive the form and the group plus a refit primitive it may call, which is what lets you (for example) regress out a per-cell gain and offset before measuring shape error.
No vendor fork required
The vendored FunSearch buffer is already vector-native: register_program(program, island_id, scores_per_test) takes a DICT, _reduce_score means over it for island ranking, and _get_signature uses the full tuple as a cluster key. Wheeler currently calls it with {"data": score}, a single key, which is where the vector is destroyed. Passing {group_id: score_i} recovers upstream's design at no cost and leaves vendor/ unforked, which matters given d0b05b7 frames it as a plug-in for LLM-SR rather than our code.
The buffer is a search heuristic; the winner is selected in best from submissions.jsonl, which keeps the raw vector. So the buffer can take a coarse signal without costing final-selection precision.
Related defect surfaced while designing this
fit.py hardcodes multi-start BFGS (_N_RESTARTS = 6). BFGS uses a numerical gradient, identically zero on a piecewise-constant objective, so it terminates at x0. Measured on a representative event-count objective:
BFGS starts that MOVED off x0: 0/13
best value found: 1.0 (0.0 is a perfect fit)
Nelder-Mead, same starts: 0.0 exact
Spike-distance and event-count losses are piecewise constant in theta, so the spike_train shape opened by #97 is fitted by an optimizer that cannot fit it: it degrades to best-of-7 random search. Per-group refit multiplies this by the group count. Addressed in slice 3.
Slices
--group-by with per-group theta and the raw vector into scores_per_test. No vendor change. Backward compatible: no --group-by means one group, byte-identical to today.- The
score(form, group, fit) inversion with the refit primitive, plus a declared cluster_by quantizer (raw floats make every signature unique, so island clustering degenerates; this is already true today with a single float score).
- Declarable
optimize with a gradient-free default for non-smooth objectives, groups fanned out in parallel to recover the cost.
Open question for slice 1
Failed groups. A form that nails 38 of 40 and fails 2, versus one mediocre on all 40: whichever way the NaN policy falls decides the winner. That is a scientific call, not a default worth inventing. It is also where the #98 hard constraints land, since a guard may now fail on a subset of groups.
Follow-on to #96, #97, #98. Full design doc lands with slice 1 at
docs/llmsr-objective-formulation.md.Problem
Symbolic regression looks for a functional FORM. The constants are usually a nuisance. The fit path conflates them:
One pooled dataset, one theta, one scalar. If the same law governs 40 cells with 40 different constant sets, this scores the form against a single compromise theta and charges the FORM for a mismatch that belongs to the PARAMETERS. A correct law loses to a wrong-but-flexible one.
Wanted:
Each group (cell, trial, subject, condition) refits its own constants under the same form. This is profile likelihood: per-group constants are nuisance parameters profiled out, so what remains measures only the form.
What #96 to #98 shipped, and the gap
Shipped: an arbitrary callable
(y_pred, y_true) -> floatregistered from the scientist's own module,data_shapedispatch, andguardfor hard accept/reject.Not expressible: per-group refit, a score vector, any scorer that must refit or rescale as part of computing the error, per-point weights, or parameter-dependent penalties. The scorer sees neither
Xnortheta, althoughConstraint.checkdoes seetheta, so chi-squared with per-point sigma cannot be written at all. The only workaround is closing over module-level data, which welds a metric to one dataset.Formulation
Five plug-in points, defaults reproducing current behavior:
The load-bearing change is the control inversion. Today the pipeline fits and then hands the metric a finished prediction, so a metric can never refit and therefore can never profile out a nuisance parameter. The scorer should instead receive the form and the group plus a refit primitive it may call, which is what lets you (for example) regress out a per-cell gain and offset before measuring shape error.
No vendor fork required
The vendored FunSearch buffer is already vector-native:
register_program(program, island_id, scores_per_test)takes a DICT,_reduce_scoremeans over it for island ranking, and_get_signatureuses the full tuple as a cluster key. Wheeler currently calls it with{"data": score}, a single key, which is where the vector is destroyed. Passing{group_id: score_i}recovers upstream's design at no cost and leavesvendor/unforked, which matters given d0b05b7 frames it as a plug-in for LLM-SR rather than our code.The buffer is a search heuristic; the winner is selected in
bestfromsubmissions.jsonl, which keeps the raw vector. So the buffer can take a coarse signal without costing final-selection precision.Related defect surfaced while designing this
fit.pyhardcodes multi-start BFGS (_N_RESTARTS = 6). BFGS uses a numerical gradient, identically zero on a piecewise-constant objective, so it terminates atx0. Measured on a representative event-count objective:Spike-distance and event-count losses are piecewise constant in theta, so the
spike_trainshape opened by #97 is fitted by an optimizer that cannot fit it: it degrades to best-of-7 random search. Per-group refit multiplies this by the group count. Addressed in slice 3.Slices
--group-bywith per-group theta and the raw vector intoscores_per_test. No vendor change. Backward compatible: no--group-bymeans one group, byte-identical to today.score(form, group, fit)inversion with the refit primitive, plus a declaredcluster_byquantizer (raw floats make every signature unique, so island clustering degenerates; this is already true today with a single float score).optimizewith a gradient-free default for non-smooth objectives, groups fanned out in parallel to recover the cost.Open question for slice 1
Failed groups. A form that nails 38 of 40 and fails 2, versus one mediocre on all 40: whichever way the NaN policy falls decides the winner. That is a scientific call, not a default worth inventing. It is also where the #98 hard constraints land, since a guard may now fail on a subset of groups.