Skip to content

Commit 37e366b

Browse files
mstrathmanclaude
andauthored
feat: fit()/predict() + backtest() aggregate for an agent-guessable API (#20)
* feat: fit()/predict()/backtest() SQL surface + hardened training core Reshape the tabular and evaluation surface into the scikit-learn shape agents guess, and extract and harden the native-student training core that backs it. Surface: - fit(f1, ..., fN, label [, options]) is an aggregate over the caller's rows and returns a registered model id or a serialized blob; predict(model, f1, ..., fN [, options]) is a scalar serving one prediction per row (deserialized once, cached on the model argument). The label is the last positional argument; the optional trailing '{...}' is always the options object. - backtest(ts, value, horizon [, options]) is an aggregate (matching forecast / detect_anomalies) that runs rolling-origin evaluation and returns one JSON document; backtest_rows() expands it into typed rows. The old TVF is renamed predict_batch. - The tabular training core (tree / gradient-boosted / MLP fitters) moves to predict-train.{c,h}, shared by fit() and the distill recipes. Correctness and safety hardening (a thorough review pass, fixed at the root and swept across every sibling, not only where first flagged): - Memory safety: a compile-time assert that predict()'s feature buffer holds any student the loader admits; re-fetch after sqlite3_set_auxdata to avoid a use-after-free; propagate OOM through the row-cursor mprintf paths instead of silent NULL columns; 64-bit sizing for model-scaled allocations; bounded, overflow-safe onnx io_spec arrays (a 32-bit-wasm overflow); a fit() training-set cap. - Determinism: total-order split comparators so the tree, and its content_hash, are identical across platforms/libc (no reads of uninitialized task-union fields). - Fail loud, never silently: reject non-finite features/labels/teacher cells and NULL labels; surface malformed-JSON parse errors, prepare failures, and mid-read step errors; preserve distinct error codes; cap the classifier vocabulary at PREDICT0_MAX_CLASS during interning and parsing; require a positive json-array cap rather than a silent-unbounded footgun. - The receipt tool's operation taxonomy is restructured (aggregate vs row-shaped, with the *_rows TVFs first-class and wrapper-aware inference). Tests: adversarial coverage for the new surface and every hardening path (malformed/oversized/boundary/non-finite inputs asserting concrete PREDICT_ERR_* codes); the corrupt-blob tests now supply a matching content hash so they reach the bounds-checked deserializer; the C soak drivers assert row counts and the specific error code per branch, and propagate statement failures to the exit code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: fit/predict/backtest reference, skills, and receipts Document the new surface for a first-time reader: the functions reference (fit/predict/backtest, predict_batch's optional train_query and required apply_query, the JSON-object-label convention), the models and options pages, the sqlite-predict / distill-lifecycle / interpret-backtest skills, README, and the CHANGELOG. Scope the read-only guarantee to the serving functions (registration writes to _predict_models). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent d924067 commit 37e366b

37 files changed

Lines changed: 3563 additions & 1947 deletions

‎CHANGELOG.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,24 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
66

77
## [Unreleased]
88

9+
### Changed
10+
11+
- **The tabular and evaluation surface is now the scikit-learn `fit`/`predict`
12+
shape.** `fit(f1, ..., fN, label [, options])` is a new aggregate that trains a
13+
native student over your rows (the label is the last positional argument).
14+
With the `'{"register":"name"}'` option it registers the trained model and
15+
returns its id; without it, `fit()` returns the student as a serialized blob.
16+
`predict(model, f1, ..., fN [, options])` is now a per-row **scalar** that
17+
serves a student (a registered id or a `fit()` blob) and composes with
18+
`WHERE`, joins, and ORMs. `backtest(ts, value, horizon [, options])` is now an
19+
**aggregate** over your rows like `forecast`/`detect_anomalies`, one JSON
20+
document per group, expanded with the new `backtest_rows()`. The former
21+
`predict(train_query, apply_query)` table-valued function is renamed
22+
**`predict_batch`** (the batched, in-context `knn5-incontext`, and ONNX serving
23+
path); the old query-string `backtest(query, horizon)` TVF is removed. This is
24+
a pre-1.0 break; `distill_predict`/`distill_forecast`, `forecast`, and
25+
`detect_anomalies` are unchanged.
26+
927
## [0.1.0] - 2026-07-30
1028

1129
### Changed

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ prefix?=dist
3737

3838
TARGET_LOADABLE=$(prefix)/predict0.$(LOADABLE_EXTENSION)
3939

40-
OBJS=sqlite-predict.c predict-forecast.c predict-registry.c predict-tabular.c predict-student.c predict-distill.c vendor/sha256.c
40+
OBJS=sqlite-predict.c predict-forecast.c predict-registry.c predict-tabular.c predict-student.c predict-train.c predict-distill.c vendor/sha256.c
4141
ONNX_OBJS=$(OBJS) predict-onnx.c
4242

4343
# onnxruntime is a build+runtime dependency of the loadable-onnx variant

‎README.md‎

Lines changed: 35 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -109,8 +109,7 @@ SELECT model_id, holdout_metric FROM distill_predict(
109109
'{"target":"churned","student_id":"churn-v1","student_kind":"gbt"}');
110110

111111
-- forever: serve the student per row, in microseconds, no runtime attached
112-
SELECT * FROM predict(NULL, 'SELECT id, tenure, spend FROM customers',
113-
'{"model":"churn-v1"}');
112+
SELECT id, predict('churn-v1', tenure, spend) FROM customers;
114113

115114
-- the same move for time series: distill a Chronos-class teacher's forecasts
116115
-- into a DLinear/TiDE student, then serve it through forecast()
@@ -159,25 +158,32 @@ skills-capable agent, or read them as the condensed operator's manual.
159158
| --- | --- | --- |
160159
| `forecast(ts, value, horizon [, options])` | Where is this metric going? | an aggregate over your rows: forecast steps with prediction intervals, one JSON document per group |
161160
| `detect_anomalies(ts, value [, options])` | Which points are abnormal? | an aggregate over your rows: per-point anomaly probability and interval, one JSON document per group |
162-
| `predict(train_query, apply_query [, options])` | Classify/regress unseen rows | a prediction and confidence per row, zero-shot from in-context examples |
163-
| `distill_predict(train_query [, options])` | Compress a slow teacher into a fast student | a tiny native model (decision tree or gradient-boosted forest), registered and ready for `predict()` |
164-
| `distill_forecast(train_query [, options])` | Compress a forecast foundation model into a fast student | a native DLinear/TiDE forecast net (a linear skip plus a small residual), registered and ready for `forecast()` |
165-
| `backtest(query, horizon [, options])` | How accurate is the model here? | per-fold accuracy (MAE, RMSE, MASE, sMAPE) and interval coverage from a rolling-origin evaluation |
166-
167-
One calling convention per operation. `forecast` and `detect_anomalies` are
168-
**aggregates**: your statement supplies the rows, so `WHERE`, joins, bound
169-
parameters, and `GROUP BY` all compose naturally, from the CLI or from an ORM
170-
(Drizzle, SQLAlchemy, Diesel). They are **pure functions**: nothing is ever
171-
written, so they run on read-only databases and inside views. Each group
172-
returns one JSON document; expand it with `forecast_rows()` /
173-
`anomaly_rows()`, or `JSON.parse` it in your app. Each language's
161+
| `backtest(ts, value, horizon [, options])` | How accurate is the model here? | an aggregate over your rows: per-fold MAE, RMSE, MASE, sMAPE and coverage from rolling-origin evaluation |
162+
| `fit(f1, ..., fN, label [, options])` | Train a model on labeled rows | an aggregate over your rows: a native tabular student, registered by id or returned as a blob |
163+
| `predict(model, f1, ..., fN [, options])` | Classify or regress a row | a scalar: one prediction per row from a fitted student |
164+
| `distill_predict(train_query [, options])` | Compress a slow teacher into a fast student | a tiny native model (decision tree, gradient-boosted forest, or MLP), registered and served by `predict` |
165+
| `distill_forecast(train_query [, options])` | Compress a forecast foundation model into a fast student | a native DLinear/TiDE forecast net (a linear skip plus a small residual), registered and served by `forecast` |
166+
167+
One rule for the whole surface. `forecast`, `detect_anomalies`, `backtest`, and
168+
`fit` are **aggregates**: your statement supplies the rows, so `WHERE`, joins,
169+
bound parameters, and `GROUP BY` all compose naturally, from the CLI or from an
170+
ORM (Drizzle, SQLAlchemy, Diesel). The first three are **pure functions**:
171+
nothing is written, so they run on read-only databases and inside views. Each
172+
returns one JSON document; expand it with `forecast_rows()` / `anomaly_rows()` /
173+
`backtest_rows()`, or `JSON.parse` it in your app. `predict` is a **scalar**,
174+
one prediction per row, so it drops into any `SELECT` beside your other columns.
175+
Each language's
174176
[getting-started guide](https://code.purestorage.com/sqlite-predict/getting-started/python/)
175177
shows the pattern through its native ORM (SQLAlchemy, Drizzle, Diesel).
176178

177-
`backtest`, `predict`, and the `distill_*` operations take read-only SELECT
178-
queries (training needs labeled or windowed row sets); their results are
179-
ordinary rows you can join, filter, and materialize. Options everywhere are a
180-
trailing JSON object (`'{"confidence_level":0.9}'`).
179+
`distill_predict`, `distill_forecast`, and `predict_batch` take read-only SELECT
180+
queries (training needs labeled or windowed row sets, batched serving re-runs a
181+
query); their results are ordinary rows you can join, filter, and materialize.
182+
`predict_batch(train_query, apply_query, ...)` scores the rows of `apply_query`,
183+
which is required; `train_query` is optional, used only by in-context models, and
184+
is `NULL` when serving a registered model: `predict_batch(NULL, apply_query,
185+
'{"model":"..."}')`.
186+
Options everywhere are a trailing JSON object (`'{"confidence_level":0.9}'`).
181187

182188
### Calibrated intervals, auto-selection, and backtesting
183189

@@ -200,8 +206,8 @@ trailing JSON object (`'{"confidence_level":0.9}'`).
200206

201207
```sql
202208
-- confirm the conformal band actually covers at the nominal 90%
203-
SELECT avg(coverage) FROM backtest('SELECT ts, value FROM readings', 6,
204-
'{"interval_method":"conformal","confidence_level":0.9,"folds":25}');
209+
SELECT avg(coverage) FROM backtest_rows((SELECT backtest(ts, value, 6,
210+
'{"interval_method":"conformal","confidence_level":0.9,"folds":25}') FROM readings));
205211
```
206212

207213
## Models
@@ -219,7 +225,7 @@ well-studied statistical models:
219225
subsequence-reconstruction detector (windowed PCA reconstruction error), the
220226
method family that leads the TSB-AD-U benchmark; ~2x the residual detector on
221227
the typical series there
222-
- `knn5-incontext` (z-scored 5-NN) for `predict()`
228+
- `knn5-incontext` (z-scored 5-NN) for `predict_batch()`
223229

224230
Foundation models are treated as *teachers*, not serving paths: in
225231
benchmarking they were far too slow to call per query on CPU. The path to
@@ -298,8 +304,8 @@ count), so the common case is one line:
298304

299305
```sql
300306
SELECT predict_register('churn', '/models/churn.onnx');
301-
SELECT * FROM predict(NULL, 'SELECT id, tenure, spend FROM customers',
302-
'{"model":"churn"}');
307+
SELECT * FROM predict_batch(NULL, 'SELECT id, tenure, spend FROM customers',
308+
'{"model":"churn"}');
303309
```
304310

305311
Feature columns map by position (apply-query order); pass an explicit
@@ -384,10 +390,12 @@ build `make loadable-onnx-gpu`.
384390

385391
## How it works
386392

387-
`forecast` and `detect_anomalies` are SQL aggregate functions; `backtest`,
388-
`predict`, and the `distill_*` operations are [eponymous table-valued
389-
functions][tvf] (virtual table modules) over the rows of an inner query.
390-
Statistical models run in-process on CPU and are deterministic. Registered
393+
`forecast`, `detect_anomalies`, `backtest`, and `fit` are SQL aggregate
394+
functions; `predict` is a scalar; `distill_predict`, `distill_forecast`, and
395+
`predict_batch` are [eponymous table-valued functions][tvf] (virtual table
396+
modules) over the rows of an inner query (`predict_batch` takes two: an optional
397+
`train_query` and the required `apply_query`). Statistical models run in-process on
398+
CPU and are deterministic. Registered
391399
models and distilled students live in one `_predict_models` table the
392400
extension manages, content-addressed so the exact bytes are pinned and
393401
verified before deserialization.

0 commit comments

Comments
 (0)