This document describes how to size and phase Quixi Math data when mixing it into a larger general pretraining corpus. The goal is not to make Quixi Math the whole math diet. The goal is to provide dense, high-quality scratchpad examples that teach exact step discipline, arithmetic bookkeeping, verification, and canonical answer formatting.
For a serious first full run after TODO.md is complete, generate about
1-3B Quixi Math tokens, likely 5-15M examples once advanced generators
produce longer scratchpads.
Use a progressive interleaved curriculum:
- Start with mostly elementary and middle-school examples.
- Gradually raise the share of high-school, college, and graduate topics.
- Never drop earlier skills entirely; advanced math still depends on arithmetic, fractions, signs, simplification, and equation solving.
Do not train as one hard boundary sequence such as "all elementary, then all middle, then all advanced." That creates avoidable distribution shifts and can make basic skills fade during later training.
Current default repo output is small per example: a rough local sample measured
about 100 tokens/example using len(json) / 4. Advanced TODO items will be
longer because they should include CHECK steps, trial/reject/accept traces,
tables, symbolic rewrites, and multi-stage computations. Plan around
150-250 tokens/example for the mature dataset.
Recommended tiers:
| Tier | Examples per skill | Approx. total scale | Use |
|---|---|---|---|
| Smoke test | 100-500 | 10-50M tokens | Pipeline validation and format checks |
| Minimum useful | 1k-2k | 100-300M tokens | Small ablations and sanity training |
| Recommended v1 | 5k-10k | 1-3B tokens | First serious pretraining mixture |
| Stretch | 20k+ | 3-6B tokens | Only if generator diversity is strong |
Avoid padding tiny finite problem spaces with duplicates. If a generator exhausts its useful space, downweight it or treat it as garnish.
When Quixi Math is included in a larger general pretraining dataset:
| Training setting | Quixi Math share |
|---|---|
| Broad general pretraining | 0.5-3% of total tokens |
| Math-heavy midtraining or final phase | 5-10% of total tokens |
| Math-specialized model | Higher is possible, but mix with natural math text, code, textbooks, and web math |
Synthetic scratchpads are high-signal but narrow-format. They should complement natural mathematical prose, code, proofs, textbook explanations, and real problem solutions.
Use grade_level and difficulty metadata to construct phase-specific sampling
weights. The percentages below are the composition of the Quixi Math subset,
not the whole pretraining corpus.
| Training progress | Elementary | Middle | High | College | Graduate |
|---|---|---|---|---|---|
| 0-20% | 80% | 20% | 0% | 0% | 0% |
| 20-45% | 50% | 40% | 10% | 0% | 0% |
| 45-70% | 20% | 40% | 35% | 5% | 0% |
| 70-90% | 10% | 20% | 40% | 25% | 5% |
| 90-100% | 10% | 15% | 30% | 30% | 15% |
Within each grade band, phase difficulty from easy to hard:
- Early phases: mostly difficulty 1-3.
- Middle phases: mix difficulty 2-4.
- Later phases: emphasize difficulty 4-5, but keep some easy examples for arithmetic replay.
One practical starting point is:
| Phase | Difficulty 1 | Difficulty 2 | Difficulty 3 | Difficulty 4 | Difficulty 5 |
|---|---|---|---|---|---|
| 0-20% | 25% | 35% | 30% | 10% | 0% |
| 20-45% | 10% | 25% | 35% | 25% | 5% |
| 45-70% | 5% | 15% | 30% | 35% | 15% |
| 70-90% | 5% | 10% | 20% | 40% | 25% |
| 90-100% | 5% | 10% | 20% | 35% | 30% |
Reserve more of the final phases for examples that teach robust reasoning habits:
CHECKsteps from self-verification.- Trial/reject/accept traces for factoring, rational roots, search procedures, and candidate testing.
- Composite problems that chain 2-3 skills.
- Word problems with irrelevant quantities and explicit selection of useful data.
- Error-spotting and fill-in-the-missing-step formats after their record format is designed.
These examples are more valuable late because the model has already learned the basic syntax and arithmetic operations.
Do not train directly on raw JSON unless the model is expected to emit JSON. Use a stable text serialization that keeps metadata available without making the JSON syntax the main task.
Example:
<math grade="middle" difficulty="4" skill="TwoStepEquationGenerator">
Problem: Solve 3x + 5 = 20.
Work:
SETUP|3x + 5 = 20
SUBTRACT|20|5|15
REWRITE|3x = 15
DIVIDE|15|3|5
CHECK|3*5 + 5 = 20
Answer: x = 5
</math>
Keep the record shape stable across phases. The model should learn one consistent scratchpad interface.
Before mixing data into training, reserve holdouts:
- Hold out 1-2% per skill and difficulty.
- Keep a separate adversarial holdout for edge cases: carries, borrows, negative coefficients, extraneous roots, undefined denominators, unit mismatches, and multi-solution answers.
- Deduplicate holdouts against training by exact
(operation, problem)and, for word problems, by normalized numeric structure where possible.
Track:
- Exact
final_answermatch. - Last-step validity: final step equals
Z|<final_answer>in source records. - Per-skill accuracy.
- Per-grade and per-difficulty accuracy.
- Step-format validity.
- Presence and correctness of
CHECKsteps where expected.
Do not commit the full budget before measuring. Run small ablations first:
- 100M Quixi tokens: verifies that the serialization, tokenizer behavior, and phase mixer are not broken.
- 300M Quixi tokens: enough to see early skill movement and regressions.
- 1B Quixi tokens: first meaningful comparison of curriculum schedules.
- 1-3B Quixi tokens: recommended v1 production scale if ablations help.
Compare at least:
- Randomly mixed Quixi examples.
- Strict easy-to-hard ordering.
- Progressive interleaving as described above.
- Progressive interleaving plus late-phase
CHECK/composite upweighting.
The expected winner is progressive interleaving. Strict easy-to-hard may help early but is more likely to create phase-boundary artifacts.
The generator already stamps grade_level and difficulty from
curriculum.py. A training mixer should use those fields directly.
Useful generation commands:
source .venv/bin/activate
python quixi_math_datagen.py --sample
python quixi_math_datagen.py -n 1000000 -o /tmp/quixi_math_1m.jsonl -s 123
python quixi_math_datagen.py -n 1000000 -o /tmp/quixi_math_1m.jsonl --weights "QuadraticGenerator=3,MeanGenerator=0.5" -s 123For large builds:
- Write outputs outside the repo, for example under
/tmpor a dataset volume. - Save the seed, generator commit, command, and weight config.
- Save per-generator stats from the build output.
- Keep the exact train/validation split manifests.
These are scale reference points, not direct prescriptions:
- DeepSeekMath continued pretraining used 120B math-related tokens mixed with natural language and code: https://arxiv.org/abs/2402.03300
- OpenMathInstruct-2 built 14M question-solution pairs and emphasized question diversity and solution format: https://arxiv.org/abs/2410.01560
- Llemma continued pretraining used a broad math mixture rather than a single synthetic format: https://arxiv.org/abs/2310.10631
- Chinchilla-style dense-model planning is often summarized around 20-25 tokens per parameter, but Quixi Math should be a targeted slice of that budget: https://arxiv.org/html/2404.10102v1
- Continued-pretraining work supports shifting toward more targeted data later in training while preserving enough base-distribution replay: https://arxiv.org/html/2407.07263v1
The proofs program (quixi_proofs_datagen.py; plans/proofs_plan.md) is a
separate dataset from the same repo: blend the two programs in your sampling
mixer, never in the concept. Proof records teach warranted derivation — every
step cites its justification and its inputs — while math records teach
procedure execution; keep them distinguishable so each signal can be weighted
(a starting point: 3-4 parts math to 1 part proofs within the math slice of
the mixture).
Records serve two export shapes from one machinery: the JSONL records are the
SFT export, and proofs_common.check_proof is the RL/eval grader — it accepts
every rule-valid derivation, not just the sampled reference path. For RL use,
also validate premises against the problem text (a registered follow-up: the
per-operation premise extractors), since check_proof deliberately grades
relative to a record's own premises.
Capacity (re-measured after the parameter-envelope widening, seeded 100k
probe): every non-garnish family now clears ~10⁵ unique problems and most
clear 10⁶ — estimated ceilings run from ~9×10⁴ (graph certificates) through
10⁵–10⁶ (primality, log laws, exponent laws, the induction families) to 10⁶+
(congruence chains, factoring, recurrences) and effectively unbounded for the
nine combinatorial families (CRT, gcd/Bezout, divisibility rules, invariant
and termination games, angle chase, rewrite chains, error spotting,
inequality composition). Total unique capacity is conservatively in the
30M–300M record range (~5B–50B rough tokens); a diversity-balanced corpus of
5–20M records is comfortable with mild weighting. Two families remain
deliberate garnish with hard ceilings — TrigExactValueProofGenerator
(~100 unique problems) and TelescopingSumProofGenerator (~300) — cap or
downweight them in any sizable build. The remaining cheap multiplier, if more
is ever needed, is prompt-phrasing count (4 per variant today; each added
phrasing set multiplies every family's problem space).