v40: bounded retention portfolio, historical Fisher and experimental capacity growth

Author: Mantiee

RETENTION_RESEARCH.md

@@ -0,0 +1,71 @@
+# Continual-learning retention portfolio (v100.40)
+
+The recommended starting point for this hardware is a frozen pretrained base,
+verified historical replay, predecessor KL, immutable accepted versions and
+independent regression gates. Additional constraints should be compared as
+separate candidates. Stacking every method is not evidence of improvement:
+stability constraints can prevent useful new learning and add expensive forwards.
+No finite test establishes universal zero forgetting.
+
+## Primary research and implementation decisions
+
+| Approach | Evidence / limitation | Implementation |
+| --- | --- | --- |
+| Historical replay | LLM CL surveys identify rehearsal as a major family; sampling coverage and distribution matter. | Existing mandatory verified prior-training records remain in A/B datasets. Existing split ledger excludes validation/audit sources. New reference sampler covers historical domains round-robin, at most eight examples. |
+| Learning without Forgetting / KL | Li & Hoiem, ECCV 2016, https://arxiv.org/abs/1606.09282. Original work is vision, not a Gemma guarantee. | Existing shared frozen-base teacher adapter and supervised-token KL remain active; model chooses strength. No second 12B base allocation. |
+| EWC | Kirkpatrick et al., PNAS 2017, https://doi.org/10.1073/pnas.1611835114. Empirical diagonal Fisher is an approximation and misses cross-parameter interactions. | New per-example squared-gradient Fisher on at most eight verified historical training records, quadratic anchor penalty, bounded LoRA parameters only. Reference weights/Fisher pinned and restored across checkpoint resume. Not original full-model EWC and not online-EWC. |
+| Parameter anchoring | L2-SP, Li et al., ICML 2018, https://proceedings.mlr.press/v80/li18a.html. Parameter closeness does not imply identical outputs. | New L2 anchor to the initial predecessor adapter; frozen base unchanged. |
+| Orthogonality | O-LoRA, EMNLP 2023, https://aclanthology.org/2023.findings-emnlp.715/; OGD, AISTATS 2020, https://proceedings.mlr.press/v108/farajtabar20a.html. | Experimental delta-A overlap penalty with predecessor row directions. Initialization penalty is zero. This is inspired by orthogonal methods, **not an O-LoRA/OGD reproduction**. |
+| A-GEM | Chaudhry et al., ICLR 2019, https://arxiv.org/abs/1812.00420. Half-space protection is first-order on sampled replay; AdamW momentum/finite steps can still cause forgetting. | New conflicting-gradient projection after AMP unscale/clip and before optimizer step. Historical replay references rotate; adds one reference forward/backward per optimizer attempt. Nonfinite AMP attempts are left to the existing health gate. |
+| Capacity growth | Progressive Neural Networks, Rusu et al., https://arxiv.org/abs/1606.04671, protect old columns when old task routes stay fixed. Route errors remain a problem. | New doubling of standard LoRA rank (maximum 64 and measured budget), zero new B columns, unchanged alpha/r and old tensors. It preserves the **initial** delta, not future behavior. Normal PEFT/GGUF candidate export and all quality gates remain required. |
+| New layers / embeddings | Architectural growth can isolate interference, but changes export/tokenization and resource use. | `capacity_growth.py` supplies isolated tiny-model residual columns with zero output initialization, frozen predecessor route, and appended embedding rows with unchanged old IDs. Maximum two million old/new parameters for these prototypes. **Not live Gemma layer surgery, tokenizer expansion or a native GGUF serving feature.** |
+| External memory / RAG | Retrieval avoids some weight updates but has stale/missing retrieval and routing failure modes. | Existing CPU hybrid persistent memory remains separate from weights. It is not proof that weights retain knowledge. |
+| Evaluation / rollback | GEM describes backward transfer and continual-learning evaluation. Repeated public/development tests can overfit. | Existing case-by-case ancestor gates, official grading, one-use fresh audits and preserved accepted versions remain mandatory. Fresh audit answers are not training/replay/Fisher inputs. Old-version routing remains an available fallback, not proof a new version retained skills. |
+
+Survey: Shi et al., *Continual Learning of Large Language Models: A Comprehensive
+Survey*, https://arxiv.org/abs/2404.16789. Methods target different kinds of forgetting
+and stages of training; results on small classification networks do not establish
+performance on sequential Gemma 12B fine-tuning or financial outcomes.
+
+## What the controller can select
+
+With `resources.retention_experiments=true`, bounded A/B planning adds:
+
+- `retention_mode`: 0 replay/KL; 1 L2; 2 EWC; 3 L2 + delta-A orthogonality;
+  4 EWC + delta-A orthogonality; 5 A-GEM projection.
+- `retention_strength`: 0.001, 0.01, 0.1, 1.0 (unused for projection).
+- `retention_rank_growth`: 1 or 2, only when an existing rank can double within
+  the measured maximum rank and the absolute limit of 64.
+
+Delta-A orthogonality needs a predecessor adapter. EWC/projection are unavailable until a pinned predecessor and historical verified
+training examples exist. Anchors/Fisher are capped at 160 million adapter
+parameters. References are selected from the historical training portion of the
+immutable combined pool, never the withheld validation set. These bounded
+references are not comprehensive coverage of all pretrained capabilities.
+
+Artifacts in each trial's training directory:
+
+- `retention.json`, `retention-anchor.safetensors`: measured/reference provenance.
+- `capacity-growth.json`, `expanded-initial/`: rank expansion evidence.
+- Existing `training_health.json`, metrics, checkpoint manifests and quality
+  judgments: actual successful optimizer counters and acceptance evidence.
+- Activity events `retention-reference` and `replay-gradient-projection`.
+
+The mission evidence collector exposes current-run reference/growth reports.
+A configured method or a saved reference is **not** proof of a successful update,
+accepted weights, improved retention, new task accuracy or profit.
+
+## Experimental boundaries and omissions
+
+Full GEM quadratic programming, full 12B Fisher/Jacobian matrices, unconstrained
+layer replication, live vocabulary changes, and running many 12B model copies
+are not enabled. They have unresolved compute/export/validation costs on a 32 GiB
+V100. The implementation provides a testable portfolio, not every CL algorithm
+or all published implementations. Newer variants require separate reproduction
+and hardware measurements before being added to this pipeline.
+
+Frozen-column and embedding prototypes have CPU tests of initial equality and
+unchanged old-route/old-ID outputs after new-parameter training. LoRA growth has
+structural factor/scaling tests. Fisher, penalties, projection and tamper-sensitive
+resume have deterministic tensor tests. **Full NF4/FP16 Gemma training, retention
+improvement and native export of an expanded adapter still need on-device tests.**

V100_AUDIT.md

@@ -276,3 +276,18 @@ argument schema. It can also decline tools with a real concise answer. Task scop
 argument validation, token limits and the original input remain enforced.
 Oversized tool results are retained in SHA-256-addressed files and activity traces;
 model-visible previews explicitly mark truncation and do not certify missing facts.
+# Retention experiments v100.40
+
+`RETENTION_RESEARCH.md` distinguishes existing replay/KL/fresh regression gates
+from new bounded LoRA L2, empirical diagonal Fisher EWC, experimental delta-A
+orthogonality, A-GEM projection and exact-initial-delta rank expansion. Model
+planning can compare them; historical methods require previous verified training
+references, never audit labels. Anchors/Fisher are pinned across resume, and
+retention/capacity artifacts appear in current-run training evidence. Existing
+ancestor/public/one-use audit gates and original accepted checkpoints remain.
+
+`capacity_growth.py` implements small isolated frozen residual-column and appended
+embedding prototypes. Their old routes/IDs stay unchanged in tensor tests; this
+does not implement live Gemma layer/vocabulary surgery or prove retention on
+unseen tasks. Full NF4/FP16 training and grown-adapter native export require V100
+acceptance. The dedicated upgrade preserves setup and skips calibration sweeps.

pyproject.toml

@@ -3,7 +3,7 @@ name = "rlms"
 authors = [
     {name = "Alex Zhang", email = "altzhang@mit.edu"},
 ]
-version = "0.1.3+v100.39"
+version = "0.1.3+v100.40"
 description = "Recursive Language Models."
 readme = "README.md"
 license = "MIT"

rlm/v100/campaign.py

@@ -145,6 +145,7 @@ def stage(name, function):
 
         stage("private-desktop", lambda: prepare_desktop(root))
     resources = profile.setdefault("resources", {})
+    resources["retention_experiments"] = True
     if benchmarks and current(root) and resources.get("public_baseline"):
         from rlm.v100.protection import file_hash
 

rlm/v100/capacity_growth.py

@@ -0,0 +1,96 @@
+"""Experimental frozen-column growth primitives for isolated PyTorch pilots.
+
+These change architecture and cannot be exported as an unchanged native Gemma.
+Old routes remain exact only when callers keep the old route and frozen state.
+"""
+
+
+def residual_column(predecessor, width: int, layers: int = 1, bottleneck: int = 16):
+    import copy
+
+    import torch
+    from torch import nn
+
+    if any(type(n) is not int for n in (width, layers, bottleneck)) or not (
+        1 <= layers <= 4 and 1 <= bottleneck <= width <= 4096
+    ):
+        raise ValueError("Growth column dimensions exceed the pilot budget")
+    if layers * width * bottleneck * 2 > 2_000_000:
+        raise ValueError("Growth column exceeds two million new parameters")
+    if sum(p.numel() for p in predecessor.parameters()) > 2_000_000:
+        raise ValueError("Frozen-column primitives are bounded tiny-model pilots")
+    predecessor = copy.deepcopy(predecessor)
+
+    class FrozenColumn(nn.Module):
+        def __init__(self):
+            super().__init__()
+            self.predecessor = predecessor
+            self.predecessor.requires_grad_(False)
+            self.predecessor.eval()
+            self.blocks = nn.ModuleList()
+            parameter = next(predecessor.parameters())
+            for _ in range(layers):
+                block = nn.Sequential(
+                    nn.Linear(width, bottleneck), nn.GELU(), nn.Linear(bottleneck, width)
+                )
+                block.to(device=parameter.device, dtype=parameter.dtype)
+                nn.init.zeros_(block[-1].weight)
+                nn.init.zeros_(block[-1].bias)
+                self.blocks.append(block)
+
+        def train(self, mode=True):
+            super().train(mode)
+            self.predecessor.eval()
+            return self
+
+        def forward(self, value, route="previous"):
+            if route not in ("previous", "expanded"):
+                raise ValueError("Unknown growth route")
+            with torch.no_grad():
+                value = self.predecessor(value)
+            if route == "expanded":
+                for block in self.blocks:
+                    value = value + block(value)
+            return value
+
+    return FrozenColumn()
+
+
+def embedding_extension(predecessor, new_rows: int):
+    """Freeze original IDs; append trainable rows without renumbering any token.
+
+    Requires a matching tokenizer AND output head in an architecture pilot.
+    This primitive alone does not add vocabulary to the serving language model.
+    """
+    import torch
+    from torch import nn
+
+    if type(new_rows) is not int or not 1 <= new_rows <= 4096:
+        raise ValueError("Embedding extension needs 1..4096 rows")
+    if (
+        not isinstance(predecessor, nn.Embedding)
+        or new_rows * predecessor.embedding_dim > 2_000_000
+        or predecessor.weight.numel() > 2_000_000
+    ):
+        raise ValueError("Embedding extension exceeds its parameter budget")
+
+    class ExpandedEmbedding(nn.Module):
+        def __init__(self):
+            super().__init__()
+            self.register_buffer("original", predecessor.weight.detach().clone())
+            self.extra = nn.Embedding(new_rows, predecessor.embedding_dim).to(
+                device=predecessor.weight.device, dtype=predecessor.weight.dtype
+            )
+            self.extra.weight.data.copy_(self.original.mean(dim=0).expand_as(self.extra.weight))
+
+        def forward(self, ids):
+            if (
+                ids.dtype not in (torch.int32, torch.int64)
+                or (ids < 0).any()
+                or (ids >= len(self.original) + new_rows).any()
+            ):
+                raise ValueError("Embedding token ID outside expanded vocabulary")
+            table = torch.cat((self.original, self.extra.weight), dim=0)
+            return torch.nn.functional.embedding(ids, table)
+
+    return ExpandedEmbedding()

rlm/v100/experiments.py

@@ -95,7 +95,7 @@ def parameter_schema(profile: dict) -> dict:
         lengths = [n for n in lengths if n <= budget["max_length"]]
         if not rank or not lengths:
             raise ValueError("Parent adapter exceeds measured training budget; recalibrate")
-    return {
+    schema = {
         "learning_rate": {
             "type": "number",
             "minimum": 0.000001,
@@ -125,6 +125,25 @@ def parameter_schema(profile: dict) -> dict:
             "enum": [0.01, 0.05, 0.1, 0.2, 0.5, 1.0],
         },
     }
+    if profile.get("resources", {}).get("retention_experiments"):
+        old_rank = rank[0] if profile["training"]["init_adapter"] else None
+        max_rank = budget["max_rank"] if budget else 64
+        schema.update(
+            retention_mode={
+                "type": "integer",
+                "description": "0=replay+KL only; 1=L2 anchor; 2=empirical diagonal EWC; 3=L2+experimental delta-A orthogonality; 4=EWC+delta-A orthogonality; 5=A-GEM replay gradient projection (first-order, AdamW is not a no-forgetting guarantee). Historical modes require prior verified TRAINING data. Compare time, new learning and retention, not training loss alone.",
+                "enum": [0, 1]
+                + ([3] if old_rank else [])
+                + ([2, 4, 5] if profile["training"].get("retention_has_history") else []),
+            },
+            retention_strength={"type": "number", "enum": [0.001, 0.01, 0.1, 1.0]},
+            retention_rank_growth={
+                "type": "integer",
+                "description": "1=keep rank; 2=double accepted LoRA rank with zero new B columns and preserved alpha/r. More memory/time; candidate only and all original quality gates remain mandatory.",
+                "enum": [1] + ([2] if old_rank and old_rank * 2 <= max_rank else []),
+            },
+        )
+    return schema
 
 
 def validate_parameters(values: dict, schema: dict) -> None:
@@ -383,6 +402,10 @@ def plan_duel(
                 branch_profile["training"].update(
                     init_adapter=parent["init_adapter"], teacher_adapter=parent["teacher_adapter"]
                 )
+            historical = [row for row in train if record_id(row) in mandatory]
+            branch_profile["training"]["retention_has_history"] = bool(
+                historical and branch_profile["training"]["init_adapter"]
+            )
             decision = choose_experiment(client, branch, catalog, branch_profile, shared.recent())
             selected = set(decision["selected_ids"]) | mandatory
             branch_path = output / branch
@@ -397,6 +420,9 @@ def plan_duel(
             chosen = copy.deepcopy(branch_profile)
             chosen["runtime"]["activity_branch"] = branch
             chosen["training"].update(decision["parameters"])
+            chosen["training"]["retention_reference_groups"] = sorted(
+                {row["group"] for row in historical}
+            )
             chosen["training"].update(
                 output=str(branch_path / "training"),
                 microbatch=1,

rlm/v100/foundation.py

@@ -143,6 +143,9 @@ def trial(
             base_model=str(base),
             init_adapter="",
             teacher_adapter="",
+            retention_mode=0,
+            retention_rank_growth=1,
+            retention_reference_groups=[],
             output=str(folder / "training"),
             max_steps=25,
             eval_steps=5,

rlm/v100/mission_evidence.py

@@ -81,7 +81,20 @@ def collect(root: Path) -> dict:
                     != counters["optimizer_updates"] + counters["amp_skipped_steps"]
                 ):
                     raise ValueError("Inconsistent optimizer counters")
-                result["training_runs"].append({"path": str(path), **counters})
+                evidence = {"path": str(path), **counters}
+                retention = path.parent / "retention.json"
+                if retention.exists():
+                    record = read_record(retention)
+                    evidence["retention"] = {
+                        "path": str(retention),
+                        "configuration": record.get("configuration"),
+                        "fisher_samples": record.get("fisher_samples"),
+                        "scope": "Configured reference; not proof of retention or promotion",
+                    }
+                growth = path.parent / "capacity-growth.json"
+                if growth.exists():
+                    evidence["capacity_growth"] = {"path": str(growth), **read_record(growth)}
+                result["training_runs"].append(evidence)
             except (ValueError, OSError) as error:
                 result["errors"].append({"path": str(path), "detail": str(error)[:300]})
         if result["training_runs"]: