Skip to content

PROJECT_STATUS.md

Latest commit

 

History

History
59 lines (44 loc) · 4.24 KB

PROJECT_STATUS.md

File metadata and controls

59 lines (44 loc) · 4.24 KB
Raw

Project Status: Complete Evolution of TinyThinker (V10 to V12 Delta-Phase)

Last Updated: August 12, 2026

Genesis and Lineage: The Path to Sovereignty

This project is the culmination of a fractal evolution of sovereign algorithms:

  1. SOMA (The Macro Ancestor): External orchestrator that gave volition to the model over its context (pin/unpin/edit). Proof that the LLM must manage its own attention.
  2. COGA (El Salto al Sistema): Internalization of SOMA. Transition to a Mutable Scratchpad and Dynamic Recurrence within the inference engine.
  3. Attention Neuron (The Atomic Level): Discovery of Spectral Cores (DCT/Walsh), Complex Phase Phasors, and $O(N)$ Matrix Delta Holographic Memory (V298/V299).
  4. TinyThinker Delta-Phase TCA (The Final Synthesis): A Matrix-Free LLM with $O(N)$ complex phase memory and causal $k=4$ depthwise convolutions for infinite context and high-density associative recall on local hardware.

Milestone: V11 Fourier-ALBERT Architecture Completed & Verified

  • Champion Config (v11_e256_d1024_k512_l8 Run 2): Best Val Loss = 4.1287 (9.44M params, constant LR=0.015).
  • Key Discoveries:
    • Virtual Block Sharing (ALBERT-style): 8 virtual layers sharing weights acts as a strong regularizer that stabilizes training at high learning rates (0.015).
    • Logical Walsh Rank ($k=512$): High-rank spectral cores encode complex syntactic/logical structures without expanding parameter count.

Milestone: V12 Delta-Phase Architecture (72.41M Params) Active Pre-training & Validation

Last Updated: August 13, 2026

1. Architectural Integrations

  • Chunkwise Parallel Rank-One Delta Memory ($T_{\text{mat}}$ Triangular Solve): Replaced slow sequential token loops with GPU-parallel batch matrix multiplications. Achieved 9.5x speedup on GPU hardware (from 185.7s down to 20.8s per iteration on Tesla T4).
  • Learnable Substrate Lerp FFN Router (v328): Multi-bank orthogonal transform router (FWHT + DCT-II + DWT Haar) saving 49.4% FFN parameter weights and 38.0% total model parameters.
  • Rigorous FP64 Gradcheck Audit: Verified with PyTorch autograd.gradcheck in FP64 (scratch/test_fp64_gradcheck.py) passing natively with $7.39 \times 10^{-16}$ global L2 relative gradient error.

2. Preentrenamiento Completado (2,000 Iteraciones - 65.5M Tokens)

  • Hardware: CPU 8 threads (69.4 horas de cómputo ininterrumpido).
  • Config: configs/train_v12_colab_t4.yaml (72.41M params, dim=1024, n_layers=8, n_heads=8, vocab_size=16384, batch_size=4, grad_accum_steps=8, 32,768 tokens/iter, learning_rate=0.0004, weight_decay=0.0).
  • Métricas de Convergencia Finales:
    • Iter 0: train_loss 9.7336, val_loss 9.7402 ($\text{Entropía inicial } \ln(16384)$, $PPL = 16,986$)
    • Iter 500: train_loss 5.1204, val_loss 4.1950 ($PPL = 66.35$)
    • Iter 1000: train_loss 4.7088, val_loss 3.7314 ($PPL = 41.73$)
    • Iter 1500: train_loss 4.4220, val_loss 3.5068 ($PPL = 33.34$)
    • Iter 2000 (FINAL): train_loss 4.4875, val_loss 3.4251 ($PPL = 30.72$, guardado en checkpoints/v12_delta_phase/ckpt_pretrain_best.pt)

3. Distribución del Router Espectral de Substratos (V12 Lerp Router)

El router multibank (FWHT + DCT-II + DWT Haar) convergió a un equilibrio tripartito estable a través de las 8 capas residuales:

  • FWHT (Walsh-Hadamard binario): ~32.7%
  • DCT-II (Cosenos discretos): ~32.7%
  • DWT Haar (Ondículas multinivel): ~34.6% (Ahorro del 49.4% de pesos en los FFNs con $100%$ de gradientes estables).

Note

Siguiente Paso Operativo: El checkpoint óptimo checkpoints/v12_delta_phase/ckpt_pretrain_best.pt está listo para realizar muestreo y generación de texto (text sampling) con scripts/test_generation.py o scripts/chat.py para auditar la coherencia sintáctica del modelo.

Standalone Open-Source Repository Release

  • Standalone Package: Released independent open-source repository delta-phase at C:\Users\mrcm_\Local\proj\algorithms\delta-phase (mcarbonell/delta-phase on GitHub).
  • Formal Proposal: Documented formal research proposal in proposal_delta_phase.md.

Document updated by TinyThinker Architect. Efficiency, phase memory, and spectral routers are the path to Sovereign AI.