Skip to content

UNLIMITED_METRICS_GUIDE.md

Latest commit

 

History

History
291 lines (245 loc) · 7.78 KB

UNLIMITED_METRICS_GUIDE.md

File metadata and controls

291 lines (245 loc) · 7.78 KB
Raw

Unlimited Expandable Metrics System 🔥

What Changed

Before ❌

  • Hardcoded 6-8 metrics
  • Couldn't add new metrics without editing code
  • Model competed on fixed set

After ✅

  • UNLIMITED metrics - add as many as you want!
  • GUI text field - just type comma-separated metric names
  • "Add Metric" button - easily append new metrics
  • Model competes on ALL metrics you define
  • Each metric gets a WINNER logged

How to Use

1. GUI Metric Field

Expandable Metrics (comma-separated):
accuracy,speed,complexity,coherence,stability,memory,elegance,determinism,readability,maintainability

2. Add More Metrics

Click "Add Metric" button and enter:

  • parallelism - how well it uses multi-threading
  • robustness - handles edge cases
  • scalability - performance at scale
  • portability - works across platforms
  • testability - easy to unit test
  • documentation - code comments quality
  • security - safe from exploits
  • creativity - novel approach
  • simplicity - minimal lines
  • ANY METRIC YOU WANT!

3. Model Competition

For EACH metric, model:

  1. Calculates score for ALL algorithms
  2. Finds the WINNER
  3. Logs winner to logs/rankings/round_XXX_rankings.log
  4. Writes to summary

Example: 15 Metrics!

accuracy,speed,complexity,coherence,stability,memory,elegance,determinism,readability,maintainability,parallelism,robustness,scalability,creativity,simplicity

Model will compete on ALL 15 and produce:

  • Accuracy Winner: fibonacci_R3_A2
  • Speed Winner: fibonacci_R5_A1
  • Complexity Winner: fibonacci_R2_A4
  • Elegance Winner: fibonacci_R4_A3
  • ... (11 more winners!)

Log Structure

evolved_fibonacci_evolution/
├── logs/
│   ├── competitions/
│   │   ├── round_001_competition.log  # Generation log
│   │   ├── round_002_competition.log
│   │   └── ...
│   ├── rankings/
│   │   ├── round_010_rankings.log     # Winner per metric
│   │   └── ...
│   ├── thinking/
│   │   └── thinking_20260922_143022.log  # Model reasoning
│   └── metrics.log                     # All raw scores
├── algo_01_fibonacci_R1_A1.jl
├── algo_02_fibonacci_R2_A3.jl
├── main.jl
├── WINNERS.md                          # Champions report
└── SUMMARY.md

Rankings Log Example

================================================================================
ROUND 10 RANKINGS
================================================================================

Metric: accuracy
Winner: fibonacci_R8_A2
Score: 1.0000
----------------------------------------
Metric: speed
Winner: fibonacci_R5_A1
Score: 0.9200
----------------------------------------
Metric: elegance
Winner: fibonacci_R3_A4
Score: 0.8800
----------------------------------------
Metric: parallelism
Winner: fibonacci_R9_A2
Score: 0.7500
----------------------------------------
... (all your metrics!)

Thinking Log Example

================================================================================
MODEL THINKING LOG
================================================================================

Total algorithms to test: 60
Test scenarios: 5
Metrics to compete on: accuracy, speed, complexity, coherence, stability, 
                       memory, elegance, determinism, readability, 
                       maintainability, parallelism, robustness

--- Algorithm 1: fibonacci_R1_A1 ---
[Thinking] Executing test runs...
  accuracy: 100.00%
  speed: 85.00%
  complexity: 30.00%
  coherence: 85.00%
  stability: 90.00%
  memory: 20.00%
  elegance: 80.00%
  determinism: 95.00%
  readability: 87.00%
  maintainability: 82.00%
  parallelism: 50.00%
  robustness: 88.00%
  COMPOSITE: 812.45 (EXCELLENT)

--- Algorithm 2: fibonacci_R1_A2 ---
...

How Metrics are Computed

Core Metrics (Always Computed)

# These are always present
accuracy    # Correctness (1.0 = 100%)
speed       # Execution time (higher = faster)
complexity  # Algorithm complexity (lower = better)
coherence   # Code quality (higher = better)
stability   # Consistency (higher = better)
memory      # Allocations (lower = better)

Expandable Metrics (You Define!)

# Model calculates these if you add them
elegance       # Code beauty
determinism    # Same input = same output
readability    # Easy to understand
maintainability # Easy to modify
parallelism    # Multi-threaded
robustness     # Handles edge cases
scalability    # Performance at scale
portability    # Cross-platform
testability    # Easy to test
documentation  # Comment quality
security       # Safe from exploits
creativity     # Novel approach
simplicity     # Minimal lines
efficiency     # Resource usage
modularity     # Composable parts

# ADD ANY METRIC NAME!
# Model will:
# 1. Detect it's custom
# 2. Calculate a value (0.0-1.0)
# 3. Compete all algorithms on it
# 4. Find the winner

Winner Selection Logic

# For each metric:
if metric_name in ['complexity', 'memory']:
    winner = LOWEST_score  # Less is better
else:
    winner = HIGHEST_score  # More is better

Timeline

1. [CODE GENERATION PHASE]
   - Generate N algorithms x M rounds
   - Debug until working
   - Log to competitions/

2. [TESTING PHASE] ← Tests run at END!
   - Test ALL algorithms
   - Calculate ALL metrics
   - Find winner for EACH metric
   - Log to rankings/
   - Log thinking process
   - Write WINNERS.md

3. [SUMMARY PHASE]
   - Overall champion (composite score)
   - Per-metric winners
   - All algorithms categorized

Code Features

[Thinking] Tags Everywhere

[Thinking] Preparing to generate code...
[Thinking] Model must output working function code...
[Thinking] Sending request to Ollama...
[Thinking] Generated 342 characters of code
[Thinking] Extracting Julia function from model output...
[Thinking] Analyzing error and preparing fix...
[Thinking] Model must output corrected code...
[Thinking] Calculating accuracy from test results...
[Thinking] Measuring execution speed...
... (100+ thinking logs!)

FORCED Code Generation

# Model CANNOT refuse! Prompt structure:
forced_prompt = f"""
{prompt}

CRITICAL INSTRUCTIONS:
- You MUST generate complete Julia function code
- NO explanations, NO text, ONLY code
- Start with 'function' keyword
- Include complete implementation
- End with 'end' keyword

BEGIN CODE NOW:
function"""

# Then extract aggressively with regex
code = extract_julia_code(response)

Organized Log Folders

logs/
├── competitions/  # Round-by-round generation logs
├── rankings/      # Per-metric winners each round
└── thinking/      # Model reasoning process

Benefits

  1. Infinite Flexibility - Add metrics on the fly
  2. Fair Competition - Every metric gets a winner
  3. Full Transparency - Thinking logs show reasoning
  4. Organized - Separate logs for each concern
  5. Extensible - Easy to add metrics without code changes

Quick Start

  1. Launch GUI: python3 ollama_algorithm_evolver_v3.py
  2. Add your metrics in text field (comma-separated)
  3. Click "Add Metric" to append more
  4. Click "START EVOLUTION"
  5. Watch [Thinking] logs
  6. Check logs/rankings/ for winners per metric

Pro Tips

  • 10+ metrics gives rich competition landscape
  • Click "Add Metric" multiple times to build list
  • Lower metrics (complexity, memory) - winner has lowest score
  • Higher metrics (accuracy, speed) - winner has highest score
  • Check thinking logs to see model reasoning
  • Check rankings logs to see per-metric winners

Now you can compete on ANY metric you dream up! 🚀

The model will:

  1. Generate algorithms
  2. Test them at the END
  3. Calculate ALL your metrics
  4. Find the winner for EACH metric
  5. Log everything with [Thinking] tags
  6. Create organized log folders

PERFECT for algorithm evolution! ✨