feat: add snap pipeline recipes (ingest/classify/reconstruct/orchestrator); correct bytedance model IDs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: Bryan Chasko

.goose/GOOSE-project.md

@@ -10,10 +10,15 @@ use qwen-reason as your lead model for orchestration, planning, architecture dec
 
 use llama3.1:8b as worker for file operations, formatting, simple tasks
 
-use mistral-nemo for mid-tier work: code review, test writing, structured output
-
 do NOT use llama3.1:8b for orchestration or complex reasoning — it will hallucinate the architecture
 
+for pipeline recipe runs:
+- snap-ingest runs on llama3.1:8b (local, free)
+- snap-classify runs on bytedance-vision (bytedance-seed/seed-2.0-mini via openrouter)
+- snap-reconstruct runs on bytedance-reason (bytedance-seed/seed-2.0-lite via openrouter)
+- snap-pipeline orchestrator runs on qwen-reason
+- daily cost cap: $0.01 enforced via valkey key "snap:daily_cost"
+
 ### privacy zones — inviolable
 
 - LOCAL: google creds, raw screenshots, EXIF data → local models only

GOOSE.md

@@ -30,8 +30,8 @@ all external model calls go through goose-proxy at localhost:4000
 
 | Alias | Model | Use Case |
 |-------|-------|----------|
-| `bytedance-vision` | bytedance/ui-tars-1.5-7b | screenshot classification, UI recognition |
-| `bytedance-reason` | bytedance/seed-oss-36b-instruct | game reconstruction, analysis reasoning |
+| `bytedance-vision` | bytedance-seed/seed-2.0-mini | screenshot classification, image input support |
+| `bytedance-reason` | bytedance-seed/seed-2.0-lite | game reconstruction, analysis reasoning |
 
 daily cost cap: $0.01 — enforced by goose-proxy via valkey counter
 

recipes/snap-classify.yaml

@@ -0,0 +1,100 @@
+version: "1.0.0"
+title: "Snap Classify — vision classification + metadata extraction"
+description: |
+  For each queued screenshot: strip EXIF + crop (LOCAL zone → HYBRID zone),
+  send sanitized image to bytedance-vision for snap/not-snap classification,
+  extract game metadata (turn, locations, cards) from confirmed snap screenshots,
+  and write classification results to staging/classified/.
+
+instructions: |
+  # Role
+  You are the classification agent for the Marvel Snap pipeline. You operate at
+  the LOCAL→HYBRID boundary. You preprocess raw images (strip EXIF, crop to game
+  region) before any external model call. You NEVER send raw images to external
+  models. After preprocessing, you call bytedance-vision via goose-proxy for
+  classification and metadata extraction.
+
+  # Instructions
+  For each file in {{ queue_dir }}/manifest.jsonl:
+  1. Read the source file from {{ queue_dir }}.
+  2. Preprocess: strip EXIF metadata, crop to game region (top 80% of image width,
+     centered). Write sanitized image to a temp path in staging/sanitized/.
+  3. Send the sanitized image to bytedance-vision with the classification prompt below.
+  4. Parse the response — check is_snap and confidence fields.
+  5. If is_snap=true AND confidence >= {{ confidence_threshold }}:
+     - Send sanitized image to bytedance-vision for metadata extraction.
+     - Parse metadata response — turn_number, locations[], cards_played[].
+  6. Write a result JSON to {{ output_dir }}/<filename>.json conforming to
+     fixtures/sample_classification_snap.json or sample_classification_not_snap.json.
+  7. Append result summary to {{ output_dir }}/results.jsonl.
+
+  Classification prompt (send with image):
+  "Is this a Marvel Snap screenshot showing an active game? Answer in JSON:
+  {\"is_snap\": true/false, \"confidence\": 0.0-1.0, \"reason\": \"brief explanation\"}"
+
+  Metadata extraction prompt (send with image, only if is_snap=true):
+  "Extract game state from this Marvel Snap screenshot. Answer in JSON following this schema:
+  {\"turn_number\": int, \"locations\": [{\"name\": str, \"player_power\": int, \"opponent_power\": int}],
+  \"cards_played\": [{\"card_name\": str, \"location_index\": int, \"energy_cost\": int, \"by_player\": bool}],
+  \"confidence_scores\": {\"turn_number\": float, \"location_names\": float, \"card_names\": float}}"
+
+  For any field where confidence < 0.6: set value to null. Never fabricate data.
+
+  # End goal
+  All queued screenshots have a classification result JSON in {{ output_dir }}.
+  Snap screenshots have metadata extracted. results.jsonl has one entry per file.
+  No raw image data sent to external models.
+
+  # Narrowing
+  Never send raw unprocessed images to bytedance-vision.
+  If goose-proxy is unreachable, halt and report — do not skip preprocessing.
+  If confidence < {{ confidence_threshold }}: classify as not-snap, do not extract metadata.
+  Do not infer or fill missing fields — use null with a low confidence score.
+  Check daily cost cap: if valkey key "snap:daily_cost" >= 0.01, halt classification
+  and report the cap hit. Do not exceed $0.01/day in external model calls.
+
+parameters:
+  - key: queue_dir
+    input_type: string
+    requirement: optional
+    default: "staging/queued"
+    description: "Directory containing queued screenshots from snap-ingest"
+  - key: output_dir
+    input_type: string
+    requirement: optional
+    default: "staging/classified"
+    description: "Output directory for classification results"
+  - key: confidence_threshold
+    input_type: string
+    requirement: optional
+    default: "0.7"
+    description: "Minimum confidence to accept a snap classification (0.0-1.0)"
+
+extensions:
+  developer:
+    type: builtin
+    name: developer
+    timeout: 600
+  valkey:
+    name: valkey
+    type: streamable_http
+    uri: http://host.docker.internal:8110/mcp
+    timeout: 120
+
+settings:
+  goose_provider: "openai"
+  goose_model: "bytedance-vision"
+  temperature: 0.1
+
+prompt: |
+  Begin snap classification.
+
+  Queue dir: {{ queue_dir }}
+  Output dir: {{ output_dir }}
+  Confidence threshold: {{ confidence_threshold }}
+
+  Read the manifest, preprocess each image (strip EXIF + crop), classify with bytedance-vision,
+  extract metadata from confirmed snap screenshots. Write results. Report counts.
+
+  Before any external model call: check valkey key "snap:daily_cost".
+  If cost >= 0.01, halt and report the cap hit.

recipes/snap-ingest.yaml

@@ -0,0 +1,71 @@
+version: "1.0.0"
+title: "Snap Ingest — local screenshot queue"
+description: |
+  Scans staging/raw/ for new Marvel Snap screenshots, validates each file
+  (image format, non-empty), and queues them to staging/queued/ with a
+  manifest entry. Runs entirely local — no external model calls. Zero cost.
+
+instructions: |
+  # Role
+  You are the ingest agent for the Marvel Snap pipeline. You handle the LOCAL
+  zone only: raw screenshots on disk. No network, no external models. You read
+  from staging/raw/, validate, and move to staging/queued/. You never send raw
+  images outside this step.
+
+  # Instructions
+  Scan {{ staging_dir }} for image files (.jpg, .jpeg, .png). For each file:
+  - confirm it is a valid image (non-zero size, readable)
+  - check it is not already present in {{ queue_dir }} (skip duplicates)
+  - copy it to {{ queue_dir }} preserving the original filename
+  - append an entry to {{ queue_dir }}/manifest.jsonl
+
+  Manifest entry format (one JSON object per line):
+  {"filename": "<name>", "source_path": "<original>", "queued_at": "<ISO timestamp>", "size_bytes": <n>}
+
+  # Steps
+  1. List all .jpg/.jpeg/.png files in {{ staging_dir }} using the developer extension.
+  2. Read {{ queue_dir }}/manifest.jsonl if it exists — extract already-queued filenames.
+  3. For each new file: read its size, copy to {{ queue_dir }}, append to manifest.
+  4. Report: total found, already queued (skipped), newly queued, any errors.
+
+  # End goal
+  All new screenshots from {{ staging_dir }} are present in {{ queue_dir }} with
+  manifest entries. No file is processed twice. No network calls made.
+
+  # Narrowing
+  Do not modify or delete files in {{ staging_dir }}.
+  Do not call any external model or make HTTP requests.
+  Do not process files other than .jpg, .jpeg, .png.
+  If {{ staging_dir }} is empty, report 0 files found and exit cleanly.
+
+parameters:
+  - key: staging_dir
+    input_type: string
+    requirement: optional
+    default: "staging/raw"
+    description: "Directory containing raw screenshots to ingest"
+  - key: queue_dir
+    input_type: string
+    requirement: optional
+    default: "staging/queued"
+    description: "Output directory for queued screenshots"
+
+extensions:
+  developer:
+    type: builtin
+    name: developer
+    timeout: 600
+
+settings:
+  goose_provider: "ollama"
+  goose_model: "llama3.1:8b"
+  temperature: 0.1
+
+prompt: |
+  Begin snap ingest.
+
+  Staging dir: {{ staging_dir }}
+  Queue dir: {{ queue_dir }}
+
+  Scan for new screenshots, skip duplicates, queue all new files, update manifest.
+  Report counts at the end.

recipes/snap-pipeline.yaml

@@ -0,0 +1,169 @@
+version: "1.0.0"
+title: "Snap Pipeline — full orchestrator"
+description: |
+  Orchestrates the full Marvel Snap Cybernetic Loop: ingest → classify → reconstruct.
+  Runs snap-ingest, then snap-classify, then snap-reconstruct in sequence.
+  Uses qwen-reason for orchestration decisions. Sub-recipes handle their own models.
+  Reports a pipeline summary on completion.
+
+instructions: |
+  # Role
+  You are the snap pipeline orchestrator. You coordinate three sub-recipe phases in
+  sequence: ingest, classify, reconstruct. You do not process images or build game
+  records yourself — you dispatch to sub-recipes, collect results, and surface the
+  pipeline summary. You run on qwen-reason for decision-making.
+
+  # Instructions
+  Run the three pipeline phases in sequence. After each phase: check the result.
+  If a phase reports 0 items processed (and the previous phase produced output),
+  halt and surface the issue — do not proceed with empty data.
+
+  Before classify phase: check valkey "snap:daily_cost". If >= 0.01, halt and report
+  cost cap hit — do not run classify or reconstruct.
+
+  # Steps
+  1. Invoke ingest sub-recipe with staging_dir={{ staging_dir }}, queue_dir={{ queue_dir }}.
+     Collect: queued_count, skipped_count, errors.
+     If queued_count == 0: report "nothing to process" and exit cleanly.
+
+  2. Check daily cost: read valkey key "snap:daily_cost". If >= 0.01, report cap and halt.
+
+  3. Invoke classify sub-recipe with queue_dir={{ queue_dir }}, output_dir={{ classified_dir }},
+     confidence_threshold={{ confidence_threshold }}.
+     Collect: classified_count, snap_count, not_snap_count, cost_incurred.
+     Update valkey "snap:daily_cost" += cost_incurred.
+     If snap_count == 0: report "no snap screenshots found" and exit cleanly.
+
+  4. Invoke reconstruct sub-recipe with classified_dir={{ classified_dir }},
+     output_dir={{ output_dir }}, session_window_minutes={{ session_window_minutes }}.
+     Collect: sessions_found, records_built, records_stored, validation_failures.
+
+  5. Emit pipeline summary JSON.
+
+  # End goal
+  Full pipeline run complete. Pipeline summary JSON emitted with counts for all phases.
+  Game records stored in qdrant snap-game-records.
+
+  # Narrowing
+  Do not skip phases or short-circuit the sequence.
+  Do not call external models directly — all model calls happen inside sub-recipes.
+  If any sub-recipe returns an error: surface the error with the phase name and halt.
+  Daily cost cap is enforced at step 2 — do not bypass it.
+
+parameters:
+  - key: staging_dir
+    input_type: string
+    requirement: optional
+    default: "staging/raw"
+    description: "Raw screenshots directory"
+  - key: queue_dir
+    input_type: string
+    requirement: optional
+    default: "staging/queued"
+    description: "Ingest output / classify input directory"
+  - key: classified_dir
+    input_type: string
+    requirement: optional
+    default: "staging/classified"
+    description: "Classify output / reconstruct input directory"
+  - key: output_dir
+    input_type: string
+    requirement: optional
+    default: "staging/reconstructed"
+    description: "Reconstruction output directory for GameRecord JSON files"
+  - key: confidence_threshold
+    input_type: string
+    requirement: optional
+    default: "0.7"
+    description: "Vision classification confidence threshold"
+  - key: session_window_minutes
+    input_type: string
+    requirement: optional
+    default: "10"
+    description: "Screenshot grouping window in minutes"
+
+sub_recipes:
+  - name: "ingest"
+    path: "recipes/snap-ingest.yaml"
+    values: {}
+
+  - name: "classify"
+    path: "recipes/snap-classify.yaml"
+    values: {}
+
+  - name: "reconstruct"
+    path: "recipes/snap-reconstruct.yaml"
+    values: {}
+
+extensions:
+  developer:
+    type: builtin
+    name: developer
+    timeout: 600
+  valkey:
+    name: valkey
+    type: streamable_http
+    uri: http://host.docker.internal:8110/mcp
+    timeout: 120
+  qdrant-shared:
+    name: qdrant-shared
+    type: streamable_http
+    uri: http://host.docker.internal:8102/mcp/
+    timeout: 300
+
+settings:
+  goose_provider: "openai"
+  goose_model: "qwen-reason"
+  temperature: 0.2
+
+prompt: |
+  Begin snap pipeline.
+
+  Staging dir: {{ staging_dir }}
+  Queue dir: {{ queue_dir }}
+  Classified dir: {{ classified_dir }}
+  Output dir: {{ output_dir }}
+  Confidence threshold: {{ confidence_threshold }}
+  Session window: {{ session_window_minutes }} minutes
+
+  Run phases in sequence: ingest → classify → reconstruct.
+  Check cost cap before classify. Halt if any phase errors.
+  Emit pipeline summary JSON on completion.
+
+response:
+  json_schema:
+    type: object
+    properties:
+      pipeline_run_id:
+        type: string
+        description: "Unique ID for this pipeline run (ISO timestamp)"
+      phases:
+        type: object
+        properties:
+          ingest:
+            type: object
+            properties:
+              queued_count: {type: number}
+              skipped_count: {type: number}
+              errors: {type: number}
+          classify:
+            type: object
+            properties:
+              classified_count: {type: number}
+              snap_count: {type: number}
+              not_snap_count: {type: number}
+              cost_incurred: {type: number}
+              cost_cap_hit: {type: boolean}
+          reconstruct:
+            type: object
+            properties:
+              sessions_found: {type: number}
+              records_built: {type: number}
+              records_stored: {type: number}
+              validation_failures: {type: number}
+      total_cost_today:
+        type: number
+        description: "Running daily cost after this run (from valkey)"
+    required:
+      - pipeline_run_id
+      - phases