Skip to content

COMPREHENSIVE_IMPROVEMENT_PLAN.md

Latest commit

 

History

History
705 lines (479 loc) · 29.2 KB

COMPREHENSIVE_IMPROVEMENT_PLAN.md

File metadata and controls

705 lines (479 loc) · 29.2 KB
Raw

Comprehensive Paper Improvement Plan

Analysis Date: 2026-02-09 Analyzed: Paper, existing plan, full rlm-scheme codebase (2,100 LOC, 373 tests), PoC validation (1,467 papers), PL theory literature

Executive Summary

Key Finding: The Paper Significantly Undersells What Exists

Dimension Paper Claims Reality Gap
Implementation "Proof-of-concept Python testbed (~300 LOC)" Production Racket sandbox + MCP server (2,100 LOC) 7x larger, production-ready
Tests 83 tests (59 deterministic + 24 comparative) 373 tests across 13 files 4.5x more coverage
Features 10 scaffold bindings 25+ bindings with vision, JSON, async, retry, rate limits 15 missing from paper
Validation Synthetic adversarial inputs 1,467-paper PoC, 277 LLM calls, zero scope violations Real-world production evidence
Scope "Scope hygiene" (title focus) Metaprogramming systems, staged computation, algebraic effects, graded monads Paper scope is too narrow

Core Argument: Broaden the Scope

You are absolutely right - the paper is about much more than hygiene. The current title "Scope Hygiene in Recursive Language Models" frames this as a narrow debugging story. The actual contribution is:

A Programming Language Approach to LLM Orchestration Systems

The paper should be reframed around:

  1. Metaprogramming systems - LLM orchestration is code generation (§2)
  2. Staged computation - Generate-execute-splice is multi-stage programming (§3.4)
  3. Algebraic effects - llm-query, finish, unsafe-* are effect operations with handlers
  4. Graded monads - Token budgets are graded effects tracking resource consumption
  5. Multi-language semantics - The Scheme/Python boundary needs formal guarantees
  6. Scope hygiene - ONE application of these principles (§3.3)

The implementation validates ALL of these, not just hygiene.

Should the Scope Be Broadened?

YES - Here's Why

  1. The implementation already went broader

    • Vision support (#:image, #:images) extends scope hygiene to multimodal data
    • Token budgets implement graded monads (Katsumata 2014)
    • Retry logic implements effect masking (Biernacki et al. 2019)
    • Async dispatch implements shallow handlers (Hillerström & Lindley 2018)
    • py-set! implements boundary terms (Matthews & Findler 2007)

  2. The PoC discovered orchestration patterns beyond scope

    • Optimal batch size (8 concurrent) - resource management
    • Rate-limit aware model selection - adaptive orchestration
    • Checkpoint-restart patterns - structured backtracking
    • Multi-model routing - cross-context reasoning

  3. The PL theory supports it

    • Algebraic effects unify all four primitives (Plotkin & Pretnar 2009)
    • Graded monads formalize token budgets (Orchard et al. 2019)
    • Modal types formalize cross-context breakage (Davies & Pfenning 2001)
    • Session types formalize callback protocol (Honda et al. 1998)

Recommended New Title

"Programming Language Foundations for LLM Orchestration" or "A Metaprogramming Systems Approach to Recursive Language Models"

With subtitle: "Algebraic Effects, Graded Monads, and Multi-Language Semantics"

This positions the paper as a foundational PL contribution to AI systems, not just a debugging story.

Which Sources Have the Most Impact?

Based on analysis of the paper's claims and the evidence available:

Tier 1: Critical - Must Be Updated

  1. The implementation itself (rlm-scheme codebase)

    • Impact: Validates EVERY theoretical claim with working code
    • Gap: Paper says "future work" but it's done (§6.1)
    • Action: Replace §6.1 with implementation report showing architecture

  2. The PoC validation (1,467 papers, 277 LLM calls, AMR drug repurposing)

    • Impact: Real-world evidence that scope hygiene prevents injection in production
    • Gap: Paper has only synthetic tests
    • Action: Add §5.4 case study showing 5x parallelization speedup, zero scope violations

  3. Test coverage (373 tests vs 83 claimed)

    • Impact: Shows production maturity
    • Gap: Paper evaluation (§5) has stale numbers
    • Action: Update all test counts, add test categories table

Tier 2: High Impact - Novel Contributions

  1. Plotkin & Pretnar (2009) - Algebraic effects and handlers

    • Impact: Unifies all four scope primitives under one framework
    • Currently: Not cited in main paper (only in companion note)
    • Action: Add §2.5 "Algebraic Effects as Unifying Framework"
    • Source: Handlers of Algebraic Effects, ESOP 2009

  2. Katsumata (2014) + Orchard et al. (2019) - Graded monads

  3. The py-set! discovery (safe data transfer)

    • Impact: Validates Matthews & Findler (2007) boundary terms
    • Currently: Mentioned in Appendix C only
    • Action: Add subsection showing orchestration/computation boundary has same scope problems

Tier 3: Important - Strengthens Arguments

  1. Hillerström & Lindley (2018) - Deep vs shallow effect handlers

    • Impact: Async dispatch is shallow handlers
    • Currently: Not cited
    • Action: Note in §4.5 that llm-query-async implements shallow handlers
    • Source: "Shallow Effect Handlers" (APLAS 2018)

  2. Biernacki et al. (2019) - Effect masking

    • Impact: unsafe-* operations are effect masking with evidence
    • Currently: Not cited
    • Action: Add to §3.4 explaining escape hatches as principled rule-breaks
    • Source: "Handle with Care: Relational Interpretation of Algebraic Effects and Handlers"

  3. Sabelfeld & Myers (2003) - Information flow types

    • Impact: Syntax objects are dynamic taint tracking
    • Currently: Prompt injection mentioned but not connected to security literature
    • Action: Reframe §3.3 using information flow terminology
    • Source: "Language-Based Information-Flow Security"

  4. Honda et al. (1998) - Session types

    • Impact: Racket-Python callback protocol is a session type
    • Currently: Protocol described informally in Appendix C
    • Action: Add formal session type for callback protocol
    • Source: "Language Primitives and Type Discipline for Structured Communication-Based Programming"

High-Priority Improvements

H1. Reframe the Paper's Scope (Critical)

Current problem: Title says "Scope Hygiene" but paper is about metaprogramming systems.

Action:

  1. New title: "Programming Language Foundations for LLM Orchestration"
  2. Revise abstract to lead with "LLM orchestration is metaprogramming" not "hygiene bugs"
  3. Reorganize §2 to introduce algebraic effects first, then show hygiene as ONE application
  4. Add Table 0 (before current Table 1) showing the full conceptual framework:
PL Concept RLM Manifestation Section Scheme Primitive
Metaprogramming Code generates prompts §2 Quasiquotation
Staged computation Generate-execute-splice §3.4 Multi-level eval
Algebraic effects Orchestration operations §2 NEW Effect handlers
Graded monads Token budgets §3.5 parameterize
Scope hygiene Prompt injection §3.3 Syntax objects
Multi-language semantics Scheme/Python boundary §4.2 Boundary terms

Effort: 1 day (rewrite abstract, intro, §2) Impact: Massive - positions paper as foundational PL contribution to AI

H2. Replace §6.1 "Future Work" with "Implementation" (Critical)

Current problem: Paper says 4 implementation steps are "future work" but they're all done.

Evidence:

  • "Racket subprocess server (~200 LOC)" → racket_server.rkt is 824 lines, production sandbox
  • "Bridge client (~100 LOC)" → mcp_server.py is 1,122 lines, full MCP server
  • "LM handler integration" → OpenAI API with retry, rate limits, multi-model routing
  • "Restricted grammar enforcement" → Sandbox blocks set!, call/cc, define-syntax

Action:

  1. Delete current §6.1 "Real Scheme Runtime Integration"
  2. Add new §6.1 "Implementation and Deployment"
    • Architecture diagram (Claude Code → MCP → Racket → Python)
    • Show code locations that implement each theoretical claim
    • Discuss what was learned building it
  3. Move remaining future work to new §6.5 "Open Directions"

Effort: 2 hours (rewrite one section) Impact: Critical - eliminates factual error

H3. Update Evaluation (§5) with Real Numbers (Critical)

Current problem: Paper claims 83 tests; reality is 373 tests (4.5x more).

Action:

  1. Update Table 3 with real test counts:

    • Total: 373 tests (not 83)
    • Core sandbox: 72 tests
    • Hygiene: 26 tests
    • MCP server: 65 tests
    • Callbacks: 14 tests
    • Recursion: 17 tests
    • API params: 17 tests
    • Rate limits: 12 tests
    • Progress: 19 tests
    • Retry: 13 tests
    • Async errors: 14 tests
    • Await patterns: 8 tests
    • Map-async: 9 tests
    • Timeout config: 11 tests
    • Tool surface: 18 tests

  2. Add sentence: "Since the paper submission, test coverage expanded from 83 to 373 tests as the proof-of-concept evolved into a production implementation."

Effort: 30 minutes (update one table) Impact: Factual accuracy

H4. Add §5.4 "Production Validation: AMR Drug Repurposing PoC" (High)

Current problem: Paper has only synthetic adversarial tests. Real-world evidence exists.

Evidence: PoC processed 1,467 papers, 277 LLM calls, zero scope violations.

Action: Add new subsection after §5.3:

§5.4 Production Validation

A multi-iteration proof-of-concept applied the Scheme coordination layer to automated research: identifying antimicrobial candidates from FDA-approved drugs by analyzing 1,467 papers. The PoC ran unsupervised for hours, making 277 LLM calls across 4 iterations.

Iteration Papers LLM Calls Strategy Result
1 97 78 Sequential Proof of concept works
2 162 63 2 concurrent Timeout at 4+ concurrent
3 1,230 70 Manual checkpoints Discovered timeout recovery gap
4 1,467 277 8 concurrent, 5 batches 40 minutes, zero scope violations

Key findings:

  1. Scope hygiene prevents injection - Despite sub-model responses containing arbitrary natural language (including instruction-like text from paper abstracts), zero prompt injection occurred. The syntax object wrapping worked as designed.
  2. Parallelization via algebraic effects - Async dispatch (llm-query-async + await) achieved 5x speedup (40 min vs 3.5 hours) through shallow effect handlers.
  3. Graded monads enable cost control - Token budgets (parameterize) prevented runaway API costs across 8 concurrent branches.
  4. Boundary terms prevent escaping - py-set! eliminated quote/backslash escaping bugs that plagued earlier f-string approaches.

Full code and results available at PoC/.

Effort: 1 hour (write subsection) Impact: High - real-world validation

H5. Expand §4.2 Feature Comparison Table (High)

Current problem: Table shows 4 features; implementation has 25+ bindings.

Missing features:

  • llm-query-async + await (parallel fan-out)
  • #:json #t (structured output)
  • #:image / #:images (vision)
  • #:temperature, #:max-tokens (generation control)
  • py-set! (safe data transfer)
  • tokens-used, token-budget (resource tracking)
  • Rate limit awareness
  • Retry with backoff
  • Call cancellation
  • Recursive delegation depth limits

Action: Expand the capabilities table in §4.2 to match README's feature list.

Effort: 1 hour (expand table) Impact: Completeness

H6. Add §2.5 "Algebraic Effects as Unifying Framework" (High)

Current problem: Four primitives introduced separately. Algebraic effects unify them.

Action: Add new subsection showing:

Claim: Every orchestration operation is an algebraic effect (Plotkin & Pretnar 2009):

  • llm-query is an effect: "I need a sub-model response"
  • The MCP server is the handler: interprets the effect by calling OpenAI API
  • finish is a control effect (delimited continuation)
  • unsafe-raw-query is the same effect with a different handler (skips scope wrapping)
  • Token budgets are resource effects with decrementing handlers

Effect signatures:

effect LLMQuery : Instruction × Data × Model → Scoped Response
effect PyExec   : Code → String
effect Finish   : Value → ⊥
effect Unsafe   : UnsafeOp → Value

The Scheme coordination layer is an effect system. This provides a cleaner taxonomy than treating each primitive separately.

References:

Effort: 2 hours (write subsection) Impact: High - unifies the four scope primitives under one framework

H7. Expand §3.5 "Token Budgets as Graded Monads" (High)

Current problem: Token budgets described as "resource threading" but formal structure is graded monads.

Action: Rewrite §3.5 to show:

Claim: Token budgets implement graded monads (Katsumata 2014; Orchard et al. 2019):

In a graded monad $M_r, a$, the grade $r$ tracks a resource:

  • return : a → M_0 a — pure computation uses zero tokens
  • bind : M_r a → (a → M_s b) → M_{r+s} b — sequencing adds costs
  • llm-query has type M_k a where $k$ is the token cost of the API call
  • parameterize ([token-budget N]) imposes the constraint $r \leq N$

Implementation evidence:

  • racket_server.rkt:91 defines token-budget as a Racket parameter
  • decrement-budget! (line 93-101) checks budget, raises error if exhausted
  • Token counts come from OpenAI API (not estimates)
  • Nested parameterize creates independent budgets (tested in test_racket_server.py)

Novel contribution: This is the first graded monad for LLM orchestration. Static analysis could infer worst-case token bounds by summing grades.

References:

Effort: 1 hour (rewrite section) Impact: High - formalizes resource control

H8. Add Vision-Aware Scope Hygiene Discussion (Medium)

Current problem: Paper focuses on text scope; implementation extends to images.

Action: Add subsection to §3.3 or §4.5:

Vision creates a new scope boundary type. When a sub-model receives both text and images, provenance tracking covers two modalities:

  • Text data has scope marks from syntax objects
  • Image data has no equivalent scope annotation

This creates asymmetry: a sub-model's response might be influenced by an injected image (visual prompt injection) that the scope log doesn't track at the same granularity.

PL analogy: Multi-sorted algebra. Kohlbecker (1986) handles one sort (identifiers); extending to images requires a second sort with different scope rules.

Implementation: _resolve_image in mcp_server.py validates images (magic bytes, 10 MB limit). This metadata could feed into the scope log to track which images were sent to which sub-call, at what resolution, from what source.

Effort: 1 hour (add subsection) Impact: Medium - shows scope hygiene extends to multimodal

Medium-Priority Improvements

M1. Deepen DSPy Comparison in §7.3 (Medium)

Current problem: DSPy comparison is one paragraph. Much richer analysis exists.

Evidence: PLAN.md has 10-row concept table showing DSPy vs RLM-Scheme.

Action: Expand §7.3 with:

  1. DSPy's caller_modules stack is "the closest thing to scope tracking in any mainstream LLM framework" but tracks which module is active, not where each datum came from
  2. DSPy's ContextVar-based dspy.context() provides dynamic scoping for settings but not for data provenance
  3. The two systems compose: DSPy optimizes prompt content, syntax objects track prompt scope

Effort: 1 hour (expand section) Impact: Medium - clarifies relationship to closest prior work

M2. Add py-set! as Boundary Term Example (Medium)

Current problem: Multi-language semantics mentioned abstractly but py-set! is concrete.

Action: Add to §4.2 or §7:

Discovery: The usage report identifies py-set! as solving "the most frustrating ergonomic issue" - escaping conflicts when passing data through Scheme strings to Python code.

This validates Matthews & Findler (2007) boundary terms:

  • py-exec embeds Python code as a string in Scheme - escaping conflicts are the same problem as unhygienic interpolation
  • py-set! sends data as JSON over a pipe - a proper boundary term that marshals values without embedding one language inside another

The orchestration/computation boundary suffers from the same scope problems as the orchestration/sub-LLM boundary.

Effort: 30 minutes (add paragraph) Impact: Medium - concrete example of multi-language semantics

M3. Add Call Registry as Expansion Trace (Medium)

Current problem: Table 1 claims syntax objects "also enable" expansion traces but doesn't show evidence.

Action: Add subsection to §4.5:

The _active_calls registry in mcp_server.py tracks every in-flight sub-model call with: call_id, type (sync/async/recursive), model, depth, instruction_preview, parent_id, start_time, future, nested_repl, cancel_event.

This is an expansion trace - the term used in §2 and Table 1's "Also Enables" column. The audit trail let the PoC user "verify that every paper was actually processed" - 277 LLM calls across 1,467 papers, all logged.

Effort: 30 minutes (add subsection) Impact: Medium - shows predictions materializing

M4. Add Blame Theory for Cross-Language Errors (Medium)

Current problem: When a scope error occurs at Scheme/Python boundary, source is unclear.

Action: Apply blame theory (Wadler & Findler 2009) to the two-REPL boundary:

When py-exec returns a string that contains instruction-like text and this string is passed to llm-query #:data, the scope system can attach blame to the Python side (the data source) rather than the Scheme side (the orchestration). The _active_calls registry already tracks call provenance; blame labels would formalize this.

Effort: 1 hour (add subsection to §7 or §6) Impact: Medium - connects to contracts literature

M5. Add Gradual Typing Adoption Story (Medium)

Current problem: Transition from Python to Scheme is all-or-nothing.

Action: Present gradual typing (Siek & Taha 2006) as incremental path:

  • Level 0: Raw string interpolation (current Python RLM)
  • Level 1: Typed fields with runtime checks (SubCallResult wrapper)
  • Level 2: Syntax objects with scope tracking (current Scheme layer)
  • Level 3: Static type checking with phantom types (ROADMAP L2)

The unsafe-* escape hatches are the "dynamic" portion - they bypass static guarantees with runtime logging.

Effort: 1 hour (add subsection to §7) Impact: Medium - makes adoption story compelling

M6. Add Session Types for Callback Protocol (Low-Medium)

Current problem: Callback protocol described informally in Appendix C.

Action: Apply session types (Honda et al. 1998):

CallbackSession = !Request . ?Response . End
AsyncSession    = !Request . End
AwaitSession    = !AwaitReq . ?Response . End

The Racket server and MCP server are provably deadlock-free if they respect the session type.

Effort: 2 hours (formal protocol specification) Impact: Low-Medium - nice formalization but not critical

Low-Priority Improvements (Structural)

L1. Update Appendix B Scaffold Reference (Low)

Problem: Lists 10 bindings; implementation has 25+.

Action: Expand to include llm-query-async, await, py-set!, tokens-used, token-budget, #:image, #:images, unsafe-* operations.

Effort: 30 minutes Impact: Completeness

L2. Add "From Testbed to Implementation" Narrative (Low)

Problem: Paper arc is "identify bugs → propose fixes → validate with testbed". Missing second act.

Action: Add §5.4 or §6.1 subsection:

What the testbed predicted correctly:

  • Scope tracking prevents capture ✓
  • Delimited continuations prevent delimiter capture ✓
  • Module systems prevent namespace collision ✓

What it missed:

  • Inter-language escaping (py-set! discovery)
  • Process lifecycle (crash recovery)
  • Vision scope (images lack syntax object equivalents)

What the real implementation revealed:

  • Call registry as expansion trace
  • Graded monads for token budgets (not just "resource threading")
  • Algebraic effects unify all four primitives

Effort: 1 hour Impact: Low - nice narrative but not critical

L3. Add RLM Environment Integration Story (Low)

Problem: ROADMAP L1 describes integration path but paper doesn't mention it.

Action: Add paragraph to §6:

Integration with upstream RLM benchmarks: ROADMAP item L1 describes integrating RLM-Scheme as a drop-in SchemeREPL(Environment) for the upstream RLM library. This would enable direct A/B comparison on the original RLM benchmarks (arithmetic, reading comprehension, code generation) - the strongest possible validation.

Effort: 30 minutes Impact: Low - future work, not critical

Companion Note: "A Monadic Framing..." Improvements

CN1. Ground Abstract Claims in Implementation (High)

Problem: Companion note is purely theoretical.

Action: Add §4b "Implementation Correspondence":

Note claims... Implementation evidence...
return wraps datum in syntax object (datum->syntax #f result-text) at racket_server.rkt:480
bind is dispatch-and-wrap llm-query-callback → API → datum->syntax lines 469-480
finish is shift with empty continuation (shift k (finished-value val)) line 416
Filinski: any monad via shift/reset (reset expr) at line 719

Effort: 1 hour Impact: High - validates monadic claims

CN2. Verify Monad Laws (Medium)

Problem: Claims monadic structure but never checks monad laws.

Action: Add subsection checking:

  • Left identity: return a >>= k = k a
  • Right identity: m >>= return = m
  • Associativity: (m >>= k) >>= h = m >>= (λx. k x >>= h)

Discuss whether syntax-e stripping marks breaks left identity and whether the violation is benign.

Effort: 2 hours Impact: Medium - formal rigor

CN3. Expand from 5 Sections to 8 (Medium)

Problem: Note is compressed (97 lines). Implementation evidence enables expansion.

Action:

  1. The Formal Structure (existing §1, expanded)
  2. Modal Types and Cross-Context Breakage (existing §2)
  3. Scope Failures as Type Errors (existing §3, add test references)
  4. Why the Scheme Implementation Works (existing §4, add code locations)
  5. NEW: The Monad Transformer Stack (continuation + state + reader + writer)
  6. NEW: The Token Budget as a Graded Monad (Katsumata 2014)
  7. NEW: Async Dispatch and Effect Handlers (Hillerström & Lindley 2018)
  8. Implications for Future Work (existing §5, add verified predictions)

Effort: 4 hours Impact: Medium - thorough formalization

CN4. Add Graded Monad Formalization (High)

Problem: Token budgets not formalized as graded monads.

Action: See H7 above - same formalization for companion note.

Effort: 1 hour Impact: High - novel contribution

References to Add

Essential (Tier 1)

  1. Plotkin & Pretnar (2009). "Handlers of Algebraic Effects." ESOP. Link
  2. Katsumata (2014). "Parametric effect monads and semantics of effect systems."
  3. Orchard, Wadler, Eades (2019). "Unifying graded and parameterised monads."

Important (Tier 2)

  1. Hillerström & Lindley (2018). "Shallow Effect Handlers." APLAS.
  2. Biernacki et al. (2019). "Handle with Care: Relational Interpretation of Algebraic Effects and Handlers."
  3. Sabelfeld & Myers (2003). "Language-Based Information-Flow Security."
  4. Honda, Vasconcelos, Kubo (1998). "Language Primitives and Type Discipline for Structured Communication-Based Programming."

Nice-to-have (Tier 3)

  1. Wadler & Findler (2009). "Well-Typed Programs Can't Be Blamed."
  2. Siek & Taha (2006). "Gradual Typing for Functional Languages."
  3. Rutten (2000). "Universal Coalgebra: A Theory of Systems."

Priority Ranking Summary

Must Do (H1-H5) - 1-2 Days Total

# Item Effort Why Critical
H1 Reframe scope (title, abstract, §2) 1 day Paper is about metaprogramming, not just hygiene
H2 Replace §6.1 future work 2 hours Factual error - implementation is done
H3 Update evaluation numbers 30 min 373 tests not 83
H4 Add PoC case study (§5.4) 1 hour Real-world validation
H5 Expand feature table (§4.2) 1 hour 15 features missing

Should Do (H6-H8 + M1-M3) - 1 Day Total

# Item Effort Why Important
H6 Add algebraic effects (§2.5) 2 hours Unifies framework
H7 Graded monads (§3.5) 1 hour Formalizes token budgets
H8 Vision scope hygiene 1 hour Extends to multimodal
M1 Deepen DSPy comparison 1 hour Clarifies prior work
M2 Add py-set! boundary term 30 min Concrete example
M3 Call registry as expansion trace 30 min Shows predictions work

Nice to Have (M4-M6 + L1-L3) - 1 Day Total

Everything else - structural improvements, additional formalizations, narrative enhancements.

Recommendations for Next Steps

Immediate Actions (Today)

  1. Read this plan and decide: reframe scope? (Recommendation: YES)
  2. Start with H1 - rewrite abstract and §2 to lead with metaprogramming, not hygiene
  3. Do H2-H3 - fix factual errors (§6.1, test counts)

This Week

  1. Do H4-H5 - add PoC validation and expand feature table
  2. Do H6-H7 - add algebraic effects and graded monads sections
  3. Review all evidence - every code location that validates a claim

Before Submission

  1. Do M1-M3 - deepen DSPy comparison, add py-set! and call registry examples
  2. Add all Tier 1-2 references - Plotkin/Pretnar, Katsumata/Orchard, etc.
  3. Update companion note - ground claims in code locations (CN1, CN4)

Answers to Your Specific Questions

Q1: Should I broaden the scope?

YES. The paper is about programming language foundations for LLM orchestration, with scope hygiene as ONE application. The implementation validates:

  • Metaprogramming systems
  • Staged computation
  • Algebraic effects
  • Graded monads
  • Multi-language semantics

Rename to "Programming Language Foundations for LLM Orchestration" or similar.

Q2: Which sources have the most impact?

  1. The implementation itself - validates every claim
  2. The PoC validation - 1,467 papers, zero scope violations
  3. Plotkin & Pretnar (2009) - algebraic effects unify everything
  4. Katsumata/Orchard - graded monads formalize token budgets
  5. The test suite - 373 tests show production maturity

Q3: Are there other important papers to reference?

YES. Add:

  • Plotkin & Pretnar (2009) - algebraic effects [essential]
  • Orchard et al. (2019) - graded monads [essential]
  • Hillerström & Lindley (2018) - shallow handlers [important]
  • Biernacki et al. (2019) - effect masking [important]
  • Sabelfeld & Myers (2003) - information flow [important]
  • Honda et al. (1998) - session types [nice-to-have]

Q4: Is the paper about more than hygiene?

ABSOLUTELY YES. You're correct. The paper is about:

  • Metaprogramming systems for LLM orchestration (§2)
  • Staged computation and modal types (§3.4)
  • Algebraic effects and handlers (should be §2.5)
  • Graded monads for resource control (§3.5)
  • Multi-language semantics for Scheme/Python boundary (§4.2)
  • Scope hygiene as one application of these principles (§3.3)

The title and framing are too narrow. Broaden to "Programming Language Foundations for LLM Orchestration."

Final Recommendation

Reframe the paper around PL foundations for AI systems, not just debugging. The implementation is production-ready with 373 tests and real-world validation. Update the paper to match what actually exists, add the formal connections (algebraic effects, graded monads), and position this as a foundational PL contribution to AI orchestration systems.

Timeline: 3-4 days of focused work to do H1-H7 + M1-M3.

Impact: Transforms paper from "we fixed some bugs" to "we established PL foundations for a new domain."