Skip to content

TODO.md

Latest commit

 

History

History
225 lines (214 loc) · 13.7 KB

TODO.md

File metadata and controls

225 lines (214 loc) · 13.7 KB
Raw

Duroxide – TODOs

Active TODOs

  • Size limits

  • RaiseEvent pub/sub

  • Nested Select2 Support - Enable select2(select2(a, b), c) composition

    • Currently select2 only accepts DurableFuture, not SelectFuture
    • Options: (1) Add trait for select2 args that both types implement, (2) Add SelectFuture::into_selectable() wrapper
    • Would enable more expressive race patterns like select2(select2(fast_path, timeout), fallback)
    • Complexity: Medium - API change, need to handle nested loser tracking

  • Move shared types to dedicated types module

    • Types like InstanceFilter, PruneOptions, DeleteInstanceResult, DeleteInstancesResult, PruneResult are currently in providers/mod.rs
    • These are user-facing API types, not provider implementation details
    • Should move to src/types.rs or similar, re-export from crate root
    • Also consider: InstanceInfo, ExecutionInfo, QueueDepths, SystemMetrics
    • Benefit: Cleaner separation, users don't conceptually depend on "providers" module
    • Complexity: Low - just move definitions and update imports

  • Provider/runtime boundary leaks

    • Capture and address cases where providers must understand runtime semantics (WorkItem routing, identity extraction, metadata reconstruction, event indexing, cancellation coupling)
    • See: docs/proposals/provider-runtime-boundary-issues.md

  • Cancel unobserved sub-orchestrations (orphan cleanup)

    • Cancel sub-orchestrations when we have a confirmed unobserved DurableFuture
    • Same triggers as activity cancellations: select2 loser, unawaited future, parent continues/completes
    • Currently: orphaned sub-orchestrations keep running or sit completed in DB forever
    • Gap identified: When a sub-orchestration's DurableFuture is dropped (e.g., select2 loser), we only add its source_id to cancelled_source_ids but NOT to a cancelled_suborchestration_ids. Activities get added to cancelled_activity_ids which triggers provider-level lock stealing. See src/futures.rs lines 953-956 - explicit check for Kind::Activity.
    • Desired: runtime cancels them automatically once confirmed unobserved
    • Related: proposals/auto-pruning.md (section: Unobserved Sub-Orchestration Completions)
    • Complexity: Medium - need to track sub-orch DurableFutures same as activities, add provider API

  • History Validation

    • Need to validate histories are well-formed before replay
    • Checks: event_id ordering, source_event_id references valid scheduling events, no duplicate event_ids
    • Invalid histories should fail fast with clear error, not cause undefined behavior
    • Related to select2 loser fix: tests assume valid histories

  • Orchestration-level Metrics - Add metric recording from orchestration code

    • Similar to existing ctx.trace_*(), ctx.new_guid(), ctx.utcnow() system calls
    • Add ctx.record_counter(name, value, labels), ctx.record_histogram(name, value, labels) etc.
    • Would enable retry metrics, custom business metrics from orchestration code
    • Must be deterministic (recorded as SystemCall events for replay)
    • Complexity: Medium - extend SystemCall machinery, wire to MetricsProvider

  • Wire up suborchestration metrics, requires some linkage of names between suborch -> parent in the replayengine

  • move stress tests to cargo standard "bench"

  • separate execution loops for orchestrations and activities, communicate through channels

  • Port samples from DurableTasks and Temporal to tests/scenarios/

  • Limits everywhere, orch/activity names, input/output event sizes, history sizes etc.

  • LLM-orchestration/provider

  • Revive batching from dispatcher-batching branch, currently the perf dropped drastically. Might've been a sqlite-only issue.

  • Mgmt API feedback

    • list instances should get back version, status, and other basic info.
    • truncate API for instances
    • truncate API for executions, to be called in eternal orchestrations. Or maybe default it.

  • API endpoint for runtime, MCP server, orch status etc.

  • Metrics/logging dashboard

  • Refactor tests, figure out what are the unit tests vs not

  • some failure stress tests.

  • Infrastructure error handling from provider operations

    • Currently: Provider errors (append/enqueue failures) cause retries via lock expiration
    • Future: Implement visibility timeout to push failed instances to back of queue
    • Related: May need abandon with delay for persistent infrastructure failures
    • Priority: Medium (current retry mechanism works but could be more sophisticated)

  • Drop crates/dependencies that aren't needed

  • Add orchestrator functions

  • Macros for syntactic sugar [DESIGNED - see docs/proposals/MACRO-FINAL-DESIGN.md]

  • activity session management - cancellations, progress report, messaging.

  • reduce the junk in stress tests, make them useful

  • name the dispatchers

  • Update provider docs

  • provider layer should have retries built into it and not sprinkled across all the code

  • need to build an azure storage provider

  • should focus on simplification, refactoring and then threading for dispatchers before synatic sugars

  • pick proposal from branch macro3 - macro and macro2 branches have fuller impls but not clean

  • macros for activities, orchestrations

  • orchestration functions!

  • build node and python bindings as well

  • convert execution, instance statuses to enums

  • Introduce a provider error type with Retryable/NonRetryable classification; update runtime to use it for retries across all provider ops (not just ack_orchestration_item)

  • Proper lock / visibility timeouts

  • review duplicate orch instance ids

  • perf pass over replay engine

  • parallelize dispatcher loops

  • lock TTL for timer and worker queues and update lease

  • Reduce ornamental user code in orchestrations and acivities

  • Continue the provider simplification

  • Rename to provider

  • fault inject: "TODO : fault injection :"

  • code coverage

  • CLI tooling to manipulate history/queue state for devops

  • add cancellation and status from within the orchestration

  • example versioned crates with orchestrations and loaders

  • profiling replay and providers

  • performance improvements for runtime

  • "pub-sub" external events

  • sharding and scale out to multiple runtimes

  • strongly typed activity and orchestration calls?

  • cancellations via JoinHandles?

  • Add proper metrics.

  • Build a website to visualize execution.

  • Write an Azure Blob based provider.

  • Batch the calls to logging, don't spin up an activity per

  • Orchestration state? Monitoring? Visiblity?

  • Real world samples (provisioning resources in Azure e.g.)

DONE

  • Poison on deserialization failure — Currently, when fetch_orchestration_item hits a deserialization error (e.g. unknown EventKind from a newer duroxide version), the provider returns a permanent error and the runtime logs-and-continues indefinitely. The item's attempt_count increments each cycle but it never reaches the poison termination path because it never enters the dispatcher. Need to wire up poison termination for items that repeatedly fail deserialization. See sqlite.rs read_history_in_tx error path and orchestration.rs permanent error handler.
  • Rename "Turn" terminology to "ReplayEngine" - rename turn variable to replay_engine, TurnResult to ReplayResult, etc. in src/runtime/execution.rs and related files.
  • Provider validation test for orphan activities (issue #37)
    • Test scenario: instance deleted while activities in worker queue (without CancelInstance)
    • Blocked: Provider trait needs delete_instance or archive_instance method first
    • Related: proposals/activity-cancellation-queue-flag.md
  • implement test coverage plan
  • move away from Otel, just use standard rust metrics
  • add cancellation for unobserved futures (select losers, drops, unawaited out of scope)
  • Security: SQL Injection Audit - Review sqlite.rs for SQL injection vulnerabilities
    • Dynamic SQL with string formatting (e.g., format!("... IN ({})...")) needs parameterization
    • Affected areas: delete_instances_atomic, purge_instances, anywhere using dynamic placeholders
    • Priority: High - security issue
  • Review DurableFuture and into_*() fns for soundness
  • Longpoll support
  • Write big rocks docs.
  • Publish crate, docs
  • Github actions/pipeline
  • Docs review for duroxide, duroxide-pg.
  • [BLOCKER] Event Schema Redesign: Common Fields + Timestamps
    • Current Issue: duroxide_orchestration_duration_seconds only measures final turn duration (40ms), not total orchestration lifetime (could be hours/days)
    • Proposed Solution: Restructure Event from flat enum to struct + EventKind enum (see docs/proposals-impl/event-schema-redesign.md)
    • Common fields: event_id, instance_id, execution_id, timestamp_ms, duroxide_version (crate semver)
    • Event-specific fields: kind: EventKind enum
    • Impact: Large refactor (~40 event creation sites, ~140+ match sites), but no DB schema changes (JSON is flexible)
    • Benefits: Accurate duration metrics, self-contained events, cleaner API, easy to add common fields, forward compatible
    • Complexity: High - Touches many files, but serde provides backward compat via #[serde(default)]
    • Related: WorkItem should get same treatment (also persisted as JSON in queues)
    • Design Doc: docs/proposals-impl/event-schema-redesign.md, docs/event-schema-evolution.md
  • [FIXED] Select2 Loser Completions Block FIFO Ordering
  • Activity Retry Policy - Implement built-in activity retry logic
    • Design doc: docs/proposals-impl/activity-retry-policy.md
  • Fix up the stress tests, make them more usable by providers
  • Do a pass on registries for interface consistency
  • Disable infra logs by default
  • conventions and changes for exporting and importing orchestration and activity registries across crates
  • make the provider tests usable by outside implementors
  • Add new provider tests
  • prompt files??
  • multi threaded timer/worker dispatchers
  • separate out dispatchers so they can run independently
  • multi threaded orchestrator dispatcher
  • Provider cleanup: -- create_with_execution (ack_orchestration_item already creates new history) -- append_with_execution (unused)
  • sqlite instance table should not have a status, it should only be in the execution table
  • updated to sqlx 0.8
  • remove unused crate dependencies
  • Update all provider and API docs, get ready to push crate
  • implementing a provider requires the implementor to look deeply at the sqlite provider code
  • clean up leaky abstractions (provider should not have to compute the orchestration state and output e.g)
  • drop the polling frequence for the dispatchers
  • Cleanup the docs before making public
  • write a real world orchestrations with versioning etc
  • fix up the tracing to not use activities [DONE - tracing is now host-side only]
  • host level events (tracing, guid, time) [IN PROGRESS via RFC]
  • SQLite provider with full transactional support and e2e test parity
    • Full ACID transactional semantics
    • Provider-backed timer queue with delayed visibility
    • Handles concurrent instance execution
    • All 25 core e2e tests passing
  • Fixed trace activities to be fire-and-forget (no longer cause nondeterminism)
  • Timer acknowledgment only after firing (reliability fix)
  • Worker queue acknowledgment only after completion enqueue (reliability fix)
  • Return instance history as well
  • Update history + complete locks + enqueue in the same call
  • typed parameters for activities and orchestrations
  • proper timer implementation in the provider
  • versioning strategy
  • Support for orchestration chaining + eternal orchestrations
  • Need to understand this oneshot channel to await on
  • transactional processing - harden queue read semantics (see docs/reliability-queue-and-history.md)
  • review how active_instances in the runtime work, why are we reenqueuing events in certain cases
  • On next session: review tests covering ContinueAsNew and multi-execution IDs
    • Files: tests/e2e_continue_as_new.rs (both tests)
    • Also revisit runtime APIs to surface list_executions and get_execution_history consistently- ContinueAsNew support
  • test hung because the initial orchestration takes longer to start!
  • tests for orchestration state,
  • tests for provider <-> runtime resumption of multiple orchestrations.
  • dequeue multiple item batch
  • do a pass through the code.
  • Add orchestration registry
  • Add capability to the runtime to resume persisted orchestrations from the history provider
  • Add signalling mechanism in the provider which runtime can poll to trigger replay
  • resolve the test hang!!
  • Document all methods
  • Add proper logging
  • Write a file system based provider.
  • Crash recovery tests.
  • Error handling for bad activity or orchestration names, or accessing wrong instance IDs
  • Add a Gabbar test
  • Logging typed handlers
  • Max size of orchestration history
  • Detailed documentation in the docs folder for how the system works
  • Remove dead code, including the one with allow(dead_code)
  • Write detailed architecture and user documentation
  • Formalize a provider model for the state, queues and timers.
  • Write GUID and time deterministic helper methods.

POSTPONED

  • Full e2e Otel test
  • Fetch item batch in dispatchers
  • RaiseEvent should target any Wait for Event, even if it was to come in the future.
  • Add a Cancelled OrchestrationStatus
  • clients should be able to query registered activities and orchestrations
    • implemented in branch runtime-registry, but it needs much more than just the orch name/version
  • implement Unpin typed future wrappers for _typed adapters
  • redo the orchestration registry change with gpt5 and compare
  • mermaid diagrams for orchestrations???
  • remove the into_activity() and similar methods