| description | Defines apcore's boundary with durable execution: what hooks it guarantees for retry/replay/workflow layers, what it deliberately omits as single-call-scoped, and how downstream runtimes integrate. |
|---|
This document defines apcore's boundary with respect to durable execution — the family of concerns that includes retry across crashes, replay after restart, deduplication of logically-equivalent invocations, long-running pause/resume, and persistent task queues. These concerns are routinely required by AI Agent runtimes and workflow systems built on top of module standards like apcore, but they are not part of apcore itself.
The purpose of this document is to make explicit:
- What apcore guarantees — the stable hooks and contracts that any retry/replay/workflow layer above apcore MAY rely on.
- What apcore deliberately does NOT do — to prevent scope creep and to give downstream integrators a clear picture of which concerns they must own.
- How a downstream layer should integrate — concrete, vendor-neutral patterns using existing apcore primitives.
- Gap watchlist — known areas where apcore has deliberately deferred a decision until real-world usage provides evidence.
apcore is the module standard. Durable execution is a runtime concern. Both can coexist; this document documents the seam.
apcore defines how AI-perceivable modules are described, validated, governed, observed, and invoked through an 11-step execution pipeline. It is intentionally:
- Single-call scoped. The pipeline operates on one module invocation at a time. apcore has no concept of a "workflow" that spans multiple module calls.
- Stateless across invocations. No
PipelineContextstate is preserved between calls. Each call starts a fresh pipeline. - Standard, not runtime. apcore ships interfaces and reference behaviors. Production-grade backends (persistent task stores, distributed approval handlers, replay engines) are the responsibility of applications, framework adapters, or community packages built on top.
Workflow orchestration — sagas, fan-out/fan-in, multi-step compensation, cross-call state machines, durable pause/resume of arbitrary execution — is outside apcore's scope. See POSITIONING.md.
External retry/replay/workflow layers can and do build on apcore. This document enumerates the contracts they may rely on as stable.
The following primitives are part of apcore's normative surface. Behavior described here is guaranteed within the current major version per the compatibility matrix in protocol-spec.md §13.5.
Context is JSON-serializable across process and network boundaries. The exact serialization rules are normative in protocol-spec.md §5.7. Key guarantees:
trace_id,caller_id,call_chain,identityround-trip losslessly.dataround-trips for all serializable values; non-serializable values (functions, sockets, connections) are silently skipped with a logged warning.executor,cancel_token,servicesare runtime injects and MUST NOT be serialized.
This makes it possible to ship a Context across a queue, a job board, or a persistent task store and rehydrate it later — the foundation of any cross-process retry/replay layer.
The Approval System (protocol-spec.md §7) is currently the only suspension point built into apcore's pipeline. Its Phase B contract is:
- A module annotated
requires_approval: truecalls into the configuredApprovalHandler. - If the handler returns
pending, the executor raisesApprovalPendingErrorcarrying anapproval_id. - The caller persists
approval_idsomewhere (apcore does not), waits for an out-of-band approval signal, and re-invokes the same module with_approval_token = approval_idinjected into the arguments. - On resume, the executor re-enters the pipeline from step 1 with
_approval_tokenpresent; the approval gate callshandler.check_approval(token)instead ofrequest_approval().
This is a complete pause/resume cycle that already crosses arbitrary time gaps and process restarts, provided the application persists the approval_id and original inputs. Downstream layers wanting human-in-the-loop or external-decision-gated execution SHOULD model their suspension on this contract rather than introducing their own.
The "Resume semantics" clarification in protocol-spec.md §7 is normative: the executor re-runs from step 1 — pre-approval middleware side effects (logging, tracing) re-execute. Middleware that needs at-most-once semantics across an approval gate SHOULD inspect _approval_token itself.
AsyncTaskManager provides single-call durable task execution: submit a module call, get a task_id, look up status later, persist TaskInfo records across restarts. Its persistence layer is the TaskStore interface (five methods: save, get, list, delete, list_expired).
apcore MUST ship an InMemoryTaskStore reference implementation in every SDK. Production-grade backends (Redis, SQL, file-system, custom queues, etc.) are not part of apcore — they are the application's or community's responsibility. The interface is the contract; the storage choice is downstream.
This means a workflow runtime that wants persistent task tracking can implement a single 5-method interface and plug it in via AsyncTaskManager(executor, store=MyTaskStore()) without touching apcore internals.
Via ExtensionManager (protocol-spec.md §11), apcore exposes six pluggable slots:
| Extension point | Multi | Typical use by retry/replay layers |
|---|---|---|
discoverer |
single | Custom module discovery (rarely needed for durability) |
middleware |
multi | Primary integration point. Wrap calls with retry, dedup, span recording, replay short-circuit |
acl |
single | Custom access control |
span_exporter |
multi | Persist trace data for replay reconstruction |
module_validator |
single | Custom module validation |
approval_handler |
single | Primary suspension point. Persist approval state externally, gate on durable decisions |
A retry/replay layer typically integrates as a custom middleware plus a custom approval_handler, plus a custom TaskStore when persistent task state is needed. No spec change is required for any of this.
The following annotations are normative inputs to retry/replay decisions (see schema-system.md):
idempotent: bool— indicates repeated calls with identical arguments are safe.readonly: bool— module performs no observable mutation; trivially retry-safe.destructive: bool— module performs irreversible mutation; warrants caution before retry.requires_approval: bool— module requires explicit approval; engages the Phase B contract above.
Annotations are declarative metadata. apcore the SDK does not enforce retry decisions based on them — that's a runtime decision. But because the values are part of the module's published contract, downstream layers can rely on them being stable.
context.data is the canonical carrier for application-level and runtime-private state that needs to thread through a call without becoming part of apcore's normative schema. Per protocol-spec.md §4.6, keys prefixed with x- are reserved for extension metadata.
Common patterns supported today:
context.data["x-correlation-id"]— caller-supplied correlation identifier (formalized inprotocol-spec.md§5.7).- Any other
x-*key — reserved namespace for downstream layers.
Strings, numbers, booleans, lists, and JSON-serializable objects placed in context.data round-trip through Context.serialize / Context.deserialize reliably. Non-serializable values are silently skipped with a logged warning. This makes context.data the appropriate place to carry idempotency keys, replay flags, attempt counters, deadlines, runtime correlation IDs, and any other non-normative per-call state.
A downstream layer needing a durable identity for a logical invocation (for retry deduplication, replay short-circuit, etc.) MAY define and consume its own x-* key. apcore reserves no specific namespace for any particular runtime, and offers no normative dedup behavior of its own.
apcore deliberately does not ship the following, and is unlikely to in the foreseeable future. These are runtime concerns, not module-standard concerns.
The 11-step executor pipeline is stateless across invocations. There is no CheckpointStore, no per-step persistence, no resume from step 6. The only suspension point built into the pipeline is the approval gate (§2.2 above), and resumption re-runs the entire pipeline from step 1.
Mid-pipeline checkpointing — preserving partial state mid-execution and resuming after a crash — would invade the pipeline's stateless invariant and require every SDK to carry a generic state-machine engine. Downstream layers that need this MUST implement it externally, typically by wrapping module calls in their own state-machine and using Context.serialize to persist input snapshots.
apcore has no notion of a "workflow", "saga", "compensation chain", "fan-out/fan-in", or "DAG". A workflow is a coordinated sequence of multiple module calls — that's the runtime layer above apcore. Workflow orchestrators are downstream consumers; their concerns (step-level retry, compensation on failure, parallel branching, durable timer, signal handling) are out of scope here.
Modules carry x-cost-per-call, x-rate-limit, and similar extension annotations as informational hints for AI planners (see protocol-spec.md §4.6). apcore does not enforce them — no built-in budget tracker, no rate-limiter middleware in the standard surface, no centralized policy engine. Downstream layers own enforcement, typically as custom middleware or via an external policy engine that consumes the annotations.
apcore does not model state shared across multiple module calls. Each call is independent; there is no "session", no "conversation context", no "agent memory" at the protocol level. Such state, if required, lives in context.data (transient, per-call-chain) or in a downstream runtime that owns the state lifecycle.
apcore deliberately does not promote application-level metadata to first-class Context fields when apcore the SDK does not consume that metadata itself. Examples:
- Idempotency keys — application semantics; lives in
context.data["x-correlation-id"]or any otherx-*key. - Replay-mode hints (
is_replay,attempt_number) — runtime semantics; lives incontext.dataunder a runtime-defined namespace. - Compensation pointers / saga step IDs — workflow semantics; lives in the workflow runtime, not in apcore.
- Deadlines, budgets, retry policies — same reasoning.
Promoting any of these to a normative field would expand apcore's surface area without adding apcore-level behavior. The §4.6 x- extension mechanism is the documented home for such concerns. Future spec revisions MAY revisit this if real-world usage demonstrates that a data-key convention is insufficient — but the bar is concrete usage evidence, not speculation.
The following are recommended, vendor-neutral recipes for building a retry/replay/workflow layer on top of apcore. They use only the stable hooks listed in §2.
For retries that survive the lifetime of a single process: implement a Middleware that catches retryable errors in on_error and returns a RetrySignal. RetryMiddleware is shipped in apcore-python's middleware collection as a reference (see middleware-system.md).
No new spec surface needed.
For "submit a call, survive a process restart, retry on failure with backoff":
- Use
AsyncTaskManagerwith a customTaskStoreimplementation backed by your persistence layer (Redis, SQL, etc.). - Configure
RetryConfig(max_retries, backoff) on submission. - On process startup, the application enumerates
pendingandrunningrecords from itsTaskStoreand re-submits or marks them as appropriate. apcore does not auto-resume tasks across process boundaries; theTaskStoremakes the state durable, but resumption policy (re-submit vs. fail-and-restart vs. ignore) is the application's choice.
AsyncTaskManager handles the lifecycle, retry scheduling, and backoff math within a single process. The 5-method TaskStore is the only interface the application implements; cross-restart behavior is built on top of it by the application.
No new spec surface needed.
For execution that must pause indefinitely waiting on a human, an external service, or a scheduled trigger:
- Mark the relevant modules with
requires_approval: true. - Implement a custom
ApprovalHandlerthat:- On
request_approval(), persists the request to your durable store and returnspendingwith anapproval_id. - On
check_approval(token), looks up the persisted decision and returnsapproved/rejected/timeout/ still-pending.
- On
- The caller catches
ApprovalPendingError, persists the call's original inputs alongsideapproval_id, and re-invokes with_approval_token = approval_idonce the decision is recorded.
This is the canonical pause/resume contract (§2.2). It crosses crashes, restarts, and arbitrary time gaps.
No new spec surface needed.
For invoking modules from a worker process, a queue consumer, or a remote service:
- The originator builds the
Context(setstrace_id,identity, anyx-*keys indata) and callsContext.serialize()to JSON. - The serialized Context plus
module_idandinputsare placed on a queue / RPC channel / task store. - The worker calls
Context.deserialize(), attaches its ownexecutor/services/cancel_tokenruntime injects, and invokes the module via the executor.
Context.serialize() and Context.deserialize() are normative; the round-trip is conformance-tested. Runtime injects are reattached on the worker side per §5.7.
No new spec surface needed.
For "the same logical call MUST NOT execute side effects twice":
- The originator sets a stable identifier (UUID, ULID, business key) in
context.data["x-correlation-id"](or anotherx-*key of the runtime's choosing). - A middleware on the worker side reads the key on
beforeand consults a dedup store (Redis, DB) for prior completion. - If a prior result exists, the middleware short-circuits by returning the cached output (or re-raising the recorded error). Otherwise it executes normally and records the result on
after.
The middleware owns the dedup logic; apcore guarantees only that the key survives serialization and child-context derivation when stored in data.
No new spec surface needed.
The following items have been considered and deliberately deferred. They are documented here so downstream integrators understand which gaps are intentional and where to prototype today.
| Deferred item | Today's workaround | Why deferred |
|---|---|---|
Explicit is_replay hint on Context |
context.data["x-is-replay"] (or runtime-chosen x-* key) |
No concrete consumer exists yet; promoting before usage evidence risks getting the semantics wrong (does it apply mid-call-chain? does it propagate to children?). |
Explicit attempt_number field |
context.data["x-attempt"] |
Same. Different runtimes count attempts differently (per-call vs per-workflow). |
compensatable: bool annotation |
Application-level convention | Compensation is a workflow concept; apcore has no notion of paired forward/reverse module pairs. |
deterministic: bool annotation |
idempotent is the closest existing primitive |
Determinism (same input → same output) is stricter than idempotency and harder to verify. Speculative without a runtime that consumes it. |
Dedicated idempotency_key first-class field |
context.data["x-correlation-id"] (or runtime-chosen x-* key) |
apcore the SDK doesn't consume the key; promoting it adds spec surface without behavioral benefit. Revisit when a runtime demonstrates the data convention is insufficient. |
Mid-pipeline CheckpointStore extension point |
Wrap calls externally with state-machine + Context.serialize() |
Would invade the executor's stateless invariant. |
| Cost / budget enforcement | Application-level middleware | Cost is multi-dimensional and runtime-specific. |
Each deferred item is a candidate for a future spec proposal once a concrete downstream runtime makes a sustained, evidence-backed case.
This document is part of apcore's spec surface and is governed by protocol-spec.md §13.5 compatibility rules.
- Stable hooks listed in §2 MAY evolve only in additive, backward-compatible ways within a major version.
- Non-goals listed in §3 MAY become goals in a future major version with appropriate community discussion and evidence.
- The Gap watchlist (§5) is non-normative and reflects current judgment; entries may be promoted to either §2 (added) or §3 (rejected) over time.
protocol-spec.md(repository root) — full normative specification- §4.6 —
x-extension mechanism - §5.7 — Context structure and serialization
- §7 — Approval System (including
_approval_tokenPhase B) - §11 — Extension points
- §13.5 — Versioning and compatibility
- §4.6 —
POSITIONING.md— apcore's place in the AI stackdocs/features/async-tasks.md—AsyncTaskManagerand theTaskStoreinterfacedocs/features/approval-system.md— Approval handler protocoldocs/features/extension-system.md—ExtensionManagerand extension point lifecycledocs/features/middleware-system.md— Middleware contract andRetrySignal