feat(moonbit): implement component async bindings

Author: peter-jerry-ye

crates/moonbit/CONTEXT.md

@@ -0,0 +1,34 @@
+# MoonBit Bindings Context
+
+This context records shared language for the MoonBit binding generator. Async
+future and stream vocabulary is split into [Async Glossary](docs/async-glossary.md).
+
+## Language
+
+**MoonBit binding**:
+Generated MoonBit source that exposes WIT imports to MoonBit code or adapts
+MoonBit exports to the component ABI.
+
+**Component adapter path**:
+The current implementation path where MoonBit emits core wasm and `wasm-tools`
+converts it into a component using adapter imports and exports.
+_Avoid_: direct component generation
+
+## Async Design
+
+- [Async design contract](docs/async-design.md)
+- [Async glossary](docs/async-glossary.md)
+- [ADR 0001: FFI-boundary conversion](docs/adr/0001-async-ffi-boundary-conversion.md)
+- [ADR 0002: local Future/Promise pair](docs/adr/0002-local-future-promise.md)
+
+The implementation targets the official upstream generator architecture. WIT
+`future` and `stream` remain distinct from local MoonBit `Future` and `Stream`;
+generated code converts them only at concrete FFI positions whose intrinsic
+names are supplied by `wit-parser`. Local `Future::new()` returns a
+MoonBit-only Future/Promise pair, and local `Semaphore` coordinates coroutines;
+neither can select or own a component endpoint from `T`. Async support is always
+available in the MoonBit generator, but endpoint-free synchronous worlds do not
+emit its runtime or wrappers. Component endpoint wrappers are
+generated-code-only, enforce a single in-flight operation, and retain operation
+buffers until cancellation or completion is observed. Each top-level component
+task has its own waitable set and scheduler state.

crates/moonbit/docs/adr/0001-async-ffi-boundary-conversion.md

@@ -0,0 +1,141 @@
+# Keep MoonBit async values detached from component futures and streams
+
+Status: accepted; implemented in the current branch
+
+MoonBit `Future[T]` handles and local `Stream[T]` values are over arbitrary
+MoonBit types, while component `future<T>` and `stream<T>` are ABI values over
+WIT-representable payloads. MoonBit async bindings will keep those concepts
+separate: generated FFI-boundary code converts between local async values and
+component endpoints only at concrete WIT positions where the endpoint operations
+and payload lift/lower code are known.
+
+For MVP, component `future<T>` maps to a one-shot `Future[T]` handle. The handle
+represents a ready value or an owned local source computation with
+value-discarding drop cleanup. For an incoming component future, generated code
+supplies a source closure that captures the raw readable handle and directly
+binds the concrete WIT position's read/cancel/drop intrinsics. The generic local
+type has no CM-specific variant or operation table. A public local `Promise[T]`
+may settle the same Future state, but it is not a component writable endpoint.
+
+## Consequences
+
+- `Stream::new()` creates local MoonBit values only.
+- `Future::new()` creates a local `Future[T]` / `Promise[T]` pair only. It does
+  not select or invoke a component `future.new` intrinsic.
+- Endpoint operation tables are not part of the target design. Generated
+  position-specific helpers call canonical intrinsics directly and keep raw
+  handle state inside generated source closures or producer tasks.
+- Component intrinsic module and field names are generated through
+  `wit-parser`'s `WasmImport::{FutureIntrinsic, StreamIntrinsic}` API. The
+  MoonBit generator does not reproduce position indices, export prefixes,
+  unit-payload names, or async-lower prefixes by string formatting.
+- The Rust generator boundary owns a recursive conversion plan for each
+  function. Ordinary lift/lower code requests position-specific helpers from
+  that plan instead of naming runtime endpoint types or type-shaped tables.
+- The existing async generator boundary is retained. The static recursive
+  rewrite happens behind it without changing ordinary lift/lower ownership or
+  the endpoint-free sync path.
+- User-facing bindings expose local `Future[T]` handles and local `Stream[T]`
+  for ordinary async composition.
+- A nested WIT shape such as `future<future<stream<T>>>` maps to
+  `Future[Future[Stream[T]]]`. Only the current layer's readable handle appears
+  at each canonical payload stage. Generated recursive lift/lower functions bind
+  each layer to its own function-position intrinsic and apply generated
+  commit/reject dispositions at every transfer boundary.
+- Recursive lower retains its canonical buffer and prepared producer state until
+  the transfer reports a disposition. Commit starts producer work only after
+  ownership transfers. Stream progress commits the accepted prefix and retries
+  the same lowered suffix; an abandoned suffix is rejected exactly once.
+- A freshly-created component future is not fully rollbackable. Canonical ABI
+  permits its writable end to be dropped only after a write succeeds or a write
+  reports that the reader was dropped. If an outer transfer rejects a nested
+  future readable, generated code drops that readable but must still drive the
+  paired writer with the local future value until its write observes `dropped`.
+  A cancelled write is not settlement and is retried while the component task
+  remains alive.
+- Stream batches whose element type recursively contains a future use a
+  one-element staging window for MVP. This bounds settlement obligations created
+  before downstream acceptance without changing the public stream API.
+- Generated stream producers remain prepared when their component pair is
+  created. Commit starts normal pumping. Parent rejection drops both
+  untransferred component ends and performs state-aware local rejection: incoming
+  component sources close immediately, buffered values use configured cleanup or
+  generated cleanup as fallback, and an unstarted local producer is discarded
+  without executing user code. `Stream::produce` accepts an optional
+  `on_unstarted_drop` callback for resources captured by that branch and a
+  separate per-element cleanup for values written after it starts.
+- Canonical `backpressure.inc/dec` controls admission of new async component
+  tasks. It is not tied to future/stream bridge lifetime and is not called
+  implicitly by the MoonBit runtime. Endpoint read/write suspension provides
+  data-flow backpressure independently.
+- Async import argument settlement distinguishes `cancelled-before-started`
+  from every state in which the callee may have started. The former recursively
+  rejects owned resources and endpoints; the latter only reclaims guest-owned
+  canonical list allocations.
+- Local future MVP has consuming `Future::get()` and value-discarding async
+  `Future::drop()`. Strong cancellation belongs to `Task` and `TaskGroup`, not
+  to a future-specific result type. `Future::drop()` is explicit cleanup, not a
+  direct alias for component `future.drop-*`, and futures that may discard
+  completed WIT payloads carry generated payload cleanup logic.
+- Outgoing component `future<T>` producer tasks must settle their raw writable
+  handle by writing a real value or by attempting the write and observing reader
+  drop. The MVP does not fabricate default values to satisfy unwritten futures.
+  If user code produces a ready `T`, generated code may write immediately; if
+  user code returns a pending `Future[T]`, the bridge exists before the CM
+  `future.write` operation starts because the component boundary already needs a
+  readable end to return.
+- After an outgoing component `future<T>` readable end is committed, or after a
+  parent transfer rejects and locally drops it, the bridge shields producer work
+  from ordinary task/subtask cancellation. Component-task cancellation is
+  cooperative: it may resolve the cancelled call with `task.cancel`, but it does
+  not forcibly destroy shielded settlement work. Only instance teardown or a
+  trap can abandon the writer without settlement. Peer reader drop is
+  loss-of-interest, not task cancellation; before `future.write` has started,
+  the bridge does nothing special for it. If a later write reports `dropped`,
+  the bridge cleans the value and settles the writer.
+- If that local future never produces `T`, the Component Model provides no
+  generic close-without-value operation. This is an explicit liveness limit; the
+  binding does not fabricate a default value or pretend an idle writer can be
+  dropped safely. The same limit applies when the paired readable was created
+  for a nested payload but the outer transfer rejected it before ownership
+  crossed the boundary.
+- Local stream MVP has `Sink::close()` for graceful producer close and async
+  `Stream::drop()` or `Stream[T]` drop for consumer loss-of-interest. It does
+  not expose public `Sink::cancel()` because component `stream` has no distinct
+  generic producer-failure signal to preserve across the boundary.
+- Local stream state keeps producer close separate from reader drop. Producer
+  close preserves buffered `FixedArray[T]` chunks for draining; reader drop
+  discards unread chunks and uses an explicit `Stream::new_with_cleanup` or
+  `Stream::produce(cleanup=...)` callback for payloads that need resource
+  cleanup.
+- Local stream capacity is measured in elements. Zero is strict rendezvous and
+  a positive value is a hard bound on accepted unread values; local streams are
+  never implicitly unbounded. Waiting readers and writers use direct FIFO
+  handoff with completion-versus-cancellation race handling.
+- An incoming component stream uses a generated demand-driven source whose
+  reads directly call its concrete site intrinsics. It does not start an eager
+  pump into a local stream pipe.
+- Forwarding component endpoints without reading them is not the default path and
+  needs an explicit advanced API if we decide to support it.
+- Runtime validation includes real `wasi:cli@0.3.0` stream output and a real
+  `wasi:http@0.3.0` handler response whose body stream, trailers future, and
+  transmission completion continue after the handler returns the response.
+- Local stream validation covers strict rendezvous, bounded buffering, cancelled
+  waiter removal, completion-versus-cancellation ownership races, local
+  producer reads, and unread resource cleanup.
+- Async export stubs intentionally expose
+  `background_group : @async-core.TaskGroup[Unit]`. It adapts the mismatch
+  between MoonBit structured completion and component task return. Generated code
+  publishes component task return when the user function produces its result,
+  then keeps the underlying MoonBit task group alive for hook-style post-return
+  work. That work remains structurally owned and bounded by the component task
+  or instance lifetime, but cannot change the already-published export result.
+- Lowering local `Future[T]` and `Stream[T]` values requires an active component
+  async task scope, including recursive occurrences inside structured payloads.
+  A sync WIT import called in that scope prepares its endpoint arguments and
+  commits those same handles immediately after the core call returns. Sync export
+  results, and sync imports called without an active scope, remain unsupported.
+  Incoming lift remains lazy and does not require a producer task until user code
+  later reads in an async scope. Scope-free sync lowering requires cooperative
+  component-thread support and must not be faked by using the stackless callback
+  ABI for a non-async WIT function.

crates/moonbit/docs/adr/0002-local-future-promise.md

@@ -0,0 +1,43 @@
+# Add a local Future/Promise pair
+
+Status: accepted; implemented in the current branch
+
+MoonBit needs a producer-facing one-shot primitive for local control-flow cycles
+that cannot be expressed as `Future::from(async () -> T)`. The motivating case
+is `wasi:http@0.3.0` request handling: `consume-body` takes a Future reporting
+processing completion before it returns the request body whose later processing
+determines that completion.
+
+`Future::new()` returns `(Future[T], Promise[T])` and
+`Future::new_with_cleanup(cleanup)` adds explicit value-discard cleanup. The
+Promise can complete with a value, fail with a MoonBit error, or close without a
+value. It is local coordination state only. It does not create, wrap, or own a
+component `future` endpoint, and it works for arbitrary MoonBit `T`.
+
+## Consequences
+
+- `Promise::complete(value)` returns `true` only when the reader accepts
+  ownership. If the Future was already dropped, it returns `false` and the
+  caller retains `value`.
+- `Future::drop()` after accepted completion runs the cleanup supplied to
+  `new_with_cleanup`. The plain `new()` constructor is appropriate only when
+  discarded `T` needs no explicit cleanup.
+- `Promise::fail(error)` makes local `Future::get()` raise that error.
+  `Promise::close()` makes it raise `PromiseClosed`.
+- Settlement is one-shot. Repeating `complete`, `fail`, or `close` after a
+  successful settlement is a programmer error.
+- Dropping or otherwise abandoning a still-pending Promise does not implicitly
+  close its Future because MoonBit has no generic deterministic destructor. The
+  producer must explicitly complete, fail, or close it.
+- Task cancellation that reaches a waiting reader before settlement drops the
+  reader, so later completion returns `false`. Once completion assigns the value
+  and wakes the reader, completion wins a simultaneous cancellation race and the
+  reader receives the value.
+- Explicit `Future::drop()` follows the same race rule while `get()` is pending:
+  dropping before settlement wakes the reader with `Cancelled`, while an
+  already-assigned value or error remains owned by the waiting reader.
+- A local failure or close cannot settle an already-exposed component future
+  without a value. If such an outcome is expected across WIT, it belongs in the
+  payload type, for example `Future[Result[V, E]]`.
+- Generated FFI-boundary code remains solely responsible for creating concrete
+  component future pairs and satisfying their writable-end settlement rules.