Export assignment-scoped worker task traces to Warp and operator OTLP

[warp-agent-staging]

Export worker-side task traces using assignment OTLP configuration for REMOTE-3534. Warp export works without an operator collector, including when operator tracing is unset or disabled. Both destinations receive the same span IDs, parentage and provider-level resource.

Review guide

Start with the spec, then internal/tasktelemetry/tracing/session.go and operator.go for pipeline ownership. otlp contains task authentication and export privacy. worker.go and the backends define observation boundaries, especially preserved Kubernetes Jobs and dispatch-only Command tasks.

What changed

• An injected tracing.Handle owns the operator pipeline and shared resource. Task providers send spans through independent SDK batch processors. Tasks own Warp pipelines and borrow the operator processor; flushing reaches both destinations without shutting down the operator exporter. Resources and root spans use warp.run_id and optional warp.execution_id, without a duplicate task.id.
• Roots start at WebSocket receipt before decoding and end after the worker boundary: configured idle, synchronous cleanup, task-log flush and terminal enqueue attempt. Creators own ordinary OTel spans; setup-events wrappers own reporting and tracing. Sessions do not register or automatically finalize child spans.
• Shutdown cancels a separate local backend-observation context and waits for creator acknowledgement before flushing. Kubernetes stops its watches/polls, ends live children before roots, and preserves Jobs without reporting remote completion. Dispatched Command tasks remain remote; an in-flight dispatch command stops locally without invoking remote cancellation. The existing ten-second backend wait bounds acknowledgement. A non-cooperative backend can lose unfinished spans at that deadline rather than have them ended by another owner.
• Docker has separate create/start spans while retaining its combined setup metric. Direct workspace preparation ends before setup; setup-event helpers trace commands and cleanup traces teardown/removal. Kubernetes separates scheduling, init-image preparation, custom init execution and task startup. Preparation includes image pulls and other pre-start delay, not exact pull durations; pod status has no pull timestamps. Intervals retain observed pod timestamps and identify init containers without exporting image names.
• Logging and tracing share OTLP authentication, endpoint validation and exporter setup, with separate SDK signal exporters and queues. Both prefer a valid assignment bootstrap token and fall back to task-identity refresh credentials. Neither uses the worker key. Task clients ignore operator headers and custom OTLP certificates; HTTPS still verifies certificates against default trust roots.
• Each Warp queue holds 256 spans with 64-span batches and a five-second export budget. Finalization flushes destinations independently under one five-second budget; worker shutdown shares that deadline across tasks. Raw task errors remain available, titles/images are omitted, and exporter/authentication errors cannot reflect credentials into SDK-global errors. Both destinations receive operator OTEL_RESOURCE_ATTRIBUTES without projection. Operators must exclude sensitive values. Thin immutable exporter adapters sanitize errors and bind trace export cancellation.

Validation

Passed locally with GOMAXPROCS=2:

• go test -race -p 1 ./internal/tasktelemetry/... ./internal/worker
• go test -p 2 ./...
• go vet -p 2 ./...
• go build -p 2 ./...
• golangci-lint run (zero issues)
• Final gofmt -s -w on changed Go files, then gofmt -s -l . and git diff --check (clean)

The Kubernetes shutdown regression failed against automatic cutoff because local observers remained active. It now proves acknowledgement, child-before-root completion, Unset interrupted status, no terminal remote message, preserved Jobs and a shared export deadline. Correlation and granular Docker/Kubernetes intervals retain regression coverage. HTTPS fixtures prove both signals reject operator-only CA trust and succeed with verified task-client trust. Existing tests cover dual export, task isolation, authentication, privacy and queue overflow. No live-cluster deployment was tested.

Plans:

• Revise worker telemetry to share setup and explicit tracing ownership

[warp-agent-staging]

This PR was generated with Warp.

Comment @warp-staging-factory on this PR to send it follow-up work.