Skip to content

Lifecycle fanout can skip processors present at operation start #6977

Description

@teamleaderleo

What happened?

Steps to Reproduce

OpenTelemetry trace and log providers can skip a processor that was registered when shutdown() or forceFlush() began.

The lifecycle fanout currently invokes processors while walking a retained array:

processor A throws synchronously
    -> promise-list construction stops
    -> processor B is never called

processor A removes processor B from the retained array
    -> live iteration no longer sees B
    -> processor B is never called

TracerProvider.forceFlush() has a related timeout-cleanup problem:

create timeout
    -> processor.forceFlush() throws synchronously
    -> the operation reports failure
    -> the timeout remains armed until expiry

The trace SDK specification requires shutdown and force flush to reach all registered span processors. The logs SDK specification requires the same for all registered log record processors.

processor A fails
processor B succeeds

call A
call B
report the operation as failed through the existing result policy

Synchronous shutdown throw

Save the setup code below as index.mjs, then run:

npm install
npm start

Current output:

secondShutdownCalls: 0

Expected output:

secondShutdownCalls: 1

The same direct-throw behavior is present in trace force flush and log shutdown/force flush.

Processor-array mutation

processors = [A, B]
A runs processors.splice(1, 1)
live iteration skips B

The operation should keep the processor set that existed when it began.

Provider timeout cleanup

On current main, use:

await provider.forceFlush({ timeoutMillis: 1000 });

A processor that throws synchronously causes an immediate rejection, but its timeout remains referenced until expiry. The process can remain alive for roughly the configured timeout.

Released @opentelemetry/sdk-trace 2.10.0 uses the deprecated constructor timeout option instead of the newer per-call option.

Expected Result

opening processors = snapshot(current processors)

for each processor in opening processors:
    invoke lifecycle method
    route a direct throw into the existing promise failure path

report through the existing result policy
clear timers for processors that have finished or failed

Actual Result

A direct synchronous throw or live-array mutation can skip a later processor. TracerProvider.forceFlush() can leave a timeout armed after reporting the corresponding failure.

Additional Details

A skipped processor can miss its final export or cleanup opportunity. In Node.js, the stale timer can delay process exit.

I prepared a candidate implementation and respective regression tests in a fork. The affected paths are:

  • MultiSpanProcessor.shutdown() and forceFlush()
  • TracerProvider.forceFlush()
  • MultiLogRecordProcessor.shutdown() and forceFlush()

My proposed patch snapshots each processor list and converts direct throws to rejected promises. It keeps the current public APIs, configuration, processor order, error behavior, and timeout behavior.

OpenTelemetry Setup Code

import { TracerProvider } from '@opentelemetry/sdk-trace';

let secondShutdownCalls = 0;

const first = {
  onStart() {},
  onEnd() {},
  forceFlush: async () => {},
  shutdown() {
    throw new Error('first processor failed');
  },
};

const second = {
  onStart() {},
  onEnd() {},
  forceFlush: async () => {},
  shutdown: async () => {
    secondShutdownCalls += 1;
  },
};

const provider = new TracerProvider({
  spanProcessors: [first, second],
});

try {
  await provider.shutdown();
} catch {}

console.log({ secondShutdownCalls });

package.json

{
  "name": "otel-lifecycle-fanout-reproduction",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "node index.mjs"
  },
  "dependencies": {
    "@opentelemetry/sdk-trace": "2.10.0"
  }
}

Relevant log output

shutdown reproduction:
{ secondShutdownCalls: 0 }

provider forceFlush reproduction:
forceFlush rejected
process exit delayed until the forceFlush timeout expires

Operating System and Version

Operating-system agnostic. The behavior follows JavaScript control flow and was exercised through the repository’s CI matrix.

Runtime and Version

Node.js v22.16.0

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpkg:sdk-logspkg:sdk-tracepriority:p2Bugs and spec inconsistencies which cause telemetry to be incomplete or incorrect

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions