Repository navigation
[BUG] func_otlp_grpc segfaults during a TLS case after ctest reports everything passing, failing the job #4489
Description
Describe your environment
GitHub hosted Linux runners, main at 1c2b007a. Seen on two different maintainer mode jobs:
CMake gcc 14 (maintainer mode, async)CMake clang 18 (maintainer mode, yaml config)
Steps to reproduce
I do not have a local reproduction. It shows up on CI, and both sightings below are on pull requests that do not touch the OTLP gRPC exporter.
What is the expected behavior?
ci/do_ci.sh reports the job's result from the tests. When ctest reports every test passing, the job passes.
What is the actual behavior?
The job fails with exit code 139 after ctest has already reported everything green. The crash is in the functional gRPC binary that runs after ctest, on a TLS case:
100% tests passed, 0 tests failed out of 1593
...
Running test cert-unreadable on secure https://localhost:4317 with server https
I0000 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A
./run_test_mode.sh: line 67: 34560 Segmentation fault (core dumped) "${TEST_BIN_DIR}/${TEST_EXECUTABLE}" --debug --m
##[error]Process completed with exit code 139
Two sightings, two days apart, two compilers:
| PR | job | run |
|---|---|---|
| #4453 | CMake gcc 14 (maintainer mode, async) |
97488993921 |
| #4337 | CMake clang 18 (maintainer mode, yaml config) |
97837662928 |
Additional context
Neither pull request can be the cause, and that is checkable rather than argued. functional/otlp/CMakeLists.txt links the crashing binary against two targets:
target_link_libraries(func_otlp_grpc ${CMAKE_THREAD_LIBS_INIT}
opentelemetry_trace opentelemetry_exporter_otlp_grpc)#4337 changes only exporters/elasticsearch/, and grep -rn -i elasticsearch functional/ returns nothing. #4453 changes only exporters/otlp/src/otlp_http_client.cc and its test. Neither is in that binary's link closure.
For #4453 there is a second measurement: the head that passed this job and the head that failed it differ only in comments, 0 non-comment changed lines. One passed, one failed.
It is not constant. CMake clang 18 (maintainer mode, yaml config) was green on 11 of the last 12 main commits, so this reads as a race in the functional test or in the gRPC TLS teardown rather than something that reproduces on demand.
What it costs today is that a red job on a pull request has to be opened and read before it can be dismissed, and the log looks alarming because ctest is green immediately above it.
Activity
marcalff commentedon Aug 31, 2026
I have seen it also in CI multiple times.
Example of log:
The last part:
is the test crash.
thc1006 commentedon Aug 31, 2026
TL;DR: this looks like the same failure as #4273, and the upstream fix has already shipped. gRPC stopped registering OpenSSL's atexit handler in 1.83.0, and we pin 1.82.1 in
third_party_release, so a gRPC bump should end it. I owe a correction there, because I said the opposite in July.Thanks for the log. It is more useful than it might look, because it aborts with
double free or corruption (fasttop)and exit 134, while the one in the description segfaults with exit 139. I think they are one bug rather than two. Both land after100% tests passed, in the same binary, on a TLS case whose handshake fails, and the case that dies moves around:client-key-unreadablein yours,cert-unreadablein mine,cert-not-foundin @dbarker's #4273. One heap corruption, reported by whichever check happens to notice it first, fits that better than three separate bugs.The corruption is at process exit. OpenSSL's
OPENSSL_cleanupatexit handler frees the library's global state while gRPC EventEngine threads are still using it. That part can be seen without waiting for a crash: anatexithandler registered before gRPC first touches OpenSSL runs after OpenSSL's own, since handlers run in reverse order of registration, so it reports the thread population at a point no earlier thanOPENSSL_cleanup. On a small gRPC client with a failing handshake it printsso nothing has drained by then.
gRPC commit
784d4464b(2026-06-17, "Initialize openssl with no atexit handler", grpc/grpc#42608) changedinit_openssl()insrc/core/tsi/ssl_transport_security.cctoso the handler is never registered and there is nothing left to race. The pull request shows as closed rather than merged, which is normal for changes that land through gRPC's internal import. Grepping that file at each tag:
OPENSSL_INIT_NO_ATEXITv1.82.2 came out on 2026-08-27 with six backports and none of them touch OpenSSL, so staying on the 1.82 line does not pick it up.
third_party_releaseline 20 readsgrpc=v1.82.1, and both jobs named in the description install from it throughci/install_thirdparty.sh --tags-file third_party_release.The correction: in my July comment on #4273 I wrote that this would stop "once it ships in a gRPC release we can bump to". It had already shipped. 1.83.0 was tagged on 2026-07-22, four days before I wrote that. I had read the release notes, which do not mention it, instead of the code. Sorry for sending anyone down the wrong path.
On what to do next I would rather defer than push. Going from 1.82.1 to 1.83.x is a minor version step, and the two previous gRPC moves here (#4167 and #4255) were paired with protobuf, so the version choice looks like a maintainer call rather than something to slip in. If it is useful I am happy to open the
third_party_releasebump together with the matching submodule andMODULE.bazelupdates and let CI report on 1.83.x, or to leave it with whoever normally drives those upgrades.If a bump is not wanted yet, the smallest local workaround is to end
func_grpc_main.ccwithstd::_Exit(rc)after flushing, so the test binary runs no atexit handler at all. I have not been able to put that in front of a run that actually fails, so I would rather fix the dependency than add it on a hunch.Thanks @dbarker for the original report and the logs in #4273, and @marcalff for confirming here. Your log is what let me line the 134 and 139 cases up.
yigitcan-ozturk commentedon Sep 3, 2026
I'd like to investigate this. I'll focus first on the TLS functional-test teardown path around
func_otlp_grpc, reproduce the double-free/segfault under sanitizers if possible, and then narrow the fix to the smallest lifecycle/cleanup race with regression coverage.thc1006 commentedon Sep 8, 2026
@yigitcan-ozturk before you spend time on that, two things have changed, and one of them is that my comment above is out of date.
I wrote that we pin gRPC 1.82.1 in
third_party_release. That was true at 09:27 UTC on 31 August, and #4505 moved it to 1.83.1 at 20:52 UTC the same day, about eleven hours later.func_otlp_grpcis built by CMake only, infunctional/otlp/CMakeLists.txt, so it now builds against a gRPC that carries grpc#42608, which is the change that stops OpenSSL's atexit handler being registered.Since that merge,
mainhas had 272 workflow runs with two failures, and neither is this one. ADocker Imagejob on 5 September, andBasicCurlHttpTests.ElegantQuitQuickatcost ms: 247on the bump commit itself. Nofunc_otlp_grpcfailure in the eight days since.That is evidence rather than proof, because it was intermittent to begin with, and @marcalff may have seen it more recently than the run list reaches. But it seems worth checking whether it still reproduces before looking for a lifecycle race in our own teardown, because if it does still happen at 1.83.1 then what I wrote above is wrong and that is useful to know early.
Separately, and not this test:
MODULE.bazelstill pins gRPC 1.80.0, so Bazel builds do not have that fix. #4279 is the open bump for it.func_otlp_grpcis not built there, so it is a different exposure rather than the same failure.thc1006 commentedon Sep 8, 2026
Correcting myself: it still reproduces at 1.83.1, so the check I suggested has already come back negative. @yigitcan-ozturk, the teardown path is worth looking at after all.
#4405 at
31e0e9ba, rebased onto a main that carriesgrpc=v1.83.1, failedCMake clang 18 (maintainer mode, sync)a few hours ago with the same signature as the reports above:The same log shows
-- - grpc: v1.83.1coming fromci/install_thirdparty.sh --tags-file third_party_release, so the build did carry grpc#42608 and crashed anyway.Two things I checked before posting this, because a red on my own pull request is the least trustworthy evidence I could offer. That branch touches only
ext/.../http/client/curl/and no gRPC or OTLP file, andfunc_otlp_grpcdoes not go through the curl client, so its diff cannot reach the crashing path. And the same job was green on #4406, #4457, #4297, #4331 and #4453, all sitting on the samemain, so this is still intermittent rather than something one branch made deterministic.What that means for my earlier comment: the 272 clean runs were a real count, but counting an intermittent failure not happening is not evidence that it was fixed, and I presented it as though it were. The atexit explanation should go back to being a hypothesis. It may still be part of the story, since the crash lands in the same place, but 1.83.1 contains the change and the crash survives it.
The
MODULE.bazelpin at 1.80.0 is unrelated to this and can be ignored for now.Retrigger CI (func_otlp_grpc TLS segfault matches known issue open-te…
Retrigger CI (func_otlp_grpc TLS segfault matches known issue open-te…
Retrigger CI (func_otlp_grpc TLS segfault matches known issue open-te…