[BUG] Exception escaping noexcept Session::SendRequest in curl HTTP client calls std::terminate, killing the host process
Maintainer thường phản hồi trong vòng 1 ngày
Chưa có ai nhận issue này.
Đánh giá
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức phù hợp với người mới
- 45/100
- Loại issue
- Lỗi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức độ hoạt động
- Sôi nổi
- Công nghệ
- cpp
- Lĩnh vực
- backend-api-design
Hướng nghiên cứu
Start with ext/src/http/client/curl/http_client_curl.cc, especially Session::SendRequest and HttpClient::MaybeSpawnBackgroundThread; read the SendRequest contract in ext/include/opentelemetry/ext/http/client/http_client.h and the existing cleanup path in exporters/otlp/src/otlp_http_client.cc. Trace how a failed send releases the session and wakes synchronous exports, then check the curl client tests for coverage. Done means exceptions from these noexcept paths are handled without terminating the host process, with session cleanup preserved and failures reported through the error handler.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Mô tả
Describe your environment
- opentelemetry-cpp v1.22.0. Also verified present on
mainas of 2026-11-16 — see "Additional context". - Linux x86-64, Clang, libstdc++, static libs.
- Build options:
WITH_OTLP_HTTP=ON,OPENTELEMETRY_ABI_VERSION_NO=2,OPENTELEMETRY_STL_VERSION=2023,OPENTELEMETRY_NO_DEPRECATED_CODE,HAVE_ABSEIL.ENABLE_ASYNC_EXPORTnot defined, so the synchronousOtlpHttpClient::Export(message)path is used. - Exporter: OTLP/HTTP metric exporter behind
PeriodicExportingMetricReader.
Steps to reproduce
Session::SendRequest is declared noexcept but performs several operations that can throw.
Any of them terminates the process. The most deterministic to demonstrate is the lazy creation
of the curl background thread, which throws std::system_error when pthread_create fails.
repro.cc:
#include <opentelemetry/exporters/otlp/otlp_http_metric_exporter_factory.h>
#include <opentelemetry/exporters/otlp/otlp_http_metric_exporter_options.h>
#include <opentelemetry/sdk/metrics/export/periodic_exporting_metric_reader.h>
#include <opentelemetry/sdk/metrics/export/periodic_exporting_metric_reader_options.h>
#include <opentelemetry/sdk/metrics/meter_provider.h>
#include <opentelemetry/sdk/metrics/view/view_registry.h>
#include <chrono>
#include <memory>
#include <thread>
int main()
{
namespace otlp = opentelemetry::exporter::otlp;
namespace metrics = opentelemetry::sdk::metrics;
// Default endpoint; nothing needs to be listening for this repro.
otlp::OtlpHttpMetricExporterOptions exporter_options;
auto exporter = otlp::OtlpHttpMetricExporterFactory::Create(exporter_options);
metrics::PeriodicExportingMetricReaderOptions reader_options;
reader_options.export_interval_millis = std::chrono::milliseconds(10000);
reader_options.export_timeout_millis = std::chrono::milliseconds(5000);
auto reader = std::make_shared<metrics::PeriodicExportingMetricReader>(
std::move(exporter), reader_options);
auto provider = std::make_shared<metrics::MeterProvider>(
std::make_unique<metrics::ViewRegistry>());
provider->AddMetricReader(reader);
auto counter = provider->GetMeter("repro")->CreateUInt64Counter("repro_counter");
counter->Add(1);
std::this_thread::sleep_for(std::chrono::seconds(20)); // allow one export
return 0;
}
fail_pthread_create.c — makes pthread_create fail once the SDK's own threads are up, so
only the curl client's lazily-created background thread is affected:
#define _GNU_SOURCE
#include <dlfcn.h>
#include <errno.h>
#include <pthread.h>
#include <time.h>
static time_t start;
__attribute__((constructor)) static void init(void) { start = time(NULL); }
int pthread_create(pthread_t *t, const pthread_attr_t *a,
void *(*fn)(void *), void *arg)
{
static int (*real)(pthread_t *, const pthread_attr_t *, void *(*)(void *), void *);
if (!real) real = dlsym(RTLD_NEXT, "pthread_create");
/* SDK reader threads are created during startup; the curl background thread is
created lazily on the first export, at ~10s. Fail anything after 5s. */
if (time(NULL) - start >= 5)
return EAGAIN;
return real(t, a, fn, arg);
}
cc -shared -fPIC -o fail_pthread_create.so fail_pthread_create.c -ldl
LD_PRELOAD=./fail_pthread_create.so ./repro
The process aborts at the first export, roughly 10 seconds in.
The same crash is reachable without any fault injection under genuine memory pressure, via the
allocating operations listed in "Additional context".
What is the expected behavior?
The export fails, the failure is reported through GlobalLogHandler, and the application keeps
running.
The error-handling specification
requires this. Four of its requirements apply:
- Implementations must not throw unhandled exceptions at runtime.
- The SDK must not throw unhandled exceptions for failures in its own operations — the spec's own
illustrating example is an exporter that cannot reach the endpoint it sends telemetry to. - The API or SDK may fail fast at initialization, but must not cause the application to fail later
at runtime. - Background tasks should run in the context of a global error handler so their exceptions do not
affect the end user application.
PeriodicExportingMetricReader already implements this correctly:
sdk/src/metrics/export/periodic_exporting_metric_reader.cc:160-206 wraps the whole
collect-and-export call in try { ... } catch (std::exception &e) { OTEL_INTERNAL_LOG_ERROR(...); return false; } catch (...) { ... }.
What is the actual behavior?
std::terminate is called and the host process dies.
Frames #21 → #16 below are exactly the region inside that SDK try block. Because
Session::SendRequest at frame #11 is noexcept, the runtime terminates instead of unwinding,
so the SDK's handler never runs. The recovery code exists and is unreachable.
#8 __cxxabiv1::__terminate (handler=<optimized out>) at libsupc++/eh_terminate.cc:48
#9 std::terminate () at libsupc++/eh_terminate.cc:58
#10 __clang_call_terminate ()
#11 opentelemetry::v2::ext::http::client::curl::Session::SendRequest (this=0x7f4693211210, callback=...)
at ext/src/http/client/curl/http_client_curl.cc:145
#12 opentelemetry::v2::exporter::otlp::OtlpHttpClient::addSession (this=0x7f46a4ee1280, session_data=...)
at exporters/otlp/src/otlp_http_client.cc:1028
#13 opentelemetry::v2::exporter::otlp::OtlpHttpClient::Export (this=0x7f46a4ee1280, message=...,
result_callback=..., max_running_requests=0)
at exporters/otlp/src/otlp_http_client.cc:759
#14 opentelemetry::v2::exporter::otlp::OtlpHttpClient::Export (this=0x7f46a4ee1280, message=...)
at exporters/otlp/src/otlp_http_client.cc:725
#15 opentelemetry::v2::exporter::otlp::OtlpHttpMetricExporter::Export (this=0x7f46a4e0c780, data=...)
at exporters/otlp/src/otlp_http_metric_exporter.cc:204
#16 opentelemetry::v2::sdk::metrics::PeriodicExportingMetricReader::CollectAndExportOnce()::$_0::operator()(
opentelemetry::v2::sdk::metrics::ResourceMetrics&) const (this=0x7f46a057a120, metric_data=...)
at sdk/src/metrics/export/periodic_exporting_metric_reader.cc:181
#17 opentelemetry::v2::nostd::function_ref<...>::BindTo<...>(...)::{lambda(void*, ResourceMetrics&)#1}::operator()
at api/include/opentelemetry/nostd/function_ref.h:41
#18 opentelemetry::v2::nostd::function_ref<...>::BindTo<...>(...)::{lambda(void*, ResourceMetrics&)#1}::__invoke
at api/include/opentelemetry/nostd/function_ref.h:40
#19 opentelemetry::v2::nostd::function_ref<bool(ResourceMetrics&)>::operator() (args=..., this=<optimized out>)
at api/include/opentelemetry/nostd/function_ref.h:89
#20 opentelemetry::v2::sdk::metrics::MetricReader::Collect (this=<optimized out>, callback=...)
at sdk/src/metrics/metric_reader.cc:46
#21 opentelemetry::v2::sdk::metrics::PeriodicExportingMetricReader::CollectAndExportOnce (this=0x7f46a4e65510)
at sdk/src/metrics/export/periodic_exporting_metric_reader.cc:172
#22 opentelemetry::v2::sdk::metrics::PeriodicExportingMetricReader::DoBackgroundWork (this=0x7f46a4e65510)
at sdk/src/metrics/export/periodic_exporting_metric_reader.cc:107
#23 std::execute_native_thread_routine (__p=<optimized out>) at libstdc++-v3/src/c++11/thread.cc:104
#24 start_thread () from /lib64/libpthread.so.0
#25 clone () from /lib64/libc.so.6
Frames #0-#7 were the host application's own std::set_terminate handler writing a core dump;
they are omitted as they carry no information about this bug.
Additional context
Note on the line attribution in frame #11. It reads http_client_curl.cc:145, which is
const auto &url = host_ + http_request_->uri_;. That is misleading. Clang emits one shared
__clang_call_terminate landing pad per noexcept function and its line-table entry points into
the function body rather than the throw site; line 145 is just the first potentially-throwing
expression. In our dump curl_operation_ was already non-null and session_id_ == 1, proving
new HttpOperation(...) at line 227 had completed and the throw came from line 232 or later.
Unguarded operations in Session::SendRequest (ext/src/http/client/curl/http_client_curl.cc:141-242).
None of these is wrapped, and the function is noexcept:
| Operation | Location | Throws |
|---|---|---|
std::function constructed from the lambda capturing this + a shared_ptr |
http_client_curl.cc:233 |
std::bad_alloc — the capture holds a non-trivially-copyable shared_ptr, so libstdc++ cannot apply its small-object optimization and must heap-allocate |
async_data_.reset(new AsyncData()); AsyncData holds a std::promise and std::future |
http_operation_curl.cc:1286, type at http_operation_curl.h:331-340 |
std::bad_alloc; std::promise construction allocates shared state and can also throw std::system_error |
pending_to_add_session_ids_.insert(...) under std::lock_guard<std::recursive_mutex> |
http_client_curl.cc:635-645 |
std::bad_alloc from the node allocation; std::system_error from lock() |
background_thread_.reset(new std::thread(...)) |
http_client_curl.cc:438-446 |
std::system_error when pthread_create fails (EAGAIN at a thread or RLIMIT_NPROC limit); also std::bad_alloc |
There is no try or catch anywhere in ext/src/http/client/curl/http_client_curl.cc or
ext/src/http/client/curl/http_operation_curl.cc, on v1.22.0 or on main.
The same defect on the trace path. We have a second occurrence from an earlier version, ABI v1,
with an identical shape: BatchSpanProcessor::DoBackgroundWork → BatchSpanProcessor::Export →
OtlpHttpExporter::Export → OtlpHttpClient::Export → OtlpHttpClient::addSession →
Session::SendRequest → __clang_call_terminate → std::terminate. So this is not specific to
the metrics exporter; it is the shared curl HTTP client.
Suggested fix, and why it is not a breaking change. noexcept is part of the public
pure-virtual contract:
// ext/include/opentelemetry/ext/http/client/http_client.h:359
virtual void SendRequest(std::shared_ptr<EventHandler>) noexcept = 0;
So the fix should honour the noexcept by catching internally, not remove it. No signature
change, no ABI impact, no change required of any implementor of the interface.
The correct recovery already exists inside the function. Lines 234-241 handle SendAsync
returning non-OK by emitting SessionState::CreateFailed through the callback and clearing
is_session_active_. That is a complete cleanup rather than just a log: ResponseHandler::OnEvent
sets need_stop (exporters/otlp/src/otlp_http_client.cc:155-163), which at :310-317 calls
Unbind(kFailure) (:320-340), which calls OtlpHttpClient::ReleaseSession (:858-882), moving
the session out of running_sessions_ and calling session_waker_.notify_all() to release the
synchronous export waiting at :772-777. A catch handler doing exactly what that else branch
does would restore spec-compliant behaviour.
Two cautions for whoever implements it:
- Calling
OTEL_INTERNAL_LOG_ERRORfrom the handler is risky. It dispatches to a user-supplied
LogHandlerand allocates, so in abad_alloc-triggered handler it can throw again and return
straight toterminate. The same applies to thestd::stringstreambuilt in
ResponseHandler::OnEvent'sCreateFailedcase. - The handler must leave
curl_operation_andasync_data_in a state theSessiondestructor
andHttpClient::CleanupSessioncan handle, since the throw may land partway through
SendAsyncwith a live curl easy handle that was never added to the multi handle. This is
adjacent to #4397.
HttpClient::MaybeSpawnBackgroundThread (http_client_curl.cc:438-446) should also guard its own
thread creation and return false on failure, mirroring what was done for the file exporter in
#3012.
Other unguarded noexcept functions on the same path. Same defect class, worth fixing together:
OtlpHttpClient::Export(exporters/otlp/src/otlp_http_client.cc:748-787) — best single outer
net, since it has a naturalreturn ExportResult::kFailure. It is not a substitute for the
SendRequestfix, though: catching only here would strand a session inrunning_sessions_and
hangForceFlush(:789-812waits forrunning_sessions_.empty()).OtlpHttpClient::addSession(:1008-1029) — therunning_sessions_[session.get()]insert at
:1022allocates.HttpClient::CreateSession(ext/src/http/client/curl/http_client_curl.cc:323-341) —
std::make_shared<Session>at:329/:332andsessions_.insertat:337. This function
already carries a// FIXME: Session may leak if it does not call SendRequestat line 339.
Related:
- #4397 — an asynchronous curl request whose
Setupfails leaks its easy handle and header list.
Same function family, resource cleanup rather than thenoexceptescape, but a fix for
SendRequestshould account for it. - #3012 — "Ignore exception when create thread in OTLP file exporter", the precedent for this
change, applied toexporters/otlp/src/otlp_file_client.cc:1546-1551only. The curl client was
missed.
- Ngôn ngữ chính
- C++
- Star
- 1.4k
- Fork
- 663
- Merge trung bình
- 1 ngày 10 giờ
- Pull request đã merge (30 ngày)
- 44
Chuẩn bị môi trường
Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.
- Không có Dockerfile hay tệp Docker Compose
- Có mẫu pull request
- Đọc hướng dẫn đóng góp
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Issue khác của open-telemetry/opentelemetry-cpp
-
Baggage::ToHeader/FromHeader use form-style `+` for space, which other W3C Baggage implementations read as a literal plusCó thể đã có người làm Có pull request liên kết đang mở hoặc đã được merge. Đang mởbug triage/accepted
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 88/100
open-telemetry/opentelemetry-cpp#4745 · 1 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
[CMake] Strengthen pkg-config installation tests to detect missing metadata and dependenciesĐang mởneeds-triage
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100
open-telemetry/opentelemetry-cpp#4697 · 1 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
TracerContext construction might result in a terminateCó thể làm lại được @akshat-lakhera đã nhận 37 ngày trước và không có pull request nào đang mở. Đang mởtriage/accepted
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 62/100
open-telemetry/opentelemetry-cpp#4684 · 2 bình luận · 2 reaction ·
Maintainer thường phản hồi trong vòng 1 ngày
-
[CI] Add Ubuntu 26.04 runners to the CI workflowCó thể làm lại được Pull request cho issue này đã bị đóng mà không được merge. Đang mởtriage/accepted
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 72/100
open-telemetry/opentelemetry-cpp#4596 · 2 bình luận · 1 reaction ·
Maintainer thường phản hồi trong vòng 1 ngày
-
[BUG] Resource::Create() throws bad_variant_access if process.executable.name isn't a stringCó thể làm lại được Pull request cho issue này đã bị đóng mà không được merge. Đang mởbug help wanted Stale triage/accepted
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 84/100
open-telemetry/opentelemetry-cpp#4535 · 2 bình luận · 2 reaction ·
Maintainer thường phản hồi trong vòng 1 ngày
Tất cả issue của open-telemetry/opentelemetry-cpp
Issue tương tự
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100
HarperFast/rocksdb-js#1118 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
bug triage
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 75/100
Maintainer thường phản hồi trong vòng 1 ngày
-
26.04 bug dark-mode Easy Hack impress symptom: visual
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100
CollaboraOnline/online#16421 · 1 bình luận ·
-
enhancement
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 63/100
telegramdesktop/tdesktop#31653 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
area/cs bug
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
ydb-platform/ydb#59881 · 1 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày