Activity/Workflow TaskProcessor opens a brand-new gRPC channel per task, causing native grpc-core crashes (SIGABRT) under load

Leggi lib/temporal/activity/poller.rb intorno alle righe 110-114 e lib/temporal/activity/task_processor.rb intorno alle righe 71-73, poi traccia come il poller e il task processor ottengono le loro connessioni. Il lavoro è completato quando i task processors possono riutilizzare la connessione di lunga durata del poller senza modificare il comportamento esistente per i callers che non ne forniscono una; esegui la suite di test pertinente e verifica che l'elaborazione concorrente dei task sia ancora coperta.

Environment

• temporal-ruby: current master (b5efd2cef8)
• grpc gem: 1.66.0
• Ruby 3.3.6, Rails 7

What happened

Our Rails-based Temporal workers (Temporal::Worker, activity_thread_pool_size: 10, workflow_thread_pool_size: 6) crash intermittently with exit code 134 (SIGABRT) — a native abort inside the grpc gem's C++ core, uncatchable from Ruby. Two different internal failure signatures observed on different occasions:

terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_S_construct null not valid
terminate called recursively
Aborted (core dumped)

and, separately:

F0000 ... work_stealing_thread_pool.cc:186] Check failed: pool_->IsQuiesced()
*** Check failure stack trace: ***
Aborted (core dumped)

Root cause

Activity::Poller#process(task) / Workflow::Poller#process(task) construct a brand-new TaskProcessor for every polled task:

https://github.com/coinbase/temporal-ruby/blob/b5efd2cef802be2fa97d5bab04839413726ac06e/lib/temporal/activity/poller.rb#L110-L114

TaskProcessor#connection memoizes its own Temporal::Connection::GRPC:

https://github.com/coinbase/temporal-ruby/blob/b5efd2cef802be2fa97d5bab04839413726ac06e/lib/temporal/activity/task_processor.rb#L71-L73

So every single activity/workflow task opens a brand-new gRPC channel (fresh DNS resolution, fresh TLS handshake, fresh subchannels/LB policy) and tears it down again right after finishing. Under load (several tasks/sec with a non-trivial activity_thread_pool_size), this produces heavy concurrent gRPC channel churn. We confirmed this directly via GRPC_TRACE=call_error,client_channel: 48 distinct channel handles, 33 creating client_channel / 89 destroying subchannel wrapper lines in a single ~3 minute window on one worker pod.

That churn races grpc-core's internal C++ lifecycle bookkeeping (subchannel refcounting, LB policy teardown, and the EventEngine thread pool's shutdown/quiescence accounting) and trips different fatal internal assertions depending on timing — which is why we saw two different crash signatures for what appears to be the same underlying stressor.

This may also explain, or be related to, #291 ("Unable to poll" / GRPC::Unavailable: Socket closed errors happening frequently under similar thread-pool concurrency), and possibly #280.

Proposed fix

TaskProcessor should reuse the Poller's own long-lived connection instead of building its own per task. gRPC channels are explicitly designed to be shared across concurrent calls, so this is safe even with several TaskProcessors running concurrently on the poller's thread pool — the connection's only Mutex (poll_mutex) guards solely the long-poll bookkeeping (poll_activity_task_queue/poll_workflow_task_queue/cancel_polling_request), not the respond_*_task_completed/respond_*_task_failed calls concurrent task processors make, so no new lock contention is introduced by sharing.

We've deployed this exact fix downstream (as a monkeypatch, since we can't modify the gem source directly in our app) and confirmed 0 crashes over a multi-day soak in an environment that was previously crash-looping every ~10 minutes to a few hours.

Happy to open a PR with this fix — backward-compatible, adds an optional connection: keyword arg to TaskProcessor#initialize defaulting to nil, so existing behavior is unchanged for anyone not passing it. Let me know if that's welcome.