Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Activity/Workflow TaskProcessor opens a brand-new gRPC channel per task, causing native grpc-core crashes (SIGABRT) under load

Abierto Apto para principiantes
  • ¿Algo no está bien?
#361 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
82/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
grpc, ruby
Área
backend-api-design

Línea de trabajo

Lee lib/temporal/activity/poller.rb alrededor de las líneas 110-114 y lib/temporal/activity/task_processor.rb alrededor de las líneas 71-73; después, sigue el flujo para ver cómo el poller y el task processor obtienen sus conexiones. Se considera terminado cuando los task processors pueden reutilizar la conexión de larga duración del poller sin cambiar el comportamiento existente para los callers que no proporcionen una; ejecuta la suite de tests relevante y verifica que el procesamiento concurrente de tareas siga estando cubierto.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Environment

  • temporal-ruby: current master (b5efd2cef8)
  • grpc gem: 1.66.0
  • Ruby 3.3.6, Rails 7

What happened

Our Rails-based Temporal workers (Temporal::Worker, activity_thread_pool_size: 10, workflow_thread_pool_size: 6) crash intermittently with exit code 134 (SIGABRT) — a native abort inside the grpc gem's C++ core, uncatchable from Ruby. Two different internal failure signatures observed on different occasions:

terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_S_construct null not valid
terminate called recursively
Aborted (core dumped)

and, separately:

F0000 ... work_stealing_thread_pool.cc:186] Check failed: pool_->IsQuiesced()
*** Check failure stack trace: ***
Aborted (core dumped)

Root cause

Activity::Poller#process(task) / Workflow::Poller#process(task) construct a brand-new TaskProcessor for every polled task:

https://github.com/coinbase/temporal-ruby/blob/b5efd2cef802be2fa97d5bab04839413726ac06e/lib/temporal/activity/poller.rb#L110-L114

TaskProcessor#connection memoizes its own Temporal::Connection::GRPC:

https://github.com/coinbase/temporal-ruby/blob/b5efd2cef802be2fa97d5bab04839413726ac06e/lib/temporal/activity/task_processor.rb#L71-L73

So every single activity/workflow task opens a brand-new gRPC channel (fresh DNS resolution, fresh TLS handshake, fresh subchannels/LB policy) and tears it down again right after finishing. Under load (several tasks/sec with a non-trivial activity_thread_pool_size), this produces heavy concurrent gRPC channel churn. We confirmed this directly via GRPC_TRACE=call_error,client_channel: 48 distinct channel handles, 33 creating client_channel / 89 destroying subchannel wrapper lines in a single ~3 minute window on one worker pod.

That churn races grpc-core's internal C++ lifecycle bookkeeping (subchannel refcounting, LB policy teardown, and the EventEngine thread pool's shutdown/quiescence accounting) and trips different fatal internal assertions depending on timing — which is why we saw two different crash signatures for what appears to be the same underlying stressor.

This may also explain, or be related to, #291 ("Unable to poll" / GRPC::Unavailable: Socket closed errors happening frequently under similar thread-pool concurrency), and possibly #280.

Proposed fix

TaskProcessor should reuse the Poller's own long-lived connection instead of building its own per task. gRPC channels are explicitly designed to be shared across concurrent calls, so this is safe even with several TaskProcessors running concurrently on the poller's thread pool — the connection's only Mutex (poll_mutex) guards solely the long-poll bookkeeping (poll_activity_task_queue/poll_workflow_task_queue/cancel_polling_request), not the respond_*_task_completed/respond_*_task_failed calls concurrent task processors make, so no new lock contention is introduced by sharing.

We've deployed this exact fix downstream (as a monkeypatch, since we can't modify the gem source directly in our app) and confirmed 0 crashes over a multi-day soak in an environment that was previously crash-looping every ~10 minutes to a few hours.

Happy to open a PR with this fix — backward-compatible, adds an optional connection: keyword arg to TaskProcessor#initialize defaulting to nil, so existing behavior is unchanged for anyone not passing it. Let me know if that's welcome.

Lenguaje dominante
Ruby
Estrellas
288
Forks
113
Merge medio
10 d 15 h
PR fusionados (30 d)
2

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de coinbase/temporal-ruby

Todos los issues de coinbase/temporal-ruby

Issues similares

Más issues de Ruby

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.