Humble image: rclcpp with cherry-picked #2109 aborts action servers ("Taking data from action server but no ready event") - needs iron's take_data workaround (#2531)

Aperta
#245 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

@SIDDARTHAREDDY8 ci sta già lavorando.

Dal 13/11/2026.

  • #247 di @SIDDARTHAREDDY8 — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
backend, devtools

Direzione di ricerca

Start with docker/Dockerfile.ros2_humble and inspect the rclcpp build around the reported lines 168–186 to see how commit #2109 is applied. Compare the Humble rclcpp_action sources with ros2/rclcpp#2531, focusing on ServerBase::take_data(), ClientBase::take_data(), and their execute() methods. Done means the image includes the no-ready-event handling while retaining #2109; validate it against the linked reproduction and evidence.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

docker/Dockerfile.ros2_humble rebuilds rclcpp with ros2/rclcpp@232262c (#2109, "Avoid losing waitable handles
while using MultiThreadedExecutor") - present since Isaac ROS 0.31.0 (DP3.1) through release-3.2 / v3.2-15
(lines 168-186 on release-3.2); not present in 4.x (Jazzy). Upstream Humble does not have #2109. With it, a
MultiThreadedExecutor can dispatch an action server whose event was already taken, and Humble's rclcpp_action
throws in that case (current message since 16.0.11, ros2/rclcpp#2635), so the process terminates:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Taking data from action server but no ready event
  ... rclcpp::executors::MultiThreadedExecutor::run -> Executor::get_next_ready_executable_from_map
      -> rclcpp_action::ServerBase::take_data()

Iron, which has #2109 upstream, avoids this with ros2/rclcpp#2531 (take_data() returns nullptr and execute()
ignores null, server and client; iron's comment: "there is a known bug in iron, that take_data might be called
multiple times"). The Humble image has #2109 but not that workaround.

Environment
  • isaac_ros_common v3.2-14 (fcf4d9e), image built with the standard Dockerfile.ros2_humble layers;
    /opt/ros/humble/VERSION lists rclcpp https://github.com/ros2/rclcpp/commit/232262c02a1265830c7785b7547bd51e1124fcd8;
    ros-humble-rclcpp 16.0.15-0jammy, ros-humble-rclcpp-action 16.0.15-1jammy
  • Jetson Orin NX 16 GB, JetPack 6.2.2 (L4T R36.5.0), aarch64
  • controller_manager 2.54.2, joint_trajectory_controller 2.54.0 (apt), rmw_fastrtps_cpp 6.2.9
Reproduction

controller_manager's stock ros2_control_node + mock hardware + stock joint_trajectory_controller
(e.g. ur_robot_driver launch with use_fake_hardware:=true), and a client sending short FollowJointTrajectory
goals back to back (~170/min, every 10th pre-empted). Script below; logs for each row are in the gist linked under Evidence. Tested on aarch64 (Jetson); the x86 image
uses the same Dockerfile but was not tested.

Libraries loaded by the node (same image, same test) Duration Goals Aborts
image's rclcpp (16.0.15 + #2109) and rclcpp_action 60 min 10,306 6
image's rclcpp, rclcpp_action 16.0.15 rebuilt from the upstream tag 60 min 10,463 6
rclcpp + rclcpp_action 16.0.15 from the upstream tag (no #2109) 60 min 10,226 0
rclcpp + rclcpp_action 16.0.21 from the upstream tag (no #2109) 60 min 10,281 0
image's rclcpp, rclcpp_action 16.0.15 with ServerBase::take_data() returning nullptr (debug build with logging) 120 min 20,315 0; the no-event call happened 252 times, all 18,284 awaited goals succeeded

In a real application (ur_robot_driver 2.14.0 + MoveIt + URSim, ~8 goals/min) it aborted the driver 8 times in
16,192 accepted goals; with the server-side workaround, 0 aborts in a 24 h / 11,385-goal run.

Mechanism (diagnostic build, 120 min, 20,415 goals: 232 no-event take_data() calls; the first 40 fully dumped, all show the same sequence)

With #2109, AllocatorMemoryStrategy::remove_null_handles() moves every ready waitable into
waitable_triggered_handles_, get_next_waitable() takes from that list, but clear_handles() does not clear it.

  1. Thread A's wait_for_work marks the action server ready (a result request or a goal expiry), so it is appended
    to the list. A does not take it. Our inference: A's next executable is a timer (timers are checked before
    waitables), probably JTC's goal-handle timer in the same mutually exclusive callback group, so the server stays
    in the list.
  2. Thread B's wait marks the same server ready again and appends it a second time; B takes the first copy and
    executes the event.
  3. Thread C later takes the stale duplicate and calls take_data() with no ready event -> throw.

Event types: result request (27/40) and goal expiry (13/40). The dumps are in the gist. Note: in the dumps, TAKE values were
printed with the READY name table; a TAKE shown as CancelService is ResultService.

Related

NVIDIA forum topic 358399 (Jan 2026, UR + MoveIt, same crash) requests backporting ros2/rclcpp@82ec3f0 (#2635).
That is already in 16.0.15 and only changes the message; with #2109 present the abort remains.
Likely the same cause: isaac_ros_visual_slam#155, isaac_ros_nvblox#114 (action client variant).

Suggested fix (any of)
  1. (Recommended; keeps #2109's fix for ros2/rclcpp#2012) Also apply iron's behaviour to rclcpp_action in the image:
    ServerBase::take_data() / ClientBase::take_data() return nullptr on no ready event and execute() returns
    on null (ros2/rclcpp#2531). We run the server-side part of this as an overlay; the diff below (~25 lines,
    against ros2/rclcpp humble) also covers the client.
  2. Or clear / de-duplicate waitable_triggered_handles_ in the backport.
  3. Dropping #2109 would remove the abort, but would likely bring back ros2/rclcpp#2012 (missed waitables /
    intra-process messages in MultiThreadedExecutor), so it is not recommended unless that is no longer needed.
Evidence

Logs for each table row and the diagnostic dumps: https://gist.github.com/butt2rflow/05dd59d0875a4ef54cb9455e0336fec2

Suggested diff for rclcpp_action (server and client)
diff --git a/rclcpp_action/src/client.cpp b/rclcpp_action/src/client.cpp
index 9c644f8..ca69697 100644
--- a/rclcpp_action/src/client.cpp
+++ b/rclcpp_action/src/client.cpp
@@ -23,6 +23,8 @@
 
 #include "rcl_action/action_client.h"
 #include "rcl_action/wait.h"
+#include "rclcpp/clock.hpp"
+#include "rclcpp/logging.hpp"
 #include "rclcpp/node_interfaces/node_base_interface.hpp"
 #include "rclcpp/node_interfaces/node_logging_interface.hpp"
 
@@ -652,7 +654,15 @@ ClientBase::take_data()
   size_t next_ready_event = pimpl_->next_ready_event.exchange(ClientBaseImpl::NO_EVENT_READY);
 
   if (next_ready_event == ClientBaseImpl::NO_EVENT_READY) {
-    throw std::runtime_error("Taking data from action client but no ready event");
+    // On humble a MultiThreadedExecutor has been observed to call take_data() with no event
+    // pending. Throwing there terminates the process, so log it and return nullptr;
+    // execute() ignores it. Iron does the same without logging (#2531).
+    // A steady clock keeps the warning visible under use_sim_time.
+    static rclcpp::Clock steady_clock(RCL_STEADY_TIME);
+    RCLCPP_WARN_THROTTLE(
+      get_logger(), steady_clock, 5000,
+      "Taking data from action client but no ready event; ignoring");
+    return nullptr;
   }
 
   return take_data_by_entity_id(next_ready_event);
@@ -753,7 +763,8 @@ void
 ClientBase::execute(std::shared_ptr<void> & data_in)
 {
   if (!data_in) {
-    throw std::runtime_error("Executing action client but 'data' is empty");
+    // take_data() returned nullptr (no ready event); nothing to execute.
+    return;
   }
 
   std::shared_ptr<ClientBaseData> data_ptr = std::static_pointer_cast<ClientBaseData>(data_in);
diff --git a/rclcpp_action/src/server.cpp b/rclcpp_action/src/server.cpp
index 065e40b..158f707 100644
--- a/rclcpp_action/src/server.cpp
+++ b/rclcpp_action/src/server.cpp
@@ -28,7 +28,9 @@
 
 #include "action_msgs/msg/goal_status_array.hpp"
 #include "action_msgs/srv/cancel_goal.hpp"
+#include "rclcpp/clock.hpp"
 #include "rclcpp/exceptions.hpp"
+#include "rclcpp/logging.hpp"
 #include "rclcpp_action/server.hpp"
 
 using rclcpp_action::ServerBase;
@@ -275,7 +277,15 @@ ServerBase::take_data()
   size_t next_ready_event = pimpl_->next_ready_event.exchange(ServerBaseImpl::NO_EVENT_READY);
 
   if (next_ready_event == ServerBaseImpl::NO_EVENT_READY) {
-    throw std::runtime_error("Taking data from action server but no ready event");
+    // On humble a MultiThreadedExecutor has been observed to call take_data() with no event
+    // pending. Throwing there terminates the process, so log it and return nullptr;
+    // execute() ignores it. Iron does the same without logging (#2531).
+    // A steady clock keeps the warning visible under use_sim_time.
+    static rclcpp::Clock steady_clock(RCL_STEADY_TIME);
+    RCLCPP_WARN_THROTTLE(
+      pimpl_->logger_, steady_clock, 5000,
+      "Taking data from action server but no ready event; ignoring");
+    return nullptr;
   }
 
   return take_data_by_entity_id(next_ready_event);
@@ -364,7 +374,8 @@ void
 ServerBase::execute(std::shared_ptr<void> & data_in)
 {
   if (!data_in) {
-    throw std::runtime_error("Executing action server but 'data' is empty");
+    // take_data() returned nullptr (no ready event); nothing to execute.
+    return;
   }
 
   std::shared_ptr<ServerBaseData> data_ptr = std::static_pointer_cast<ServerBaseData>(data_in);
fakehw_goals.py (goal client used for the table)
"""Goal hammer for the fake-hardware repro (2026-10-10): stream FollowJointTrajectory goals at the STOCK
joint_trajectory_controller of a stock ros2_control_node (mock hardware), as fast as they complete, with a pre-empting
goal every PREEMPT-th goal (a new goal sent while one is executing). No UR code, no robot.
    python3 fakehw_goals.py <out.csv> [minutes=180] [action=/joint_trajectory_controller/follow_joint_trajectory]
Writes one CSV row per goal (time, n, status, accept_s, total_s) and prints a progress line every 500 goals."""
import sys, time, csv, threading, rclpy
from rclpy.node import Node
from rclpy.action import ActionClient
from rclpy.executors import MultiThreadedExecutor
from control_msgs.action import FollowJointTrajectory
from trajectory_msgs.msg import JointTrajectoryPoint
from builtin_interfaces.msg import Duration

OUT = sys.argv[1]; MINUTES = float(sys.argv[2]) if len(sys.argv) > 2 else 180.0
ACTION = sys.argv[3] if len(sys.argv) > 3 else "/joint_trajectory_controller/follow_joint_trajectory"
PREEMPT = 10
J = ["shoulder_pan_joint", "shoulder_lift_joint", "elbow_joint", "wrist_1_joint", "wrist_2_joint", "wrist_3_joint"]
A = [0.0, -1.57, 1.57, -1.57, -1.57, 0.0]

rclpy.init(); n = Node("fakehw_goals"); ac = ActionClient(n, FollowJointTrajectory, ACTION)
ex = MultiThreadedExecutor(); ex.add_node(n); threading.Thread(target=ex.spin, daemon=True).start()

def wait(fut, to):
    t = time.time()
    while not fut.done() and time.time() - t < to: time.sleep(0.002)
    return fut.result() if fut.done() else None

def goal(k):
    g = FollowJointTrajectory.Goal(); g.trajectory.joint_names = J
    for i, dt in enumerate((0.15, 0.30)):
        p = JointTrajectoryPoint(); s = 0.05 * (1 if (k + i) % 2 else -1)
        p.positions = [a + s for a in A]; p.time_from_start = Duration(sec=0, nanosec=int(dt * 1e9)); g.trajectory.points.append(p)
    return g

if not ac.wait_for_server(timeout_sec=60):
    print("NO ACTION SERVER " + ACTION, flush=True); sys.exit(2)
f = open(OUT, "w", newline=""); w = csv.writer(f); w.writerow(["time", "n", "status", "accept_s", "total_s"])
t_end = time.time() + MINUTES * 60; k = 0; ok = 0; bad = 0
while time.time() < t_end:
    k += 1; t = time.time(); h = wait(ac.send_goal_async(goal(k)), 5.0); ta = time.time() - t
    if h is None or not h.accepted:
        st = "no_accept" if h is None else "rejected"; bad += 1
        w.writerow([round(t, 3), k, st, round(ta, 3), round(time.time() - t, 3)]); f.flush()
        if h is None and not ac.server_is_ready():            # server gone (node died): wait for it to come back
            print("%s server gone after %d goals" % (time.strftime("%H:%M:%S"), k), flush=True)
            ac.wait_for_server(timeout_sec=120)
        continue
    if k % PREEMPT == 0:                                       # pre-empt: next goal while this one executes
        time.sleep(0.05); continue
    r = wait(h.get_result_async(), 5.0)
    st = "ok" if r is not None else "no_result"; ok += st == "ok"; bad += st != "ok"
    w.writerow([round(t, 3), k, st, round(ta, 3), round(time.time() - t, 3)])
    if k % 500 == 0:
        f.flush(); print("%s goals %d ok %d bad %d" % (time.strftime("%H:%M:%S"), k, ok, bad), flush=True)
f.close(); print("DONE goals %d ok %d bad %d" % (k, ok, bad), flush=True)

Generated by Generative AI: drafted with Claude (Anthropic, Claude Opus 5.5); data and testing by the reporter.

Lingua principale
C++
Stelle
324
Fork
229
Merge medio
1h 4m
PR unite (30g)
1

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA-ISAAC-ROS/isaac_ros_common

Tutte le issue di NVIDIA-ISAAC-ROS/isaac_ros_common

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.