Humble image: rclcpp with cherry-picked #2109 aborts action servers ("Taking data from action server but no ready event") - needs iron's take_data workaround (#2531)
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 38/100
調査の方向性
Start with docker/Dockerfile.ros2_humble and inspect the rclcpp build around the reported lines 168–186 to see how commit #2109 is applied. Compare the Humble rclcpp_action sources with ros2/rclcpp#2531, focusing on ServerBase::take_data(), ClientBase::take_data(), and their execute() methods. Done means the image includes the no-ready-event handling while retaining #2109; validate it against the linked reproduction and evidence.
索引モデルが issue の本文から書いたものです。
説明
Summary
docker/Dockerfile.ros2_humble rebuilds rclcpp with ros2/rclcpp@232262c (#2109, "Avoid losing waitable handles
while using MultiThreadedExecutor") - present since Isaac ROS 0.31.0 (DP3.1) through release-3.2 / v3.2-15
(lines 168-186 on release-3.2); not present in 4.x (Jazzy). Upstream Humble does not have #2109. With it, a
MultiThreadedExecutor can dispatch an action server whose event was already taken, and Humble's rclcpp_action
throws in that case (current message since 16.0.11, ros2/rclcpp#2635), so the process terminates:
terminate called after throwing an instance of 'std::runtime_error'
what(): Taking data from action server but no ready event
... rclcpp::executors::MultiThreadedExecutor::run -> Executor::get_next_ready_executable_from_map
-> rclcpp_action::ServerBase::take_data()
Iron, which has #2109 upstream, avoids this with ros2/rclcpp#2531 (take_data() returns nullptr and execute()
ignores null, server and client; iron's comment: "there is a known bug in iron, that take_data might be called
multiple times"). The Humble image has #2109 but not that workaround.
Environment
- isaac_ros_common v3.2-14 (fcf4d9e), image built with the standard
Dockerfile.ros2_humblelayers;
/opt/ros/humble/VERSIONlistsrclcpp https://github.com/ros2/rclcpp/commit/232262c02a1265830c7785b7547bd51e1124fcd8;
ros-humble-rclcpp 16.0.15-0jammy,ros-humble-rclcpp-action 16.0.15-1jammy - Jetson Orin NX 16 GB, JetPack 6.2.2 (L4T R36.5.0), aarch64
- controller_manager 2.54.2, joint_trajectory_controller 2.54.0 (apt), rmw_fastrtps_cpp 6.2.9
Reproduction
controller_manager's stock ros2_control_node + mock hardware + stock joint_trajectory_controller
(e.g. ur_robot_driver launch with use_fake_hardware:=true), and a client sending short FollowJointTrajectory
goals back to back (~170/min, every 10th pre-empted). Script below; logs for each row are in the gist linked under Evidence. Tested on aarch64 (Jetson); the x86 image
uses the same Dockerfile but was not tested.
| Libraries loaded by the node (same image, same test) | Duration | Goals | Aborts |
|---|---|---|---|
| image's rclcpp (16.0.15 + #2109) and rclcpp_action | 60 min | 10,306 | 6 |
| image's rclcpp, rclcpp_action 16.0.15 rebuilt from the upstream tag | 60 min | 10,463 | 6 |
| rclcpp + rclcpp_action 16.0.15 from the upstream tag (no #2109) | 60 min | 10,226 | 0 |
| rclcpp + rclcpp_action 16.0.21 from the upstream tag (no #2109) | 60 min | 10,281 | 0 |
image's rclcpp, rclcpp_action 16.0.15 with ServerBase::take_data() returning nullptr (debug build with logging) |
120 min | 20,315 | 0; the no-event call happened 252 times, all 18,284 awaited goals succeeded |
In a real application (ur_robot_driver 2.14.0 + MoveIt + URSim, ~8 goals/min) it aborted the driver 8 times in
16,192 accepted goals; with the server-side workaround, 0 aborts in a 24 h / 11,385-goal run.
Mechanism (diagnostic build, 120 min, 20,415 goals: 232 no-event take_data() calls; the first 40 fully dumped, all show the same sequence)
With #2109, AllocatorMemoryStrategy::remove_null_handles() moves every ready waitable into
waitable_triggered_handles_, get_next_waitable() takes from that list, but clear_handles() does not clear it.
- Thread A's
wait_for_workmarks the action server ready (a result request or a goal expiry), so it is appended
to the list. A does not take it. Our inference: A's next executable is a timer (timers are checked before
waitables), probably JTC's goal-handle timer in the same mutually exclusive callback group, so the server stays
in the list. - Thread B's wait marks the same server ready again and appends it a second time; B takes the first copy and
executes the event. - Thread C later takes the stale duplicate and calls
take_data()with no ready event -> throw.
Event types: result request (27/40) and goal expiry (13/40). The dumps are in the gist. Note: in the dumps, TAKE values were
printed with the READY name table; a TAKE shown as CancelService is ResultService.
Related
NVIDIA forum topic 358399 (Jan 2026, UR + MoveIt, same crash) requests backporting ros2/rclcpp@82ec3f0 (#2635).
That is already in 16.0.15 and only changes the message; with #2109 present the abort remains.
Likely the same cause: isaac_ros_visual_slam#155, isaac_ros_nvblox#114 (action client variant).
Suggested fix (any of)
- (Recommended; keeps #2109's fix for ros2/rclcpp#2012) Also apply iron's behaviour to rclcpp_action in the image:
ServerBase::take_data()/ClientBase::take_data()returnnullptron no ready event andexecute()returns
on null (ros2/rclcpp#2531). We run the server-side part of this as an overlay; the diff below (~25 lines,
against ros2/rclcpphumble) also covers the client. - Or clear / de-duplicate
waitable_triggered_handles_in the backport. - Dropping #2109 would remove the abort, but would likely bring back ros2/rclcpp#2012 (missed waitables /
intra-process messages in MultiThreadedExecutor), so it is not recommended unless that is no longer needed.
Evidence
Logs for each table row and the diagnostic dumps: https://gist.github.com/butt2rflow/05dd59d0875a4ef54cb9455e0336fec2
Suggested diff for rclcpp_action (server and client)
diff --git a/rclcpp_action/src/client.cpp b/rclcpp_action/src/client.cpp
index 9c644f8..ca69697 100644
--- a/rclcpp_action/src/client.cpp
+++ b/rclcpp_action/src/client.cpp
@@ -23,6 +23,8 @@
#include "rcl_action/action_client.h"
#include "rcl_action/wait.h"
+#include "rclcpp/clock.hpp"
+#include "rclcpp/logging.hpp"
#include "rclcpp/node_interfaces/node_base_interface.hpp"
#include "rclcpp/node_interfaces/node_logging_interface.hpp"
@@ -652,7 +654,15 @@ ClientBase::take_data()
size_t next_ready_event = pimpl_->next_ready_event.exchange(ClientBaseImpl::NO_EVENT_READY);
if (next_ready_event == ClientBaseImpl::NO_EVENT_READY) {
- throw std::runtime_error("Taking data from action client but no ready event");
+ // On humble a MultiThreadedExecutor has been observed to call take_data() with no event
+ // pending. Throwing there terminates the process, so log it and return nullptr;
+ // execute() ignores it. Iron does the same without logging (#2531).
+ // A steady clock keeps the warning visible under use_sim_time.
+ static rclcpp::Clock steady_clock(RCL_STEADY_TIME);
+ RCLCPP_WARN_THROTTLE(
+ get_logger(), steady_clock, 5000,
+ "Taking data from action client but no ready event; ignoring");
+ return nullptr;
}
return take_data_by_entity_id(next_ready_event);
@@ -753,7 +763,8 @@ void
ClientBase::execute(std::shared_ptr<void> & data_in)
{
if (!data_in) {
- throw std::runtime_error("Executing action client but 'data' is empty");
+ // take_data() returned nullptr (no ready event); nothing to execute.
+ return;
}
std::shared_ptr<ClientBaseData> data_ptr = std::static_pointer_cast<ClientBaseData>(data_in);
diff --git a/rclcpp_action/src/server.cpp b/rclcpp_action/src/server.cpp
index 065e40b..158f707 100644
--- a/rclcpp_action/src/server.cpp
+++ b/rclcpp_action/src/server.cpp
@@ -28,7 +28,9 @@
#include "action_msgs/msg/goal_status_array.hpp"
#include "action_msgs/srv/cancel_goal.hpp"
+#include "rclcpp/clock.hpp"
#include "rclcpp/exceptions.hpp"
+#include "rclcpp/logging.hpp"
#include "rclcpp_action/server.hpp"
using rclcpp_action::ServerBase;
@@ -275,7 +277,15 @@ ServerBase::take_data()
size_t next_ready_event = pimpl_->next_ready_event.exchange(ServerBaseImpl::NO_EVENT_READY);
if (next_ready_event == ServerBaseImpl::NO_EVENT_READY) {
- throw std::runtime_error("Taking data from action server but no ready event");
+ // On humble a MultiThreadedExecutor has been observed to call take_data() with no event
+ // pending. Throwing there terminates the process, so log it and return nullptr;
+ // execute() ignores it. Iron does the same without logging (#2531).
+ // A steady clock keeps the warning visible under use_sim_time.
+ static rclcpp::Clock steady_clock(RCL_STEADY_TIME);
+ RCLCPP_WARN_THROTTLE(
+ pimpl_->logger_, steady_clock, 5000,
+ "Taking data from action server but no ready event; ignoring");
+ return nullptr;
}
return take_data_by_entity_id(next_ready_event);
@@ -364,7 +374,8 @@ void
ServerBase::execute(std::shared_ptr<void> & data_in)
{
if (!data_in) {
- throw std::runtime_error("Executing action server but 'data' is empty");
+ // take_data() returned nullptr (no ready event); nothing to execute.
+ return;
}
std::shared_ptr<ServerBaseData> data_ptr = std::static_pointer_cast<ServerBaseData>(data_in);
fakehw_goals.py (goal client used for the table)
"""Goal hammer for the fake-hardware repro (2026-10-10): stream FollowJointTrajectory goals at the STOCK
joint_trajectory_controller of a stock ros2_control_node (mock hardware), as fast as they complete, with a pre-empting
goal every PREEMPT-th goal (a new goal sent while one is executing). No UR code, no robot.
python3 fakehw_goals.py <out.csv> [minutes=180] [action=/joint_trajectory_controller/follow_joint_trajectory]
Writes one CSV row per goal (time, n, status, accept_s, total_s) and prints a progress line every 500 goals."""
import sys, time, csv, threading, rclpy
from rclpy.node import Node
from rclpy.action import ActionClient
from rclpy.executors import MultiThreadedExecutor
from control_msgs.action import FollowJointTrajectory
from trajectory_msgs.msg import JointTrajectoryPoint
from builtin_interfaces.msg import Duration
OUT = sys.argv[1]; MINUTES = float(sys.argv[2]) if len(sys.argv) > 2 else 180.0
ACTION = sys.argv[3] if len(sys.argv) > 3 else "/joint_trajectory_controller/follow_joint_trajectory"
PREEMPT = 10
J = ["shoulder_pan_joint", "shoulder_lift_joint", "elbow_joint", "wrist_1_joint", "wrist_2_joint", "wrist_3_joint"]
A = [0.0, -1.57, 1.57, -1.57, -1.57, 0.0]
rclpy.init(); n = Node("fakehw_goals"); ac = ActionClient(n, FollowJointTrajectory, ACTION)
ex = MultiThreadedExecutor(); ex.add_node(n); threading.Thread(target=ex.spin, daemon=True).start()
def wait(fut, to):
t = time.time()
while not fut.done() and time.time() - t < to: time.sleep(0.002)
return fut.result() if fut.done() else None
def goal(k):
g = FollowJointTrajectory.Goal(); g.trajectory.joint_names = J
for i, dt in enumerate((0.15, 0.30)):
p = JointTrajectoryPoint(); s = 0.05 * (1 if (k + i) % 2 else -1)
p.positions = [a + s for a in A]; p.time_from_start = Duration(sec=0, nanosec=int(dt * 1e9)); g.trajectory.points.append(p)
return g
if not ac.wait_for_server(timeout_sec=60):
print("NO ACTION SERVER " + ACTION, flush=True); sys.exit(2)
f = open(OUT, "w", newline=""); w = csv.writer(f); w.writerow(["time", "n", "status", "accept_s", "total_s"])
t_end = time.time() + MINUTES * 60; k = 0; ok = 0; bad = 0
while time.time() < t_end:
k += 1; t = time.time(); h = wait(ac.send_goal_async(goal(k)), 5.0); ta = time.time() - t
if h is None or not h.accepted:
st = "no_accept" if h is None else "rejected"; bad += 1
w.writerow([round(t, 3), k, st, round(ta, 3), round(time.time() - t, 3)]); f.flush()
if h is None and not ac.server_is_ready(): # server gone (node died): wait for it to come back
print("%s server gone after %d goals" % (time.strftime("%H:%M:%S"), k), flush=True)
ac.wait_for_server(timeout_sec=120)
continue
if k % PREEMPT == 0: # pre-empt: next goal while this one executes
time.sleep(0.05); continue
r = wait(h.get_result_async(), 5.0)
st = "ok" if r is not None else "no_result"; ok += st == "ok"; bad += st != "ok"
w.writerow([round(t, 3), k, st, round(ta, 3), round(time.time() - t, 3)])
if k % 500 == 0:
f.flush(); print("%s goals %d ok %d bad %d" % (time.strftime("%H:%M:%S"), k, ok, bad), flush=True)
f.close(); print("DONE goals %d ok %d bad %d" % (k, ok, bad), flush=True)
Generated by Generative AI: drafted with Claude (Anthropic, Claude Opus 5.5); data and testing by the reporter.
- 主要言語
- C++
- スター
- 324
- フォーク
- 229
- 平均マージ
- 1時間 4分
- マージ済み PR(30日)
- 1
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
NVIDIA-ISAAC-ROS/isaac_ros_common のほかの issue
-
`version_info.yaml` is written into the source tree for `ament_python` packages対応中かも @Daniiiil1 が 105 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 86/100
-
難易度 4/5 3〜5日 初心者へのやさしさ 45/100
-
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
-
Release 3.2 Dockerfile.base numpy version mismatch再び着手できるかも @swapnesh-wani-nvidia が 204 日前に担当しましたが、オープン中のプルリクエストはありません。 オープンbug
NVIDIA-ISAAC-ROS/isaac_ros_common#235 · コメント 1 件 · 担当者 1 名 ·
-
Isaac ROS Dev Base Catalog Page says it's Humble not Jazzy再び着手できるかも @chengronglai が 254 日前に担当しましたが、オープン中のプルリクエストはありません。 オープンdocumentation
NVIDIA-ISAAC-ROS/isaac_ros_common#232 · 担当者 1 名 ·
NVIDIA-ISAAC-ROS/isaac_ros_common の issue をすべて見る
似ている issue
-
bug
難易度 1/5 1時間未満 初心者へのやさしさ 85/100
メンテナーはふだん 1 日以内に返信
-
external
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
ydb-platform/ydb#60347 ·
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
BZFlag-Dev/bzflag#405 ·
-
bug: unconfirmed status: needs triage
難易度 2/5 1〜3時間 初心者へのやさしさ 65/100
PrismLauncher/PrismLauncher#6525 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
Bug
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
Warzone2100/warzone2100#5163 ·
メンテナーはふだん 1 日以内に返信