Run Isaac ROS Foundationpose on custom dataset with ground truth segmentations

Open
  • Is something wrong?
#87 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
docker
Domain
computer-vision, robotics

Research direction

Start with the isaac_ros_foundationpose.launch.py entry point and the documented input topics: pose_estimation/image, pose_estimation/depth_image, pose_estimation/segmentation, and pose_estimation/camera_info. Reproduce the launch using the listed TensorRT engines and static-image scripts, then trace whether the node receives synchronized inputs; the issue is done when it produces pose output or reports a clear input/configuration error.

Written by the indexing model from the issue text.

Description

Hello! I've been trying to run Foundationpose for Isaac following the tutorial for Isaac 3.1 on a custom dataset, using the Isaac Docker image with tag aarch64-ros2_humble on a Jetson with Jetpack 6.2. I was able to run the examples using the rosbag successfully after converting the score, refine models to TensorRT engines but I am having trouble running the node for my dataset. More specifically, I am running the commands below (they are all executed in the same docker container spawned with ${ISAAC_ROS_WS}/src/isaac_ros_common/scripts/run_dev.sh) :

  • Run realsense camera: ros2 launch realsense2_camera rs_launch.py rgb_camera.profile:=640x480x30 depth_module.profile:=640x480x30
  • Run foundationpose ros node: ros2 launch isaac_ros_foundationpose isaac_ros_foundationpose.launch.py refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/IndustryShapes/textured_models/obj_000001/obj_000001.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/IndustryShapes/textured_models/obj_000001/texture.png launch_rviz:=true
  • Run a script to remap camera_info topic /camera/color/camera_info to /pose_estimation/camera_info.
  • Run a script to stream static images to the topics that foundationpose subscribes to, according to the documentation. More specifically I publish the RGB to pose_estimation/image, the depth image to pose_estimation/depth_image, and the ground truth mask to pose_estimation/segmentation (so I am not currently using DETR or any other detector/segmentation model).

Also the origin frame of the 3D mesh of my object is at the center of the mesh, as instructed. And I can see that the rgb, depth, segmentation topics are published correctly, as their visualizations on rviz are what I expect. However, I am not getting any pose (or error for that matter), on rviz or in any of the output topics (/output, pose_estimation/pose_matrix_output). Could anyone please indicate what I could be missing?

Dominant language
C++
Stars
517
Forks
57
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation

All issues in NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.