RobotecAI/rai

integrate with LeRobot

Open

#345 opened on Dec 11, 2024

 (14 comments) (2 reactions) (0 assignees)Python (73 forks)auto 404
enhancementhelp wantedpriority/minor

Repository metrics

Stars
 (554 stars)
PR merge metrics
 (PR metrics pending)

Description

Is your feature request related to a problem? Please describe. Currently rai-manipulation-demo uses ros2 moveit and a fusion of segmentation with depth camera to control Panda robotic arm. This approach is good to for reasoning and planning, but might be limited when it comes to more complex movements of the gripper. For such task, policies implemented in LeRobot can be more useful.

LeRobot currently offers:

  • 4 pre-trained policies listed in Models section: link
  • scripts for policies training: link
  • dataset format for training policies: link

Describe the solution you'd like I find the integration of all LeRobot features mentioned above useful in rai, but I think a step-by-step should be applied.

  • First logical step would be to attach pre-trained policy to the rai agent as a tool.
  • Next would be to determine what policies would be useful for manipulation agent and to train them on datasets curated in simulation.
  • Next step would be to generate and record trajectories for training using rai and text descriptions

Describe alternatives you've considered

Additional context

Contributor guide