MicroROS suddenly stopped working. Not enough memory in the buffer stream AND vector larger than max_size()

Open
#748 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
docker, linux
Domain
cli, embedded-iot

Research direction

Start with the micro-ros-agent serial --dev /dev/ttyACM0 baudrate=115200 -v6 entry point and compare its native behavior with the working microros/micro-ros-agent:jazzy Docker command. Trace the failure after datawriter created; done means the native agent establishes the session and creates the datawriter without either exception.

Written by the indexing model from the issue text.

Description

The problem

I was using microROS to test some hardware. I was publishing some int32 values with ros2 topic pub and the microcontroller (Raspberry Pi Pico) decoded those values and acted accordingly.
All had been working well for several days of short tests. Then the day I started running longer tests, the microcontroller had been working all morning without interrupting the connection to microROS.
Then when I went for lunch and I didn't interrupt the connection on purpose, I just left the computer on because in the real-life scenario the hardware will be working for hours so if there was an issue I wanted to know. Careful what you wish for.
When I came back the connection had stopped with a buffer message error. Without thinking much of it I tried to restart the connection but now this error appears.

terminate called after throwing an instance of 'eprosima::fastcdr::exception::NotEnoughMemoryException'
  what():  Not enough memory in the buffer stream
Aborted

Ever since I get the same error no matter the microcontroller or the code. I restarted the computer, changed the microcontroller, even changed the code to other code that had been tested before without issues and even tried the example code. Nothing works anymore. They all throw the same memory error after showing "datawriter created".
I connect using the command
micro-ros-agent serial --dev /dev/ttyACM0 baudrate=115200
I thought of trying a different method with
ros2 run micro_ros_agent micro_ros_agent serial --dev /dev/ttyACM0
This generates a different error after "session established" it throws

terminate called after throwing an instance of 'std::length_error'
  what():  cannot create std::vector larger than max_size()
[ros2run]: Aborted

Any ideas on how to solve this? even a quick fix so I can finish the tests would be greatly appreciated.

System description: Lenovo intel core i7 running ROS2 Humble, no RTOS.

UPDATE:
I tried re-installing the microROS agent with:

sudo snap remove micro-ros-agent
sudo snap install micro-ros-agent
sudo snap set core experimental.hotplug=true
sudo systemctl restart snapd
snap connect micro-ros-agent:serial-port snapd:pico

but it didn´t work.
I tried using the micro-ros-agent Docker:

docker run -it --rm -v /dev:/dev --privileged --net=host microros/micro-ros-agent:jazzy serial --dev /dev/ttyACM0 -b 115200

And this works. So what is the issue here? any ideas on what to re-install to have it working without docker?

This is what it shows after
sudo micro-ros-agent serial --dev /dev/ttyACM0 baudrate=115200 -v6

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
[1733135915.478349] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[1733135915.478578] info     | Root.cpp           | set_verbose_level        | logger setup           | verbose_level: 6
[1733135915.604772] info     | Root.cpp           | create_client            | create                 | client_key: 0x50C0CFC1, session_id: 0x81
[1733135915.604883] info     | SessionManager.hpp | establish_session        | session established    | client_key: 0x50C0CFC1, address: 0
[1733135915.605051] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50C0CFC1, len: 19, data: 
0000: 81 00 00 00 04 01 0B 00 00 00 58 52 43 45 01 00 01 0F 00
[1733135915.607049] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x50C0CFC1, len: 40, data: 
0000: 81 80 00 00 01 07 20 00 00 0A 00 01 01 03 00 00 12 00 00 00 00 01 00 20 0A 00 00 00 70 69 63 6F
0020: 5F 6E 6F 64 65 00 00 00
[1733135915.618144] info     | ProxyClient.cpp    | create_participant       | participant created    | client_key: 0x50C0CFC1, participant_id: 0x000(1)
[1733135915.618203] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50C0CFC1, len: 14, data: 
0000: 81 80 00 00 05 01 06 00 00 0A 00 01 00 00
[1733135915.618221] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50C0CFC1, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 01 00 00 00 80
[1733135915.619209] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x50C0CFC1, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 01 00 00 00 80
[1733135915.621683] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x50C0CFC1, len: 80, data: 
0000: 81 80 01 00 01 07 46 00 00 0B 00 02 02 03 00 00 38 00 00 00 12 00 00 00 72 74 2F 70 69 63 6F 5F
0020: 70 75 62 6C 69 73 68 65 72 00 00 01 1C 00 00 00 73 74 64 5F 6D 73 67 73 3A 3A 6D 73 67 3A 3A 64
0040: 64 73 5F 3A 3A 49 6E 74 33 32 5F 00 00 01 00 00
[1733135915.621779] info     | ProxyClient.cpp    | create_topic             | topic created          | client_key: 0x50C0CFC1, topic_id: 0x000(2), participant_id: 0x000(1)
[1733135915.621815] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50C0CFC1, len: 14, data: 
0000: 81 80 01 00 05 01 06 00 00 0B 00 02 00 00
[1733135915.621834] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50C0CFC1, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 02 00 00 00 80
[1733135915.623364] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x50C0CFC1, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 02 00 00 00 80
[1733135915.623797] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x50C0CFC1, len: 24, data: 
0000: 81 80 02 00 01 07 10 00 00 0C 00 03 03 03 00 00 02 00 00 00 00 00 00 01
[1733135915.623858] info     | ProxyClient.cpp    | create_publisher         | publisher created      | client_key: 0x50C0CFC1, publisher_id: 0x000(3), participant_id: 0x000(1)
[1733135915.623883] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50C0CFC1, len: 14, data: 
0000: 81 80 02 00 05 01 06 00 00 0C 00 03 00 00
[1733135915.623903] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50C0CFC1, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 03 00 00 00 80
[1733135915.625225] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x50C0CFC1, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 03 00 00 00 80
[1733135915.626267] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x50C0CFC1, len: 36, data: 
0000: 81 80 03 00 01 07 1C 00 00 0D 00 05 05 03 00 00 0E 00 00 00 00 02 01 10 03 00 01 00 0A 00 00 00
0020: 00 00 00 03
[1733135915.626542] info     | ProxyClient.cpp    | create_datawriter        | datawriter created     | client_key: 0x50C0CFC1, datawriter_id: 0x000(5), publisher_id: 0x000(3)
[1733135915.626578] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50C0CFC1, len: 14, data: 
0000: 81 80 03 00 05 01 06 00 00 0D 00 05 00 00
[1733135915.626599] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50C0CFC1, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 04 00 00 00 80
[1733135915.627597] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x50C0CFC1, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 04 00 00 00 80
terminate called after throwing an instance of 'eprosima::fastcdr::exception::NotEnoughMemoryException'
  what():  Not enough memory in the buffer stream
Aborted
Dominant language
Shell
Stars
509
Forks
183
Avg merge
22h 26m
Merged PRs (30d)
7

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from micro-ROS/micro_ros_setup

All issues in micro-ROS/micro_ros_setup

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.