Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Extremely slow message creation for large arrays in Python

Aperta
  • Qualcosa non va?
#156 3 commenti 1 reazione 1 assegnatario Vedi su GitHub

@hidmic ci sta già lavorando.

Dal 22/4/2022.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

The message serialization is very slow (> hundred seconds for large point clouds), when setting the data property of a PointCloud2 message in Python by passing a byte string (e.g. by NumPy asbytes()).

This does not happen if the PointCloud2 message constructor is used (see #155 for the inconsistent behavior) or if a array.array is passed (does not work with the constructor because of #176).

Bug report

Required Info:

  • Operating System:
    • Ubuntu 20.04
  • Installation type:
    • apt
  • Version or commit hash:
    • rolling
  • Client library (if applicable):
    • rclpy
Steps to reproduce issue
  • Create a large Numpy Array called nparr
  • Create a matching PointCloud2 message called msg with parameters matching the NumPy array format
  • Set the data msg.data = nparr.tobytes()
  • Time this process and compare it to setting the data directly in the constructor

The second one is much slower:

  • 2.7484822273254395 seconds # byte string set via the constructor
  • 412.08923530578613 seconds # byte string set via the property
Expected behavior

Both should be equally fast

Actual behavior

The second one is much slower, because an additional assertion is performed.
This assertion is skipped in the constructor because of the bug in #155. It therefore performs much better.

The constructor directly casts the data and sends it to the setter

self.data = array.array('B', kwargs.get('data', []))

The setter itself checks if the cast is already done and skips the casing, including the following assertion

assert \
                ((isinstance(value, Sequence) or
                  isinstance(value, Set) or
                  isinstance(value, UserList)) and
                 not isinstance(value, str) and
                 not isinstance(value, UserString) and
                 all(isinstance(v, int) for v in value) and
                 all(val >= 0 and val < 256 for val in value)), \
                "The 'data' field must be a set or sequence and each value of type 'int' and each unsigned integer in [0, 255]"

The following part of the assertion is critical because it massively slows it down by iterating >2x over all values.

all(isinstance(v, int) for v in value)  and
all(val >= 0 and val < 256 for val in value))

This happens only in __debug__ is set, but that is the case for many dev machines, and this effectively crashes the code by freezing for many seconds.

Implementation considerations

Drop the check for byte strings. As it does not make any sense in the case of this datatype ether way.

I honestly don't know how to change this since the templating seems quite complex. Some advice would be appreciated.

Issue moved from https://github.com/ros2/common_interfaces/issues/177

Lingua principale
EmberScript
Stelle
26
Fork
68
Merge medio
1g 9h
PR unite (30g)
3

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ros2/rosidl_python

Tutte le issue di ros2/rosidl_python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.