cartographer-project/cartographer

Serialized SparsePoseGraphs can easily be too big for default proto deserialization

Open

#433 opened on Jul 27, 2017

View on GitHub
 (3 comments) (0 reactions) (0 assignees)C++ (6,665 stars) (2,188 forks)batch import
enhancementhelp wanted

Description

I'm working with some very large multi-trajectory SLAM problems (10s of kms, indoor), and the resulting serialized SparsePoseGraph objects are too big to deserialize, as they're much larger than the default 64MB limit. (Seriously: I'm up to about 1.3GB for the SPG alone.)

There are three solutions:

  1. Modify the serialization of SPG to use the pbstream, writing the pieces out one-by-one.
  2. Improve pbstream to use a google::protobuf::io::CodedInputStream directly, so we can call SetTotalBytesLimit with a bigger value.
  3. "Mac, you're out of your mind, make yourself a branch and leave the rest of us out of this."

(I've currently implemented 3, via 2, in macmason/cartographer and macmason/cartographer_ros, in branches both called mass_map.)

Thoughts?

FYI @brandon-northcutt

Contributor guide

Tech stack
cpp
Domain
backendperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
C++ programmingProtocol BuffersCartographer internals
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
The issue proposes three solutions: (1) modify serialization to use pbstream writing pieces individually, (2) improve pbstream to use CodedInputStream with SetTotalBytesLimit, or (3) fork. A working prototype is in the `mass map` branches at macmason/cartographer and macmason/cartographer ros. To proceed, examine the existing implementation of SparsePoseGraph serialization in `cartographer/mapping/sparse pose graph.cc` and the pbstream reader/writer to determine the cleanest approach. Engage with the community to decide on a direction before implementing.