isl-org/Open3D

Integrate DNN runtime for running inference in C++

Open

#6,778 创建于 2024年4月29日

在 GitHub 查看
 (6 评论) (0 反应) (1 负责人)C++ (2,151 fork)batch import
help wantedml

仓库指标

Star
 (10,036 star)
PR 合并指标
 (平均合并 23天 3小时) (30 天内合并 5 个 PR)

描述

Enable 3DML processing in C++

This allows an increasing number of 3D processing methods to be available to the Open3D C++ API.

Options:

  • libtorch [preferred]
  • onnx

Docs:

High level plan:

  • The workflow is that you can train a model in PyTorch, then torch.export() it to a .pt file on disk. This can then be loaded from a C++ program for inferece. See AOTInductor example above.
  • Add open3d::ml::model class with methods load_model and forward to load a model from disk and run the forward pass for inference.
  • The load_model function should:
    • dlopen libtorch, so that libtorch functions can be called from Open3D. This ensures that libtorch remains an optional requirement.
    • Follow the AOTInductor example. Note that the inputs will actually be Open3D tensors (on CPU or GPU). We will use DLPack to wrap these to PyTorch Tensors and provide to the run() function. The outputs will similarly be converted from PyTorch to Open3D with DLPack.
  • Test the integration with a very simple model, say just a small linear layer initialized with known weights (e.g. all ones). Check the output in PyTorch and in Open3D with a known input tensor (say all ones). Add this as a C++ unit test.
  • Next add a real world model. GeDI is a good candidate - this is the SoTA point cloud registration feature point descriptor and uses Open3D for processing. See demo.py. Port this example to C++. You will have to check that this model can be torch.export() ed. If not, we will have to pick a different model.

贡献者指南