Feature: Add point-cloud embedding task and Uni3D model family

オープン
  • 何か問題がありますか？
#1,929 コメント 3 件 リアクション 0 件 担当者 2 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@Moviw がすでに取り組んでいます。

2026年11月4日 から。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python, pytorch
領域
ai, machine-learning

調査の方向性

Start by reading the existing image_to_embedding task contract and the PointNet family implementation, then trace how families build bundles and run through the native TensorRT runtime. The issue names modelzoo/uni3d-b/model.pt as the model source and calls for an E2E comparing embeddings with the official PyTorch reference and checking ModelNet40 top-1 agreement. Done means the points_to_embedding contract and Uni3D family are supported, with those E2E checks passing.

索引モデルが issue の本文から書いたものです。

説明

enhancement
Request type

New model or checkpoint support

Problem and use case

TensorRT-Model-Connect now has a point-cloud task (points_to_semantic_segmentation, added with the PointNet family in #1384), but no general-purpose point-cloud encoder. The repository already exposes image_to_embedding and text_to_embedding for CLIP-style models, while 3D point clouds have no equivalent embedding path.

3D-CLIP encoders such as Uni3D map a colored point cloud into the same embedding space as a CLIP text/image encoder. That enables zero-shot 3D classification, text-to-3D and image-to-3D retrieval, and a general 3D feature extractor for robotics and scene understanding, without training a task-specific head.

Proposed outcome
  • Add a points_to_embedding task contract that follows the existing image_to_embedding shape: points [N, C] in, one embedding vector out.
  • Add a uni3d family that builds the Uni3D point encoder into a bundle and runs it through the native runtime.
  • Add an E2E that compares the TensorRT embedding against the official PyTorch reference (cosine similarity) and checks zero-shot ModelNet40 top-1 agreement, using class-prompt text embeddings precomputed once and stored as a fixture. The CLIP text encoder stays outside the engine.
Model and target details
  • Model: Uni3D-B (BAAI/Uni3D, modelzoo/uni3d-b/model.pt, revision 3d8233b76aa350d72f6213ecd2123c2026b42355; code at baaivision/Uni3D 64e03c3c)
  • Input: 10,000 points with XYZ + RGB; 512 groups of 64 points
  • Output: 1024-d embedding in the EVA02-E-14-plus CLIP space
  • Precision: FP16 first; FP8 for the ViT blocks as a follow-up
  • Target: NVIDIA GPU, TensorRT native runtime

The encoder is a PointBERT-style tokenizer (farthest point sampling, kNN grouping, a mini-PointNet) followed by a standard EVA ViT. kNN is already torch.topk. The only custom CUDA op upstream is pointnet2_ops farthest point sampling, which I will replace with a plain PyTorch implementation that starts from index 0 like pointnet2_ops, keeping the whole network in standard ONNX ops with no plugins or family-owned kernels.

Alternatives considered

No response

Additional context

License: the upstream project is MIT licensed (baaivision/Uni3D#8).

Submission checks
  • I searched open and closed issues and found no duplicate request.
  • I removed secrets, private/internal evidence, personal paths, and restricted artifacts.
主要言語
Python
スター
288
フォーク
73
平均マージ
1日 7時間
マージ済み PR（30日）
275

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/TensorRT-Model-Connect のほかの issue

NVIDIA/TensorRT-Model-Connect の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。