open-mmlab/mmaction2

[Feature] Can work on Depth map from Depth Camera?

Open

#2566 opened on Jun 27, 2023

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Python (3,648 stars) (1,137 forks)batch import
good first issue

Description

What is the problem this feature will solve?

MMAction2 is so perfect to predict the kone points and the person position. But the kone points detection method for Depth Camera is not good so that i can't wholly trust the result of the old method like Kinect. I want MMAction2 to work on Depth Camera.

What is the feature?

Consider Depth map and RGB photo as input to detect kone point and person.

What alternatives have you considered?

No response

Contributor guide

Tech stack
pythonpytorch
Domain
machine learningai
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Familiarity with MMAction2 codebaseUnderstanding of depth camerasKnowledge of PyTorch and model input modifications
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
First, examine how MMAction2 handles RGB frames for keypoint detection in configs and models (e.g., configs/detection/). Then, research methods for fusing depth maps with RGB, such as adding a depth channel or using a separate stream. Modify the data pipeline (e.g., mmaction/datasets/pipelines/) and model forward method to accept depth maps. Validate with depth camera datasets like NYU Depth V2 or custom Kinect data.