TODO

Description

This is a half-semester project, an ongoing attempt to extend the original work by:

1. Using LIDAR point cloud data instead of RGB-D data
2. Using bird's eye view representation of LIDAR data to generate additional 3D bounding box proposals from the top view

The below figure shows the proposed modification to the original work. The components (at the top) connected by red arrows are what we are going to add.

The purpose is to experiment on and try using the bird's eye view LIDAR data to help improving the performance in some extreme condition, such as low light or occlusion, where it may be difficult to detect the 2D bounding boxes in the image (and hence 3D object in the original framework). By using the bird's eye view LIDAR data, we expect to obtain additional 3D box proposals, which will be combined with the original proposals to improve the performance.

TODO

Please note that there is a priority (the number in square box) which indicates what urgent the most.

• 1. Read related papers and get the ideas out of it. If still unclear about how/why we are going to combine the ideas from these 2 papers, we can have a short Q&A or short presentation as @luulinh90s suggested.
  • Frustum PointNets for 3D Object Detection from RGB-D Data
  • PIXOR: Real-Time 3D Object Detection From Point Clouds
• 2. Download KITTI dataset.
  • 2.1 Download
    • Velodyne point clouds (29 GB)
    • Training labels of object data set (5 MB)
    • Camera calibration matrices of object data set (16 MB)
    • Left color images of object data set (12 GB)
  • 2.2 Explore the dataset format #7
• 3. Get the code up and running (e.g., test that the library is working fine and run some quick demo).
  • Dataset
  • mayavi
  • TensorFlow custom operators
• 4. Try to understand the codebase of Frustum PointNet; just getting the general ideas out of it.
  • 4.1 Understand the data preparation pipeline.
  • 4.2 Understand how the model is evaluated.
  • 4.3 Understand the differences between the level of "difficulties" defined in KITTI: easy, hard, moderate.
• 5. Check the PIXOR implementations and make it work. We have to either implement a similar model in TensorFlow at step 6.2, or just use the PyTorch version (decide later).
  • 5.1 https://github.com/philip-huang/PIXOR :one:
  • 5.2 https://github.com/ankita-kalra/PIXOR :two:
  • 5.3 https://github.com/overfitover/pixor_pytorch :three: @witignite
• 6. Train a network to propose 3D bounding boxes from the bird's eye view (BEV) representation of LIDAR point cloud data.
  • 6.1 Project the LIDAR point cloud into BEV. :four:
  • 6.2 Construct and train a single-stage network (either in TensorFlow or PyTorch) to generate 3D bounding boxes from BEV of point cloud. This will be difficult. :five:
  • 6.3 Combine the result from 6.2 to our baseline network model. :six:
  • 6.4 Evaluate the performance.

Important Dates

• 18 Oct: Proposal submission
• 14 Nov: Progress presentation
• ? Dec: Final presentation

Environment

• The current code is tested with the following environment
  • g++ 5.4.0
  • CUDA 9.0
  • Boost 1.77.0 (I built from source, although you can install via sudo apt-get)
  • Anaconda 4.6.11
  • Python 3.6.9
  • tensorflow-gpu 1.12.0
  • NumPy 1.17.2
  • mayavi 4.6.2
• From environment.yaml I don't recommend to install Python libraries from my environment.yaml file because there are some other components not used in this project; it's not a clean environment. Also, I mix conda with pip, which is not safe.
• Manual The following installation command should suffice:

  conda create -n py36 python=3.6 numpy scipy matplotlib statsmodels pandas pyyaml mayavi
  

  And in this environment, install:

  conda install opencv -c conda-forge
  pip install tensorflow-gpu==1.12.0
  

• Important Note:
  • Mixing pip with conda packages is not safe.
  • You need to install the correct cudatoolkit version. Do check your GPU compatibility and CUDA version.

---

Google Sheets

For more thorough details such as who is working on which step, progress status, etc. https://docs.google.com/spreadsheets/d/1XSqLmRnz4X9YVjgq7cah004tj2bg_MMFn16CfM0oWJI/edit#gid=0