Livestream anonymisation: blur buildings and optimise the pipeline

Abierto
  • ¿Algo no está bien?
#41 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
20/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
audio-video-rtc, computer-vision

Línea de trabajo

Start with the linked pyro-engine issues #465, #466, and #467, which divide this work into study, model training and deployment, and pipeline optimisation. Read the listed datasets and model options, then measure the current 640x360, 10 fps stream on a station. Done means choosing blur classes, matching current FPS with the new model, measuring and merging useful pipeline optimisations, and closing the linked issues.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

computer-vision engine epic P2

Related to pyronear/roadmap#37 (Privacy mask: a fixed mask per camera. This epic is about blurring objects live).

Context & Objective

During a doubt removal, the station streams the camera live and blurs it on the fly. A YOLO11n (pyronear/yolov11n on Hugging Face, NCNN on the Pi) detects persons and vehicles, and we paint a black box on them. Buildings are not blurred: YOLO has no such class.

We want to blur buildings too, and other classes if useful (houses, gardens, pools), and keep the stream real time on a Raspberry Pi. Two parts: train a new model with these classes, as fast as today, and optimise the stream code in pyro-engine, which is simple today.

What has to be done

Issues in pyro-engine: pyronear/pyro-engine#465 (study), pyronear/pyro-engine#466 (train and deploy), pyronear/pyro-engine#467 (optimise the pipeline).

1. Study: models and datasets
  • Datasets with a building class: ADE20K (https://datasetninja.com/ade20k#class-sizes), COCO-Stuff, Cityscapes, Mapillary Vistas, Open Images. Our cameras look at landscapes from a tower, buildings are small and far. Most datasets are street level.
  • Models: YOLO11n retrained on a merged dataset, YOLO11n-seg or a small segmentation model, or a big open-vocabulary model to label our own images and train a small YOLO on them.
  • Choose the classes to blur and the target speed on the Pi (today: 640x360 at 10 fps, model input 640).
2. Train and deploy the new model
  • Build the dataset, train, export NCNN and ONNX, publish on Hugging Face.
  • Update Anonymizer in pyro-engine. Measure FPS and latency on a station.
3. Optimise the stream pipeline

Today: one ffmpeg decodes the RTSP to raw frames, Python runs the model on the last frame in a thread, and an encoder thread paints the boxes and sends the frames to a second ffmpeg (x264 to SRT). Leads:

  • Fewer copies per frame: give the raw frame to the model directly, paint the boxes in place.
  • Boxes are painted on the current frame, not on the frame they come from. A moving object can show for a few frames. Delay the encoder, or keep the boxes longer and make them bigger.
  • Hardware decode and encode on the Pi, or use a smaller camera substream.
  • Smaller model input, or run the model one frame out of N.
  • Measure before and after.

Done when

  • The study is done and the classes to blur are chosen
  • The new model runs on a station at the same FPS as today
  • The pipeline is measured and the useful optimisations are merged
  • The listed issues are closed

Theme Engine · Priority P2 · Period to be set

Lenguaje dominante
Sin datos de lenguaje
Estrellas
0
Forks
0
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pyronear/roadmap

Todos los issues de pyronear/roadmap

Issues similares

Más issues de Audio/Video & RTC

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.