Livestream anonymisation: blur buildings and optimise the pipeline

Aperta
  • Qualcosa non va?
#41 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
audio-video-rtc, computer-vision

Direzione di ricerca

Start with the linked pyro-engine issues #465, #466, and #467, which divide this work into study, model training and deployment, and pipeline optimisation. Read the listed datasets and model options, then measure the current 640x360, 10 fps stream on a station. Done means choosing blur classes, matching current FPS with the new model, measuring and merging useful pipeline optimisations, and closing the linked issues.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

computer-vision engine epic P2

Related to pyronear/roadmap#37 (Privacy mask: a fixed mask per camera. This epic is about blurring objects live).

Context & Objective

During a doubt removal, the station streams the camera live and blurs it on the fly. A YOLO11n (pyronear/yolov11n on Hugging Face, NCNN on the Pi) detects persons and vehicles, and we paint a black box on them. Buildings are not blurred: YOLO has no such class.

We want to blur buildings too, and other classes if useful (houses, gardens, pools), and keep the stream real time on a Raspberry Pi. Two parts: train a new model with these classes, as fast as today, and optimise the stream code in pyro-engine, which is simple today.

What has to be done

Issues in pyro-engine: pyronear/pyro-engine#465 (study), pyronear/pyro-engine#466 (train and deploy), pyronear/pyro-engine#467 (optimise the pipeline).

1. Study: models and datasets
  • Datasets with a building class: ADE20K (https://datasetninja.com/ade20k#class-sizes), COCO-Stuff, Cityscapes, Mapillary Vistas, Open Images. Our cameras look at landscapes from a tower, buildings are small and far. Most datasets are street level.
  • Models: YOLO11n retrained on a merged dataset, YOLO11n-seg or a small segmentation model, or a big open-vocabulary model to label our own images and train a small YOLO on them.
  • Choose the classes to blur and the target speed on the Pi (today: 640x360 at 10 fps, model input 640).
2. Train and deploy the new model
  • Build the dataset, train, export NCNN and ONNX, publish on Hugging Face.
  • Update Anonymizer in pyro-engine. Measure FPS and latency on a station.
3. Optimise the stream pipeline

Today: one ffmpeg decodes the RTSP to raw frames, Python runs the model on the last frame in a thread, and an encoder thread paints the boxes and sends the frames to a second ffmpeg (x264 to SRT). Leads:

  • Fewer copies per frame: give the raw frame to the model directly, paint the boxes in place.
  • Boxes are painted on the current frame, not on the frame they come from. A moving object can show for a few frames. Delay the encoder, or keep the boxes longer and make them bigger.
  • Hardware decode and encode on the Pi, or use a smaller camera substream.
  • Smaller model input, or run the model one frame out of N.
  • Measure before and after.

Done when

  • The study is done and the classes to blur are chosen
  • The new model runs on a station at the same FPS as today
  • The pipeline is measured and the useful optimisations are merged
  • The listed issues are closed

Theme Engine · Priority P2 · Period to be set

Lingua principale
Nessun dato sulla lingua
Stelle
0
Fork
0
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pyronear/roadmap

Tutte le issue di pyronear/roadmap

Issue simili

Altre issue su Audio/Video & RTC

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.