Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Custom video dataset encoding/serialize uses all memory, process killed. How to fix?

Abierto
  • ¿Algo no está bien?
#5,499 10 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Tranquilo
Stack tecnológico
jupyter-notebook, python
Área
data-engineering, machine-learning

Línea de trabajo

Reproduce el crecimiento de memoria con los Colab notebooks enlazados e inspecciona dgs_corpus.py, el VideoFeature personalizado de config.py y las rutas citadas split_builder.py y writer.py. Rastrea dónde se extraen, codifican y serializan los fotogramas de vídeo y, a continuación, define un cambio específico con una prueba de memoria reproducible y un resultado claro de memoria acotada.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

help

What I need help with / What I was wondering

I want to load a dataset containing these
image
without this happening (Colab notebook for replicating)
image

...How can I edit my dataset loader to use less memory when encoding videos?

Background:
I am trying to load a custom dataset with a Video feature.
When I try to tfds.load() it, or even just download_and_prepare, RAM usage goes up very high and then the process gets killed.
For example this notebook will crash if allowed to run, though with a High-RAM instance it may not.
It seems it is using over 30GB of memory to encode one or two 10 MB videos.
I would like to know how to edit/update this custom dataset so that it will not use so much memory.

What I've tried so far
image

I did a bunch of debugging and tracing of the problem with memray, etc. See this notebook and this issue for detailed analysis including a copy of the memray report.

Tried various different ideas in the notebook, including loading just a slice, editing buffer size, and switching from .load() to download_and_prepare()

Finally I traced the problem to serializing and encoding steps under the
See this comment, which was allocating many GiB of memory to encode even one 10MB video.

I discovered that even one 10MB video was extracted to over 13k video frames, taking up nearly 5GiB of space. And then the
serializing would take up 14-15 GiB, and the encoding would take another 14-15, and so the process would be killed.

Relevant items:

It would be nice if...

  • ...there were more examples of how to efficiently load video datasets, and explanations of why they are more efficient.
  • ...there were a way to do this in some sort of streaming fashion that used less memory, e.g. loading in a batch of frames, using a sliding window, etc.
  • ...there were some way to set a memory limit, and just have it process more slowly within that limit.
  • ...there were a way to separate the download and prepare processes. A download_only option, like --download_only in the CLI
  • ...there were a warning that the dataset was using a lot of memory in processing, before the OS kills the process.
  • ...for saving disk space, a way to encode and serialize videos without extracting thousands of individual frames, ballooning the size from 10MB to multiple GiB. Maybe there is and I just don't know.
  • ...it was possible to download only part of a dataset. It's possible to load a slice, but only after download_and_prepare does its whole thing.
  • ...more explanation of what serialization and encoding are for, maybe? What are they?

Environment information
I've tested it on Colab and a few other Ubuntu workstations. High-Ram Colab Instances seem to have enough memory to get past this.

Lenguaje dominante
Python
Estrellas
4.6k
Forks
1.6k
Merge medio
9 h
PR fusionados (30 d)
8

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de tensorflow/datasets

Todos los issues de tensorflow/datasets

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.