Make Add-On scratch a multi-part upload

Abierto
  • ¿Algo no está bien?
#521 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
django, postgresql, python
Área
api, backend

Línea de trabajo

Start with documentcloud/addons/serializers.py and the AddOnEventSerializer, then inspect the AddOn.store_event_data() implementation in python-documentcloud. Add ScratchField and tests for a multipart PATCH over 2.5 MB and a normal JSON PATCH; update the client to send scratch as a file part and release a new version. Done means both PATCH cases work within the nginx upload limit.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Issue

Something broke during the DocumentCloud dependency update today and it's really only been working coincidentally in some cases. store_event_data which calls a PATCH to /api/addon_events/pk with a JSON blob previously was working without a limit.

This is why I started working on enforcing a limit via nginx in this PR:
https://github.com/MuckRock/documentcloud/pull/496

It turns out Django does have a native limit which is DATA_UPLOAD_MAX_MEMORY_SIZE. This is set to 2.5MB by default. This wasn't evaluated correctly in older versions of django-rest-framework for JSON blobs and was the subject of a CVE here:
https://fieldguide.lutrasecurity.com/CVE-2026-73228/

This was patched in our most recent update of django-rest-framework, which was a minor revision bump.

We do currently use scratch > 2.5MB on some Add-Ons and I initially thought up to 25MB would be ok, since it's just there for the occasional store and load call.

Resolution

Since scratch is already deferred and only loaded in the detail view, keeping it in Postgres is fine. To restore support for scratch over 2.5 MB, we can send it as a multipart file upload instead of a JSON body. The nginx 25MB will be the limit that caps this.

In app:

  • Add ScratchField to documentcloud/addons/serializers.py and declare scratch = ScratchField(required=False, help_text=_("Field to store data for add-on between events")) on AddOnEventSerializer.
  • Add tests for a multipart file PATCH over 2.5 MB and a normal JSON PATCH.
    No parser changes are needed, because the default parsers already include MultiPartParser.

In python-documentcloud:

  • Change AddOn.store_event_data() to send scratch as a real file part with a filename, for example files={"scratch": ("scratch.json", json.dumps(data), "application/json")}. A plain form field still counts against the limit.
  • Release a new version.
Lenguaje dominante
Python
Estrellas
55
Forks
10
Merge medio
1 d 13 h
PR fusionados (30 d)
22

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de MuckRock/documentcloud

Todos los issues de MuckRock/documentcloud

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.