longblob silently stores str(array) under DataJoint 2.x — 16 affected attributes, no error raised
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 35/100
Línea de trabajo
Comienza con las 12 declaraciones de facial_behavior_estimation.py y las 4 de facemap_inference.py, y compáralas después con el datajoint-2.x branch. Verifica que los atributos afectados permitan hacer round-trip de arrays con DataJoint 2.x sin cambiar el tipo de columna MySQL subyacente, y confirma el resultado frente a la restricción de compatibilidad 0.14.9 indicada.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
longblob silently stores str(array) under DataJoint 2.x — the data is destroyed, nothing raises
element-facemap declares 16 longblob attributes. Under DataJoint 2.x every one of
them silently corrupts the data written to it. I lost some time to this and it
seems worth writing down, because the failure has no symptom at the point where
it happens.
DataJoint 2.0.0 shipped 2026-02-03 as a complete rewrite; 2.3.2 is current.
Because install_requires says datajoint>=0.13, a fresh pip install element-facemap
today resolves 2.x. (It then fails at import on dj.schema — that part is loud,
and is what the accompanying PR fixes. This issue is about what happens after
you get past it.)
What 2.x changed
In 0.14.x, longblob meant "a DataJoint blob": pack numpy on write, unpack on
read. In 2.x, blob handling moved into an extensible codec system (<blob>), and
a bare longblob is now treated as a raw native MySQL column. The write path
coerces whatever you hand it to a string; the read path returns the raw bytes.
Reproduction
import numpy as np, datajoint as dj # datajoint 2.3.2, MySQL 9.7.1
schema = dj.Schema("probe")
@schema
class Legacy(dj.Manual): # how this element declares its blobs
definition = "id : int\n---\ntrace : longblob\n"
@schema
class Codec(dj.Manual): # the 2.x codec
definition = "id : int\n---\ntrace : <blob>\n"
a = np.arange(5000, dtype="float32") / 7.0 # 20,000 bytes
Legacy().insert1({"id": 1, "trace": a})
Codec().insert1({"id": 1, "trace": a})
longblob -> bytes 90 bytes
b'[0.0000000e+00 1.4285715e-01 2.8571430e-01 ... 7.1385712e+02 7.1400000e+02\n 7.1414288e+02]'
<blob> -> ndarray dtype=float32 shape=(5000,) array_equal: True
Read that stored value again: it is str(array). NumPy's own abbreviated text
repr, ellipsis and all. 4,994 of the 5,000 values no longer exist anywhere.
The insert did not raise. The fetch did not raise. information_schema reports
the column as longblob in both cases, so nothing about the schema looks wrong.
Smaller arrays are text too, just not yet truncated:
np.arange(4, dtype='float32')/7 -> b'[0. 0.14285715 0.2857143 0.42857143]'
np.arange(9).reshape(3,3) -> b'[[0. 1. 2.]\n [3. 4. 5.]\n [6. 7. 8.]]'
dtype and shape are gone in every case; dj.blob.unpack() on the returned bytes
returns None rather than failing, because it was never a DataJoint blob.
There is one signal, at declaration time only, and it does not mention data
loss:
[WARNING]: Native type 'longblob' is used in attribute 'trace'.
Consider using a core DataJoint type for better portability.
What is affected in element-facemap
facial_behavior_estimation.py (12), facemap_inference.py (4) — including FacialSignal.MotionSVD.{motmask,projection}, FacialSignal.Summary.{avgframe,avgmotion}, FacialSignal.Region.motion and FacemapProcessing's facemap_params
Fix
Declare <blob> instead of longblob. That restores 0.14.x round-trip
behaviour exactly, and the underlying MySQL column type is unchanged
(longblob either way), so it is a definition edit and not a table migration —
as long as it happens before real rows exist. Once a table has been populated
under 2.x with longblob, the original arrays are not recoverable from the
database.
I cannot send this one as a PR: 0.14.9 rejects <blob> with
Support for Adapted Attribute types is disabled, so it would break every
current user. It is on a branch instead, split so the backward-compatible part
can be taken separately:
compat-fixes—dj.schema→dj.Schema,boolean→bool,
enum("x")→enum('x'). Works on both 0.14.9 and 2.3.2. Sent as a PR.datajoint-2.x— the above + all 16longblob→<blob>. 2.x only.
https://github.com/akshay-jaggi/element-facemap/tree/datajoint-2.x
Every change on both branches was applied mechanically with regexes and reviewed
line by line; no element behaviour is altered, only attribute-type spellings.
Environment
datajoint 2.3.2 · MySQL 9.7.1 · Python 3.11 · element-facemap at main
(last commit 2025-05-20). Cross-checked against datajoint 0.14.9 on the same
server.
Related PR (the backward-compatible subset): #44
- Lenguaje dominante
- Jupyter Notebook
- Estrellas
- 3
- Forks
- 14
- Merge medio
- 1 h 14 min
- PR fusionados (30 d)
- 1
Preparar el entorno
Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de datajoint/element-facemap
-
FacialSignal.MotionSVD is built from the wrong array axis, and empty motSv raises IndexErrorPosiblemente ocupada @akshay-jaggi la tomó hace 101 días. Abierto
Dificultad 3/5 1-2 días Aptitud para principiantes 72/100
datajoint/element-facemap#46 · 2 comentarios ·
Todos los issues de datajoint/element-facemap
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 5 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
Infisical/infisical#9737 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
datajoint/datajoint-python#1703 ·
Los mantenedores suelen responder en 6 días
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 92/100
liberation-data/drivine#826 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 1 día