longblob silently stores str(array) under DataJoint 2.x — 16 affected attributes, no error raised

Abierto
#45 4 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
mysql, numpy, python
Área
database

Línea de trabajo

Comienza con las 12 declaraciones de facial_behavior_estimation.py y las 4 de facemap_inference.py, y compáralas después con el datajoint-2.x branch. Verifica que los atributos afectados permitan hacer round-trip de arrays con DataJoint 2.x sin cambiar el tipo de columna MySQL subyacente, y confirma el resultado frente a la restricción de compatibilidad 0.14.9 indicada.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

longblob silently stores str(array) under DataJoint 2.x — the data is destroyed, nothing raises

element-facemap declares 16 longblob attributes. Under DataJoint 2.x every one of
them silently corrupts the data written to it. I lost some time to this and it
seems worth writing down, because the failure has no symptom at the point where
it happens.

DataJoint 2.0.0 shipped 2026-02-03 as a complete rewrite; 2.3.2 is current.
Because install_requires says datajoint>=0.13, a fresh pip install element-facemap
today resolves 2.x. (It then fails at import on dj.schema — that part is loud,
and is what the accompanying PR fixes. This issue is about what happens after
you get past it.)

What 2.x changed

In 0.14.x, longblob meant "a DataJoint blob": pack numpy on write, unpack on
read. In 2.x, blob handling moved into an extensible codec system (<blob>), and
a bare longblob is now treated as a raw native MySQL column. The write path
coerces whatever you hand it to a string; the read path returns the raw bytes.

Reproduction
import numpy as np, datajoint as dj          # datajoint 2.3.2, MySQL 9.7.1

schema = dj.Schema("probe")

@schema
class Legacy(dj.Manual):                     # how this element declares its blobs
    definition = "id : int\n---\ntrace : longblob\n"

@schema
class Codec(dj.Manual):                      # the 2.x codec
    definition = "id : int\n---\ntrace : <blob>\n"

a = np.arange(5000, dtype="float32") / 7.0   # 20,000 bytes
Legacy().insert1({"id": 1, "trace": a})
Codec().insert1({"id": 1, "trace": a})
longblob -> bytes      90 bytes
            b'[0.0000000e+00 1.4285715e-01 2.8571430e-01 ... 7.1385712e+02 7.1400000e+02\n 7.1414288e+02]'
<blob>   -> ndarray    dtype=float32 shape=(5000,)   array_equal: True

Read that stored value again: it is str(array). NumPy's own abbreviated text
repr, ellipsis and all. 4,994 of the 5,000 values no longer exist anywhere.
The insert did not raise. The fetch did not raise. information_schema reports
the column as longblob in both cases, so nothing about the schema looks wrong.

Smaller arrays are text too, just not yet truncated:

np.arange(4, dtype='float32')/7   ->  b'[0.         0.14285715 0.2857143  0.42857143]'
np.arange(9).reshape(3,3)         ->  b'[[0. 1. 2.]\n [3. 4. 5.]\n [6. 7. 8.]]'

dtype and shape are gone in every case; dj.blob.unpack() on the returned bytes
returns None rather than failing, because it was never a DataJoint blob.

There is one signal, at declaration time only, and it does not mention data
loss:

[WARNING]: Native type 'longblob' is used in attribute 'trace'.
           Consider using a core DataJoint type for better portability.
What is affected in element-facemap

facial_behavior_estimation.py (12), facemap_inference.py (4) — including FacialSignal.MotionSVD.{motmask,projection}, FacialSignal.Summary.{avgframe,avgmotion}, FacialSignal.Region.motion and FacemapProcessing's facemap_params

Fix

Declare <blob> instead of longblob. That restores 0.14.x round-trip
behaviour exactly, and the underlying MySQL column type is unchanged
(longblob either way), so it is a definition edit and not a table migration —
as long as it happens before real rows exist. Once a table has been populated
under 2.x with longblob, the original arrays are not recoverable from the
database.

I cannot send this one as a PR: 0.14.9 rejects <blob> with
Support for Adapted Attribute types is disabled, so it would break every
current user. It is on a branch instead, split so the backward-compatible part
can be taken separately:

Every change on both branches was applied mechanically with regexes and reviewed
line by line; no element behaviour is altered, only attribute-type spellings.

Environment

datajoint 2.3.2 · MySQL 9.7.1 · Python 3.11 · element-facemap at main
(last commit 2025-05-20). Cross-checked against datajoint 0.14.9 on the same
server.


Related PR (the backward-compatible subset): #44

Lenguaje dominante
Jupyter Notebook
Estrellas
3
Forks
14
Merge medio
1 h 14 min
PR fusionados (30 d)
1

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de datajoint/element-facemap

Todos los issues de datajoint/element-facemap

Issues similares

Más issues de Databases

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.