Migrating 0.14 externally stored blobs and attachments leaves columns that 2.x cannot read

オープン
#1,698 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 6 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
mysql, python

調査の方向性

Start with src/datajoint/migrate.py, src/datajoint/hash_registry.py, and the migration guide at how-to/migrate-to-v20.md; trace migrate_external, migrate_external_pointers_v2, migrate_filepath, and the verification functions. The report identifies failures across legacy path and hash conversion, column comments, and verification. Done means the documented migration lets 2.x fetch legacy blob and attachment values and completes its checks.

索引モデルが issue の本文から書いたものです。

説明

bug

After the documented migration of a 0.14 schema, columns declared blob@store or attach@store cannot be fetched in 2.x, although the migration reports success.

Found on master (eefe74b4) with MySQL, on a schema laid out as 0.14.6 lays it out: an ~external_legacy table, and a binary(16) column signal with the comment :blob@legacy:raw signal, whose blob file is at {location}/{database}/ec/88/<hash hex>.

What happens

  1. migrate_external(schema, dry_run=False) fills the new column with
    {"url": null, "hash": "EC88A4468AB852D319309178A2E34955", "size": 109}
    
    • url is built from e.filepath (migrate.py#L1096-L1104). 0.14 leaves filepath NULL for blobs and attachments, because their path is derived from the hash.
    • The value has no path key and no store key.
    • The hash is the 32-digit hex value, not the 26-character base32 hash that 2.x uses.
  2. migrate_external(schema, dry_run=False, finalize=True) writes the column comment as :blob@legacy: (migrate.py#L1004), not :<blob@legacy>:. The heading then reads the column as plain json, and fetch1("signal") returns the dict above instead of the array.
  3. Correcting the comment by hand to :<blob@legacy>: doesn't help:
    • fetch1 then raises KeyError: 'path', because get_hash reads metadata["path"] (hash_registry.py#L284-L293).
    • With a path added as well, it raises Hash mismatch: expected EC88A446…, got 5sekiruk….

No later step rewrites these rows. analyze_columns and migrate_columns skip the column as external storage.

Related defects in the same flow

  • migrate_external_pointers_v2, Option A in the migration guide:
    • On the 0.14 binary(16) column it fails with DataError 1406: Data too long for column.
    • With the column widened, it writes {"path": null, "hash": "<32 hex digits>", ...} with no store, which fails the same way when read.
  • migrate_filepath finds no columns. _find_filepath_columns looks for DATA_TYPE = 'varchar' (migrate.py#L878-L882), but 0.14 filepath@store columns are binary(16).
  • verify_external_integrity raises TypeError on blob rows, from os.path.join(location, None) (migrate.py#L1350).
  • verify_schema_v20 always raises TypeError: not enough arguments for format string on MySQL. Its query has an unescaped LIKE '%blob' and also takes arguments (migrate.py#L2128).
  • The migration guide (how-to/migrate-to-v20.md) finalizes with migrate_external(schema, finalize=True). That runs with the default dry_run=True and changes nothing.

Expected

After the documented steps, a 0.14 blob@store or attach@store column returns its values in 2.x.

Suggested fix

  • Build each path the way 0.14 derived it from the hash, {database}/{subfolders}/{hash hex}, with the attachment name appended for attachments.
  • Write {path, store, size, hash}, and let reads accept the legacy hex hash or skip the check for it.
  • Write the comment as :<codec@store>:.
  • Match binary(16) filepath columns.
  • Escape % in the verify_schema_v20 query.
  • Handle a NULL filepath in verify_external_integrity.
  • Pass dry_run=False in the guide's finalize example.
主要言語
Python
スター
198
フォーク
103
平均マージ
6日 1時間
マージ済み PR(30日)
10

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

datajoint/datajoint-python のほかの issue

datajoint/datajoint-python の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。