Codec encode/decode: separate connection context from primary key in the `key` argument
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 重构
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- python
调研方向
首先跟踪 SchemaCodec.encode/decode 和 _extract_context,了解主键和连接上下文如何流入 _build_path 和 _get_backend。检查内置的 object、npy、hash、filepath、attach 和 blob codecs,以及所引用的第三方 codecs。当签名和调用方使用分开的键值与上下文值,并且所需配置在不使用全局 fallback 的情况下显式传入时,即视为完成。
由索引模型根据 Issue 内容生成。
描述
Background
`SchemaCodec.encode(self, value, *, key=None, store_name=None)` and `.decode(self, stored, *, key=None)` overload a single `key` dict with two different kinds of data:
- primary key values (the row this codec call is writing/reading)
- connection context — `_schema`, `_table`, `_field`, and critically `_config` (the calling connection's `Config`, carrying that connection's store credentials)
`_extract_context` separates these by convention — anything with a leading underscore is context, everything else is primary key:
```python
def _extract_context(self, key: dict | None) -> tuple[str, str, str, dict]:
key = dict(key) if key else {}
schema = key.pop("_schema", "unknown")
table = key.pop("_table", "unknown")
field = key.pop("field", "data")
primary_key = {k: v for k, v in key.items() if not k.startswith("")}
return schema, table, field, primary_key
```
`config` is read directly by codec authors as `(key or {}).get("_config")` and threaded manually into `_build_path`/`_get_backend`.
Two problems this causes
1. Naming collision risk. The underscore-prefix convention is not structurally enforced — it works today only because DataJoint's identifier grammar disallows attribute names starting with `_`. That's an implicit, undocumented invariant; nothing raises if it's ever violated (a future internal/system column, a future relaxation of the naming grammar), and a colliding key would silently vanish from `primary_key` rather than error.
2. Wrong failure mode for a required dependency. `config` is functionally required for correct store resolution in any multi-connection process (e.g., a dashboard serving several users), but it's passed as an optional dict key a codec author must remember to `.get()` out and thread through by hand. Forgetting to do so doesn't fail loudly — it silently falls back to global `dj.config`, which either raises a confusing `DataJointError: Missing S3 configuration` deep inside `StorageBackend._validate_spec`, or worse, resolves a different-but-valid store with no error at all.
That second failure mode is what actually happened, twice, independently: `dj-figpack-codecs#6` and `dj-canvasxpress-codecs#3` both omitted `config=` on `_build_path`/`_get_backend`, both following the pattern shown in `SchemaCodec`'s own docstring example (fixed in #1549, docs-only).
Proposed direction
Make connection context an explicit, separately named parameter rather than a dict key:
```python
def encode(self, value, *, key=None, context=None, store_name=None): ...
def decode(self, stored, *, key=None, context=None): ...
```
`key` reverts to meaning exactly what it means everywhere else in DataJoint (fetch, insert, `make()`) — the primary key dict, nothing else. `context` (a dict or small structure) carries `schema`, `table`, `field`, `config`.
If `_build_path`/`_get_backend` also made `config` a required parameter rather than defaulting to `None` + global fallback, forgetting to pass it becomes a loud `TypeError` at the call site instead of a quiet wrong-store bug discovered later in production — a strictly better failure mode than the current one, and better than just nesting context under a single `key["_context"]` entry (which fixes the collision risk but not the silent-fallback problem).
Cost
Breaking change to `Codec.encode`/`decode` and every codec reading `key["_config"]` or calling `_extract_context` — built-ins (`object`, `npy`, `hash`, `filepath`, `attach`, `blob`) plus third-party codecs (`dj-figpack-codecs`, `dj-canvasxpress-codecs`). Worth doing now rather than later — the codec system is still being shaken out, evidenced by this same-week cluster of config-threading fixes, so the migration surface is as small as it will ever be.
Raised following review of #1549, which fixes the docstring example to match current (flawed) behavior — this issue proposes revisiting the underlying signature instead.
- 主要语言
- Python
- 星标
- 197
- 派生
- 98
- 平均合并
- 6 天 7 小时
- 30 天内合并 PR
- 1
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
datajoint/datajoint-python 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 76/100
datajoint/datajoint-python#1539 · 3 条评论 ·
-
dj.Diagram SVG output is not byte-reproducible: set iteration order leaks into node emission order 未关闭bug
难度 3/5 1-2 天 新手友好度 78/100
datajoint/datajoint-python#1551 ·
-
难度 5/5 一周以上 新手友好度 35/100
datajoint/datajoint-python#1547 ·
-
难度 4/5 3-5 天 新手友好度 52/100
datajoint/datajoint-python#1546 · 1 条评论 ·
-
Make key_source restrict-only: add key_source_restriction, deprecate parent-redefining overrides 未关闭
难度 5/5 一周以上 新手友好度 35/100
datajoint/datajoint-python#1523 · 1 条评论 ·
查看 datajoint/datajoint-python 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 78/100
syfoud/Simulated_Scepter#172 ·
-
A cancelled tests run makes the coverage comment workflow fail and reports it as a red check on main 未关闭area: ci bug perceived difficulty: 3
难度 2/5 1-3 小时 新手友好度 78/100
Nitjsefnie-Harness-Commons/daedalus#921 · 1 条评论 ·
-
难度 2/5 1-3 小时 新手友好度 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
难度 1/5 1 小时以内 新手友好度 92/100
-
难度 2/5 1-3 小时 新手友好度 78/100
ClickHouse/clickhouse-connect#1057 ·