FileResultStore.delete_expired crashes on JSON-materialized cache results

未关闭 适合新手
#1,813 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 3 天内回复

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
75/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
backend, testing

调研方向

从 hamilton/caching/stores/file.py 开始,重点查看 FileResultStore.delete_expired() 和 _load_result_from_path(),然后检查文件存储的测试中是否覆盖了清理功能。运行针对性测试,并使用 JSONDataSaver 和 JSONDataLoader 添加一个回归用例;完成的标准是清理操作顺利完成,且未过期的结果仍可读取。

由索引模型根据 Issue 内容生成。

描述

Current behavior

FileResultStore.delete_expired() raises UnpicklingError after storing a result with the built-in JSON saver/loader. The result can be read successfully before cleanup, and nothing has expired.

This prevents cleanup of a store containing valid JSON-materialized results.

Steps to reproduce

from pathlib import Path
from tempfile import TemporaryDirectory

from hamilton.caching.stores.file import FileResultStore
from hamilton.io.default_data_loaders import JSONDataLoader, JSONDataSaver

with TemporaryDirectory() as directory:
    store = FileResultStore(directory)
    store.set(
        "example",
        {"value": 1},
        saver_cls=JSONDataSaver,
        loader_cls=JSONDataLoader,
    )
    assert store.get("example") == {"value": 1}
    print(sorted(path.name for path in Path(directory).iterdir()), flush=True)
    store.delete_expired()
    assert store.get("example") == {"value": 1}

Observed output and traceback (paths shortened):

['example', 'example.json']
Traceback (most recent call last):
  File "repro.py", line 17, in <module>
    store.delete_expired()
  File "hamilton/caching/stores/file.py", line 138, in delete_expired
    stored_result = self._load_result_from_path(file_path)
  File "hamilton/caching/stores/file.py", line 55, in _load_result_from_path
    return StoredResult.load(data)
  File "hamilton/caching/stores/base.py", line 234, in load
    loaded = pickle.loads(raw)
_pickle.UnpicklingError: invalid load key, '{'.

Expected behavior

Cleanup should complete without attempting to unpickle the JSON payload, and the unexpired result should remain readable.

Library and system information

  • Hamilton 1.90.0, editable checkout at 2bd9dbc7ee22e1c2a4f6cd687a353aa0079d4d81
  • Python 3.11.17, macOS arm64
  • Compared both affected files (caching/stores/file.py and caching/stores/base.py) with upstream main at ef88b1e96140a01f3d72967316c2335f8b62a067; they are byte-for-byte identical.

Proposed contribution

set() writes a pickled loader reference and a separate .json payload into the same directory. delete_expired() iterates every directory entry and passes each to StoredResult.load(), including the JSON payload.

I would like to contribute a focused fix that distinguishes stored-result entries from materialized payloads, with a regression test showing cleanup preserves a readable JSON result. Would you be open to me taking this on, and is there a preferred way to identify those entries? I would keep any broader expiration/serialization redesign separate.

主要语言
Jupyter Notebook
星标
2.6k
派生
230
平均合并
3 天 3 小时
30 天内合并 PR
22

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

apache/hamilton 的其他 Issue

查看 apache/hamilton 的全部 Issue

相似的 Issue

更多 Backend & API Design Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。