kedro-org/kedro

Improve `%load_node` for node that doesn't have persisted dataset

开放

#4,169 创建于 2024年9月16日

 (2 条评论) (0 个反应) (1 位负责人)Python (864 个派生)batch import
HacktoberfestIssue: Feature Request

仓库指标

星标
 (9,146 个星标)
PR 合并指标
 (平均合并 8天 11小时) (30 天内合并 30 个 PR)

描述

Description

Originated from https://github.com/kedro-org/vscode-kedro/issues/140

%load_node works great but it doesn't work on MemoryDataset, I don't save every node and it's not easy to figure out which nodes I need to run again to produce the data

Context

  • This is much more powerful because not every node has persisted data and this limited the usage of the feature.
  • This enable much more powerful slicing feature in kedro-viz, which is currently limited because we do not want the slicing generate a command that Kedro does not know how to run.
  • Same as above, if we are able to support more flexible slicing, we may end up expanding the API of kedro run as well since we can support different combinations of slicing

Possible Implementation

Check out the suggest_resume_scanerio in SequentialRunner: https://github.com/kedro-org/kedro/blob/91765e3e70f8a4ba7ef3931a7855aed0ec97eb82/kedro/runner/runner.py#L200-L210

This roughly has the logic to resume pipeline but it's hidden in a private API, we need to surface this for more generic usage.

贡献者指南