kedro-org/kedro
Improve `%load_node` for node that doesn't have persisted dataset
Aperta
#4169 aperta il 16 set 2024
HacktoberfestIssue: Feature Request
Metriche repository
- Star
- (9146 stelle)
- Metriche merge PR
- (Merge medio 8g 11h) (30 PR mergiate in 30 g)
Descrizione
Description
Originated from https://github.com/kedro-org/vscode-kedro/issues/140
%load_node works great but it doesn't work on MemoryDataset, I don't save every node and it's not easy to figure out which nodes I need to run again to produce the data
Context
- This is much more powerful because not every node has persisted data and this limited the usage of the feature.
- This enable much more powerful slicing feature in
kedro-viz, which is currently limited because we do not want the slicing generate a command that Kedro does not know how to run. - Same as above, if we are able to support more flexible slicing, we may end up expanding the API of
kedro runas well since we can support different combinations of slicing
Possible Implementation
Check out the suggest_resume_scanerio in SequentialRunner:
https://github.com/kedro-org/kedro/blob/91765e3e70f8a4ba7ef3931a7855aed0ec97eb82/kedro/runner/runner.py#L200-L210
This roughly has the logic to resume pipeline but it's hidden in a private API, we need to surface this for more generic usage.