Diagram.trace: descend through master-part boundaries (Merge-table upstream tracing)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- data-engineering, databases
Research direction
Start with reference/specs/trace.md §2, Diagram.trace(), and the existing test test_trace_stops_at_master_no_part_down_collection. Read _propagate_part_to_master and discussion #1232 to understand the proposed upward analog, then resolve the default-versus-opt-in behavior and add coverage showing stable OR-merged tracing through Master.Part, including strict_provenance effects.
Written by the indexing model from the issue text.
Description
Summary
Diagram.trace() (and therefore self.upstream) walks ancestor FK edges only. It does not descend from an ancestor Master into that Master's Parts. In the merge-table shape used by Spyglass —
Parent → Master.Part → Master → Child
— trace(Child & key) reaches Master, but trace[Master.Part] and trace[Parent] raise DataJointError. The true upstream source (Parent) is unreachable through the merge point.
This matches the shipped spec (the Upstream Trace spec, reference/specs/trace.md §2 "Allowed table set"; formerly provenance.md, renamed in 2.3: an ancestor's Part is included only when the Part itself lies on an FK path to the seed) and is now pinned by test_trace_stops_at_master_no_part_down_collection. However, the design comment on discussion #1232 described a symmetric down-collection ("when a Master is reached, add its Parts to the trace, and continue upward from the Parts' FK parents") that was never implemented — a correction has been posted there. This issue tracks actually building that capability.
Proposed behavior (opt-in or default — design question)
When the upward walk reaches a Master, additionally:
- Restrict the Master's Parts downward from the Master's restriction (the existing forward rules).
- Continue upward from the Parts' other FK parents (the upward rules), OR-merging into the trace.
- Multi-pass until stable (mirror of the downstream
part_integrity="cascade"mechanics — this is its upward analog, cf._propagate_part_to_master).
Design questions
- Default-on (a Part is semantically an extension of its Master, so "what contributed" arguably includes contributions through Parts) vs. opt-in flag (
trace(expr, through_parts=True)) to preserve current semantics. - OR-merge and termination through alias nodes; interaction with the (part, master)-pair dedup introduced for the downstream analog.
- Effect on
self.upstream's allowed-table set understrict_provenance— the merge pattern is exactly where downstreammake()s need to read through the merge point.
Motivation
- Spyglass Merge tables (discussion #1232, @CBroz1): versioned pipelines route provenance through Master/Part merge points; upstream tracing that stops at the merge master cannot answer "which Parent produced this result."
- Without it, strict-provenance mode makes such reads impossible rather than merely unergonomic (the true parent is not in the allowed set).
- Dominant language
- Python
- Stars
- 197
- Forks
- 98
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from datajoint/datajoint-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
datajoint/datajoint-python#1539 · 3 comments ·
-
dj.Diagram SVG output is not byte-reproducible: set iteration order leaks into node emission order Openbug
Difficulty 3/5 1-2 days Newbie friendliness 78/100
datajoint/datajoint-python#1551 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
datajoint/datajoint-python#1550 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
datajoint/datajoint-python#1547 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
datajoint/datajoint-python#1546 · 1 comment ·
All issues in datajoint/datajoint-python
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
learningequality/ricecooker#747 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
BSData/horus-heresy-3rd-edition#3171 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
run-llama/llama_index#23199 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
KhronosGroup/glTF-Blender-IO#2769 ·