Expose terminal failed-column and seed-row provenance in public results
Maintainer thường phản hồi trong vòng 1 ngày
Chưa có ai nhận issue này.
Đánh giá
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức phù hợp với người mới
- 45/100
Hướng nghiên cứu
Bắt đầu với DatasetCreationResults trong packages/data-designer/src/data_designer/interface/results.py, sau đó lần theo việc xử lý lỗi kết thúc qua async_scheduler.py, task_model.py, dataset_builder.py và data_designer.py. Xác định cách danh tính seed và các bản ghi lỗi của lần chạy hiện tại đi qua create(), preview() và các ngoại lệ có kiểu. Hoàn tất khi hợp đồng kết quả công khai đáp ứng các ví dụ chấp nhận được liệt kê mà không yêu cầu async_trace hoặc phân tích log.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Mô tả
Priority Level
High (Major improvement)
Is your feature request related to a problem? Please describe.
Consumers that compose many columns into one Data Designer graph cannot determine, from the public run result, which column terminally failed for each omitted seed row.
Data Designer internally knows the task column, row group, and row index when a task fails. DatasetCreationResults.task_traces can expose those fields when full async tracing is enabled, but that is a heavy per-task diagnostic mechanism and the row position is not a stable caller-provided record identity. On a partially successful run, a consumer can diff input and output IDs to discover that a row disappeared, but cannot attribute that loss to the originating column without parsing logs.
This blocks an integration in NVIDIA-NeMo/Anonymizer#237 and PR #240. Anonymizer's legacy rewrite implementation uses several Data Designer runs, so a missing row can be attributed to a stage such as replacement-map generation, initial rewrite, evaluation, or repair. The combined conditional graph uses one Data Designer run. Without column-level failure provenance, every omitted row can only be reported as failing the coarse rewrite-combined boundary.
The required contract is a stable mapping from each terminally omitted seed row to its originating failed column, plus a safe failure category or reason. It must be available without enabling full task tracing.
Describe the solution you'd like
Expose lightweight, structured terminal row failures through the public result API for create() and preview(). If a run raises instead of returning a result, make the same provenance available on the typed exception.
The exact class and field names are flexible, but consumers need this information for each terminal failure:
@dataclass(frozen=True)
class TerminalRowFailure:
seed_record_id: str | int
column: str
failure_kind: str
reason: str | None = None
The public API should also provide a way for the caller to designate a seed identity column, for example _anonymizer_record_id, whose value is copied into seed_record_id. The returned identity must be directly joinable to the original input and must not require reconstructing it from (row_group, row_index). If no identity column is configured, Data Designer may expose a documented stable seed-row locator as a fallback.
Required semantics:
- Capture terminal row omissions after retry and salvage behavior has completed.
- Attribute the root column that failed, not downstream columns that were skipped or cancelled because their dependency was missing.
- Do not report
skip.whendecisions as failures. - Do not report transient failures that later recovered.
- Work for LLM, custom, expression, and other row-producing column types that can omit a row.
- Remain available when only part of a dataset succeeds and when every row fails.
- Be collected independently of
async_trace; do not retain prompts, completions, row contents, or successful task traces. - Document whether failure records cover the current invocation or the full resumed run.
- Preserve deterministic input ordering where practical.
A result shape such as DatasetCreationResults.failed_records and the equivalent preview result field would satisfy the need. A lazy public artifact referenced by the result would also work if it provides the same typed, current-run contract without requiring log parsing.
Acceptance examples:
- Given two seed rows with designated IDs, if column
evaluate_0terminally fails for one row, the result contains exactly that seed ID andcolumn="evaluate_0". - If downstream
repair_0is not scheduled becauseevaluate_0failed, no separaterepair_0failure is reported. - If
repair_0is skipped becauseskip.whenis true, no failure is reported. - If a retry succeeds, no terminal failure is reported.
- The same information is available without
async_trace=Trueand for an all-rows-failed typed exception.
Describe alternatives you've considered
- Keep the Anonymizer pipeline split across multiple Data Designer runs. This preserves coarse stage attribution but retains custom orchestration and prevents a single conditional graph from becoming the default.
- Diff input and output record IDs. This identifies omitted rows but not the column that caused the omission.
- Enable
async_trace=Trueand map(row_group, row_index)back to the seed. This retains every task trace, is not appropriate as a scalable production result API, and does not expose the caller's stable record ID. - Parse warning logs, or consume proposed structured log records from #386. Logs and telemetry are useful diagnostics, but they are not a reliable post-run outcome contract and can be filtered or missed by handlers.
- Preserve failed fields as
nullas proposed in #362. That changes row-retention semantics; this request only asks Data Designer to report the provenance of its existing terminal row omissions.
Agent Investigation
Investigation of current main found:
packages/data-designer/src/data_designer/interface/results.py:DatasetCreationResultsexposestask_traces, but no terminal failure collection.packages/data-designer-engine/src/data_designer/engine/dataset_builders/scheduling/task_model.py:TaskTracecontainscolumn,row_group,row_index,status, anderror, but traces are only created when tracing is enabled and do not contain a caller-provided seed identity.packages/data-designer-engine/src/data_designer/engine/dataset_builders/async_scheduler.py: the scheduler has the originatingTaskwhen recording terminal failures and already distinguishes failures, retries, and skips internally.packages/data-designer-engine/src/data_designer/engine/dataset_builders/dataset_builder.py: the builder exposes only the first non-retryable exception for the special zero-record error path, not per-row terminal failure provenance.packages/data-designer/src/data_designer/interface/data_designer.py: task traces are copied toDatasetCreationResults, while terminal failure records are not returned.
Existing issues #386 and #362 are adjacent but do not provide a typed result mapping terminally omitted rows to both the originating column and the caller's seed ID.
Additional context
This is not a request for full tracing, model-call telemetry, or preservation of failed row contents. The intended surface is a lightweight correctness and failure-attribution contract for applications that embed Data Designer graphs.
Checklist
- I've reviewed existing issues and the documentation
- This is a design proposal, not a "please build this" request
- Ngôn ngữ chính
- Python
- Star
- 2.3k
- Fork
- 211
- Merge trung bình
- 3 ngày 12 giờ
- Pull request đã merge (30 ngày)
- 45
Chuẩn bị môi trường
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Issue khác của NVIDIA-NeMo/DataDesigner
-
task
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
NVIDIA-NeMo/DataDesigner#760 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Harden Slurm inference routing, backpressure, and failoverCó thể đã có người làm @nabinchha đã nhận 1 ngày trước. Đang mởtask
NVIDIA-NeMo/DataDesigner#966 · 1 người được giao ·
Maintainer thường phản hồi trong vòng 1 ngày
-
enhancement triaged
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 40/100
NVIDIA-NeMo/DataDesigner#956 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
task
Độ khó 5/5 Hơn một tuần Mức phù hợp với người mới 42/100
NVIDIA-NeMo/DataDesigner#947 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
bug
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 52/100
NVIDIA-NeMo/DataDesigner#946 · 1 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
Tất cả issue của NVIDIA-NeMo/DataDesigner
Issue tương tự
-
Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 72/100
letsencrypt/cp-cps#353 ·
-
Marble Madness II is missingĐang mở
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 84/100
PedestrianDynamics/pyFDS-Evac#394 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 78/100
DOI-USGS/pywatershed#421 ·
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 78/100
python-pillow/Pillow#10087 · 1 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày