Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Public async runs cannot tune row-group horizon/adaptive admission

Đang mở
  • Có gì không ổn?
#741 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@chethanuk đang làm issue này rồi.

Từ ngày 6/10/2026.

  • #982 của @chethanuk — đã đóng, không merge

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Ít trao đổi
Công nghệ
python
Lĩnh vực
api, backend, distributed-systems, performance

Hướng nghiên cứu

Bắt đầu bằng cách đọc surface công khai của RunConfig và đường dẫn builder công khai tạo async scheduler, sau đó truy vết cách internal row-group horizon của scheduler và các adaptive admission controls hiện đang được chọn. Hoàn thành khi có các public configuration hoặc policy controls được hỗ trợ, cùng với diagnostics cho first checkpoint, oldest open row group, active row groups, admission blocking và configured so với effective horizon.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug
Priority Level

High

Describe the bug

Public async fire-and-forget settings do not expose a row-group horizon or adaptive row-group admission policy. Users can tune RunConfig.buffer_size, RunConfig.max_in_flight_tasks, and model/provider request caps, but they cannot control how many row groups are actively admitted or whether that horizon adapts to DAG shape, latency tails, and endpoint capacity.

As a result, large async runs can leave fast endpoints idle or delay durable row completion even when public concurrency settings look generous. The useful setting is workload-dependent: a smaller fixed horizon tends to checkpoint earlier but under-exposes ready work, while a wider/adaptive horizon can improve endpoint occupancy but may delay first checkpoint or leave too many row groups open.

This makes it hard for public API users to choose a setting that reliably maximizes durable record completion and endpoint utilization across dependency chains, fan-in/fan-out DAGs, mixed latency, and retry-heavy workloads.

Steps/Code to reproduce bug
  1. Configure a large async dataset with many records and multiple LLM/model columns.
  2. Use either a dependency chain or fan-in/fan-out DAG so later columns depend on earlier model output.
  3. Set only public fire-and-forget controls, for example:
run_config = RunConfig(
    buffer_size=16,
    max_in_flight_tasks=2048,
)

# Configure model/provider caps through the public model config surface,
# then run DataDesigner.create(...) with the run config above.
  1. Increase buffer_size, max_in_flight_tasks, and model/provider request caps across a few runs.
  2. Observe that public settings do not provide a way to directly choose or adapt the active row-group horizon. Depending on workload shape, the run may either checkpoint slowly with low endpoint occupancy or expose more work while delaying durable row completion.
Expected behavior

Users running large async synthetic data jobs should have a documented, supported way to let DataDesigner choose, expose, or adapt the active row-group horizon from declared workload shape and model capacity.

A public fire-and-forget user should be able to reason about the tradeoff between early durable checkpoints and endpoint occupancy without reaching into private scheduler internals.

Agent Diagnostic / Prior Investigation

Important observations from bounded large-job probes:

  • A default/fixed small row-group horizon can leave fast endpoints idle even when buffer_size, max_in_flight_tasks, and model caps are set high enough to suggest more work should run.
  • Wider/adaptive row-group admission can improve endpoint occupancy, especially when some columns are slow, long-tail, or cooling down after retryable responses, but it may delay first checkpoint and keep older row groups open longer.
  • In a same-workload comparison, a default-like fixed small horizon checkpointed records but did not maximize endpoint occupancy; a wider/adaptive policy improved occupancy but could produce no full row-group checkpoint inside the same short observation window.
  • In a practical fan-in DAG, a fixed small horizon made visible record progress while leaving a fast endpoint mostly idle; a wider horizon fed endpoints better but produced no checkpointed records in the same timebox.
  • Public-only sweeps over buffer_size, max_in_flight_tasks, and model caps did not find a setting that consistently achieved both high endpoint utilization and high durable row throughput.

API/config surface check:

  • RunConfig exposes buffer_size and max_in_flight_tasks, but not max_concurrent_row_groups, an adaptive row-group admission toggle, an adaptive initial target, or a max-admitted-row guard.
  • The public builder path constructs the async scheduler without passing row-group horizon/adaptive admission arguments, so users inherit the hidden scheduler default.
  • The scheduler has internal controls for row-group horizon/adaptive admission, but those are not part of the supported public run configuration.
Additional context

Related open issues exist but do not appear to be clear duplicates:

  • #727 covers durable async capacity diagnostics for explaining endpoint idle time.
  • #700 covers disambiguating buffer_size from scheduler work admission semantics.
  • #645 tracks the broader async scheduling/resource metadata epic.

This issue is specifically about the missing public row-group horizon/adaptive admission control and default policy.

Suggested fix

Expose row-group horizon/adaptive row-group admission in RunConfig, for example through a supported policy field rather than requiring direct scheduler construction.

Add an adaptive default or diagnostics-driven policy that can balance endpoint occupancy against first-checkpoint latency and oldest-open-row-group age for common large async DAG shapes.

Surface first-checkpoint, oldest-open-row-group, active-row-group count, row-group admission blocked reason, and configured/effective horizon diagnostics so users can tune without private scheduler instrumentation.

Checklist
  • Searched existing open issues for a clear duplicate.
  • Kept the report anonymized and excluded local paths, hostnames, usernames, branch names, temp harness names, output directory names, commit hashes, and machine-specific setup details.
  • Used the repo bug report section format.
  • Included a concrete suggested fix.
Ngôn ngữ chính
Python
Star
2.4k
Fork
234
Merge trung bình
1 ngày 22 phút
Pull request đã merge (30 ngày)
83

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA-NeMo/DataDesigner

Tất cả issue của NVIDIA-NeMo/DataDesigner

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.