CustomContainerTrainingJob.run drops max_wait_duration=0 instead of requesting indefinite DWS wait

未关闭 适合新手
#7,067 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
78/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
冷清
技术栈
python

调研方向

从 google/cloud/aiplatform/training_jobs.py 中的 _prepare_training_task_inputs_and_output_dir 开始,然后对比 issue 中引用的较低层 CustomJob 实现。验证省略、零值和正值 max_wait_duration 的调度输入;完成标准是零被序列化为 0s,同时省略状态仍保持区别,正时长得到保留。

由索引模型根据 Issue 内容生成。

描述

api: vertex-ai

Thanks for maintaining this library. CustomContainerTrainingJob.run(max_wait_duration=0) drops the value because the high-level training-job implementation uses a truthiness check. This prevents callers from requesting the documented indefinite wait for DWS Flex Start.

Environment details
  • OS type and version: macOS
  • Python version: 3.12
  • google-cloud-aiplatform versions: reproduced in 1.148.1; source inspection confirms the same behavior in 1.150.0, 1.163.0, and current main
Steps to reproduce
  1. Create a CustomContainerTrainingJob using any valid project, staging bucket, and container image.
  2. Run it with Flex Start and max_wait_duration=0.
  3. Inspect the scheduling inputs produced by _prepare_training_task_inputs_and_output_dir or the resulting TrainingPipeline request.
Code example
job.run(
    scheduling_strategy=custom_job.Scheduling.Strategy.FLEX_START,
    max_wait_duration=0,
)

Observed scheduling value:

max_wait_duration: None

Expected scheduling value:

max_wait_duration: 0s

For comparison, a positive input such as 14400 is preserved as 14400s.

Cause

The high-level training-job path converts the duration using a truthiness check, so zero follows the same branch as None:

https://github.com/googleapis/python-aiplatform/blob/v1.163.0/google/cloud/aiplatform/training_jobs.py#L1663-L1667

The generated API contract says that explicit zero means indefinite waiting, while omission defaults to 24 hours:

https://github.com/googleapis/python-aiplatform/blob/v1.163.0/google/cloud/aiplatform_v1/types/custom_job.py#L577-L582

max_wait_duration is a presence-aware google.protobuf.Duration, so explicit zero and an omitted field are distinct requests.

A previous fix correctly changed the lower-level CustomJob and hyperparameter-tuning paths to use is not None, explicitly noting that 0 is valid:

https://github.com/googleapis/python-aiplatform/commit/d9675fdf051233539f478187143f2833fd6e6af0

That commit did not update the CustomContainerTrainingJob path in training_jobs.py.

Suggested fix

Use an explicit is not None check in _prepare_training_task_inputs_and_output_dir, consistent with the lower-level implementation, and add tests covering omitted, zero, and positive durations.

Stack trace

No exception is raised. The problem is a silent request-semantics change: 0 is serialized as omission.

主要语言
Python
星标
907
派生
467
平均合并
1 天 13 小时
30 天内合并 PR
44

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

googleapis/python-aiplatform 的其他 Issue

查看 googleapis/python-aiplatform 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。