CustomContainerTrainingJob.run drops max_wait_duration=0 instead of requesting indefinite DWS wait
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start in google/cloud/aiplatform/training_jobs.py at _prepare_training_task_inputs_and_output_dir, then compare the lower-level CustomJob implementation referenced in the issue. Verify the scheduling inputs for omitted, zero, and positive max_wait_duration values; done means zero is serialized as 0s while omission remains distinct and positive durations are preserved.
Written by the indexing model from the issue text.
Description
Thanks for maintaining this library. CustomContainerTrainingJob.run(max_wait_duration=0) drops the value because the high-level training-job implementation uses a truthiness check. This prevents callers from requesting the documented indefinite wait for DWS Flex Start.
Environment details
- OS type and version: macOS
- Python version: 3.12
google-cloud-aiplatformversions: reproduced in 1.148.1; source inspection confirms the same behavior in 1.150.0, 1.163.0, and currentmain
Steps to reproduce
- Create a
CustomContainerTrainingJobusing any valid project, staging bucket, and container image. - Run it with Flex Start and
max_wait_duration=0. - Inspect the scheduling inputs produced by
_prepare_training_task_inputs_and_output_diror the resultingTrainingPipelinerequest.
Code example
job.run(
scheduling_strategy=custom_job.Scheduling.Strategy.FLEX_START,
max_wait_duration=0,
)
Observed scheduling value:
max_wait_duration: None
Expected scheduling value:
max_wait_duration: 0s
For comparison, a positive input such as 14400 is preserved as 14400s.
Cause
The high-level training-job path converts the duration using a truthiness check, so zero follows the same branch as None:
The generated API contract says that explicit zero means indefinite waiting, while omission defaults to 24 hours:
max_wait_duration is a presence-aware google.protobuf.Duration, so explicit zero and an omitted field are distinct requests.
A previous fix correctly changed the lower-level CustomJob and hyperparameter-tuning paths to use is not None, explicitly noting that 0 is valid:
https://github.com/googleapis/python-aiplatform/commit/d9675fdf051233539f478187143f2833fd6e6af0
That commit did not update the CustomContainerTrainingJob path in training_jobs.py.
Suggested fix
Use an explicit is not None check in _prepare_training_task_inputs_and_output_dir, consistent with the lower-level implementation, and add tests covering omitted, zero, and positive durations.
Stack trace
No exception is raised. The problem is a silent request-semantics change: 0 is serialized as omission.
- Dominant language
- Python
- Stars
- 907
- Forks
- 467
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 44
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from googleapis/python-aiplatform
-
api: vertex-ai
Difficulty 1/5 Under an hour Newbie friendliness 85/100
googleapis/python-aiplatform#7132 ·
-
api: vertex-ai
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
googleapis/python-aiplatform#7097 · 1 comment ·
-
api: vertex-ai
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
googleapis/python-aiplatform#6877 ·
-
api: vertex-ai
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
googleapis/python-aiplatform#6865 · 1 comment ·
-
api: vertex-ai
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
googleapis/python-aiplatform#6822 · 2 reactions ·
All issues in googleapis/python-aiplatform
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100