Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

`HyperparameterTuner` drops `ModelTrainer` checkpoint and retry configuration

Chiusa Adatta ai principianti
  • Qualcosa non va?
#6,500 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
85/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
aws, python
Ambito
backend-api-design, machine-learning

Direzione di ricerca

Inizia da HyperparameterTuner._build_training_job_definition() e verifica come costruisce definition_kwargs a partire da ModelTrainer. Confronta i campi con HyperParameterTrainingJobDefinition e con i test esistenti per le definizioni del tuner. Il lavoro è completo quando le impostazioni di checkpoint e di nuovo tentativo vengono mantenute se specificate e omesse correttamente se assenti, incluso l’addestramento spot gestito.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

PySDK Version

  • PySDK V2 (2.x)
  • PySDK V3 (3.x)

Describe the bug

HyperparameterTuner builds a
HyperParameterTrainingJobDefinition from its ModelTrainer.

The generated shape supports checkpoint_config and retry_strategy.
However, _build_training_job_definition() does not copy either value from
the trainer.

As a result, tuned training jobs lose:

  • The checkpoint S3 location and local checkpoint path.
  • The training retry strategy.

This behavior is especially risky for managed spot tuning. An interrupted job
cannot resume from the configured checkpoint, and transient failures do not use
the configured retry count.

This defect is present in SageMaker Python SDK v3.25.0.

To reproduce

Run this script. It does not call a live AWS service.

from unittest.mock import patch

import boto3
from sagemaker.core.parameter import IntegerParameter
from sagemaker.core.training.configs import (
    CheckpointConfig,
    Compute,
    OutputDataConfig,
    RetryStrategy,
)
from sagemaker.core.workflow.pipeline_context import PipelineSession
from sagemaker.train import ModelTrainer
from sagemaker.train.tuner import HyperparameterTuner

role = "arn:aws:iam::123456789012:role/example"
boto_session = boto3.Session(
    aws_access_key_id="test",
    aws_secret_access_key="test",
    region_name="us-east-2",
)
pipeline_session = PipelineSession(boto_session=boto_session)

with patch(
    "sagemaker.train.defaults.resolve_and_validate_role",
    return_value=role,
):
    trainer = ModelTrainer(
        training_image="123456789012.dkr.ecr.us-east-2.amazonaws.com/example:latest",
        role=role,
        sagemaker_session=pipeline_session,
        compute=Compute(
            instance_type="ml.m5.large",
            instance_count=1,
            enable_managed_spot_training=True,
        ),
        checkpoint_config=CheckpointConfig(
            s3_uri="s3://example-bucket/checkpoints/",
            local_path="/opt/ml/checkpoints",
        ),
        output_data_config=OutputDataConfig(
            s3_output_path="s3://example-bucket/output/",
        ),
    ).with_retry_strategy(
        RetryStrategy(maximum_retry_attempts=10)
    )

tuner = HyperparameterTuner(
    model_trainer=trainer,
    objective_metric_name="validation:accuracy",
    objective_type="Maximize",
    hyperparameter_ranges={
        "epochs": IntegerParameter(1, 5),
    },
    max_jobs=2,
    max_parallel_jobs=1,
)

definition = tuner._build_training_job_definition([])

print("Trainer checkpoint:", trainer.checkpoint_config)
print("Tuning checkpoint:", definition.checkpoint_config)
print("Trainer retry:", trainer.retry_strategy)
print("Tuning retry:", definition.retry_strategy)

assert definition.checkpoint_config == trainer.checkpoint_config
assert definition.retry_strategy == trainer.retry_strategy

Both assertions fail because the tuning definition fields remain unassigned.

Expected behavior

The tuning job definition must preserve the trainer's checkpoint and retry
configuration.

Screenshots or logs

The generated shape accepts both fields:

HyperParameterTrainingJobDefinition(
    # ...
    checkpoint_config=...,
    retry_strategy=...,
)

HyperparameterTuner._build_training_job_definition() does not pass them in
definition_kwargs.

System information

  • SageMaker Python SDK version: 3.25.0
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): Framework-independent
  • Framework version: Not applicable
  • Python version: 3.12
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): Reproduces with either

Additional context

A possible fix is to add the assigned trainer values to
definition_kwargs:

if model_trainer.checkpoint_config is not None:
    definition_kwargs["checkpoint_config"] = model_trainer.checkpoint_config

if model_trainer.retry_strategy is not None:
    definition_kwargs["retry_strategy"] = model_trainer.retry_strategy

Tests must cover normal and managed spot trainers, explicit values, and omitted
values.

Lingua principale
Python
Stelle
2.3k
Fork
1.3k
Merge medio
2g 20h
PR unite (30g)
101

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di aws/sagemaker-python-sdk

Tutte le issue di aws/sagemaker-python-sdk

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.