Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

`HyperparameterTuner` drops `ModelTrainer` checkpoint and retry configuration

Fechada Para iniciantes
  • Algo errado?
#6,500 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 2 dias

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
2/5
Tempo estimado
1-3 horas
Facilidade para iniciantes
85/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
aws, python
Domínio
backend-api-design, machine-learning

Direção de pesquisa

Comece em HyperparameterTuner._build_training_job_definition() e examine como ele constrói definition_kwargs a partir de ModelTrainer. Compare os campos com HyperParameterTrainingJobDefinition e com os testes existentes para definições do tuner. O trabalho estará concluído quando as configurações de checkpoint e de nova tentativa forem preservadas quando fornecidas e omitidas corretamente quando estiverem ausentes, inclusive no treinamento spot gerenciado.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

PySDK Version

  • PySDK V2 (2.x)
  • PySDK V3 (3.x)

Describe the bug

HyperparameterTuner builds a
HyperParameterTrainingJobDefinition from its ModelTrainer.

The generated shape supports checkpoint_config and retry_strategy.
However, _build_training_job_definition() does not copy either value from
the trainer.

As a result, tuned training jobs lose:

  • The checkpoint S3 location and local checkpoint path.
  • The training retry strategy.

This behavior is especially risky for managed spot tuning. An interrupted job
cannot resume from the configured checkpoint, and transient failures do not use
the configured retry count.

This defect is present in SageMaker Python SDK v3.25.0.

To reproduce

Run this script. It does not call a live AWS service.

from unittest.mock import patch

import boto3
from sagemaker.core.parameter import IntegerParameter
from sagemaker.core.training.configs import (
    CheckpointConfig,
    Compute,
    OutputDataConfig,
    RetryStrategy,
)
from sagemaker.core.workflow.pipeline_context import PipelineSession
from sagemaker.train import ModelTrainer
from sagemaker.train.tuner import HyperparameterTuner

role = "arn:aws:iam::123456789012:role/example"
boto_session = boto3.Session(
    aws_access_key_id="test",
    aws_secret_access_key="test",
    region_name="us-east-2",
)
pipeline_session = PipelineSession(boto_session=boto_session)

with patch(
    "sagemaker.train.defaults.resolve_and_validate_role",
    return_value=role,
):
    trainer = ModelTrainer(
        training_image="123456789012.dkr.ecr.us-east-2.amazonaws.com/example:latest",
        role=role,
        sagemaker_session=pipeline_session,
        compute=Compute(
            instance_type="ml.m5.large",
            instance_count=1,
            enable_managed_spot_training=True,
        ),
        checkpoint_config=CheckpointConfig(
            s3_uri="s3://example-bucket/checkpoints/",
            local_path="/opt/ml/checkpoints",
        ),
        output_data_config=OutputDataConfig(
            s3_output_path="s3://example-bucket/output/",
        ),
    ).with_retry_strategy(
        RetryStrategy(maximum_retry_attempts=10)
    )

tuner = HyperparameterTuner(
    model_trainer=trainer,
    objective_metric_name="validation:accuracy",
    objective_type="Maximize",
    hyperparameter_ranges={
        "epochs": IntegerParameter(1, 5),
    },
    max_jobs=2,
    max_parallel_jobs=1,
)

definition = tuner._build_training_job_definition([])

print("Trainer checkpoint:", trainer.checkpoint_config)
print("Tuning checkpoint:", definition.checkpoint_config)
print("Trainer retry:", trainer.retry_strategy)
print("Tuning retry:", definition.retry_strategy)

assert definition.checkpoint_config == trainer.checkpoint_config
assert definition.retry_strategy == trainer.retry_strategy

Both assertions fail because the tuning definition fields remain unassigned.

Expected behavior

The tuning job definition must preserve the trainer's checkpoint and retry
configuration.

Screenshots or logs

The generated shape accepts both fields:

HyperParameterTrainingJobDefinition(
    # ...
    checkpoint_config=...,
    retry_strategy=...,
)

HyperparameterTuner._build_training_job_definition() does not pass them in
definition_kwargs.

System information

  • SageMaker Python SDK version: 3.25.0
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): Framework-independent
  • Framework version: Not applicable
  • Python version: 3.12
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): Reproduces with either

Additional context

A possible fix is to add the assigned trainer values to
definition_kwargs:

if model_trainer.checkpoint_config is not None:
    definition_kwargs["checkpoint_config"] = model_trainer.checkpoint_config

if model_trainer.retry_strategy is not None:
    definition_kwargs["retry_strategy"] = model_trainer.retry_strategy

Tests must cover normal and managed spot trainers, explicit values, and omitted
values.

Linguagem predominante
Python
Estrelas
2.3k
Forks
1.3k
Merge médio
2d 13h
PRs com merge (30d)
97

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de aws/sagemaker-python-sdk

Todas as issues de aws/sagemaker-python-sdk

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.