Windows host writes sm_train.sh with CRLF in SDK v3, causing SageMaker training job bootstrap failure ($'\r': command not found)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- aws, python, pytorch
- Domain
- cloud, machine-learning
Research direction
Inspect the script-writing code in sagemaker/train/model_trainer.py, especially the open call shown in the issue, and reproduce the generated file on Windows. Done means sm_train.sh contains LF line endings and the SageMaker container logs show the training entry point starting without the reported bash parsing errors.
Written by the indexing model from the issue text.
Description
PySDK Version
- PySDK V2 (2.x)
- PySDK V3 (3.x)
Describe the bug
When launching a SageMaker training job from Windows using sagemaker.train.ModelTrainer (SDK v3), the generated bootstrap script sm_train.sh is written with CRLF line endings. Inside the Linux training container, bash fails to parse it and the job exits before user training code starts.
The script appears to be written in sagemaker/train/model_trainer.py with:
with open(os.path.join(tmp_dir.name, TRAIN_SCRIPT), "w") as f:
which applies platform newline conversion on Windows (\r\n).
To reproduce
- Use Windows host with SageMaker Python SDK v3.
- Run this complete script (replace
ROLE_ARNandS3_INPUT):
from sagemaker.core import image_uris
from sagemaker.core.helper.session_helper import Session
from sagemaker.core.training.configs import SourceCode, Compute, InputData
from sagemaker.train import ModelTrainer
session = Session()
region = session.boto_region_name
training_image = image_uris.retrieve(
framework="pytorch",
region=region,
version="2.4.0",
py_version="py311",
instance_type="ml.g4dn.xlarge",
image_scope="training",
)
trainer = ModelTrainer(
sagemaker_session=session,
role="ROLE_ARN",
training_image=training_image,
source_code=SourceCode(
source_dir=".",
entry_script="sagemaker_entry.py",
requirements="requirements/sagemaker_train.txt",
),
compute=Compute(instance_type="ml.g4dn.xlarge", instance_count=1),
)
trainer.train(
input_data_config=[InputData(channel_name="train", data_source="S3_INPUT")],
wait=True,
logs=True,
)
- Check CloudWatch logs for the training job.
Expected behavior
sm_train.sh should be written with LF (\n) and execute correctly in the Linux container, allowing the training entry point to start.
Screenshots or logs
Observed logs:
/opt/ml/input/data/sm_drivers/sm_train.sh: line 1: $'\r': command not found
Starting training script#015
/opt/ml/input/data/sm_drivers/sm_train.sh: line 3: set: -#015: invalid option
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--]
/opt/ml/input/data/sm_drivers/sm_train.sh: line 5: $'\r': command not found
/opt/ml/input/data/sm_drivers/sm_train.sh: line 6: syntax error near unexpected token `$'{\r''
/opt/ml/input/data/sm_drivers/sm_train.sh: line 6: `handle_error() {#015'
System information
- SageMaker Python SDK version: 3.12.0
- Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch
- Framework version: 2.4.0 (DLC)
- Python version: 3.11
- CPU or GPU: GPU (
ml.g4dn.xlarge) - Custom Docker image (Y/N): N
Additional context
- Reproduced from Windows 10 host.
- Workaround: launching from Linux (WSL/Studio/EC2) avoids CRLF in
sm_train.sh. - Suggested fix in SDK: force LF on script write, e.g.
open(..., "w", newline="\n").
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 42
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aws/sagemaker-python-sdk
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
aws/sagemaker-python-sdk#6278 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
aws/sagemaker-python-sdk#6253 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 93/100
aws/sagemaker-python-sdk#6232 ·
-
component: model monitor type: bug
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
aws/sagemaker-python-sdk#6206 ·
-
component: model builder type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
aws/sagemaker-python-sdk#6199 ·
All issues in aws/sagemaker-python-sdk
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100