SageMakerClient CustomSession uses default values instead of passed user session
维护者通常 2 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 新手友好度
- 68/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 冷清
- 技术栈
- aws, python
调研方向
从 sagemaker-core/src/sagemaker/core/utils/utils.py 中 SageMakerClient.init 附近（大约第 395–406 行）开始，然后比较 sagemaker-runtime、featurestore 和 metrics 客户端如何使用提供的会话。运行提供的多配置文件复现，并验证 CreateTrainingJob 使用的是指定的账户和区域，而不是默认配置文件。
由索引模型根据 Issue 内容生成。
描述
PySDK Version
- PySDK V2 (2.x)
- PySDK V3 (3.x)
Describe the bug
SageMakerClient builds the sagemaker client from botocore.session.get_session(), ignoring the provided user session resulting in cross-account CreateTrainingJob failures.
In sagemaker-core 2.16.0, SageMakerClient.init (sagemaker/core/utils/utils.py, ~L395–406) creates self.sagemaker_client from a fresh default botocore session (botocore.session.get_session() → custom_session) rather than the session argument, under a # TODO: Remove post-launch custom service-model loader. The sagemaker-runtime/featurestore/metrics clients correctly use session. Because the class is a SingletonMeta singleton, the first (default-credential) client is reused process-wide ignoring passed in user sessions.
Impact: callers that pass an explicit boto3 session (e.g. ModelTrainer(sagemaker_session=…) → TrainingJob.create(session=…)) have the control-plane call issued under the ambient default AWS profile instead. When the execution role is in a different account than the default profile, CreateTrainingJob fails with ValidationException: RoleArn: Cross-account pass role is not allowed.
To reproduce
If a user has multiple AWS profiles (all for different accounts) set up in ~/.aws/config and the default in ~/.aws/credentials is NOT the same account as the intended profile that is used to create the SageMaker Session then CreateTrainingJob will fail with ValidationException: RoleArn: Cross-account pass role is not allowed.
from sagemaker.core.helper.session_helper import Session
from sagemaker.train import ModelTrainer
from sagemaker.train.configs import SourceCode, Compute, InputData
import boto3
sess = Session(boto_session=boto3.Session(profile_name="acct-A", region_name="us-east-2"))
ModelTrainer(
sagemaker_session=sess, # account A
role="arn:aws:iam::<ACCOUNT_A>:role/<your-exec-role>", # account A
training_image="<ACCOUNT_A>.dkr.ecr.us-east-2.amazonaws.com/img:latest",
compute=Compute(instance_type="ml.m5.xlarge", instance_count=1),
source_code=SourceCode(source_dir="src", entry_script="train.py"),
).train(input_data_config=[InputData(channel_name="train", data_source="s3://bucket/train/")])
# -> ValidationException: RoleArn: Cross-account pass role is not allowed
# (only when your *default* AWS profile is a different account than A)
Expected behavior
A training job to be created and executed in the specified account. This is a migration from v2 to v3 and the v2 implementation of this works without issue creating training jobs in the targeted accounts based on the profile passed to the Session.
Screenshots or logs
If applicable, add screenshots or logs to help explain your problem.
System information
A description of your system. Please provide:
- SageMaker Python SDK version: 3.15.1
- Framework name (eg. PyTorch) or algorithm (eg. KMeans): SKLearn
- Framework version: 1.4-2
- Python version: py3
- CPU or GPU: CPU
- Custom Docker image (Y/N): N
Additional context
This issue also seems to persist to the latest version of 3.16 since the same custom_session is also used to initiate the sagemaker_client.
- 主要语言
- Python
- 星标
- 2.3k
- 派生
- 1.3k
- 平均合并
- 3 天 10 小时
- 30 天内合并 PR
- 88
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
aws/sagemaker-python-sdk 的其他 Issue
-
Cannot use spark_event_logs_s3_uri in PySparkProcessor job可能已有人在做 @rsareddy0329 于 9 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
aws/sagemaker-python-sdk#6253 ·
维护者通常 2 天内回复
-
[Bug] V3 Hyperparameter Tuning Pipeline page labelled "Download Data" in navigation due to missing title cell可能已有人在做 @admivsn 于 38 天前认领。 未关闭
难度 1/5 1 小时以内 新手友好度 93/100
aws/sagemaker-python-sdk#6232 ·
维护者通常 2 天内回复
-
[Bug] ModelTrainer with no input channels emits InputDataConfig: [], which CreatePipeline rejects (min=1) — v2 omitted the key可能已有人在做 @sagemaker-bot 于 9 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 76/100
aws/sagemaker-python-sdk#6156 · 2 条评论 ·
维护者通常 2 天内回复
-
sagemaker-train should depend on mlflow-skinny, following sagemaker-mlflow 0.5.0可能已有人在做 @mohamedzeidan2021 于 10 天前认领。 未关闭
难度 2/5 半天 新手友好度 72/100
aws/sagemaker-python-sdk#6152 ·
维护者通常 2 天内回复
-
ModelTrainer generates sm_train.sh with CRLF line endings on Windows causing training job failure可能已有人在做 @MohammedAlkindi 于 29 天前认领。 未关闭
难度 1/5 1 小时以内 新手友好度 88/100
aws/sagemaker-python-sdk#5904 · 1 个 reaction ·
维护者通常 2 天内回复
查看 aws/sagemaker-python-sdk 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 70/100
NVIDIA/earth2studio#1241 ·
维护者通常 3 天内回复
-
docs(types): update the collection binding note now that typed collections shipped in pycubrid 1.9.0未关闭documentation priority: low size: S
难度 2/5 1-3 小时 新手友好度 75/100
cubrid-lab/sqlalchemy-cubrid#768 ·
维护者通常 1 天内回复
-
bug help wanted
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
documentation
难度 1/5 1 小时以内 新手友好度 65/100
ansys/pydpf-core#3547 ·
维护者通常 1 天内回复
-
good first issue
难度 2/5 1-3 小时 新手友好度 78/100
OktoLabsAI/okto-pulse#114 ·
维护者通常 1 天内回复