ModelBuilder.build() ignores the configured session when it reloads a model
維護者通常 2 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 2/5
- 預估耗時
- 1-3 小時
- 新手友好度
- 72/100
- Issue 類型
- 缺陷
- 描述清晰度
- 描述清楚
- 活躍度
- 活躍
- 技術堆疊
- aws, python
研究方向
錯誤位於 sagemaker/serve/model_builder.py 中 ModelBuilder.build() 的最終回傳處,那裡在呼叫 Model.get() 時沒有傳入已設定的工作階段。請閱讀該類別中 self.sagemaker_session 的設定方式,然後檢查 Model.get() 是否接受工作階段參數。新增一個單元測試,使用不同的預設 boto 工作階段與目標 boto 工作階段,並斷言在 CreateModel 之後 Model.get() 收到的是目標工作階段。當該測試通過且現有的 ModelBuilder 測試仍然通過時,即完成。
由索引模型根據 Issue 內容生成。
描述
PySDK Version
- PySDK V2 (2.x)
- PySDK V3 (3.x)
Describe the bug
ModelBuilder.build() creates a SageMaker model with its configured
sagemaker_session. It then reloads the model with Model.get() without
passing that session.
If the configured session and default boto session use different AWS accounts,
the create request succeeds in the configured account. The subsequent describe
request uses the default account and fails because the model does not exist
there.
The relevant call in sagemaker/serve/model_builder.py is:
return Model.get(model_name=self.model_name, region=self.region)
To reproduce
Configure two AWS CLI profiles that use different AWS accounts. The target
profile must have SageMaker permissions and permission to pass the execution
role.
Set these environment variables:
export TARGET_PROFILE="<profile-for-target-account>"
export DEFAULT_PROFILE="<profile-for-different-account>"
export TARGET_ROLE_ARN="arn:aws:iam::<target-account-id>:role/<sagemaker-execution-role>"
export TARGET_BUCKET="<existing-bucket-in-target-account>"
export IMAGE_URI="<inference-image-uri-accessible-from-target-account>"
export AWS_REGION="us-east-2"
Run this script:
import os
import uuid
import boto3
from botocore.exceptions import ClientError
from sagemaker.core.helper.session_helper import Session
from sagemaker.core.resources import Model
from sagemaker.core.shapes import ContainerDefinition
from sagemaker.serve import ModelBuilder
region = os.environ["AWS_REGION"]
target_profile = os.environ["TARGET_PROFILE"]
default_profile = os.environ["DEFAULT_PROFILE"]
role_arn = os.environ["TARGET_ROLE_ARN"]
bucket = os.environ["TARGET_BUCKET"]
image_uri = os.environ["IMAGE_URI"]
model_name = f"modelbuilder-session-repro-{uuid.uuid4().hex[:8]}"
# Make the process default session use a different account.
boto3.setup_default_session(profile_name=default_profile, region_name=region)
target_boto_session = boto3.Session(
profile_name=target_profile,
region_name=region,
)
target_sagemaker_session = Session(
boto_session=target_boto_session,
default_bucket=bucket,
)
target_account = target_boto_session.client("sts").get_caller_identity()["Account"]
default_account = boto3.client("sts", region_name=region).get_caller_identity()["Account"]
print(f"Target account: {target_account}")
print(f"Default account: {default_account}")
source_model = Model(
model_name=f"{model_name}-source",
primary_container=ContainerDefinition(image=image_uri),
execution_role_arn=role_arn,
)
builder = ModelBuilder(
model=[source_model],
role_arn=role_arn,
sagemaker_session=target_sagemaker_session,
instance_type="ml.m5.large",
)
try:
builder.build(model_name=model_name)
finally:
# The create request succeeds before build() raises.
try:
target_sagemaker_session.sagemaker_client.delete_model(
ModelName=model_name,
)
except ClientError as error:
if error.response["Error"]["Code"] != "ValidationException":
raise
The script raises an error similar to:
botocore.exceptions.ClientError: An error occurred (ValidationException)
when calling the DescribeModel operation:
Could not find model "modelbuilder-session-repro-1234abcd".
Before cleanup, DescribeModel succeeds through target_sagemaker_session.
The model is present in the target account.
Expected behavior
ModelBuilder.build() must use the configured session for all operations.
The method must return the created Model resource from the target account.
The reload call can pass the boto session:
return Model.get(
model_name=self.model_name,
session=self.sagemaker_session.boto_session,
region=self.region,
)
Suggested patch:
diff --git a/src/sagemaker/serve/model_builder.py b/src/sagemaker/serve/model_builder.py
--- a/src/sagemaker/serve/model_builder.py
+++ b/src/sagemaker/serve/model_builder.py
@@
- return Model.get(model_name=self.model_name, region=self.region)
+ return Model.get(
+ model_name=self.model_name,
+ session=self.sagemaker_session.boto_session,
+ region=self.region,
+ )
The regression test must configure different default and target boto sessions.
It must make sure that Model.get() receives the target boto session after
CreateModel.
Screenshots or logs
Target account: <target-account-id>
Default account: <different-account-id>
Creating model with name: modelbuilder-session-repro-1234abcd
Model has been created: modelbuilder-session-repro-1234abcd
ValidationException: Could not find model
"modelbuilder-session-repro-1234abcd".
CloudTrail shows CreateModel in the target account. The failing
DescribeModel request uses the default account.
System information
- SageMaker Python SDK version: 3.24.0
- Framework name (eg. PyTorch) or algorithm (eg. KMeans): Not framework-specific
- Framework version: N/A
- Python version: 3.12.0
- CPU or GPU: CPU, Apple Silicon (arm64)
- Custom Docker image (Y/N): N
Additional context
The failure is hidden when the default boto session uses the same account as
the configured SageMaker session.
This also affects multi-container inference pipelines. CreateModel completes
in the target account, but endpoint creation never starts because the model
reload fails first.
- 主要語言
- Python
- 星號
- 2.3k
- 分支
- 1.3k
- 平均合併
- 3 天 4 小時
- 30 天內合併 PR
- 90
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 沒有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
aws/sagemaker-python-sdk 的其他 Issue
-
[Bug] ModelTrainer with no input channels emits InputDataConfig: [], which CreatePipeline rejects (min=1) — v2 omitted the key可能已有人在做 @sagemaker-bot 於 15 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 76/100
aws/sagemaker-python-sdk#6156 · 2 則留言 ·
維護者通常 2 天內回覆
-
sagemaker-train should depend on mlflow-skinny, following sagemaker-mlflow 0.5.0可能已有人在做 @mohamedzeidan2021 於 16 天前認領。 未關閉
難度 2/5 半天 新手友好度 72/100
aws/sagemaker-python-sdk#6152 ·
維護者通常 2 天內回覆
-
[Bug] ModelTrainer drops TrainingJobName for PipelineSession, breaking use_custom_job_prefix on TrainingStep可能已有人在做 @mohamedzeidan2021 於 20 天前認領。 未關閉component: pipelines type: bug
難度 2/5 1-3 小時 新手友好度 72/100
aws/sagemaker-python-sdk#5776 · 1 則留言 · 1 個 reaction ·
維護者通常 2 天內回覆
-
`ModelTrainer` bug with method to load hyperparameters from file for Amazon Nova Recipe可能已有人在做 @sagemaker-bot 於 15 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 82/100
aws/sagemaker-python-sdk#5770 ·
維護者通常 2 天內回覆
-
難度 2/5 1-3 小時 新手友好度 68/100
aws/sagemaker-python-sdk#5588 · 1 則留言 ·
維護者通常 2 天內回覆
查看 aws/sagemaker-python-sdk 的全部 Issue
相似的 Issue
-
documentation pkg: sphinx-needs
難度 1/5 1 小時以內 新手友好度 82/100
useblocks/sphinx-needs#2255 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 60/100
home-assistant/core#186538 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 72/100
btclib-org/btclib-secp256k1#1181 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 65/100
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 78/100
pyrite-wiki/pyrite#917 ·
維護者通常 1 天內回覆