sagemaker-core 2.15.0: role validation raises false-positive `RoleValidationError` under condition-based SCPs (IAM simulator can't evaluate conditional SCPs)

Bắt đầu với sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py, đặc biệt là resolve_and_validate_role và _evaluate_permissions, sau đó tái hiện phản hồi của trình mô phỏng IAM bằng lệnh aws iam simulate-principal-policy được cung cấp. Được xem là hoàn tất khi các kết quả SCP dựa trên điều kiện không còn tạo ra RoleValidationError giả, các trường hợp từ chối quyền thực sự vẫn được xử lý đúng cách và hành vi warn-and-proceed hiện có được giữ nguyên.

PySDK Version

• PySDK V2 (2.x)
• PySDK V3 (3.x)

Reported against the sagemaker-core distribution, version 2.15.0 (repo tag v3.15.0).

Describe the bug

sagemaker-core 2.15.0 added a client-side permission pre-check that runs during high-level construction (e.g. ModelTrainer(...) → TrainDefaults.get_role) before any training job is submitted: resolve_and_validate_role → _evaluate_permissions → iam:SimulatePrincipalPolicy. It raises RoleValidationError on any non-allowed simulate verdict — and that verdict includes the AWS Organizations / SCP layer (OrganizationsDecisionDetail.AllowedByOrganizations).

Per AWS docs, the IAM policy simulator does not evaluate SCPs that have any conditions. So in an account whose organization uses condition-based SCPs, SimulatePrincipalPolicy returns AllowedByOrganizations: false (with EvalDecision: implicitDeny, MatchedStatements: []) for actions that are actually permitted at run time. The pre-check treats this as a definitive denial and raises — a false positive — even though the execution role is correctly configured and the real API call would succeed.

Two observable consequences:

1. Creating a brand-new, fully-permissioned role does not help — the simulate is denied at the org layer regardless of the role's own policies.
2. The same role works fine from a notebook / via a direct create_training_job call, because those paths don't run this client-side pre-check.

2.15.0 is currently the latest published release, so there is no fixed version to upgrade to.

To reproduce

Prerequisites: an AWS account under an organization with at least one condition-based SCP; a training execution role that trusts sagemaker.amazonaws.com and grants the training smoke-test actions at Resource: *; a calling identity that can call iam:SimulatePrincipalPolicy.

pip install 'sagemaker-core==2.15.0'

from sagemaker.core.helper.iam_role_resolver import IamRoleResolver

# Also reproducible via ModelTrainer(...) construction with role_arn set to the same role.
IamRoleResolver().resolve_and_validate_role(
    role_arn="arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role>",
    role_type="training",
)

Result:

RoleValidationError: IAM role 'arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role>' cannot be used for 'training' workloads.
Missing permissions: cloudwatch:PutMetricData, ec2:CreateNetworkInterface,
ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission, ec2:DescribeDhcpOptions, ec2:DescribeNetworkInterfaces,
ec2:DescribeSecurityGroups, ec2:DescribeSubnets, ec2:DescribeVpcs,
ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:GetAuthorizationToken,
ecr:GetDownloadUrlForLayer

Confirm the verdict is an org-layer artifact rather than a real permission gap:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role> \
  --action-names cloudwatch:PutMetricData ec2:CreateNetworkInterface sagemaker:CreateTrainingJob

{
  "EvalActionName": "cloudwatch:PutMetricData",
  "EvalDecision": "implicitDeny",
  "MatchedStatements": [],
  "OrganizationsDecisionDetail": { "AllowedByOrganizations": "false" }
  // ...same for the other actions, including sagemaker:CreateTrainingJob —
  // yet CreateTrainingJob calls from this role succeed at run time (visible in CloudTrail).
}

The identical role runs the same workload successfully from a notebook / via direct API, so the real run-time evaluation permits these actions.

Expected behavior

The pre-check should not hard-fail on an Organizations/SCP-layer denial that the IAM policy simulator cannot faithfully evaluate. Because the simulator ignores condition-based SCPs, an AllowedByOrganizations: false result (with no matched explicit identity Deny) is unverifiable, not authoritative — it should be treated the same as the existing "caller can't call simulate → warn and proceed" path, letting the real API call be the source of truth. An explicit opt-out (e.g. validate_role=False or an env var) would also let users bypass the client-side check without modifying IAM.

Screenshots or logs

.../site-packages/sagemaker/core/helper/iam_role_resolver.py:573 in resolve_and_validate_role
570   # Permission check (definitive denial blocks; unverifiable warns)
571   verdict, denied = _evaluate_permissions(iam_client, role_arn, rol...
572   if verdict is False:
> 573       raise RoleValidationError(
574           _build_validation_error_message(role_arn, role_type, miss...

System information

• SageMaker Python SDK version: sagemaker-core 2.15.0 (repo tag v3.15.0)
• Framework name or algorithm: N/A — fails during role validation, before framework/job selection (framework-agnostic)
• Framework version: N/A
• Python version: 3.12
• CPU or GPU: N/A (fails before job submission)
• Custom Docker image (Y/N): N/A

Additional context

Introduced in sagemaker-core 2.15.0 — file sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py, added in commit dba1127a ("New release (#5969)"), first tag v3.15.0. Authoring PRs: #2041 (added SimulatePrincipalPolicy-based resolve_or_create_role) → #2080 (replaced it with the raising resolve_and_validate_role). #2080 notes it gates only on *-resource "smoke test" actions to avoid false denials on resource-scoped actions, but does not account for the Organizations/SCP layer the simulate call implicitly evaluates — which is the source of this false positive.

Suggested fix: in _evaluate_permissions, when an action is implicitDeny with no matched identity/SCP statement and OrganizationsDecisionDetail.AllowedByOrganizations == false, treat it as unverifiable (warn + proceed) rather than a missing permission; and/or add an explicit opt-out.

Relevant docs:

• IAM policy simulator can't test SCPs with conditions
• IAM policy evaluation logic (explicit Deny overrides Allow)

Workarounds (both verified):

1. Attach an explicit Deny on iam:SimulatePrincipalPolicy to the identity running the SDK — it then skips the pre-check, warns, and proceeds (an explicit Deny is needed to override any existing Allow).
2. Pin sagemaker-core<2.15.0, which predates the pre-check.