Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

sagemaker-core 2.15.0: role validation raises false-positive `RoleValidationError` under condition-based SCPs (IAM simulator can't evaluate conditional SCPs)

Aperta
  • Qualcosa non va?
#6,019 2 commenti 3 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
64/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Tranquilla
Stack tecnologico
aws, python
Ambito
cloud, security

Direzione di ricerca

Inizia da sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py, in particolare da resolve_and_validate_role e _evaluate_permissions, quindi riproduci la risposta del simulatore IAM usando il comando fornito aws iam simulate-principal-policy. Il lavoro è completo quando i risultati SCP basati su condizioni non producono più un RoleValidationError falso, i dinieghi di autorizzazione autentici continuano a essere gestiti correttamente e il comportamento esistente di warn-and-proceed viene preservato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

component: pysdk-team type: bug

PySDK Version

  • PySDK V2 (2.x)
  • PySDK V3 (3.x)

Reported against the sagemaker-core distribution, version 2.15.0 (repo tag v3.15.0).

Describe the bug

sagemaker-core 2.15.0 added a client-side permission pre-check that runs during high-level construction (e.g. ModelTrainer(...) → TrainDefaults.get_role) before any training job is submitted: resolve_and_validate_role → _evaluate_permissions → iam:SimulatePrincipalPolicy. It raises RoleValidationError on any non-allowed simulate verdict — and that verdict includes the AWS Organizations / SCP layer (OrganizationsDecisionDetail.AllowedByOrganizations).

Per AWS docs, the IAM policy simulator does not evaluate SCPs that have any conditions. So in an account whose organization uses condition-based SCPs, SimulatePrincipalPolicy returns AllowedByOrganizations: false (with EvalDecision: implicitDeny, MatchedStatements: []) for actions that are actually permitted at run time. The pre-check treats this as a definitive denial and raises — a false positive — even though the execution role is correctly configured and the real API call would succeed.

Two observable consequences:

  1. Creating a brand-new, fully-permissioned role does not help — the simulate is denied at the org layer regardless of the role's own policies.
  2. The same role works fine from a notebook / via a direct create_training_job call, because those paths don't run this client-side pre-check.

2.15.0 is currently the latest published release, so there is no fixed version to upgrade to.

To reproduce

Prerequisites: an AWS account under an organization with at least one condition-based SCP; a training execution role that trusts sagemaker.amazonaws.com and grants the training smoke-test actions at Resource: *; a calling identity that can call iam:SimulatePrincipalPolicy.

pip install 'sagemaker-core==2.15.0'

from sagemaker.core.helper.iam_role_resolver import IamRoleResolver

# Also reproducible via ModelTrainer(...) construction with role_arn set to the same role.
IamRoleResolver().resolve_and_validate_role(
    role_arn="arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role>",
    role_type="training",
)

Result:

RoleValidationError: IAM role 'arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role>' cannot be used for 'training' workloads.
Missing permissions: cloudwatch:PutMetricData, ec2:CreateNetworkInterface,
ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission, ec2:DescribeDhcpOptions, ec2:DescribeNetworkInterfaces,
ec2:DescribeSecurityGroups, ec2:DescribeSubnets, ec2:DescribeVpcs,
ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:GetAuthorizationToken,
ecr:GetDownloadUrlForLayer

Confirm the verdict is an org-layer artifact rather than a real permission gap:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<ACCOUNT_ID>:role/<training-exec-role> \
  --action-names cloudwatch:PutMetricData ec2:CreateNetworkInterface sagemaker:CreateTrainingJob

{
  "EvalActionName": "cloudwatch:PutMetricData",
  "EvalDecision": "implicitDeny",
  "MatchedStatements": [],
  "OrganizationsDecisionDetail": { "AllowedByOrganizations": "false" }
  // ...same for the other actions, including sagemaker:CreateTrainingJob —
  // yet CreateTrainingJob calls from this role succeed at run time (visible in CloudTrail).
}

The identical role runs the same workload successfully from a notebook / via direct API, so the real run-time evaluation permits these actions.

Expected behavior

The pre-check should not hard-fail on an Organizations/SCP-layer denial that the IAM policy simulator cannot faithfully evaluate. Because the simulator ignores condition-based SCPs, an AllowedByOrganizations: false result (with no matched explicit identity Deny) is unverifiable, not authoritative — it should be treated the same as the existing "caller can't call simulate → warn and proceed" path, letting the real API call be the source of truth. An explicit opt-out (e.g. validate_role=False or an env var) would also let users bypass the client-side check without modifying IAM.

Screenshots or logs

.../site-packages/sagemaker/core/helper/iam_role_resolver.py:573 in resolve_and_validate_role
570   # Permission check (definitive denial blocks; unverifiable warns)
571   verdict, denied = _evaluate_permissions(iam_client, role_arn, rol...
572   if verdict is False:
> 573       raise RoleValidationError(
574           _build_validation_error_message(role_arn, role_type, miss...

System information

  • SageMaker Python SDK version: sagemaker-core 2.15.0 (repo tag v3.15.0)
  • Framework name or algorithm: N/A — fails during role validation, before framework/job selection (framework-agnostic)
  • Framework version: N/A
  • Python version: 3.12
  • CPU or GPU: N/A (fails before job submission)
  • Custom Docker image (Y/N): N/A

Additional context

Introduced in sagemaker-core 2.15.0 — file sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py, added in commit dba1127a ("New release (#5969)"), first tag v3.15.0. Authoring PRs: #2041 (added SimulatePrincipalPolicy-based resolve_or_create_role) → #2080 (replaced it with the raising resolve_and_validate_role). #2080 notes it gates only on *-resource "smoke test" actions to avoid false denials on resource-scoped actions, but does not account for the Organizations/SCP layer the simulate call implicitly evaluates — which is the source of this false positive.

Suggested fix: in _evaluate_permissions, when an action is implicitDeny with no matched identity/SCP statement and OrganizationsDecisionDetail.AllowedByOrganizations == false, treat it as unverifiable (warn + proceed) rather than a missing permission; and/or add an explicit opt-out.

Relevant docs:

Workarounds (both verified):

  1. Attach an explicit Deny on iam:SimulatePrincipalPolicy to the identity running the SDK — it then skips the pre-check, warns, and proceeds (an explicit Deny is needed to override any existing Allow).
  2. Pin sagemaker-core<2.15.0, which predates the pre-check.
Lingua principale
Python
Stelle
2.3k
Fork
1.3k
Merge medio
3g 2h
PR unite (30g)
70

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di aws/sagemaker-python-sdk

Tutte le issue di aws/sagemaker-python-sdk

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.