Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

insert --watch does not recover from stale IRSA credentials after IAM role recreation

Aperta
  • Qualcosa non va?
#202 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
50/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
aws, java
Ambito
backend, cli, cloud

Direzione di ricerca

Inizia da InsertWatch e segui lo stack trace attraverso Insert.processFile, BaseS3File e S3InputFile per tracciare come vengono creati e memorizzati nella cache il client S3 e le credenziali di web identity. Riproduci o testa un errore di autenticazione durante un watch di lunga durata, quindi verifica che un nuovo tentativo con credenziali aggiornate elabori il messaggio ed esegua il commit dello snapshot Iceberg senza riavviare il pod.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

A long-running ice insert --watch process on EKS can get stuck retrying
S3 operations with 403 Forbidden after its IAM/IRSA environment is
reconciled or the IAM role is recreated. The effective IAM policy is correct,
but the process does not recover until its Kubernetes pod is recreated.

After the pod restart, the same SQS messages and S3 objects are processed
successfully and Iceberg snapshots are committed. This suggests that the
long-running process keeps stale AWS credentials or an S3 client/session that
is not refreshed after the underlying IRSA identity changes.

Environment

  • ice: 0.16.0
  • Image: cr-enam.altinity.io/altinity/ice:0.16.0
  • AWS authentication: IRSA / web identity ServiceAccount annotation
  • Java: 21
  • Source: S3 ObjectCreated events delivered through SQS
  • Destination: Iceberg REST Catalog
  • Partitioning: hour(timestamp)

Command shape

The watcher is run as a long-lived process equivalent to:

ice insert \
  -p \
  --force-no-copy \
  --skip-duplicates \
  --partition='[{"column":"timestamp","transform":"hour"}]' \
  's3://<bucket>/cold/metrics-v5/**/*.parquet' \
  --watch='<SQS queue URL>'

Observed behavior

After an infrastructure reconciliation, the watcher continues receiving SQS
messages but fails while reading S3 object metadata:

ERROR c.a.i.c.i.cmd.InsertWatch > Failed to process batch of messages (retry in PT20S)
java.io.IOException: Error processing file(s)
...
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Forbidden
(Service: S3, Status Code: 403)
...
at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(...)
at org.apache.iceberg.aws.s3.BaseS3File.getObjectMetadata(...)
at org.apache.iceberg.aws.s3.S3InputFile.getLength(...)
at com.altinity.ice.cli.internal.iceberg.parquet.Metadata.readFooter(...)
at com.altinity.ice.cli.internal.cmd.Insert.processFile(...)

The watcher retries every 20 seconds but does not recover by itself.

The live inline role policy was checked while the failure was happening and
contained the required permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::<bucket>",
        "arn:aws:s3:::<bucket>/cold/metrics-v5/*"
      ]
    }
  ]
}

The failing object was under the allowed cold/metrics-v5/ prefix.

Recovery and evidence

Recreating only the watcher pod fixes the problem without changing the IAM
policy, SQS queue, S3 object, Iceberg table, or watcher arguments:

kubectl rollout restart deployment/vector-metrics-ice-watch-v5

Immediately after restart, the new process successfully processes the queued
objects:

INFO o.a.i.SnapshotProducer > Committed snapshot 6236223900153511381 (MergeAppend)
INFO c.a.i.c.i.cmd.InsertWatch > Received S3 event: ObjectCreated:Put -> s3://<bucket>/cold/metrics-v5/dt=2026-07-27/hour=18/<uuid>.parquet
...
INFO o.a.i.SnapshotProducer > Committed snapshot 197678787631752742 (MergeAppend)

It then drains the accumulated SQS backlog. No Forbidden or
Failed to process messages appear after the restart. --skip-duplicates
prevents already registered files from being added twice.

Timeline from this incident:

  • 2026-07-28 13:40:46 UTC: repeated S3 403 Forbidden from headObject
  • IAM inline policy verified and found correct
  • watcher Deployment restarted; no policy change
  • 2026-07-28 14:08:53 UTC: successful MergeAppend snapshot commit
  • 2026-07-28 14:09:15 UTC: another successful snapshot commit while draining
    old events

Expected behavior

A long-running watcher using IRSA should recover when temporary web-identity
credentials expire or when the role behind the same ServiceAccount is
reconciled/recreated. It should not require an external pod restart.

At minimum, after an S3 authentication failure the watcher should
invalidate/rebuild the AWS credentials/client state and retry the message with
freshly resolved web-identity credentials.

Suggested investigation

  • Verify whether the S3 client/FileIO is constructed with a refreshable
    web-identity credentials provider or with credentials resolved once at
    startup.
  • Check whether the S3 client is cached for the entire lifetime of
    InsertWatch without credential invalidation.
  • On 401/403, consider rebuilding the S3 client/FileIO and retrying once with
    newly resolved credentials before returning to the normal 20-second retry
    loop.
  • Add an integration test that rotates/recreates the IRSA role or otherwise
    invalidates temporary credentials while insert --watch remains running.

Current workaround

Restart the watcher Deployment to force a new web-identity session:

kubectl rollout restart deployment/<ice-watch-deployment>
Lingua principale
Java
Stelle
35
Fork
7
Merge medio
6g 8h
PR unite (30g)
5

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Altinity/ice

Tutte le issue di Altinity/ice

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.