Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

insert --watch does not recover from stale IRSA credentials after IAM role recreation

Abierto
  • ¿Algo no está bien?
#202 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
50/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
aws, java
Área
backend, cli, cloud

Línea de trabajo

Comienza con InsertWatch y sigue el stack trace a través de Insert.processFile, BaseS3File y S3InputFile para rastrear cómo se crean y almacenan en caché el cliente S3 y las credenciales de identidad web. Reproduce o prueba un fallo de autenticación durante una supervisión de larga duración y, después, verifica que reintentar con credenciales renovadas procese el mensaje y haga commit del snapshot de Iceberg sin reiniciar el pod.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

A long-running ice insert --watch process on EKS can get stuck retrying
S3 operations with 403 Forbidden after its IAM/IRSA environment is
reconciled or the IAM role is recreated. The effective IAM policy is correct,
but the process does not recover until its Kubernetes pod is recreated.

After the pod restart, the same SQS messages and S3 objects are processed
successfully and Iceberg snapshots are committed. This suggests that the
long-running process keeps stale AWS credentials or an S3 client/session that
is not refreshed after the underlying IRSA identity changes.

Environment

  • ice: 0.16.0
  • Image: cr-enam.altinity.io/altinity/ice:0.16.0
  • AWS authentication: IRSA / web identity ServiceAccount annotation
  • Java: 21
  • Source: S3 ObjectCreated events delivered through SQS
  • Destination: Iceberg REST Catalog
  • Partitioning: hour(timestamp)

Command shape

The watcher is run as a long-lived process equivalent to:

ice insert \
  -p \
  --force-no-copy \
  --skip-duplicates \
  --partition='[{"column":"timestamp","transform":"hour"}]' \
  's3://<bucket>/cold/metrics-v5/**/*.parquet' \
  --watch='<SQS queue URL>'

Observed behavior

After an infrastructure reconciliation, the watcher continues receiving SQS
messages but fails while reading S3 object metadata:

ERROR c.a.i.c.i.cmd.InsertWatch > Failed to process batch of messages (retry in PT20S)
java.io.IOException: Error processing file(s)
...
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Forbidden
(Service: S3, Status Code: 403)
...
at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(...)
at org.apache.iceberg.aws.s3.BaseS3File.getObjectMetadata(...)
at org.apache.iceberg.aws.s3.S3InputFile.getLength(...)
at com.altinity.ice.cli.internal.iceberg.parquet.Metadata.readFooter(...)
at com.altinity.ice.cli.internal.cmd.Insert.processFile(...)

The watcher retries every 20 seconds but does not recover by itself.

The live inline role policy was checked while the failure was happening and
contained the required permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::<bucket>",
        "arn:aws:s3:::<bucket>/cold/metrics-v5/*"
      ]
    }
  ]
}

The failing object was under the allowed cold/metrics-v5/ prefix.

Recovery and evidence

Recreating only the watcher pod fixes the problem without changing the IAM
policy, SQS queue, S3 object, Iceberg table, or watcher arguments:

kubectl rollout restart deployment/vector-metrics-ice-watch-v5

Immediately after restart, the new process successfully processes the queued
objects:

INFO o.a.i.SnapshotProducer > Committed snapshot 6236223900153511381 (MergeAppend)
INFO c.a.i.c.i.cmd.InsertWatch > Received S3 event: ObjectCreated:Put -> s3://<bucket>/cold/metrics-v5/dt=2026-07-27/hour=18/<uuid>.parquet
...
INFO o.a.i.SnapshotProducer > Committed snapshot 197678787631752742 (MergeAppend)

It then drains the accumulated SQS backlog. No Forbidden or
Failed to process messages appear after the restart. --skip-duplicates
prevents already registered files from being added twice.

Timeline from this incident:

  • 2026-07-28 13:40:46 UTC: repeated S3 403 Forbidden from headObject
  • IAM inline policy verified and found correct
  • watcher Deployment restarted; no policy change
  • 2026-07-28 14:08:53 UTC: successful MergeAppend snapshot commit
  • 2026-07-28 14:09:15 UTC: another successful snapshot commit while draining
    old events

Expected behavior

A long-running watcher using IRSA should recover when temporary web-identity
credentials expire or when the role behind the same ServiceAccount is
reconciled/recreated. It should not require an external pod restart.

At minimum, after an S3 authentication failure the watcher should
invalidate/rebuild the AWS credentials/client state and retry the message with
freshly resolved web-identity credentials.

Suggested investigation

  • Verify whether the S3 client/FileIO is constructed with a refreshable
    web-identity credentials provider or with credentials resolved once at
    startup.
  • Check whether the S3 client is cached for the entire lifetime of
    InsertWatch without credential invalidation.
  • On 401/403, consider rebuilding the S3 client/FileIO and retrying once with
    newly resolved credentials before returning to the normal 20-second retry
    loop.
  • Add an integration test that rotates/recreates the IRSA role or otherwise
    invalidates temporary credentials while insert --watch remains running.

Current workaround

Restart the watcher Deployment to force a new web-identity session:

kubectl rollout restart deployment/<ice-watch-deployment>
Lenguaje dominante
Java
Estrellas
35
Forks
7
Merge medio
9 min
PR fusionados (30 d)
1

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Altinity/ice

Todos los issues de Altinity/ice

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.