[EKS] [bug]: Nitro Sandboxes silently reorder or drop exec stdin above ~64 KiB

Aperta
  • Qualcosa non va?
#2,913 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
43/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
aws, kubernetes, linux
Ambito
backend, networking

Direzione di ricerca

The report narrows the fault to after containerd-shim-nitro-sandbox-v2 reads the exec FIFO, possibly in the shim, vsock transport, or guest agent, but names no source files or tests. Start by locating the shim's exec stdin path and tracing how it sends data and signals close; compare the Nitro path with runc. Done means large stdin arrives in order and completely before close, with hashes matching the source across the reported sizes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

EKS Nitro Sandboxes
Community Note
  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

Exec stdin sent to a Nitro Sandbox pod (runtime handler isolated-sandbox) can arrive out of order, or with a 32 KiB chunk missing, once it's larger than about 64 KiB. The command still exits 0, so nothing reports it. Please deliver exec stdin in order, and all of it before the close.

Which service(s) is this request for?

EKS (Nitro Sandboxes, isolated mode)

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Anything that copies files into a pod over exec stdin gets a corrupted file with no error: kubectl exec -i … < file, tar … | kubectl exec -i … tar x, or Inspect AI's sandbox().write_file(). The EKS user guide says isolated mode allows exec and stdin, "so interactive debugging works the same as for a standard pod".

Repro, from a pod in the same cluster:

head -c 16777216 /dev/urandom > /tmp/s; sha256sum /tmp/s
kubectl exec -i <sandbox-pod> -- sh -c 'cat > /tmp/x; sha256sum /tmp/x; wc -c < /tmp/x' < /tmp/s

The hashes differ, and sometimes the file is 32,768 bytes short. Corrupted sends out of total, with kubectl exec -i v1.36.6:

stdin size 64 KiB 128 KiB 1 MiB 16 MiB
Nitro, c8i.4xlarge (Intel) 0/540 2/540 6/40 40/40
Nitro, c8a.4xlarge (AMD) 0/540 4/540 66/190 40/40
runc, same nodes 0/80 0/80 0/80 0/80

What goes wrong, from sending numbered bytes:

  • Usually every byte arrives once, but one 32 KiB chunk arrives after up to seven later chunks.
  • Otherwise, the chunk still in flight when stdin closes (usually the last) never arrives. Closing stdin 1 s after the last byte stops this, but not the reordering.

It isn't the client or the pod settings:

  • Client: the same with kubectl over WebSocket or SPDY, and with the Python client at any frame size from 4 KiB to 1 MiB.
  • Reader and pod settings: the same with cat or sha256sum reading stdin, with throttles on or off, with network-isolated, and at 1 to 4 vCPUs.
  • Sender speed: a slower sender corrupts less often, but still corrupts: at 1 MiB/s, 7 of 30 16 MiB sends.
  • Stdout: exec stdout from the same pod is intact, with 0 of 170 reads corrupted up to 16 MiB.

Where it happens: the node's containerd serves the exec stream and writes stdin into a FIFO, just as for the runc pods on the same node, which stay intact. So the fault comes after containerd-shim-nitro-sandbox-v2 reads that FIFO: in the shim, the vsock transport or the guest agent.

Are you currently working around this issue?

We send at most 64 KiB per exec and hash-check each piece; 0 of 40 4 MiB files were corrupted that way. It's slow: 20–40 s per 16 MiB.

Additional context

  • EKS 1.36.5 on self-managed nodes (Karpenter), with kernel 6.18.56, containerd 2.3.7 and Firecracker v1.18.0. Nitro Enclaves are enabled.
  • Runtime 0.1.4274.0 and the 1.36.5/2026-11-06 guest image.
  • Reproduced on two node setups:
    • amazon-eks-ami's install-pod-isolation.sh on the AL2023 EKS AMI v20261031;
    • the user guide's "Create and update a Nitro Sandbox AMI" page on v20261107.
  • Pods run ubuntu:24.04 with 2 CPU and 2 GiB.
  • We couldn't test cpu_overcommit = false: with it, every sandbox failed to start (Failed to start enclave: I/O error (os error 5)).
Lingua principale
Shell
Stelle
5.4k
Fork
332
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di aws/containers-roadmap

Tutte le issue di aws/containers-roadmap

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.