json-file logging does not recover after ENOSPC and can block container output
メンテナーはふだん 1 日以内に返信
評価
調査の方向性
Start with the reproducer docs/testing/repro-json-log-enospc.sh and its explanation in docs/testing/json-log-enospc.md, then read the JSON log consumer and its json.Encoder write path. Done means logging resumes after a transient ENOSPC without blocking output, with regression tests for both async consumers, persistent failure, partial writes and synchronous recovery. Open PR #5249 already implements this, so coordinate with it before starting.
索引モデルが issue の本文から書いたものです。
説明
Description
A transient JSON log write failure can permanently stop logging, even after disk space is available again. With nerdctl 2.2.2, the asynchronous JSON consumer exits on the first failed write; the producer channel and stdout/stderr FIFO then fill, blocking container output. On the PR base (add513266aea6883b60a3e54ba2ef4ced948c903), JSON logging uses synchronous writes, but its json.Encoder retains the first writer error and never resumes writing later entries.
This report uses only a synthetic Alpine container and a private 8 MiB tmpfs. No host filesystem is filled and no existing workload is modified.
Steps to reproduce the issue
On a disposable Linux host with root access, containerd, Bash and standard mount/dd/grep tools:
curl -fsSL https://raw.githubusercontent.com/alongubkin/nerdctl/afcad2e3/docs/testing/repro-json-log-enospc.sh -o repro-json-log-enospc.sh
sudo bash repro-json-log-enospc.sh /path/to/nerdctl
The script:
- Starts a fresh container in a unique namespace. Its JSON log is on a private 8 MiB tmpfs; its progress counter is on a different filesystem.
- Fills only that tmpfs and verifies the filler write fails with ENOSPC.
- Produces 30,000 lines on each output stream, exceeding the old per-stream producer channel capacity.
- Frees space and checks that the workload finishes and both recovery markers reach the JSON log.
- Removes its container and temporary mount/files. Pulled image content can remain cached.
Rotation is bounded (max-size=1m,max-file=2) to prevent immediate re-exhaustion after the filler is removed. The default snapshotter is overlayfs; use sudo env SNAPSHOTTER=native bash ... if native is configured. CONTAINERD_ADDRESS selects another daemon socket. The selected snapshotter must be supported by containerd's image unpack configuration.
Describe the results you received and expected
Observed with the same synthetic workload:
| Binary | Result after freeing space |
|---|---|
| nerdctl 2.2.2 | Container output stays blocked at approximately 10,000 iterations; script exits 1. |
Unmodified PR base add51326 |
Workload completes, but neither recovery marker reaches the JSON log; script exits 1. |
| PR #5249 | Workload completes all 30,000 writes per stream; both markers are logged; script exits 0. |
Expected: continue consuming output while writes fail, dropping unwritable records, and resume logging subsequent records after the sink recovers. Lost records cannot be recovered. A partial record left by a failed write is a separate corruption concern; it must not be joined to the next valid record.
The PR also includes regression tests for both asynchronous consumers, persistent write failure, partial writes followed by recovery, and synchronous recovery. All three fail against the unchanged implementation and pass with the fix.
What version of nerdctl are you using?
- Released nerdctl 2.2.2.
- Unmodified PR base
add513266aea6883b60a3e54ba2ef4ced948c903, built withmake BUILDTAGS=no_ipfs. - Fixed implementation from PR #5249, with reproducer at
afcad2e3.
Are you using a variant of nerdctl?
No. Official release binaries and builds from upstream source.
Host information
Disposable rootful Linux arm64 test environment; kernel 7.0.14-orbstack-00374-gbbca68e8d741; native snapshotter. The released 2.2.2 test used containerd 2.2.1; the source-base/fixed comparison used containerd 2.4.1 with native unpack configured. runc 1.4.0. Only the logging implementation differed between the source-base and fixed comparison.
Related fix: #5249. Prepared with Codex assistance; the failure, recovery, and regression tests were run locally.
- 主要言語
- Go
- スター
- 10.4k
- フォーク
- 834
- 平均マージ
- 2日 18時間
- マージ済み PR（30日）
- 37
環境構築
- Dockerfile または Docker Compose ファイルあり
- プルリクエストのテンプレートなし
- コントリビューションガイドなし
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
containerd/nerdctl のほかの issue
-
`nerdctl kill` shouldn't set `containerd.io/restart.explicitly-stopped` label for non-stop signals再び着手できるかも このイシューのプルリクエストはマージされずにクローズされました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
containerd/nerdctl#5171 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信
-
area/project
難易度 4/5 3〜5日 初心者へのやさしさ 45/100
containerd/nerdctl#5244 ·
メンテナーはふだん 1 日以内に返信
-
container cannot request other container when not set networks対応中かも @manenim が 14 日前に担当しました。 オープンbug
難易度 4/5 3〜5日 初心者へのやさしさ 48/100
containerd/nerdctl#5228 · コメント 3 件 · 担当者 1 名 ·
メンテナーはふだん 1 日以内に返信
-
area/ci
難易度 3/5 1〜2日 初心者へのやさしさ 10/100
containerd/nerdctl#5202 · コメント 5 件 · リアクション 1 件 ·
メンテナーはふだん 1 日以内に返信
-
platform/Windows/Non-WSL2
難易度 4/5 3〜5日 初心者へのやさしさ 55/100
containerd/nerdctl#5065 ·
メンテナーはふだん 1 日以内に返信
containerd/nerdctl の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 81/100
apache/rocketmq-dashboard#6895 ·
メンテナーはふだん 3 日以内に返信
-
agent-butler-finding chore
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
jordansmall/spindrift#5466 ·
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 87/100
777genius/agent-notifications#490 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
devantler-tech/ksail#7836 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
keyxmakerx/Chronicle#1469 ·
メンテナーはふだん 1 日以内に返信