"Compaction failed" errors in Windows container when TSDB (Data directory) is stored in Docker volume · prometheus/prometheus#18308

(1 留言) (0 反應) (0 負責人)Go (64,042 star) (10,408 fork)batch import

component/tsdbhelp wanted

描述

What did you do?

It appears that Prometheus has issues running in a Windows container when its Data directory (TSDB) is stored in a Docker volume. When the container is using "Process" isolation, Prometheus is unable to run "compaction" against its TSDB. I was unable to find any reports regarding this particular problem.

These errors are logged and TSDB stays "uncompacted" forever:

time=2026-03-16T16:20:52.875+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVCWYJVA6GQK2FVTE7X2YBZ.tmp-for-creation data\\01KKVCWYJVA6GQK2FVTE7X2YBZ: The system cannot find the path specified."

This problem doesn't occur when TSDB is not stored in a Docker volume or some other Docker mount.

I've tried to use Hyper-V isolation mode for that Windows container, but encountered another issue which I think is covered in #14331. Prometheus simply crashes when Hyper-V isolation is used and there is a Docker volume. To add to #14331, the problem occurs with local Docker volumes as well.

Due to these problems, it appears that it's not possible to persist TSDB data when Prometheus is running in a Windows container. And in fact it seems that Prometheus doesn't simply support running in Windows containers. And if it's true, it would be great to mention this in the documentation somewhere. Sorry if I'm missing something here.

What did you expect to see?

My expectation is that Prometheus can be run within a Windows Docker container and persist data via volumes. I am operating under the assumption that Prometheus officially supports Windows container deployment.

What did you see instead? Under which circumstances?

I see that Prometheus cannot persist TSDB data when running in a Windows container which makes it impossible to run it in Windows containers beyond testing scenarios.

System information

Windows 11 24H2 [Version 10.0.26100.8037], Docker version 29.2.1, build a5c7197

Prometheus version

prometheus, version 3.10.0 (branch: HEAD, revision: 54e010926b0a270cadb22be1113ad45fe9bcb90a)
  build user:       root@a38ea3653297
  build date:       20260225-21:17:09
  go version:       go1.26.0
  platform:         windows/amd64
  tags:             builtinassets

Logs

Process isolation:

time=2026-03-16T16:27:53.131+03:00 level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1773400610784 maxt=1773403200000 ulid=01KKVD9S1BPXCYDESCK59Y5AFK
time=2026-03-16T16:27:53.170+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVD9S1BPXCYDESCK59Y5AFK.tmp-for-creation data\\01KKVD9S1BPXCYDESCK59Y5AFK: The system cannot find the path specified."
time=2026-03-16T16:28:53.170+03:00 level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1773400610784 maxt=1773403200000 ulid=01KKVDBKNJTV5T8CVR0VK9S09R
time=2026-03-16T16:28:53.211+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVDBKNJTV5T8CVR0VK9S09R.tmp-for-creation data\\01KKVDBKNJTV5T8CVR0VK9S09R: The system cannot find the path specified."
time=2026-03-16T16:29:53.212+03:00 level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1773400610784 maxt=1773403200000 ulid=01KKVDDE9WXX2QAMNZ6R4H17RC
time=2026-03-16T16:29:53.254+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVDDE9WXX2QAMNZ6R4H17RC.tmp-for-creation data\\01KKVDDE9WXX2QAMNZ6R4H17RC: The system cannot find the path specified."
time=2026-03-16T16:30:53.255+03:00 level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1773400610784 maxt=1773403200000 ulid=01KKVDF8Y7SFMKQPVX0559AZZ9
time=2026-03-16T16:30:53.297+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVDF8Y7SFMKQPVX0559AZZ9.tmp-for-creation data\\01KKVDF8Y7SFMKQPVX0559AZZ9: The system cannot find the path specified."
time=2026-03-16T16:31:53.297+03:00 level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1773400610784 maxt=1773403200000 ulid=01KKVDH3JHGM2Y62QDZ5EZTN8H
time=2026-03-16T16:31:53.336+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVDH3JHGM2Y62QDZ5EZTN8H.tmp-for-creation data\\01KKVDH3JHGM2Y62QDZ5EZTN8H: The system cannot find the path specified."
time=2026-03-16T16:32:53.336+03:00 level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1773400610784 maxt=1773403200000 ulid=01KKVDJY6R3G6VAP02ZAYNZFBN
time=2026-03-16T16:32:53.376+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVDJY6R3G6VAP02ZAYNZFBN.tmp-for-creation data\\01KKVDJY6R3G6VAP02ZAYNZFBN: The system cannot find the path specified."
time=2026-03-16T16:33:53.377+03:00 level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1773400610784 maxt=1773403200000 ulid=01KKVDMRV126K9A2RQVS6P4V0G
time=2026-03-16T16:33:53.433+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVDMRV126K9A2RQVS6P4V0G.tmp-for-creation data\\01KKVDMRV126K9A2RQVS6P4V0G: The system cannot find the path specified."
time=2026-03-16T16:34:53.434+03:00 level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1773400610784 maxt=1773403200000 ulid=01KKVDPKFTVV3Z78NS52R609VR
time=2026-03-16T16:34:53.488+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVDPKFTVV3Z78NS52R609VR.tmp-for-creation data\\01KKVDPKFTVV3Z78NS52R609VR: The system cannot find the path specified."
time=2026-03-16T16:35:53.489+03:00 level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1773400610784 maxt=1773403200000 ulid=01KKVDRE4HY6XHMAYJ54NJT52W
time=2026-03-16T16:35:53.533+03:00 level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact head: persist head block: rename block dir: rename data\\01KKVDRE4HY6XHMAYJ54NJT52W.tmp-for-creation data\\01KKVDRE4HY6XHMAYJ54NJT52W: The system cannot find the path specified."

貢獻者指南

技術棧: godocker
領域: infrastructure
議題類型: bug
難度: 4
預計時間: over 1 week
活動狀態: fresh
清晰度: clear
前置要求: Windows container experienceDocker volumesPrometheus TSDB internalsGo file operations
新手友善度: 15
研究方向: Investigate the rename operation failure in Windows containers when TSDB is on a Docker volume. Focus on the compact.go and db.go files (referenced in logs). Determine if the issue is caused by Docker volume path handling or Windows filesystem restrictions. Test with Process and Hyper V isolation, and consider whether a workaround like using symlinks or adjusting permissions could apply. Coordinate with maintainers via the linked issue #14331 to better understand the Hyper V isolation crash.