KVM agent: VDDK auto-detection runs find / repeatedly, so the agent misses the Ready timeout and gets stuck in a reconnect loop

Đang mở
  • Có gì không ổn?
#14,585 4 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
58/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
java
Lĩnh vực
backend, infrastructure

Hướng nghiên cứu

Start by reading LibvirtComputingResource.resolveVddkLibDir(), detectVddkLibDir() and the VDDK capability checks named in the issue, then trace their calls from initialize() and LibvirtReadyCommandWrapper. Check how empty or invalid vddk.lib.dir values are handled and how often detection runs on hosts without virt-v2v. Done means the scans no longer delay ReadyCommand past its 60-second wait or trigger repeated agent reinitialization.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug
problem

Since VDDK support was added for VMware to KVM migration (#12970, extended in #13656 and restored in #14256), the KVM agent scans the whole filesystem many times while it starts up and while it handles ReadyCommand:

find / -type d -name 'vmware-vix-disklib-distrib' 2>/dev/null | head -n 1

On our hosts each scan takes about 20 seconds. The agent runs it repeatedly:
configure() (agent start) -> 2 two scans -> ~47s
initialize() (before StartupRoutingCommand) -> 5 scans -> ~108s

The management server sends ReadyCommand with "wait":"60", but the agent needs about 107 s to answer it. This sets off a loop that keeps repeating:

  1. The agent sends Startup, and the management server sends ReadyCommand.
  2. The ReadyAnswer is still not sent when the next ping goes out. The PingAnswer comes ba("Management server requested startup command to reinitialize the agent"), so the agentruns initialize() again, which means 5 more scans in parallel.
  3. The agent sends a second Startup, and the management server closes the connection righost connection to host ... Attempting reconnection while we still have 1 commands inprogress).
  4. The agent reconnects and step 1 starts again, roughly every 5 minutes.

The agent never gets to a stable connected state. A full agent restart takes more than a minute before it even sends its first Startup.

Root cause

In LibvirtComputingResource:

  • resolveVddkLibDir() calls detectVddkLibDir() (the find /) whenever vddkLibDir is blank out VDDK this is always the case, so the (empty) result is never cached and every callscans again.
  • hostSupportsVddk(String) calls resolveVddkLibDir() before it checks hostSupportsInstancerefore runs even on hosts where virt-v2v isn't installed (in our log: Failed to detectvirt-v2v password option: Cannot run program "virt-v2v").
  • hostSupportsVddk(), hostSupportsVddkBlockCopy(), hostSupportsVddkBlockCopyInPlaceFinaliBlockCopyRbd() and hostSupportsVddkRbdDirectImport() all end up in hostSupportsVddk(). They are called one after another in initialize() and in LibvirtReadyCommandWrapper, so each of them runs its own scan.
10:18:41,716 DEBUG [cloud.agent.Agent] Sending Startup: Seq 19-0
10:18:42,815 INFO  [cloud.agent.Agent] Processing agent ready command, agent id = 19
10:18:43,403 DEBUG [utils.script.Script] Executing command [/bin/bash -c find / -type d -name 'vmware-vix-disklib-distrib' 2>/dev/null | head -n 1 ].
10:19:03,098 DEBUG [utils.script.Script] Successfully executed process [...] for command [/bin/bash -c find / -type d -name 'vmware-vix-disklib-distrib' ...].
10:19:03,115 DEBUG [utils.script.Script] Executing command [/bin/bash -c find / -type d -name 'vmware-vix-disklib-distrib' 2>/dev/null | head -n 1 ].
10:19:24,828 DEBUG [utils.script.Script] Successfully executed process [...] for command [/bin/bash -c find / -type d -name 'vmware-vix-disklib-distrib' ...].
10:19:41,918 DEBUG [cloud.agent.Agent] Received response: Seq 19-1: { ... "PingAnswer": { ... "sendStartup":"true" ... } }
10:19:41,919 INFO  [cloud.agent.Agent] Management server requested startup command to reinitialize the agent
10:20:29,354 DEBUG [cloud.agent.Agent] ... ReadyAnswer ...
10:21:31,629 DEBUG [cloud.agent.Agent] Sending Startup: Seq 19-3
10:21:31,659 DEBUG [utils.nio.NioClient] Closing socket Socket[addr=/<mgmt-ip>,port=8250,...]
10:21:36,661 INFO  [cloud.agent.Agent] Lost connection to host: <mgmt-ip>. Attempting reconnection while we still have 1 commands in progress.
10:23:28,740 DEBUG [cloud.agent.Agent] Sending Startup: Seq 0-4
10:24:28,922 INFO  [cloud.agent.Agent] Management server requested startup command to reinitialize the agent
10:26:14,006 DEBUG [utils.nio.NioClient] Closing socket Socket[addr=/<mgmt-ip>,port=8250,...]
... (the cycle repeats)

Workaround

None that is reliable. Setting vddk.lib.dir in agent.properties stops the scans only if it points to a valid VDDK installation (lib64/libvixDiskLib.so*). An invalid or empty value still falls back to find /.

versions
  • CloudStack: latest main
  • Hypervisor: KVM, no VDDK and no virt-v2v installed
The steps to reproduce the bug
  1. Use a KVM host without VMware VDDK (and without virt-v2v), running a build that includes the latest main
  2. Restart cloudstack-agent.
  3. Check agent.log: find / runs many times, the ReadyAnswer comes long after the 60s wait, and the agent keeps reconnecting.
What to do about it?

No response

Ngôn ngữ chính
Java
Star
3.1k
Fork
1.4k
Merge trung bình
7 ngày 16 giờ
Pull request đã merge (30 ngày)
15

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/cloudstack

Tất cả issue của apache/cloudstack

Issue tương tự

Thêm issue về Java

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.