VR command aggregation drops commands and fails with NPE when two jobs aggregate on the same router

Đang mở
  • Có gì không ổn?
#14,639 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@bhouse-nexthop đang làm issue này rồi.

Từ ngày 9/11/2026.

  • #14640 của @bhouse-nexthop — đang mở

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
55/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
java
Lĩnh vực
cloud, infrastructure

Hướng nghiên cứu

Start with core/.../agent/resource/virtualnetwork/VirtualRoutingResource.java and trace how AggregationControlCommand Start and Finish access the per-router queue, _vrLockMap, and aggregation map. Check the agent request threading and related tests before changing these shared maps. Done means concurrent aggregation for one router does not replace or misapply queued commands, create inconsistent locks, or fail when Finish finds no queue; add coverage for those cases.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

problem

Command aggregation on a virtual router isn't safe when two jobs aggregate commands for the same router at the same time. Parallel tier deploys in one VPC do this. Commands get dropped, and jobs fail with a NullPointerException.

VirtualRoutingResource (core/.../agent/resource/virtualnetwork/VirtualRoutingResource.java) keeps one aggregation queue per router name in a HashMap:

  • AggregationControlCommand Start puts a new queue for the router. A second Start while one exists replaces it, so the commands the first job has queued so far are never applied. Their answers were already successful, because a queued command is answered with success at once.
  • Every other NetworkElementCommand for that router goes into whichever queue is current.
  • Finish applies the current queue and removes it. The first job's Finish applies the second job's queue, and the second job's Finish then finds no queue. The loop and then queue.clear() in the finally block throw java.lang.NullPointerException: Cannot invoke "java.util.Queue.clear()" because "queue" is null.

The result is that one job fails, and another reports success although some of its configuration never reached the router.

The per-router lock map (_vrLockMap) and the aggregation map are also plain HashMaps changed from parallel agent request threads. The lock is created with a check-then-put, so two threads can get different locks for the same router.

This may be one cause of #11710: with redundant routers, the two routers of a VPC can end up with different configurations.

versions

4.22 (the code is the same on main and 4.20).

The steps to reproduce the bug
  1. Create a VPC with redundant routers on KVM.
  2. Create several tiers and deploy VMs into them in parallel, for example 24 deploys across the tiers at once.
  3. Some deploys fail. The management server logs Failed to re-program the network as a part of network ... implement due to aggregated commands execution failure!, and the router's host agent answers an AggregationControlCommand Finish with java.lang.NullPointerException: Cannot invoke "java.util.Queue.clear()" because "queue" is null.
  4. In our lab run, 12 of 24 parallel tier deploys failed. In two of three rounds, the two routers ended up with different keepalived VIP sets, and both were PRIMARY.
What to do about it?
  • On the agent:
    • do not replace an existing queue on Start;
    • do not fail with an NPE on a Finish whose queue was already applied;
    • make the maps thread-safe.
  • A complete fix would serialise aggregated applies per router on the management server, or tag each aggregation with an id. That is a larger change.
Ngôn ngữ chính
Java
Star
3.1k
Fork
1.4k
Merge trung bình
7 ngày 16 giờ
Pull request đã merge (30 ngày)
15

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/cloudstack

Tất cả issue của apache/cloudstack

Issue tương tự

Thêm issue về Java

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.