Duplicate Management Servers entries

Start by tracing how the Management Server registers itself and how the listmanagementservers command reports records, then inspect the related persistence code and tests. The report identifies CloudStack 4.23.0 and shows multiple records with the same hostname and IP after an outage, but gives no source files or reproducible outage sequence. Done means a recovery scenario reuses the existing record and a regression test covers it.

problem

Title

Duplicate Management Server Entries Created After Host and Management Server Outage

Description

Specially following an outage affecting the hosts and the Management Server, CloudStack creates a new Management Server entry instead of reusing the existing one. The previous entries remain in the database with state=Down, resulting in multiple entries for the same Management Server, all sharing the same hostname and IP address.

We have observed this behavior in CloudStack 4.23.0. So far, we have been unable to reproduce it by simply restarting the Management Server or agent services.

The issue appears to leave stale Management Server records in the database, requiring manual cleanup.

Steps to Reproduce

1. Experience an outage affecting the hosts and the Management Server.
2. Restore the infrastructure and allow the Management Server to start.
3. Run cmk listmanagementservers to inspect the registered Management Servers.

Observed result: Multiple entries exist for the same Management Server, with identical hostnames and IP addresses. Previous entries remain in the Down state, while the latest entry is Up.

Expected Behavior

CloudStack should identify and reuse the existing Management Server entry when the Management Server restarts after an outage, rather than creating duplicate entries for the same instance.

Workaround

Manually remove the stale Management Server entries using the cmk tool. These entries cannot currently be deleted through the CloudStack UI.

Environment

• CloudStack version: 4.23.0
• OS: Red Hat Enterprise Linux 9.8
• Java: OpenJDK 17.0.20.1+1-LTS

Additional Information

The output of cmk listmanagementservers shows five entries for the same Management Server (acs01v, IP address 108.145.117.211): four entries in the Down state and one in the Up state.

The entries have different UUIDs and boot/start timestamps, suggesting that CloudStack registers a new Management Server record after certain recovery scenarios rather than updating the existing record.

The attached command output provides evidence of the duplicate entries.

# cmk listmanagementservers
{
  "count": 5,
  "managementserver": [
    {
      "agents": [],
      "agentscount": 0,
      "id": "2a4899aa-c94a-4a45-b175-6e382afd2908",
      "ipaddress": "108.145.117.211",
      "javadistribution": "Red Hat, Inc.",
      "javaversion": "17.0.20.1+1-LTS",
      "lastagents": [],
      "lastboottime": "2026-08-04T17:11:55+0000",
      "lastserverstart": "2026-09-07T14:28:10+0000",
      "lastserverstop": "2026-09-11T11:54:00+0000",
      "name": "acs01v",
      "osdistribution": "Red Hat Enterprise Linux 9.8 (Plow)",
      "pendingjobscount": 0,
      "serviceip": "108.145.117.211",
      "state": "Down",
      "version": "4.23.0.0"
    },
    {
      "agents": [],
      "agentscount": 0,
      "id": "b426d612-69cf-4076-b9f7-fd51f7f09848",
      "ipaddress": "108.145.117.211",
      "javadistribution": "Red Hat, Inc.",
      "javaversion": "17.0.20.1+1-LTS",
      "lastagents": [],
      "lastboottime": "2026-09-11T15:00:02+0000",
      "lastserverstart": "2026-09-11T15:03:10+0000",
      "name": "acs01v",
      "osdistribution": "Red Hat Enterprise Linux 9.8 (Plow)",
      "pendingjobscount": 0,
      "serviceip": "108.145.117.211",
      "state": "Down",
      "version": "4.23.0.0"
    },
    {
      "agents": [],
      "agentscount": 0,
      "id": "36d2aea4-de92-4144-8092-b85e6b5a0bc6",
      "ipaddress": "108.145.117.211",
      "javadistribution": "Red Hat, Inc.",
      "javaversion": "17.0.20.1+1-LTS",
      "lastagents": [],
      "lastboottime": "2026-09-22T13:09:55+0000",
      "lastserverstart": "2026-09-22T13:12:20+0000",
      "name": "acs01v",
      "osdistribution": "Red Hat Enterprise Linux 9.8 (Plow)",
      "pendingjobscount": 0,
      "serviceip": "108.145.117.211",
      "state": "Down",
      "version": "4.23.0.0"
    },
    {
      "agents": [],
      "agentscount": 0,
      "id": "f8106e4a-ae0d-4379-9455-1db3022406df",
      "ipaddress": "108.145.117.211",
      "javadistribution": "Red Hat, Inc.",
      "javaversion": "17.0.20.1+1-LTS",
      "lastagents": [],
      "lastboottime": "2026-10-16T00:45:02+0000",
      "lastserverstart": "2026-10-16T00:45:19+0000",
      "name": "acs01v",
      "osdistribution": "Red Hat Enterprise Linux 9.8 (Plow)",
      "pendingjobscount": 0,
      "serviceip": "108.145.117.211",
      "state": "Down",
      "version": "4.23.0.0"
    },
    {
      "agents": [
        "ddf6a499-b749-43d4-9bf2-b48750d7a60e",
        "5991fc17-8524-47bd-aea8-08bd02992e21",
        "d77e251f-e3f3-4287-beee-b54f2ec9de4e",
        "29b219f5-4e3d-4df9-b4f1-a837c17b00d8",
        "27c1f440-6c52-4868-9c80-65704c4e13a0",
        "18cb67d7-e201-4a07-8f38-ab97282c197a",
        "fa70337a-5adc-47f4-8911-f5e628aa9540",
        "2e79a92d-a842-4713-aa68-3ab4050af8b9",
        "8aa046f3-85a6-45ab-8a0f-39312899e84d",
        "f300c7ab-ec0f-49c6-be68-e2167baa4ce0",
        "7bbec373-a643-4dc7-ac8b-3dcc35381191",
        "f9a87e6d-8afd-4837-bd06-7994f87e0868",
        "bd2c3d29-2c73-482f-83d5-479ed18bbd0a",
        "9ee31367-3f4b-496c-9f53-0f825ff65782",
        "b344da4e-123f-4376-8e7d-ce6405647b59",
        "4a208773-a87b-4757-b1f0-a0b777edbdb7",
        "d3858ead-b752-48c6-8682-40527153456f",
        "2782f1a7-3f03-4249-ba73-11afa118e960",
        "f9d490cd-4718-4838-ada7-5cab656222aa",
        "ec0b57af-cb03-41bd-9606-c0336697d145",
        "9b804eb2-b3d9-4b84-89db-06fe217b029b",
        "41814525-d029-45a5-939a-3e306a3fc3d0",
        "5af03a0c-85c3-4e03-b9dc-37e4f2d330f1",
        "26be2dcf-c4db-465c-a323-1729ba793d74",
        "f2076754-6950-49c6-991f-4452d5ffbc83",
        "ae987858-0d31-49d4-acfb-8ea4e9ebcdec",
        "e2888a12-c26f-46f2-8366-70b9503b0a09",
        "8011c492-42ce-4297-a142-97926abd7cef"
      ],
      "agentscount": 28,
      "id": "5ddba450-94bb-4013-9d76-bb0215ab91e7",
      "ipaddress": "108.145.117.211",
      "javadistribution": "Red Hat, Inc.",
      "javaversion": "17.0.20.1+1-LTS",
      "lastagents": [
        "ddf6a499-b749-43d4-9bf2-b48750d7a60e",
        "5991fc17-8524-47bd-aea8-08bd02992e21",
        "d77e251f-e3f3-4287-beee-b54f2ec9de4e",
        "29b219f5-4e3d-4df9-b4f1-a837c17b00d8",
        "27c1f440-6c52-4868-9c80-65704c4e13a0",
        "18cb67d7-e201-4a07-8f38-ab97282c197a",
        "fa70337a-5adc-47f4-8911-f5e628aa9540",
        "2e79a92d-a842-4713-aa68-3ab4050af8b9",
        "8aa046f3-85a6-45ab-8a0f-39312899e84d",
        "f300c7ab-ec0f-49c6-be68-e2167baa4ce0",
        "7bbec373-a643-4dc7-ac8b-3dcc35381191",
        "f9a87e6d-8afd-4837-bd06-7994f87e0868",
        "bd2c3d29-2c73-482f-83d5-479ed18bbd0a",
        "9ee31367-3f4b-496c-9f53-0f825ff65782",
        "b344da4e-123f-4376-8e7d-ce6405647b59",
        "4a208773-a87b-4757-b1f0-a0b777edbdb7",
        "d3858ead-b752-48c6-8682-40527153456f",
        "2782f1a7-3f03-4249-ba73-11afa118e960",
        "f9d490cd-4718-4838-ada7-5cab656222aa",
        "ec0b57af-cb03-41bd-9606-c0336697d145",
        "9b804eb2-b3d9-4b84-89db-06fe217b029b",
        "41814525-d029-45a5-939a-3e306a3fc3d0",
        "5af03a0c-85c3-4e03-b9dc-37e4f2d330f1",
        "26be2dcf-c4db-465c-a323-1729ba793d74",
        "f2076754-6950-49c6-991f-4452d5ffbc83",
        "ae987858-0d31-49d4-acfb-8ea4e9ebcdec"
      ],
      "lastboottime": "2026-10-16T13:20:49+0000",
      "lastserverstart": "2026-10-19T19:08:59+0000",
      "lastserverstop": "2026-10-19T19:08:58+0000",
      "name": "acs01v",
      "osdistribution": "Red Hat Enterprise Linux 9.8 (Plow)",
      "pendingjobscount": 0,
      "serviceip": "108.145.117.211",
      "state": "Up",
      "version": "4.23.0.0"
    }
  ]
}
#

versions

The versions of ACS, hypervisors, storage, network etc..

The steps to reproduce the bug

...

What to do about it?

No response