SBE update succeeds on hardware but update never reports success - "Initialization failure" in SBEUpdateOemInformation (ECE state stamp fails)

Start by reviewing the existing troubleshooting guides under TSG/SolutionExtension/ and the reported SBEUpdateOemInformation failure signature. The issue proposes documenting validation and recovery for a missing ECE state stamp; confirm the supported scope with maintainers before documenting manual state-store writes. Done means an agreed TSG covers the failure, safe diagnostic steps, and supported next actions.

Summary

On an Azure Local 24H2+ cluster running an HPE ProLiant Solution Builder Extension (SBE_HPE_ProLiant-Standard 5.0.2604.21 → 5.0.2608.10), the SBE update run completed every hardware stage successfully (CAU run green on all nodes, drive firmware installed, cleanup done) but then failed at the very last step — Finalize hardware update / SBEUpdateOemInformation — with an unhelpful exception. The cluster was fully patched and verified, but:

• the update could never report success,
• Get-SolutionUpdateEnvironment kept showing the old CurrentSbeVersion,
• and every later update (both SBE and solution updates) refused to start / did not appear, because the environment view was anchored to the old version.

This looks like success from the outside (hardware is updated) but behaves like a hard failure from the update engine's perspective. No existing TSG in this repo covers the finalize/stamp step, so we are sharing the signature and the recovery that worked.

Environment

• Azure Local, 2607.x era (also observed this behaviour matter on earlier 25xx builds)
• HPE ProLiant (Premier Solution), SBE family HPE-ProLiant-Standard
• SBE package SBE_HPE_ProLiant-Standard_5.0.2608.10 installed standalone (sideloaded via Add-SolutionUpdate), after a prior combined solution update
• Four-node cluster, orchestrator / Update Service cluster role subject to node moves during CAU reboots

Symptoms

Update run steps all green through hardware, then:

Type 'SBEUpdateOemInformation' of Role 'SBE' raised an exception:
Exception calling "GetResult" with "0" argument(s): "Initialization failure "
at SBEUpdateOemInformation, ...Classes\SBE\SBE.psm1: line 746
at Invoke-EceInterfaceInternal, ...ECEWinService\InvokeInterfaceInternal.psm1: line 165

Update run summary reports failures on SBE Package Update and Finalize hardware update despite the hardware work having completed. Get-SolutionUpdate may still show the SBE State: Installed with HealthState: Success for the package, while Get-SolutionUpdateEnvironment keeps showing the previous CurrentSbeVersion. Later Start-SolutionUpdate attempts return no run or fail quickly.

The exception is misleading: "Initialization failure" is not a validation error and does not identify what failed to initialize. In our case the same ECE client calls worked perfectly when executed interactively by hand — the client itself initialized fine; the in-context call from the action plan did not.

Cause (as we understand it)

Finalize hardware update is bookkeeping, not hardware. The SBE module calls three methods on the Enterprise Cloud Engine (ECE) client to stamp the new SBE version into the cluster's state store:

• SetOemFamily
• SetPackageVersion (for the SBE package)
• SetOemVersion

When those calls fail (here: an exception from inside the running action plan, likely related to the Update Service cluster role having moved between nodes during the CAU reboots), the cluster is patched but the state store is never stamped, so the update can never report success and discovery/eligibility stays anchored to the old version.

Validation

The ECE client module ships with the SBE package. Import it and build the same client the action plan builds (run on the node where the update orchestrates, with the NugetStore copy matching the installed SBE version):

Import-Module 'C:\NugetStore\Microsoft.AzureStack.Role.SBE.<sbe-version>\content\Helpers\Microsoft.AzureStack.Solution.LCMControllerWinService\ECEClient.psm1'
$c = Create-ECEClusterServiceClient -MaxRetryTimeInSec 60

Read before you write. Every getter returns a Task; unwrap the way the SBE module does:

"SBE package : $($c.GetPackageVersions().Result['SBE'])"
"OEM version : $($c.GetOemVersion().GetAwaiter().GetResult())"
"OEM family  : $($c.GetOemFamily().GetAwaiter().GetResult())"

In our case the store still held the old version while the staged SBE metadata (oemMetadata.xml under the staged SBE package) named the target family. That made the failure unambiguous: hardware patched, stamp missing.

Mitigation

Caution: the following writes the cluster's ECE state store by hand. We found it low-risk precisely because it is the identical operation the failed step performs, but it is still state surgery — take note of all current values first, and if you are uncomfortable, open a support case and reference this issue.

With the interactive client from above (which, unlike the in-context call, initializes successfully):

$c.SetOemFamily('<family-from-oemMetadata.xml>').GetAwaiter().GetResult()
$c.SetPackageVersion('SBE', '<new-sbe-version>').GetAwaiter().GetResult()
$c.SetOemVersion('<new-sbe-version>').GetAwaiter().GetResult()
# read back
"SBE now: $($c.GetPackageVersions().Result['SBE'])"

Values come from the staged package's oemMetadata.xml (OEM family) and the SBE package version. If any Set call throws from the working client, stop and read the exception — that is new information, not a retry signal.

After stamping, the update object may still be stuck because the Update Service environment view is a cached copy that only refreshes on certain events. Escalate through:

1. Restart the orchestrator cluster resource (no Restart-ClusterResource in Windows PowerShell 5.1 — stop/start):

Stop-ClusterResource 'Azure Stack HCI Orchestrator Service'
Start-ClusterResource 'Azure Stack HCI Orchestrator Service'

2. Restart the Update Service cluster resource the same way, then confirm the environment view picked up the stamped version:

Get-SolutionUpdateEnvironment | fl CurrentSbeVersion, LastUpdated, State, HealthState

3. Restart ECEAgent on every node (the agent executes SBE interface steps and can be days old after repeated failed runs).
4. If it persists, move the orchestrator role to a different node and retry.

In our case, after the stamp + orchestrator/Update Service restarts, Get-SolutionUpdateEnvironment reported the new CurrentSbeVersion (5.0.2608.10) with State: UpdateAvailable, and the previously invisible solution update reappeared and started cleanly.

Ask

• Is there a supported way to re-run just the finalize/stamp step, or to make SBEUpdateOemInformation retry/repair rather than strand a patched cluster?
• Would Microsoft accept a full TSG for the finalize-stamp failure signature in TSG/SolutionExtension/? Happy to contribute one.
• Root-cause question: what makes Create-ECEClusterServiceClient fail inside the action plan context but succeed interactively minutes later (Update Service role move? stale ECE connection pool in the action plan process?). "Initialization failure" with no inner message cost us several days of diagnosis.