User Overrides for OpenCHAMI Boot Service (boot-service)

This is a feature specification for layered boot parameter overrides (global, group, node by MAC) in the OpenCHAMI Boot Script Service, with no code files or entry points named. Start with the open questions and the contradictory list-merge criteria (AC-004 and AC-005), which need a maintainer decision before any implementation. Done would mean the precedence rules are settled and the Ansible-managed YAML override files merge as specified.

User Overrides for Boot Script Service (BSS)

---

Summary

The Infrastructure Administrator (Infra Admin) can manage boot parameters for nodes at global, group, and individual levels, enabling customized configurations and troubleshooting without requiring manual intervention on each node.

---

Problem Statement

Currently, administrators cannot use Omnia to manage heterogeneous clusters where nodes in a functional group consist of different server brands, generations, or internal components (e.g., NICs, CPUs). The primary pain point arises when an administrator attempts to manually modify the Boot Script Service (BSS) configuration to accommodate this diversity; any manual changes are overwritten the next time Omnia auto-generates the configuration. This forces administrators into a frustrating and inefficient cycle of trial and error, requiring multiple cluster reboots and causing significant downtime. This limitation makes it nearly impossible to manage mixed-hardware environments effectively with Omnia.

---

Value Proposition

Customer Value

Customer Administrators can now independently and safely manage boot configurations for diverse hardware, eliminating the frustrating cycle of manual edits being overwritten by Omnia. This empowerment accelerates the process of bringing new or mixed hardware online and reduces cluster downtime, allowing critical research and computational work to proceed with minimal interruption. The feature provides a reliable, persistent, and auditable method for managing essential boot parameters, increasing administrator confidence and satisfaction with the platform.

---

User Personas

Infrastructure Administrator (Infra Admin)

• Role Context: Manages the cluster post-handover. Owns the logical cluster (Slurm, K8s, OpenCHAMI config, functional groups).
• Interaction with Features: Will define and apply boot parameter overrides at the global, group, and individual node levels to enforce policies, troubleshoot issues, and customize node behavior.
• Needs and Pain Points: Needs a centralized and scalable way to manage boot configurations without physical access to nodes. Current methods are manual, error-prone, and time-consuming.

Deployment Engineers

• Role Context: Responsible for initial cluster setup and configuration before handover to the customer.
• Interaction with Features: May use the override mechanism during initial deployment to apply site-specific configurations or work around hardware-specific issues.
• Needs and Pain Points: Needs to reduce per-node configuration time and enable faster cluster delivery.

---

Use Cases

Use Case 1: Applying a Global Security Policy

Actor: Infrastructure Administrator (Infra Admin)
Context: A new security mandate requires SELinux to be enabled on all nodes.
Action: The administrator defines a global override to set security.selinux to true.
Result: All nodes in the cluster boot with SELinux enabled, ensuring compliance with the new policy.

Use Case 2: Troubleshooting a Faulty Node

Actor: Infrastructure Administrator (Infra Admin)
Context: A specific node is failing to boot due to a problematic driver.
Action: The administrator applies a node-specific override to blacklist the faulty driver for that node's MAC address.
Result: The node successfully boots, allowing for further diagnostics without impacting the rest of the cluster.

Use Case 3: Correcting a Garbled Serial Console

Actor: Infrastructure Administrator (Infra Admin)
Context: During a hardware refresh, a new generation of servers is added to an existing functional group. These new servers have a different default serial console baud rate, causing all console output during the boot process to be unreadable, which prevents any troubleshooting.
Action: The administrator applies node-specific overrides for the new servers, setting the console.baud_rate to the correct value for that hardware.
Result: The serial console output for the new servers becomes readable, allowing for proper monitoring and diagnostics during boot-up without affecting the existing servers in the group.

---

Scope Definition

Goals (Execution Outcomes)

• Goal 1: Provide a mechanism to define boot parameter overrides at three levels: global, functional group, and individual node (by MAC address).
• Goal 2: Ensure that overrides are merged in the correct order of precedence (Node > Group > Global).
• Goal 3: Support adding, editing, and removing boot parameters through the override mechanism.

Non-Goals (Scope Boundaries)

• Non-Goal 1: This feature will not provide a graphical user interface for managing overrides. All interactions will be via configuration files or APIs.
• Non-Goal 2: The override mechanism is for boot parameters provided by the BSS only; it does not extend to other configuration services.

---

Requirements

Requirement: The system must allow defining global boot parameter overrides. [Req 1]

Scenario: [Happy path]

• When a global override for the console baud rate is applied, all nodes booting without a more specific override use the new baud rate.

• When the BSS attempts to apply overrides that contain errors, the system logs an error and falls back to the default configuration.

Requirement: The system must allow defining boot parameter overrides for functional groups. [Req 2]

Scenario: [Happy path]

• When a group-specific override is applied for the same parameter, nodes within that group use the group-specific value, while other nodes use the global value.

Requirement: The system must allow defining node-specific boot parameter overrides. [Req 3]

Scenario: [Happy path]

• When a node-specific override is applied for the same blacklist (removing one driver), the specific node uses the shorter, node-specific blacklist, while other nodes in the group use the group-level list.

• When the BSS attempts to apply overrides that contain errors, the system logs an error and the affected node falls back to its group or global configuration.

---

Usability

• CLI Standards: Any new CLI commands for managing overrides must include clear help text (-h, --help) and provide examples of common usage.
• Error Messaging: Error messages related to override application (e.g., syntax errors in the override file) must be clear, concise, and indicate the location of the error.

---

Reliability

Failure Mode	System Behavior
Invalid override file syntax	The system will log an error and ignore the invalid override, falling back to the next level of configuration (e.g., group to global).
BSS service unavailable	Nodes will fail to boot, as this is core infrastructure. Standard cluster recovery procedures apply.

Availability: The BSS is a critical component and must match the availability targets of the overall Omnia control plane.
Recovery: Changes to overrides should be auditable and version-controlled to allow for easy rollback.

---

Security

• Authorization: Changes to the override configuration must be restricted to authorized personnel (e.g., Customer Administrators) via standard RBAC.
• Secrets Management: If any sensitive values are included in boot parameters, they must be managed via secure encrypted storage or a similar secrets management tool.
• Data Integrity: The override mechanism must not be able to inject arbitrary commands or compromise the integrity of the boot process.

---

Performance

Metric	Target
Override application overhead	The process of merging and applying overrides should add no more than 100ms to the node boot time.

---

Documentation

• User Guide: Update the Omnia user guide to include a section on how to configure and use the BSS override feature.
• Release Notes: Announce the new feature in the release notes.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Incorrect override causes boot failure	Technical	High	Medium	Provide clear documentation and examples. Implement a "dry run" mode to validate override changes before applying.
Complex overrides are difficult to manage	Adoption	Medium	Medium	Provide clear documentation on precedence rules. Encourage keeping overrides as simple as possible.
Unauthorized changes to overrides	Security	High	Low	Enforce strict RBAC on the configuration files and any APIs used to manage overrides.

---

Dependencies

• OpenCHAMI BSS: This feature is an enhancement to the existing Boot Script Service.
• Configuration Management: A mechanism (e.g., Ansible, GitOps) is required to manage and apply the override configuration files.
• BR-36094 (Kernel Boot Parameter Management): Boot parameter management via the BSS override system should share the same underlying architecture to avoid conflicting UX mechanisms for administrators.

---

Acceptance Criteria

• AC-001: A global override for a boot parameter is correctly applied to all nodes. [Req 1]
• AC-002: A group-level override correctly takes precedence over a global override. [Req 2]
• AC-003: A node-level override correctly takes precedence over group and global overrides. [Req 1]
• AC-004: The system correctly merges list-based parameters (e.g., driver blacklists) as described in the concept. [Req 1]
• AC-005: The system correctly merges list-based parameters (e.g., driver blacklists) by replacing the parent list with the child list, as described in the concept. [Req 1]
• AC-006: An error is logged when an invalid override file (e.g., incorrect YAML syntax) is processed, and the system falls back to the next valid configuration level. [Req 1]
• AC-007: The system correctly applies overrides based on the documented precedence (Node > Group > Global). [Req 1]
• AC-008: An error is logged for a node-specific override that uses an invalid or unrecognized MAC address format. [Req 3]

---

Scope & Sequencing

• MVP: The initial release will support overrides via configuration files managed by Ansible.
• Future: A future release may include a REST API for programmatic management of overrides.

---

Assumptions / Open Questions

Assumptions

• A1: Users will have the necessary permissions to modify the override configuration files.
• A2: A version control system (e.g., Git) will be used to manage changes to override configurations.

Open Questions

• Q1: (Resolved) The override configuration files will use the YAML format.
• Q2: Will there be a need for a "dry run" or validation feature to test override changes before they are applied?

---