Orchestrate architecture-aware distributed Splunk upgrades

Open
  • Is something wrong?
#96 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
ansible, python
Domain
devops, infrastructure

Research direction

Start with ansible/upgrade_distributed.yml, then read ansible/upgrade_idxc_rolling.yml, ansible/upgrade_shc_rolling.yml and ansible/upgrade_splunk.yml to see how they are called, and the tier order in docs/upgrade.md. Tier progress labels are set in lib/spa/runlog.py, and the test to replace is test_upgrade_distributed_uses_manager_group_and_version_check. Done means the playbook runs tiers in the documented order, skips absent roles, stops on a failed tier, and the static test asserts that order.

Written by the indexing model from the issue text.

Description

enhancement

Summary

spa run upgrade_distributed does not upgrade a complex Splunk environment. ansible/upgrade_distributed.yml still follows a 6.6/7.0 comment and runs only two plays: every role_cluster_manager at once, then every role_search_head with serial: 1, both through splunk_software upgrade_software.

Indexer-cluster peers are not upgraded. Deployment server, license manager, monitoring console, SHC deployer, heavy forwarders, and universal forwarders are not upgraded. Search-head cluster members are upgraded as ordinary search heads instead of through the existing rolling playbook.

The per-tier procedure already exists in docs/upgrade.md, and the rolling plays already exist:

  • ansible/upgrade_idxc_rolling.yml (spa run upgrade_idxc_rolling)
  • ansible/upgrade_shc_rolling.yml (spa run upgrade_shc_rolling)
  • ansible/upgrade_splunk.yml (spa run upgrade_splunk --hosts …) for a single role or host

Ship

Rework ansible/upgrade_distributed.yml into one architecture-aware orchestration. Keep the operator command spa run upgrade_distributed --yes. It stays mutating and provisioned. Do not duplicate the rolling-upgrade task files; import or call the existing IDXC and SHC playbooks for clustered tiers, and use the single-host upgrade path for the other tiers.

Walk the roles that are actually present and skip the rest. For an environment that has them, use the 8.0+ order already documented in docs/upgrade.md (confirm it against the current Splunk upgrade order of operations for the supported versions; do not keep the 6.6/7.0 comment as the procedure):

  1. Deployment server
  2. License manager
  3. Cluster manager, after the indexer-cluster preflight
  4. Monitoring console
  5. Standalone search heads (not SHC members)
  6. SHC deployer
  7. Search-head cluster members, through upgrade_shc_rolling (captain and KV store checks stay with that playbook)
  8. Indexer-cluster peers, through upgrade_idxc_rolling
  9. Heavy forwarders
  10. Intermediate universal forwarders, one at a time when they are a pair
  11. Universal forwarders
  12. Version check across the upgraded hosts

Also cover the non-cluster cases the current play ignores: a standalone indexer uses the single-host upgrade, not the indexer rolling play; a search head that is not in an SHC uses the single-host upgrade, not upgrade_shc_rolling. Multiple idxcluster_* / shcluster_* groups stay inside the existing rolling playbooks.

Stop the run when a tier fails. Do not start the next tier after a failed upgrade.

Update compact progress in lib/spa/runlog.py. Today every upgrade_distributed play is forced into upgrade_distributed_cm or upgrade_distributed_sh, so a longer sequence would be mislabeled. Give each tier (and an imported rolling play) its own group. Richer stored transcripts stay in #93.

Replace test_upgrade_distributed_uses_manager_group_and_version_check, which currently locks the two-play shape. Assert the tier order, that clustered tiers call the rolling playbooks, that absent roles are skipped, and that an intermediate-forwarder pair is not upgraded together. A static playbook/fixture test is enough; do not require a live Splunk deployment.

Update docs/upgrade.md so spa run upgrade_distributed --yes is the full-environment path. Keep the existing per-tier commands for a partial upgrade.

Out of scope

  • Changing splunk_version in config and having that trigger the upgrade (separate Lifecycle roadmap item).
  • Windows universal forwarders (separate Lifecycle roadmap item).
  • Rewriting the internals of upgrade_idxc_rolling or upgrade_shc_rolling.
  • Storing full Ansible task results (#93) or credential redaction (#88).

Related

Notes

Pickup-ready enhancement. Skills and agents call spa only. Never print secrets.

Dominant language
Python
Stars
138
Forks
51
Avg merge
1h 38m
Merged PRs (30d)
48

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from splunk/splunk-platform-automator

All issues in splunk/splunk-platform-automator

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.