Splunk Operator: Missing Hedless service or incorrect use of ClusterIP service.

Start by tracing checkLicenseRelatedPodFailures and how the LicenseManager StatefulSet gets its spec.serviceName; compare that with the service creation paths for Standalone, IndexerCluster, and MonitoringConsole. Reproduce the missing DNS name in a Kubernetes deployment and identify whether the controller should create the referenced headless service or query the existing ClusterIP service. Done means the license check succeeds without recurring DNS errors for LicenseManager and ClusterManager.

Please select the type of request

Bug

Tell us more

Describe the request

Bug: In Splunk Operator 3.2.0 the LicenseManager controller logs recurring errors from
checkLicenseRelatedPodFailures, because it queries the license manager pod through its
headless-service DNS name (splunk-lm-license-manager-0.splunk-lm-license-manager-headless...).
The operator does not create a headless service for LicenseManager, so the name never resolves:

{"level":"ERROR","msg":"failed to get license information from Splunk API","controller":"LicenseManager",
 "func":"checkLicenseRelatedPodFailures","error":"Get \"https://splunk-lm-license-manager-0.splunk-lm-license-manager-headless.<namespace>.svc.cluster.local:8089/services/licenser/licenses?output_mode=json\":
 dial tcp: lookup splunk-lm-license-manager-0.splunk-lm-license-manager-headless.<namespace>.svc.cluster.local: no such host"}

The LicenseManager StatefulSet references the headless service in spec.serviceName, but only the
ClusterIP service exists. This matches the documented service set (docs/Ingress.md lists only
*-license-manager-service and *-cluster-manager-service, no headless services for these components):

$ kubectl get sts splunk-lm-license-manager -o jsonpath='{.spec.serviceName}'
splunk-lm-license-manager-headless

$ kubectl get svc | grep -E "lm-|cm-"
splunk-cm-cluster-manager-service   ClusterIP   ...   8000/TCP,8089/TCP
splunk-lm-license-manager-service   ClusterIP   ...   8000/TCP,8089/TCP

The same gap exists for ClusterManager (spec.serviceName: splunk-cm-cluster-manager-headless, service
not created), though we have not observed errors caused by it yet. Headless services are created for
Standalone, IndexerCluster and MonitoringConsole.

Expected behavior

The license check succeeds without errors. Either:

1. The operator creates the headless service referenced in the StatefulSet's spec.serviceName for
   LicenseManager (and ClusterManager), or
2. checkLicenseRelatedPodFailures reaches the license manager via splunk-<name>-license-manager-service
   instead of the pod's headless DNS name.

Splunk setup on K8S

• Splunk Operator 3.2.0 (Helm chart; CRDs applied from the 3.2.0 release splunk-operator-crds.yaml)
• Splunk Enterprise 9.4.16 and 10.2.8 (behaviour identical on both)
• CRs: LicenseManager, ClusterManager, IndexerCluster (3 peers, RF2/SF2), MonitoringConsole,
  Standalone (search head) and Standalone (3 heavy forwarders)
• TLS on splunkd with certificates from cert-manager

Reproduction/Testing steps

1. Deploy a LicenseManager CR with Splunk Operator 3.2.0.
2. Check the StatefulSet: kubectl get sts splunk-<name>-license-manager -o jsonpath='{.spec.serviceName}'
   → splunk-<name>-license-manager-headless
3. Check services: kubectl get svc | grep license-manager → only splunk-<name>-license-manager-service
4. Watch the operator log:
   kubectl logs deploy/splunk-operator-controller-manager -n <operator-ns> | grep checkLicenseRelatedPodFailures
   → no such host errors.

K8s environment

• OpenShift 4.22.15 (v1.35.6)
• Operator deployed via Helm, managed by Argo CD
• Reproduced on two separate clusters

Proposed changes(optional)

Either create the headless service for LicenseManager/ClusterManager (consistent with spec.serviceName),
or have checkLicenseRelatedPodFailures use the existing ClusterIP service.

K8s collector data(optional)

Can be provided on request.

Additional context(optional)

Workaround: creating the headless service manually resolves the error (verified: no further
no such host errors after creation):

apiVersion: v1
kind: Service
metadata:
  name: splunk-lm-license-manager-headless
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/component: license-manager
    app.kubernetes.io/instance: splunk-lm-license-manager
    app.kubernetes.io/managed-by: splunk-operator
    app.kubernetes.io/name: license-manager
    app.kubernetes.io/part-of: splunk-lm-license-manager
  ports:
    - name: https-splunkd
      port: 8089
      targetPort: 8089

A manually created service could conflict if a future operator version starts creating this service
itself, so an upstream fix would be preferred.

Impact: licensing itself works (all components use the ClusterIP service). Only the operator's
license-related pod failure check is non-functional and produces recurring ERROR log entries.