Splunk Operator: Missing Hedless service or incorrect use of ClusterIP service.

Open
  • Is something wrong?
#1,885 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
66/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes
Domain
backend, devops

Research direction

Start by tracing checkLicenseRelatedPodFailures and how the LicenseManager StatefulSet gets its spec.serviceName; compare that with the service creation paths for Standalone, IndexerCluster, and MonitoringConsole. Reproduce the missing DNS name in a Kubernetes deployment and identify whether the controller should create the referenced headless service or query the existing ClusterIP service. Done means the license check succeeds without recurring DNS errors for LicenseManager and ClusterManager.

Written by the indexing model from the issue text.

Description

Please select the type of request

Bug

Tell us more

Describe the request

Bug: In Splunk Operator 3.2.0 the LicenseManager controller logs recurring errors from
checkLicenseRelatedPodFailures, because it queries the license manager pod through its
headless-service DNS name (splunk-lm-license-manager-0.splunk-lm-license-manager-headless...).
The operator does not create a headless service for LicenseManager, so the name never resolves:

{"level":"ERROR","msg":"failed to get license information from Splunk API","controller":"LicenseManager",
 "func":"checkLicenseRelatedPodFailures","error":"Get \"https://splunk-lm-license-manager-0.splunk-lm-license-manager-headless.<namespace>.svc.cluster.local:8089/services/licenser/licenses?output_mode=json\":
 dial tcp: lookup splunk-lm-license-manager-0.splunk-lm-license-manager-headless.<namespace>.svc.cluster.local: no such host"}

The LicenseManager StatefulSet references the headless service in spec.serviceName, but only the
ClusterIP service exists. This matches the documented service set (docs/Ingress.md lists only
*-license-manager-service and *-cluster-manager-service, no headless services for these components):

$ kubectl get sts splunk-lm-license-manager -o jsonpath='{.spec.serviceName}'
splunk-lm-license-manager-headless

$ kubectl get svc | grep -E "lm-|cm-"
splunk-cm-cluster-manager-service   ClusterIP   ...   8000/TCP,8089/TCP
splunk-lm-license-manager-service   ClusterIP   ...   8000/TCP,8089/TCP

The same gap exists for ClusterManager (spec.serviceName: splunk-cm-cluster-manager-headless, service
not created), though we have not observed errors caused by it yet. Headless services are created for
Standalone, IndexerCluster and MonitoringConsole.

Expected behavior

The license check succeeds without errors. Either:

  1. The operator creates the headless service referenced in the StatefulSet's spec.serviceName for
    LicenseManager (and ClusterManager), or
  2. checkLicenseRelatedPodFailures reaches the license manager via splunk-<name>-license-manager-service
    instead of the pod's headless DNS name.

Splunk setup on K8S

  • Splunk Operator 3.2.0 (Helm chart; CRDs applied from the 3.2.0 release splunk-operator-crds.yaml)
  • Splunk Enterprise 9.4.16 and 10.2.8 (behaviour identical on both)
  • CRs: LicenseManager, ClusterManager, IndexerCluster (3 peers, RF2/SF2), MonitoringConsole,
    Standalone (search head) and Standalone (3 heavy forwarders)
  • TLS on splunkd with certificates from cert-manager

Reproduction/Testing steps

  1. Deploy a LicenseManager CR with Splunk Operator 3.2.0.
  2. Check the StatefulSet: kubectl get sts splunk-<name>-license-manager -o jsonpath='{.spec.serviceName}'
    → splunk-<name>-license-manager-headless
  3. Check services: kubectl get svc | grep license-manager → only splunk-<name>-license-manager-service
  4. Watch the operator log:
    kubectl logs deploy/splunk-operator-controller-manager -n <operator-ns> | grep checkLicenseRelatedPodFailures
    → no such host errors.

K8s environment

  • OpenShift 4.22.15 (v1.35.6)
  • Operator deployed via Helm, managed by Argo CD
  • Reproduced on two separate clusters

Proposed changes(optional)

Either create the headless service for LicenseManager/ClusterManager (consistent with spec.serviceName),
or have checkLicenseRelatedPodFailures use the existing ClusterIP service.

K8s collector data(optional)

Can be provided on request.

Additional context(optional)

Workaround: creating the headless service manually resolves the error (verified: no further
no such host errors after creation):

apiVersion: v1
kind: Service
metadata:
  name: splunk-lm-license-manager-headless
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/component: license-manager
    app.kubernetes.io/instance: splunk-lm-license-manager
    app.kubernetes.io/managed-by: splunk-operator
    app.kubernetes.io/name: license-manager
    app.kubernetes.io/part-of: splunk-lm-license-manager
  ports:
    - name: https-splunkd
      port: 8089
      targetPort: 8089

A manually created service could conflict if a future operator version starts creating this service
itself, so an upstream fix would be preferred.

Impact: licensing itself works (all components use the ClusterIP service). Only the operator's
license-related pod failure check is non-functional and produces recurring ERROR log entries.

Dominant language
Go
Stars
260
Forks
134
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from splunk/splunk-operator

All issues in splunk/splunk-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.