Using checkstate.sh as liveness probe leads to failing HEC connections
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- docker, kubernetes, shell
- Domain
- devops, infrastructure
Research direction
Start with checkstate.sh and its curl check against localhost:8089. Review how the container exposes HECs and receivers, then determine how their responses should affect liveness while retaining the current 8089 check when none are running. Done means the probe stops routing traffic when those services are unavailable without breaking the existing fallback.
Written by the indexing model from the issue text.
Description
We are using docker-splunk in k8s and therefore use checkstate.sh as liveness probe. The problem is that checkstate.sh executes the following to check if Splunk is still running: curl --max-time 30 --fail --insecure $scheme://localhost:8089/
So it checks if splunkd is still running on port 8089, but this is probably the thing that is available until the very last second if Splunk shutting down so Splunk Web, HECs, Receivers,... they are all already gone when this endpoint will still return 200. As long as this returns 200 the LoadBalancer or something like an ingress-nginx will happily send traffic to the endpoint, leading to timeouts and broken connections.
My proposal to fix this would be to apply the following logic in checkstate.sh:
- Check if there are HECs and receivers running
- If so, assess the liveness of the container based on the response of these ports and not 8089
- If not, stay with the current check and see if 8089 is still available
There may even be better ways to achieve this, maybe someone has an idea?
If we agree on a fix I would be happy to create an MR to solve this.
- Dominant language
- Python
- Stars
- 549
- Forks
- 277
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from splunk/docker-splunk
-
Difficulty 2/5 Half a day Newbie friendliness 78/100
splunk/docker-splunk#730 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 58/100
splunk/docker-splunk#737 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
splunk/docker-splunk#732 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
splunk/docker-splunk#731 · 1 reaction ·
-
Difficulty 3/5 1-2 days Newbie friendliness 38/100
splunk/docker-splunk#724 ·
All issues in splunk/docker-splunk
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100