zalando/zalenium

Kubernetes containers occasionally not removed

Open

#1,009 opened on Jul 16, 2019

View on GitHub
 (8 comments) (4 reactions) (0 assignees)Java (2,385 stars) (584 forks)batch import
help wantedwaiting-answer

Description

🐛 Bug Report

Using a Zalenium installation in kubernetes (via helm chart), occasionally containers are not removed when no tests are running despite max test sessions being set to 1 and desired containers set to 0. We currently run a few thousand tests a day and this only seems to happen about one in thousand or so tests. Max container # is set to 15 and often throughout the day all 15 slots are being consumed.

This is particularly unfortunate for us as the orphaned (probably not the right term here) containers often result in our Zalenium node pool to remain scaled up indefinitely.

As well, we have max container number set to 15 so slowly (if not noticed) the capacity is eaten up and tests run slower until the stop all together.

To Reproduce

Install Zalenium via helm into a kubernetes cluster. Have max test sessions being set to 1, desired containers set to 0, max containers set to 15. Repeatedly run tests at a rate which consume all 15 slots for some time. Eventually some containers will not be properly cleaned up.

Expected behavior

When max test sessions is set to 1 and desired containers is set to 0, all containers should be removed when no tests are running.

Environment

OS: Container Optimized OS (GKE) Zalenium Image Version(s): dosel/zalenium:3.141.59n Selenium Image: elgalu/selenium:3.141.59-p14 If using Kubernetes, specify your environment, and if relevant your manifests: GKE - v1.13.7-gke.8.

Contributor guide

Tech stack
javadockerkubernetes
Domain
devopsinfrastructurecloudtesting
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
KubernetesDockerSelenium Grid
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
Investigate the container lifecycle management in Zalenium, focusing on the Kubernetes pod cleanup logic. Look for race conditions when multiple tests finish concurrently. Examine the code that manages desired container count and identify why containers are not removed despite max test sessions being 1. Check if there is a timing issue with the Kubernetes API or the container status polling.