Flaky stream load balancing
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- docker, go
- Domain
- distributed-systems
Research direction
Check out the reproduce-load-balancing-issue branch and run the repeated E2E command in test/e2e. Start with the requireReplicatedStream stuck message, then inspect the attached logs and Docker container logs. Done means identifying the cause of the incorrect stream distribution between Mist nodes.
Written by the indexing model from the issue text.
Description
From time to time, the stream is not load balanced correctly between Mist nodes. I attach the logs of both catalyst nodes.
logs-node-1.txt
logs-node-2.txt
Steps to reproduce
1. Check out the code from the branch with additional debug logs
Check out the branch reproduce-load-balancing-issue. It adds additional logs and increases the timeout of the E2E Test
2. Run many tests sequentially to reproduce the flakiness
for i in $(seq 0 30); do go test -timeout 30m -v ./test/e2e/*; done
3. Notice that a test was stuck
You should see a message that the test is stuck.
#####
STUCK in requireReplicatedStream
4. Check the container logs
Now, you can check the logs of the containers and investigate the issue.
docker ps
- Dominant language
- Go
- Stars
- 24
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from livepeer/catalyst
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
All issues in livepeer/catalyst
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
LanternOps/breeze#8353 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
prime-radiant-inc/evener#4223 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Maintainers usually reply within 1 day
-
input.Scanner.Scan loops forever when the input reaches EOF, hanging every ConfirmAction promptPossibly taken @mlliarm claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
elastic/cloud-sdk-go#521 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
open-telemetry/opentelemetry-go-compile-instrumentation#1467 ·
Maintainers usually reply within 3 days