Losing redis connection
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 30/100
Research direction
Start with flowcept/flowceptor/consumers/document_inserter.py around _start at line 199, where pubsub.listen() raises the Redis connection error. Investigate the connection-closed path and the conditions that could cause messages or data to be lost. Done means the failure mechanism is identified and a reliable handling plan is documented or validated.
Written by the indexing model from the issue text.
Description
Sometimes, fortunately only rarely with the LLM experiment, we get the error below. We need to debug it to plan what to do. One possibility is simply to retry the connection and the failed request until it makes it. Today, if this error happens, we are likely losing data.
[flowcept][ERROR][frontier06306.frontier.olcf.ornl.gov][pid=61095][thread=140733193385728][function=_start][Connection closed by server.]
Traceback (most recent call last):
File "/lustre/orion/stf219/scratch/souzar/flowcept/flowcept/flowceptor/consumers/document_inserter.py", line 199, in _start
for message in pubsub.listen():
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/client.py", line 1653, in listen
response = self.handle_message(self.parse_response(block=True))
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/client.py", line 1531, in parse_response
response = self._execute(conn, try_read)
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/client.py", line 1507, in _execute
return conn.retry.call_with_retry(
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/retry.py", line 49, in call_with_retry
fail(error)
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/client.py", line 1509, in
lambda error: self._disconnect_raise_connect(conn, error),
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/client.py", line 1496, in _disconnect_raise_connect
raise error
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/client.py", line 1508, in
lambda: command(*args, **kwargs),
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/client.py", line 1529, in try_read
return conn.read_response()
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/connection.py", line 848, in read_response
response = self._parser.read_response(disable_decoding=disable_decoding)
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/connection.py", line 335, in read_response
result = self._read_response(disable_decoding=disable_decoding)
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/connection.py", line 383, in _read_response
response = [
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/connection.py", line 384, in
self._read_response(disable_decoding=disable_decoding)
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/connection.py", line 377, in _read_response
response = self._buffer.read(length)
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/connection.py", line 230, in read
self._read_from_socket(length - self.length)
File "/lustre/orion/stf219/scratch/souzar/miniconda/envs/llm3/lib/python3.8/site-packages/redis/connection.py", line 195, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
- Dominant language
- Python
- Stars
- 43
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ORNL/flowcept
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
DocumentInserter drops whole task batches when a payload key contains a dotPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Dependency-mgmt devops priority:low
Difficulty 3/5 1-2 days Newbie friendliness 45/100
-
Difficulty 4/5 3-5 days Newbie friendliness 50/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Netcracker/qubership-ai-packages#266 ·
Maintainers usually reply within 1 day
-
PROCESSED_DATA_FOLDER: strip whitespace and treat empty as unsetPossibly taken @TorecLuik claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
NL-BioImaging/BIOMERO.importer#63 ·
Maintainers usually reply within 1 day
-
MLX training-complete chat/merge suggestions omit --backend mlxPossibly taken @vaputa claimed this today. Openneeds-triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
MakazhanAlpamys/Soup#2913 · 1 comment ·
Maintainers usually reply within 1 day
-
area:security effort:S priority:P3 status:needs-triage type:chore
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
randomparity/kdive#3657 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
aws/mcp-proxy-for-aws#476 ·
Maintainers usually reply within 1 day