tests: two race tests share one boto3 client across threads and failed with TLSV1_ALERT_DECODE_ERROR

Open Beginner friendly
#894 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 4 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
aws, python, rust

Research direction

Read the race fixture in tests/test_update_item_creates_key.py and the worker setup in tests/test_transact_write_conflict.py, then compare them with the per-writer client setup in tests/test_index_rows_track_base_item.py and the fix described in #392. Give each racing writer its own client and run the affected tests; done when both race tests pass without sharing a client connection pool across writer threads.

Written by the indexing model from the issue text.

Description

ci:stability

Two cassandra-pytest jobs failed with this error from the test client:

botocore.exceptions.SSLError: SSL validation failed for https://127.0.0.1:18443/ [SSL: TLSV1_ALERT_DECODE_ERROR] tlsv1 alert decode error
  • job 126933918802, fix/367-table-name-validation, 4 November 23:20 UTC: tests/test_transact_write_conflict.py::test_opposite_order_transactions_cancel_instead_of_failing, the test of #879
  • job 127366989897, fix/sigv4-scope-date-first, 5 November 16:25 UTC: four tests of tests/test_update_item_creates_key.py, at the setup of the race fixture

In both files, every writer thread uses one raw_client fixture object. The race fixture in test_update_item_creates_key.py starts 2 × WRITERS threads on one raw_client. The workers in test_transact_write_conflict.py share its raw_client in the same way.

#392 found this error when threads shared one client in test_rfc0003_concurrency.py. As #392 explains, two threads that interleave on one TLS connection corrupt its record stream, and the request fails with this error. #392 gave each thread its own client. test_index_rows_track_base_item.py also builds one client per racing writer, so that the writers share no connection pool.

These were 2 failures in 297 completed cassandra-pytest jobs between 3 November 00:00 UTC and 6 November 15:07 UTC. I have not reproduced them.

Dominant language
Rust
Stars
517
Forks
60
Avg merge
5d 3h
Merged PRs (30d)
79

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ExtendDB/extenddb

All issues in ExtendDB/extenddb

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.