[Bug]: Flaky Elixir Tests

Start by reproducing the failures in test/elixir/test/support/mango_database.exs and the named Mango and DiskMonitor tests, especially the Access.get/3 errors and status-code assertion. Trace the failing setup and test conditions across repeated runs. Done means the reported Elixir tests no longer fail intermittently or require repeated CI retries.

Version

unknown

Describe the problem you're encountering

Noticed a lot of elixir tests are flaky. Almost every CI run has at least a few failed elixir tests. They retry but sometimes even 3 retries are not enough. It's not an urgent issue, of course, as retries cover it up but it does make the tests run a bit longer

05:36:45    1) test limit (BasicFindTest)
05:36:45       test/elixir/test/mango/02_basic_find_test.exs:177
05:36:45       ** (FunctionClauseError) no function clause matching in Access.get/3
05:36:45  
05:36:45       The following arguments were given to Access.get/3:
05:36:45  
05:36:45           # 1
05:36:45           {"error", "not_found"}
05:36:45  
05:36:45           # 2
05:36:45           "id"
05:36:45  
05:36:45           # 3
05:36:45           nil
05:36:45  
05:36:45       Attempted function clauses (showing 6 out of 6):
05:36:45  
05:36:45           def get(-%module{} = container-, key, default)
05:36:45           def get(map, key, default) when -is_map(map)-
05:36:45           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
05:36:45           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
05:36:45           def get(list, key, _default) when -is_list(list)-
05:36:45           def get(-nil-, _key, default)
05:36:45  
05:36:45       stacktrace:
05:36:45         (elixir 1.18.4) lib/access.ex:320: Access.get/3
05:36:45         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
05:36:45         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
05:36:45         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
05:36:45         (couchdbtest 0.1.0) test/elixir/test/support/user_docs.ex:355: UserDocs.setup/4
05:36:45         test/elixir/test/mango/02_basic_find_test.exs:13: BasicFindTest.__ex_unit__/2
05:36:45  

DiskMonitorTest [test/elixir/test/disk_monitor_test.exs]
07:33:38    * test block interactive view indexing [L#30]
07:33:38    * test block interactive view indexing (30.9ms) [L#30]
07:33:38  
07:33:38    1) test block interactive view indexing (DiskMonitorTest)
07:33:38       test/elixir/test/disk_monitor_test.exs:30
07:33:38       Assertion with == failed
07:33:38       code:  assert resp.status_code == 507
07:33:38       left:  200
07:33:38       right: 507
07:33:38       stacktrace:
07:33:38         test/elixir/test/disk_monitor_test.exs:41: (test)

In this case all 3 retries failed in a row on two separate worker (so 6 total elixir failures): https://ci-couchdb.apache.org/job/jenkins-cm1/job/FullPlatformMatrix/job/main/1871/pipeline-console/?selected-node=374

   2) test choose index most accurate (ChooseCorrectIndexForDocs)
04:56:17       test/elixir/test/mango/12_use_correct_index_test.exs:81
04:56:17       ** (FunctionClauseError) no function clause matching in Access.get/3
04:56:17  
04:56:17       The following arguments were given to Access.get/3:
04:56:17  
04:56:17           # 1
04:56:17           {"error", "not_found"}
04:56:17  
04:56:17           # 2
04:56:17           "id"
04:56:17  
04:56:17           # 3
04:56:17           nil
04:56:17  
04:56:17       Attempted function clauses (showing 6 out of 6):
04:56:17  
04:56:17           def get(-%module{} = container-, key, default)
04:56:17           def get(map, key, default) when -is_map(map)-
04:56:17           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:56:17           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:56:17           def get(list, key, _default) w

 ChooseCorrectIndexForDocs [test/elixir/test/mango/12_use_correct_index_test.exs]
04:58:13    * test choose index with id [L#128]
04:58:13    * test choose index with id (41.1ms) [L#128]
04:58:14    * test chooses idxA [L#105]
04:58:14    * test chooses idxA (19.5ms) [L#105]
04:58:14  
04:58:14    1) test chooses idxA (ChooseCorrectIndexForDocs)
04:58:14       test/elixir/test/mango/12_use_correct_index_test.exs:105
04:58:14       ** (FunctionClauseError) no function clause matching in Access.get/3
04:58:14  
04:58:14       The following arguments were given to Access.get/3:
04:58:14  
04:58:14           # 1
04:58:14           {"error", "not_found"}
04:58:14  
04:58:14           # 2
04:58:14           "id"
04:58:14  
04:58:14           # 3
04:58:14           nil
04:58:14  
04:58:14       Attempted function clauses (showing 6 out of 6):
04:58:14  
04:58:14           def get(-%module{} = container-, key, default)
04:58:14           def get(map, key, default) when -is_map(map)-
04:58:14           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:58:14           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:58:14           def get(list, key, _default) when -is_list(list)-
04:58:14           def get(-nil-, _key, default)
04:58:14  
04:58:14       stacktrace:
04:58:14         (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:58:14         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:58:14         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:58:14         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:58:14         test/elixir/test/mango/12_use_correct_index_test.exs:54: ChooseCorrectIndexForDocs.__ex_unit_setup_1/1
04:58:14         test/elixir/test/mango/12_use_correct_index_test.exs:13: ChooseCorrectIndexForDocs.__ex_unit__/2
04:58:14  

05:01:11    1) test warn on full db scan (ChooseCorrectIndexForDocs)
05:01:11       test/elixir/test/mango/12_use_correct_index_test.exs:97
05:01:11       ** (FunctionClauseError) no function clause matching in Access.get/3
05:01:11  
05:01:11       The following arguments were given to Access.get/3:
05:01:11  
05:01:11           # 1
05:01:11           {"error", "not_found"}
05:01:11  
05:01:11           # 2
05:01:11           "id"
05:01:11  
05:01:11           # 3
05:01:11           nil
05:01:11  
05:01:11       Attempted function clauses (showing 6 out of 6):
05:01:11  
05:01:11           def get(-%module{} = container-, key, default)
05:01:11           def get(map, key, default) when -is_map(map)-
05:01:11           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
05:01:11           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
05:01:11           def get(list, key, _default) when -is_list(list)-
05:01:11           def get(-nil-, _key, default)
05:01:11  
05:01:11       stacktrace:
05:01:11         (elixir 1.18.4) lib/access.ex:320: Access.get/3
05:01:11         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
05:01:11         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
05:01:11         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
05:01:11         test/elixir/test/mango/12_use_correct_index_test.exs:54: ChooseCorrectIndexForDocs.__ex_unit_setup_1/1
05:01:11         test/elixir/test/mango/12_use_correct_index_test.exs:13: ChooseCorrectIndexForDocs.__ex_unit__/2

04:51:48    1) test json range (BeginsWithOperator)
04:51:48       test/elixir/test/mango/25_beginswith_test.exs:60
04:51:48       ** (FunctionClauseError) no function clause matching in Access.get/3
04:51:48  
04:51:48       The following arguments were given to Access.get/3:
04:51:48  
04:51:48           # 1
04:51:48           {"error", "not_found"}
04:51:48  
04:51:48           # 2
04:51:48           "id"
04:51:48  
04:51:48           # 3
04:51:48           nil
04:51:48  
04:51:48       Attempted function clauses (showing 6 out of 6):
04:51:48  
04:51:48           def get(-%module{} = container-, key, default)
04:51:48           def get(map, key, default) when -is_map(map)-
04:51:48           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:51:48           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:51:48           def get(list, key, _default) when -is_list(list)-
04:51:48           def get(-nil-, _key, default)
04:51:48  
04:51:48       stacktrace:
04:51:48         (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:51:48         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:51:48         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:51:48         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:51:48         test/elixir/test/mango/25_beginswith_test.exs:28: BeginsWithOperator.__ex_unit_setup_1/1
04:51:48         test/elixir/test/mango/25_beginswith_test.exs:13: BeginsWithOperator.__ex_unit__/2

) test use index without fallback fails for empty index (JSONIndexSelectionTest)
04:53:33       test/elixir/test/mango/05_index_selection_test.exs:284
04:53:33       ** (FunctionClauseError) no function clause matching in Access.get/3
04:53:33  
04:53:33       The following arguments were given to Access.get/3:
04:53:33  
04:53:33           # 1
04:53:33           {"error", "not_found"}
04:53:33  
04:53:33           # 2
04:53:33           "id"
04:53:33  
04:53:33           # 3
04:53:33           nil
04:53:33  
04:53:33       Attempted function clauses (showing 6 out of 6):
04:53:33  
04:53:33           def get(-%module{} = container-, key, default)
04:53:33           def get(map, key, default) when -is_map(map)-
04:53:33           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:53:33           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:53:33           def get(list, key, _default) when -is_list(list)-
04:53:33           def get(-nil-, _key, default)
04:53:33  
04:53:33       stacktrace:
04:53:33         (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:53:33         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:53:33         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:53:33         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:53:33         (couchdbtest 0.1.0) test/elixir/test/support/user_docs.ex:355: UserDocs.setup/4
04:53:33         test/elixir/test/mango/05_index_selection_test.exs:274: JSONIndexSelectionTest.__ex_unit__/2

04:57:04    1) test json range (BeginsWithOperator)
04:57:04       test/elixir/test/mango/25_beginswith_test.exs:60
04:57:04       ** (FunctionClauseError) no function clause matching in Access.get/3
04:57:04  
04:57:04       The following arguments were given to Access.get/3:
04:57:04  
04:57:04           # 1
04:57:04           {"error", "not_found"}
04:57:04  
04:57:04           # 2
04:57:04           "id"
04:57:04  
04:57:04           # 3
04:57:04           nil
04:57:04  
04:57:04       Attempted function clauses (showing 6 out of 6):
04:57:04  
04:57:04           def get(-%module{} = container-, key, default)
04:57:04           def get(map, key, default) when -is_map(map)-
04:57:04           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:57:04           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:57:04           def get(list, key, _default) when -is_list(list)-
04:57:04           def get(-nil-, _key, default)
04:57:04  
04:57:04       stacktrace:
04:57:04         (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:57:04         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:57:04         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:57:04         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:57:04         test/elixir/test/mango/25_beginswith_test.exs:28: BeginsWithOperator.__ex_unit_setup_1/1
04:57:04         test/elixir/test/mango/25_beginswith_test.exs:13: BeginsWithOperator.__ex_unit__/2
04:57:04  
04:57:04  

I think a common theme is around {error, not_found}. Maybe setup needs a tweak or need an extra clause just for not found with an internal retry?

Expected Behaviour

Flaky failures would be a bit more rare.

Steps to Reproduce

No response

Your Environment

No response

Additional Context

No response