3.6.4.0 on python3.9: SemLock __exit__ throws '[WinError 6] The handle is invalid'

Open
  • Is something wrong?
#335 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
backend, operating-systems

Research direction

Start by reading SimpleQueue.get_payload and SemLock.exit, then reproduce the failure under Python 3.9.2 on Windows using the pytest setup described. Use the shutdown log to identify where the invalid handle is raised; done means the exception no longer occurs during pool shutdown and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Description

When the semLock acquired in SimpleQueue.get_payload exits, I get an exception: [WinError 6] The handle is invalid
I'm calling billiard from pytest unit tests running under pyCharm 2019.2.2 (Professional Edition)
The problem has appeared after upgrading from python3.6 to python3.9.2
Log extract:

2021-06-16 15:56:45,628 Setting up handler pool
[DEBUG/MainProcess] created semlock with handle 612
[DEBUG/MainProcess] created semlock with handle 1052
[DEBUG/MainProcess] created semlock with handle 860
[DEBUG/MainProcess] worker handler starting
[DEBUG/MainProcess] result handler starting
2021-06-16 15:56:45,637 Worker process started worker_id=0
[DEBUG/MainProcess] closing pool
[DEBUG/MainProcess] task handler got sentinel
[DEBUG/MainProcess] task handler sending sentinel to result handler
[DEBUG/MainProcess] task handler sending sentinel to workers
[DEBUG/MainProcess] task handler exiting
[DEBUG/MainProcess] result handler got sentinel
[DEBUG/MainProcess] recreated blocker with handle 20
[DEBUG/MainProcess] recreated blocker with handle 24
[DEBUG/MainProcess] worker handler exiting
[DEBUG/MainProcess] joining worker handler
[DEBUG/MainProcess] joining task handler
[DEBUG/MainProcess] joining result handler
[DEBUG/MainProcess] recreated blocker with handle 28
[DEBUG/MainProcess] recreated blocker with handle 32
[INFO/SpawnPoolWorker-1] child process 12964 calling self.run()
[DEBUG/SpawnPoolWorker-1] worker got OSError -- exiting
[DEBUG/MainProcess] Supervisor: cleaning up worker 0
[DEBUG/MainProcess] Supervisor: worked 0 joined
[DEBUG/MainProcess] result handler: all workers terminated, timeout in 4.0s
[DEBUG/MainProcess] result handler: all workers terminated, timeout in 3.0s
[DEBUG/MainProcess] result handler: all workers terminated, timeout in 2.0s
[DEBUG/MainProcess] result handler: all workers terminated, timeout in 1.0s
[DEBUG/MainProcess] result handler: all workers terminated, timeout in 0.0s
[DEBUG/MainProcess] result handler exiting: timed out
[DEBUG/MainProcess] ensuring that outqueue is not full
[DEBUG/MainProcess] result handler exiting: len(cache)=1, thread._state=0
[DEBUG/MainProcess] result handler joined
[DEBUG/MainProcess] pool join complete
Dominant language
Python
Stars
436
Forks
273
Avg merge
4d 19h
Merged PRs (30d)
5

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from celery/billiard

All issues in celery/billiard

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.