pypa/pipx

parallel execution of pipx

Open

#1,056 opened on Sep 13, 2023

View on GitHub
 (2 comments) (4 reactions) (0 assignees)Python (7,844 stars) (363 forks)batch import
enhancementhelp wanted

Description

Not sure if this is a bug or a feature request, I just wanted to note this here since I haven't seen a pertaining issue.

Describe the bug Executing pipx commands in parallel can result in unforeseen issues. I have noticed two main types of issues:

  1. pipx list fails if pipx install is ongoing due to missing metadata
  2. if shared libraries are outdated, any pipx command will trigger a shared libraries upgrade, which will cause concurrent commands to fail

How to reproduce

Scenario 1

import subprocess
from concurrent.futures import ThreadPoolExecutor
from typing import List

cmd = ["pipx", "list"]
cmd_list = [cmd] * 5 + ["pipx", "install", "pycowsay"]
with ThreadPoolExecutor() as executor:
    results: List[subprocess.CompletedProcess] = list(executor.map(subprocess.run, cmd_list))

print(results)

Scenario 2 as above, but remove the pipx install and ensure shared libraries are outdated, for example by running in a docker container with outdated libraries

Expected behavior Don't fail, but wait for the concurrent process to finish (debatable)

At the very least I'd expect pipx list not to fail with exit code 1 when a pipx install is ongoing

Contributor guide

Tech stack
python
Domain
clideveloper experience
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Basic PythonUnderstanding of concurrency
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
45
Research direction
Investigate the pipx source code to understand how metadata and shared libraries are handled. Look at files like `pipx/commands/` and `pipx/shared libs.py`. Implement file level locking (e.g., using `fcntl` or a lockfile) to prevent concurrent accesses that cause the reported failures. Check existing comments or PRs for proposed solutions and discuss with maintainers on the issue thread.