Missing data for processes started with ProcessPoolExecutor

Open
#49 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws, flask, python
Domain
backend, observability-sre

Research direction

Start by reproducing the Flask and Gunicorn setup with a long-running ProcessPoolExecutor and its initializer on AWS Fargate. Investigate the profiler lifecycle in worker processes; done means profiling data from executor workers is collected without triggering the duplicate-profiler-start error.

Written by the indexing model from the issue text.

Description

We have a Flask application with Gunicorn running on AWS Fargate. Our application has part of the business logic executed in parallel, using ProcessPoolExecutor. The executor, including the worker processes are reused and long running.
We have noticed that we are missing profiling data for the logic executed in worker processes. We have attempted to start new profiler for worker processes using ProcessPoolExecutor initializer, but this attempt has failed with message that we can't start the profiler twice within the same process.

Dominant language
Python
Stars
16
Forks
12
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aws/amazon-codeguru-profiler-python-agent

All issues in aws/amazon-codeguru-profiler-python-agent

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.