Support for asyncio based workers

Open
#167 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
28/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
backend, distributed-systems, performance

Research direction

Begin by reviewing the current multiprocessing worker implementation and its blocking task-execution path. Before coding, clarify the desired asyncio/threaded process-and-event-loop model, performance goal, and acceptance criteria; the issue currently describes a design discussion rather than a bounded change.

Written by the indexing model from the issue text.

Description

enhancement

The current worker implementation model uses multiprocessing for parallel execution but each process blocks on task execution (though it might be waiting on I/O). Is there a possibility to use asyncio/threaded approach to alleviate this problem and think of a model where processes could be used for parallelization (using multiple cores) and each process has an event loop/ThreadPool to poll and execute tasks to maximize performance on each core ?

I'd be happy to discuss more on this and submit a PR if required as well. Thanks for all the awesome work so far 👍

Dominant language
Python
Stars
104
Forks
43
Avg merge
1d 13h
Merged PRs (30d)
4

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from conductor-oss/python-sdk

All issues in conductor-oss/python-sdk

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.