josdejong/workerpool

Concurrent async tasks on a single worker

Open

#371 opened on 2023年1月13日

GitHub で見る
 (1 comment) (0 reactions) (0 assignees)JavaScript (164 forks)github user discovery
enhancementhelp wanted

Repository metrics

Stars
 (2,297 stars)
PR merge metrics
 (PR metrics pending)

説明

Workerpool allows each thread to process only a single task at a time, which means that each task must be strictly CPU-bound, and any IO makes the pool work inefficiently.

At the same time, there are some traits of supporting multiple parallel tasks per threads in the code: https://github.com/josdejong/workerpool/blob/master/src/WorkerHandler.js#L332 - I think this is for the case when the thread is in "wait for ready" state only though, and due to this check: https://github.com/josdejong/workerpool/blob/master/src/Pool.js#L248 - it never schedules more than 1 task to a "busy" thread.

Piscina has a feature of scheduling multiple tasks to the same thread.

Are there any plans to implement the similar thing in workerpool (considering that the most of the infrastructure for this is already in the code)?

Repro of the serialization:

cat <<'EOT' > main.js
const workerpool = require("workerpool");
async function main() {
  const pool = workerpool.pool(`${__dirname}/worker.js`, { minWorkers: 1, maxWorkers: 1 });
  await Promise.all([
    pool.exec("test", [1]),
    pool.exec("test", [2]),
    pool.exec("test", [3]),
    pool.exec("test", [4])
  ]);
}
main();
EOT

cat <<'EOT' > worker.js
const workerpool = require("workerpool");
const delay = require("delay");
workerpool.worker({
  test: async (v) => {
    console.log(`[${v}] started`);
    await delay(1000);
    console.log(`[${v}] finished`);
  }
});
EOT

node main.js

[1] started
[1] finished
[2] started
[2] finished
[3] started
[3] finished
[4] started
[4] finished

コントリビューターガイド