Chunks

Open
#199 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
javascript
Domain
cli

Research direction

Start with the bolt workspaces exec CLI path and its --parallel handling. Review how the existing workspace execution modes launch processes, then determine the design for --chunks and --max-chunk-size; done means the proposed chunked parallel mode is supported by the CLI examples in this issue.

Written by the indexing model from the issue text.

Description

There are generally a couple different ways to run tools in a Bolt repo:

  • Globally (across all workspaces at once)
  • By Workspace
    • Parallel
    • Chunks by dependency tree
    • Serial

When you have a tool that could either run globally or by workspace, you have to
make a tradeoff decision:

  • Does the tool take a long startup time? By workspace is probably slower, having to startup over and over.
  • Does the tool have a long process time? Globally is probably slower, would benefit from parallelization.

Right now we don't offer anything in between these two extremes. The different
"by workspace" modes don't help at all because they all spawn a new process for
every workspace.

(./) $ tool "./packages/*"

(./packages/a) $ tool "."
(./packages/b) $ tool "."
(./packages/c) $ tool "."
(./packages/d) $ tool "."

But what if we could break workspaces into "chunks" and spawn a smaller number
of processes?

(./) $ tool "./{packages/a,packages/b}"
(./) $ tool "./{packages/c,packages/d}"

This could allow us to increase parallelism of a tool while also reducing the
number of processes that it starts up.

CLI

I would focus on just when the --parallel flag is on initially.

bolt workspaces exec --parallel --chunks -- tool "()"

If you wanted to limit the number of items to place in a chunk at a time:

bolt workspaces exec --parallel --chunks --max-chunk-size 10 -- tool "()"
Dominant language
JavaScript
Stars
2.4k
Forks
97
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from boltpkg/bolt

All issues in boltpkg/bolt

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.