Separate parallel backend from frontend
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 25/100
- issue の種類
- 機能追加
- 明瞭さ
- 説明が足りない
- 活発さ
- 停滞
- 技術スタック
- python
調査の方向性
No files or tests are named. Start by tracing how scatter, connect, DaskStream, and node clients interact, then inspect the client API points around submit and loop; done would mean pipeline definitions can be reused across backends with client choice and downstream execution handled consistently.
索引モデルが issue の本文から書いたものです。
説明
Recently I've been trying to split my pipelines up into chunks so the chunks can be re-used as much as possible. One issue I foresee running into is that scatter may not work as intended. scatter will send the data to the cluser for processing, but since the downstream nodes are already defined (I'm linking them via connect) they won't properly operate on the data (since they aren't DaskStream nodes).
It seems that it may be beneficial to separate the front end pipeline definition from the decision about which backend is being used, at least at the class level. This would make pipelines more modular since we don't need to re-write the same pipeline for each backend. Additionally this would allow the user executing a pipeline to decide what backend they want/are able to use.
One approach to handle this could be to specify a client for the entire pipeline, where a client mimics some of the dask client API (mostly submit and loop I think). If the client associated with any node is changed/updated then the entire pipeline shifts to use that client. This would also require nodes to know if they are downstream of a scatter but haven't been gathered yet, since then they would know to use the pipeline's client rather than standard local execution (which most likely should be a dummy client). Maybe this can be inspected when the client is set, since it could change during the pipeline build process?
- 主要言語
- Python
- スター
- 1.3k
- フォーク
- 149
- PR マージ指標
- 30日以内にマージされた PR はありません
コントリビューションガイド
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
python-streamz/streamz のほかの issue
-
難易度 3/5 1〜2日 初心者へのやさしさ 35/100
python-streamz/streamz#481 · コメント 4 件 ·
-
Combining the streamz.Stream.filenames() and streamz.Stream.from_textfile() using dask scatter? オープン
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
python-streamz/streamz#480 · コメント 2 件 ·
-
難易度 5/5 1週間以上 初心者へのやさしさ 20/100
python-streamz/streamz#479 · コメント 1 件 ·
-
難易度 5/5 1週間以上 初心者へのやさしさ 20/100
python-streamz/streamz#478 · コメント 6 件 ·
-
難易度 4/5 3〜5日 初心者へのやさしさ 10/100
python-streamz/streamz#476 · コメント 17 件 · リアクション 2 件 ·
python-streamz/streamz の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
syfoud/Simulated_Scepter#172 ·
-
A cancelled tests run makes the coverage comment workflow fail and reports it as a red check on main オープンarea: ci bug perceived difficulty: 3
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
Nitjsefnie-Harness-Commons/daedalus#921 · コメント 1 件 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
難易度 1/5 1時間未満 初心者へのやさしさ 92/100
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
ClickHouse/clickhouse-connect#1057 ·