Separate parallel backend from frontend
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 25/100
- Issue 类型
- 功能
- 描述清晰度
- 需要澄清
- 活跃度
- 停滞
- 技术栈
- python
调研方向
No files or tests are named. Start by tracing how scatter, connect, DaskStream, and node clients interact, then inspect the client API points around submit and loop; done would mean pipeline definitions can be reused across backends with client choice and downstream execution handled consistently.
由索引模型根据 Issue 内容生成。
描述
Recently I've been trying to split my pipelines up into chunks so the chunks can be re-used as much as possible. One issue I foresee running into is that scatter may not work as intended. scatter will send the data to the cluser for processing, but since the downstream nodes are already defined (I'm linking them via connect) they won't properly operate on the data (since they aren't DaskStream nodes).
It seems that it may be beneficial to separate the front end pipeline definition from the decision about which backend is being used, at least at the class level. This would make pipelines more modular since we don't need to re-write the same pipeline for each backend. Additionally this would allow the user executing a pipeline to decide what backend they want/are able to use.
One approach to handle this could be to specify a client for the entire pipeline, where a client mimics some of the dask client API (mostly submit and loop I think). If the client associated with any node is changed/updated then the entire pipeline shifts to use that client. This would also require nodes to know if they are downstream of a scatter but haven't been gathered yet, since then they would know to use the pipeline's client rather than standard local execution (which most likely should be a dummy client). Maybe this can be inspected when the client is set, since it could change during the pipeline build process?
- 主要语言
- Python
- 星标
- 1.3k
- 派生
- 151
- 平均合并
- 17 小时 39 分钟
- 30 天内合并 PR
- 1
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
python-streamz/streamz 的其他 Issue
-
难度 3/5 1-2 天 新手友好度 35/100
python-streamz/streamz#481 · 4 条评论 ·
-
难度 5/5 一周以上 新手友好度 25/100
python-streamz/streamz#480 · 2 条评论 ·
-
难度 5/5 一周以上 新手友好度 20/100
python-streamz/streamz#479 · 1 条评论 ·
-
难度 5/5 一周以上 新手友好度 20/100
python-streamz/streamz#478 · 6 条评论 ·
-
难度 4/5 3-5 天 新手友好度 10/100
python-streamz/streamz#476 · 17 条评论 · 2 个 reaction ·
查看 python-streamz/streamz 的全部 Issue
相似的 Issue
-
comp/tools duplicate P2 sweeper:risk-compatibility tool/mcp type/bug
难度 1/5 1 小时以内 新手友好度 88/100
NousResearch/hermes-agent#132042 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
deepset-ai/haystack#13092 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 86/100
EverMind-AI/Raven#845 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
AstrBotDevs/AstrBot#10340 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
BasedHardware/omi#20401 · 1 条评论 ·
维护者通常 1 天内回复