Support extras for `tasks.libraries.whl` on classic clusters (follow-up to #1602)
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start by tracing how the CLI resolves tasks.libraries.whl paths during deployment and how it passes wheel library references to classic job clusters; compare that flow with the serverless dependencies handling cited in the issue. The report gives no source files or tests, so first locate those entry points and their tests. Done means a wheel reference with [my-extra] deploys and installs the extra on a classic cluster, while plain wheel references still work.
由索引模型根据 Issue 内容生成。
描述
Hi, this is a follow-up to #1602. Extras on local wheels work now for serverless environment dependencies (#6291), which is great, but they dont work for tasks.libraries.whl, so jobs on classic job clusters cant use them.
We split our wheel into extras per domain, because we dont want every job to install libraries it does not need, and we want one pin in pyproject.toml instead of a second pin in the job config.
What we see:
whl: ./dist/*.whl[my-extra]fails at deploy withno files match pattern: dist/*.whl[my-extra]whl: /Workspace/.../my_package-0.1.0-py3-none-any.whl[my-extra]deploys, but the library install on the cluster fails with "The library file does not exist or the user does not have permission to read the library file"- the same absolute path without
[my-extra]installs fine, so it looks like the suffix is read as part of the file name
Minimal config:
artifacts:
my_package:
type: whl
build: uv build --wheel
path: .
resources:
jobs:
my_job:
tasks:
- task_key: main
spark_python_task:
python_file: ./main.py
libraries:
- whl: ./dist/*.whl[my-extra]
new_cluster:
spark_version: 18.0.x-scala2.13
node_type_id: <any>
num_workers: 0
Expected: the wheel installs with the extra, the same as ./dist/*.whl[my-extra] in environment dependencies on serverless.
Workarounds we know: move the job to serverless, or pin the extra libraries again as pypi libraries on the task, which brings back the 2 pins that drift.
If the second part is on the cluster library side and not the CLI, can u route it please?
CLI version: 1.21.0
- 主要语言
- Go
- 星标
- 416
- 派生
- 256
- 平均合并
- 1 天 13 小时
- 30 天内合并 PR
- 199
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 没有贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
databricks/cli 的其他 Issue
-
Apps commands only accept relative path to app.yml可能已有人在做 @yuri-rod 于 30 天前认领。 未关闭CLI
难度 2/5 1-3 小时 新手友好度 82/100
databricks/cli#6910 ·
维护者通常 1 天内回复
-
DABs
难度 2/5 1-3 小时 新手友好度 68/100
databricks/cli#6670 · 1 个 reaction ·
维护者通常 1 天内回复
-
`bundle plan` always plans an App update for `config.env` values, because the Apps API does not return them可能已有人在做 @darshpatel1052 今天认领。 未关闭DABs
难度 3/5 1-2 天 新手友好度 45/100
databricks/cli#7301 · 1 条评论 ·
维护者通常 1 天内回复
-
bundle validate does not catch missing trigger.table_update.condition with multiple tables可能已有人在做 @yuri-rod 于 15 天前认领。 未关闭DABs Enhancement
难度 3/5 1-2 天 新手友好度 72/100
databricks/cli#7101 ·
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 35/100
databricks/cli#7100 ·
维护者通常 1 天内回复
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 66/100
riverqueue/river#1576 ·
维护者通常 1 天内回复
-
NFS StorageProvider renders a missing nfsSubDir as "<no value>"可能已有人在做 @ldming 今天认领。 未关闭kind/bug
难度 2/5 1-3 小时 新手友好度 82/100
apecloud/kubeblocks#11067 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
emersion/go-msgauth#87 ·
-
CLI: `describe <ns>.<table>` reports "namespace does not exist" when only the table is missing可能已有人在做 @Shruthirdy 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 62/100
apache/iceberg-go#2423 ·
维护者通常 2 天内回复
-
good first issue help wanted
难度 2/5 1-3 小时 新手友好度 85/100
mafzalkalwardev/mailforge#21 ·