Support extras for `tasks.libraries.whl` on classic clusters (follow-up to #1602)

Ouverte
  • Quelque chose ne va pas ?
#7,256 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
55/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
go, python
Domaine
backend, cli

Piste de recherche

Start by tracing how the CLI resolves tasks.libraries.whl paths during deployment and how it passes wheel library references to classic job clusters; compare that flow with the serverless dependencies handling cited in the issue. The report gives no source files or tests, so first locate those entry points and their tests. Done means a wheel reference with [my-extra] deploys and installs the extra on a classic cluster, while plain wheel references still work.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

DABs Enhancement

Hi, this is a follow-up to #1602. Extras on local wheels work now for serverless environment dependencies (#6291), which is great, but they dont work for tasks.libraries.whl, so jobs on classic job clusters cant use them.

We split our wheel into extras per domain, because we dont want every job to install libraries it does not need, and we want one pin in pyproject.toml instead of a second pin in the job config.

What we see:

  • whl: ./dist/*.whl[my-extra] fails at deploy with no files match pattern: dist/*.whl[my-extra]
  • whl: /Workspace/.../my_package-0.1.0-py3-none-any.whl[my-extra] deploys, but the library install on the cluster fails with "The library file does not exist or the user does not have permission to read the library file"
  • the same absolute path without [my-extra] installs fine, so it looks like the suffix is read as part of the file name

Minimal config:

artifacts:
  my_package:
    type: whl
    build: uv build --wheel
    path: .

resources:
  jobs:
    my_job:
      tasks:
        - task_key: main
          spark_python_task:
            python_file: ./main.py
          libraries:
            - whl: ./dist/*.whl[my-extra]
          new_cluster:
            spark_version: 18.0.x-scala2.13
            node_type_id: <any>
            num_workers: 0

Expected: the wheel installs with the extra, the same as ./dist/*.whl[my-extra] in environment dependencies on serverless.

Workarounds we know: move the job to serverless, or pin the extra libraries again as pypi libraries on the task, which brings back the 2 pins that drift.

If the second part is on the cluster library side and not the CLI, can u route it please?

CLI version: 1.21.0

Langage dominant
Go
Étoiles
416
Forks
256
Merge moyen
1 j 13 h
PR mergées (30 j)
199

Préparer son environnement

  • Fournit un Dockerfile ou un fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de databricks/cli

Toutes les issues de databricks/cli

Issues similaires

Plus d'issues Go

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.