DataBricksStep has deprecated init_scripts incorrectly
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 38/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- azure, python
- Domain
- cloud, machine-learning
Research direction
Start by locating DataBricksStep and its init_scripts parameter handling, then compare the current documentation with the linked Databricks init-script guidance. The work is complete when Workspace-folder init scripts remain usable for Unity Catalog-enabled clusters and the deprecation guidance accurately describes DBFS storage rather than disabling init_scripts.
Written by the indexing model from the issue text.
Description
The argument init_script is no longer working for DataBricksStep(). The docs cite the following reason.
init_scripts | [str]Deprecated. Databricks announced the init script stored in DBFS will stop work after Dec 1, 2023. To mitigate the issue, please 1) use global init scripts in databricks following https://learn.microsoft.com/azure/databricks/init-scripts/global 2) comment out the line of init_scripts in your AzureML databricks step.
However, init_scripts have not been deprecated by DataBricks. DataBricks has deprecated storing init_scripts in dbfs:/ location, and instead suggets to store them in Workspace folders. This is what DataBricks recommends for all clusters and is, in fact, the only possible way for Unity Catalog enabled DataBricks clusters as global init scripts does not work for them.
Please enable init_scripts parameter for the DataBricksStep() as the current way of using global init_scripts does not work for Unity Catalog enabled clusters.
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/MachineLearningNotebooks
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Azure/MachineLearningNotebooks#1975 · 1 comment ·
-
duplicates Open
Difficulty 1/5 Under an hour Newbie friendliness 68/100
Azure/MachineLearningNotebooks#1960 ·
-
machine Open
Difficulty 5/5 Over a week Newbie friendliness 10/100
Azure/MachineLearningNotebooks#1987 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Azure/MachineLearningNotebooks#1985 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
Azure/MachineLearningNotebooks#1981 · 1 reaction ·
All issues in Azure/MachineLearningNotebooks
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
palladius/rails8-app-on-gcp#141 · 1 comment ·
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
documentation improve or update documentation priority/low triage
Difficulty 2/5 Half a day Newbie friendliness 86/100
warpdotdev/docs#782 · 1 comment ·
-
customer-reported question
Difficulty 1/5 Under an hour Newbie friendliness 80/100
Azure/awesome-azd#1017 ·