AutoML experiment: get model and metrics for any algorithm (not only for the best one)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- azure, jupyter-notebook, python
- Domain
- api, machine-learning
Research direction
Start with the MLflow client usage shown in the issue, including get_run, the automl_best_child_run_id tag, and pipeline_id_000. Trace how AutoML child runs are identified and how algorithm names relate to their internal run names. Done means documenting or enabling retrieval of a model and its metrics for any algorithm in the experiment, not only the best run.
Written by the indexing model from the issue text.
Description
What I'm trying to do
For an AutoML Forecasting experiment, I'd like to compare the performance of the best model with the performance of another model from the same experiment.
For an AutoML run, I understand how to get the best performing model and its metrics like this:
# ...initialize MLFlow client...
mlflow_parent_run = mlflow_client.get_run('upbeat_square_abs3942')
best_child_run_id = mlflow_parent_run.data.tags["automl_best_child_run_id"]
best_run = mlflow_client.get_run(best_child_run_id)
best_run.data.metrics
# etc...
But how can I fetch the job for any model based on the algorithm name?
Something like:
# pseudocode:
mlflow_client.get_automl_run_by_algorithm('XGBoostRegressor')
So far, I managed to figure out the following:
- list of algorithms used in the AutoML experiment
mlflow_parent_run.data.tags['pipeline_id_000']
# '__AutoML_Naive__;__AutoML_SeasonalNaive__;__AutoML_Average__;__AutoML_SeasonalAverage__;__AutoML_Ensemble__'
However, this list seems to be in an arbitrary order and I struggle to get the corresponding job names for the algorithms.
- "internal" job names for the child runs
The child runs seem to have different names than the names shown in Azure ML Studio.
They are named for instance upbeat_square_abs3942_2 - i.e. the name of the parent run upbeat_square_abs3942 followed by underscore plus a number (_2in this example).
But Azure ML Studio displays names like (no upbeat_square_abs3942_2 to be found):

So this code works:
child_run = mlflow_client.get_run('upbeat_square_abs3942_2')
but using a name shown in the screenshot above throws an exception, e.g.
child_run = mlflow_client.get_run('green_floor_0ln3tlpv')
Question
How can I obtain the model and metrics for any algorithm used in the experiment?
Thanks!
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Azure/MachineLearningNotebooks
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Azure/MachineLearningNotebooks#1975 · 1 comment ·
-
duplicates Open
Difficulty 1/5 Under an hour Newbie friendliness 68/100
Azure/MachineLearningNotebooks#1960 ·
-
machine Open
Difficulty 5/5 Over a week Newbie friendliness 10/100
Azure/MachineLearningNotebooks#1987 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Azure/MachineLearningNotebooks#1985 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
Azure/MachineLearningNotebooks#1981 · 1 reaction ·
All issues in Azure/MachineLearningNotebooks
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
canonical/paas-charm#368 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
tech debt
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
area:workflow bug ready-for-agent
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
fil-donadoni/tolaria#4409 ·
-
status/awaiting_triage
Difficulty 2/5 1-3 hours Newbie friendliness 75/100