Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

AutoML experiment: get model and metrics for any algorithm (not only for the best one)

Open
#1,885 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
azure, jupyter-notebook, python

Research direction

Start with the MLflow client usage shown in the issue, including get_run, the automl_best_child_run_id tag, and pipeline_id_000. Trace how AutoML child runs are identified and how algorithm names relate to their internal run names. Done means documenting or enabling retrieval of a model and its metrics for any algorithm in the experiment, not only the best run.

Written by the indexing model from the issue text.

Description

What I'm trying to do

For an AutoML Forecasting experiment, I'd like to compare the performance of the best model with the performance of another model from the same experiment.

For an AutoML run, I understand how to get the best performing model and its metrics like this:

# ...initialize MLFlow client...
mlflow_parent_run = mlflow_client.get_run('upbeat_square_abs3942')
best_child_run_id = mlflow_parent_run.data.tags["automl_best_child_run_id"]
best_run = mlflow_client.get_run(best_child_run_id)
best_run.data.metrics
# etc...

But how can I fetch the job for any model based on the algorithm name?
Something like:

# pseudocode:
mlflow_client.get_automl_run_by_algorithm('XGBoostRegressor')

So far, I managed to figure out the following:

  1. list of algorithms used in the AutoML experiment
mlflow_parent_run.data.tags['pipeline_id_000']
#  '__AutoML_Naive__;__AutoML_SeasonalNaive__;__AutoML_Average__;__AutoML_SeasonalAverage__;__AutoML_Ensemble__'

However, this list seems to be in an arbitrary order and I struggle to get the corresponding job names for the algorithms.

  1. "internal" job names for the child runs

The child runs seem to have different names than the names shown in Azure ML Studio.
They are named for instance upbeat_square_abs3942_2 - i.e. the name of the parent run upbeat_square_abs3942 followed by underscore plus a number (_2in this example).

But Azure ML Studio displays names like (no upbeat_square_abs3942_2 to be found):
image
So this code works:

child_run = mlflow_client.get_run('upbeat_square_abs3942_2')

but using a name shown in the screenshot above throws an exception, e.g.

child_run = mlflow_client.get_run('green_floor_0ln3tlpv')
Question

How can I obtain the model and metrics for any algorithm used in the experiment?

Thanks!

Dominant language
Jupyter Notebook
Stars
4.4k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Azure/MachineLearningNotebooks

All issues in Azure/MachineLearningNotebooks

Similar issues

More Backend & API Design issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.