AutoML experiment: get model and metrics for any algorithm (not only for the best one)
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 技术栈
- azure, jupyter-notebook, python
- 领域
- api, machine-learning
调研方向
从 issue 中展示的 MLflow 客户端用法开始,包括 get_run、automl_best_child_run_id 标签和 pipeline_id_000。跟踪 AutoML 子运行的识别方式,以及算法名称与其内部运行名称之间的关系。完成的标准是记录或支持获取实验中任意算法的模型及其指标,而不仅仅是最佳运行。
由索引模型根据 Issue 内容生成。
描述
What I'm trying to do
For an AutoML Forecasting experiment, I'd like to compare the performance of the best model with the performance of another model from the same experiment.
For an AutoML run, I understand how to get the best performing model and its metrics like this:
# ...initialize MLFlow client...
mlflow_parent_run = mlflow_client.get_run('upbeat_square_abs3942')
best_child_run_id = mlflow_parent_run.data.tags["automl_best_child_run_id"]
best_run = mlflow_client.get_run(best_child_run_id)
best_run.data.metrics
# etc...
But how can I fetch the job for any model based on the algorithm name?
Something like:
# pseudocode:
mlflow_client.get_automl_run_by_algorithm('XGBoostRegressor')
So far, I managed to figure out the following:
- list of algorithms used in the AutoML experiment
mlflow_parent_run.data.tags['pipeline_id_000']
# '__AutoML_Naive__;__AutoML_SeasonalNaive__;__AutoML_Average__;__AutoML_SeasonalAverage__;__AutoML_Ensemble__'
However, this list seems to be in an arbitrary order and I struggle to get the corresponding job names for the algorithms.
- "internal" job names for the child runs
The child runs seem to have different names than the names shown in Azure ML Studio.
They are named for instance upbeat_square_abs3942_2 - i.e. the name of the parent run upbeat_square_abs3942 followed by underscore plus a number (_2in this example).
But Azure ML Studio displays names like (no upbeat_square_abs3942_2 to be found):

So this code works:
child_run = mlflow_client.get_run('upbeat_square_abs3942_2')
but using a name shown in the screenshot above throws an exception, e.g.
child_run = mlflow_client.get_run('green_floor_0ln3tlpv')
Question
How can I obtain the model and metrics for any algorithm used in the experiment?
Thanks!
- 主要语言
- Jupyter Notebook
- 星标
- 4.4k
- 派生
- 2.6k
- PR 合并指标
- 30 天内没有已合并 PR
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
Azure/MachineLearningNotebooks 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 72/100
Azure/MachineLearningNotebooks#1975 · 1 条评论 ·
-
duplicates 未关闭
难度 1/5 1 小时以内 新手友好度 68/100
Azure/MachineLearningNotebooks#1960 ·
-
machine 未关闭
难度 5/5 一周以上 新手友好度 10/100
Azure/MachineLearningNotebooks#1987 ·
-
难度 5/5 一周以上 新手友好度 25/100
Azure/MachineLearningNotebooks#1985 · 1 条评论 ·
-
难度 3/5 1-2 天 新手友好度 35/100
Azure/MachineLearningNotebooks#1981 · 1 个 reaction ·
查看 Azure/MachineLearningNotebooks 的全部 Issue
相似的 Issue
-
Area: Excel support
难度 2/5 1-3 小时 新手友好度 75/100
orbeon/orbeon-forms#7893 ·
-
essnmx good first issue
难度 1/5 1 小时以内 新手友好度 95/100
-
难度 2/5 1-3 小时 新手友好度 75/100
yeti-platform/yeti#1380 ·
-
难度 2/5 1-3 小时 新手友好度 75/100
modelcontextprotocol/python-sdk#3566 ·
-
难度 2/5 1-3 小时 新手友好度 70/100
griptape-ai/griptape#2353 ·