microsoft/nni

Framework for Model Selection

Open

#3,376 建立於 2021年2月9日

在 GitHub 查看
 (2 留言) (0 反應) (1 負責人)Python (13,504 star) (1,830 fork)batch import
Framework Supportfeature requesthelp wantednew featureuser raised

描述

What I would you like to be added:

It would be great if NNI had an extension/framework for model selection.

Why is this needed:

Model selection is a very relevant decision for an ML project, arguably more important than hyperparameter tuning.

Without this feature, how does current nni work：

Currently, NNI seems to require you to decide on a model type in advance and then you optimise this model type. For example, you decide to use XGBoost and then you optimise hyperparameters. But you cannot compare / select from all the different possible model types that NNI supports in principle (e.g. models from scikit-learn, Keras, XGBoost etc.). In summary, it seems that NNI currently covers a little bit of feature selection and a lot of hyperparameter tuning; but it does not cover model selection at all.

I could not find any examples where NNI is used for model selection. If I missed something, I'd be thankful for your reply.

貢獻者指南

技術棧
pythonscikit learntensorflow
領域
machine learning
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
5
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
over 1 week
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
blocked
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
PythonMachine learningNNIModel training
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
10
研究方向
The issue requests a new framework for model selection within NNI. To start, review the existing NNI codebase for how hyperparameter tuning and model training are handled (e.g., the `nni` SDK, `tuner` interfaces, and experiment configuration). Examine how different model types like scikit learn, Keras, and XGBoost are currently integrated. Look at the assignee's branch or open threads for any preliminary work. The design should focus on an API that allows users to specify a set of model candidates and automatically compare them using a metric, potentially leveraging existing search algorithms. Consider reading the NNI documentation on general tuning architecture and any related feature requests or discussions in the issue tracker.