microsoft/nni

Framework for Model Selection

Open

#3,376 opened on 2021年2月9日

GitHub で見る
 (2 comments) (0 reactions) (1 assignee)Python (13,504 stars) (1,830 forks)batch import
Framework Supportfeature requesthelp wantednew featureuser raised

説明

What I would you like to be added:

It would be great if NNI had an extension/framework for model selection.

Why is this needed:

Model selection is a very relevant decision for an ML project, arguably more important than hyperparameter tuning.

Without this feature, how does current nni work：

Currently, NNI seems to require you to decide on a model type in advance and then you optimise this model type. For example, you decide to use XGBoost and then you optimise hyperparameters. But you cannot compare / select from all the different possible model types that NNI supports in principle (e.g. models from scikit-learn, Keras, XGBoost etc.). In summary, it seems that NNI currently covers a little bit of feature selection and a lot of hyperparameter tuning; but it does not cover model selection at all.

I could not find any examples where NNI is used for model selection. If I missed something, I'd be thankful for your reply.

コントリビューターガイド

技術スタック
pythonscikit learntensorflow
領域
machine learning
Issue 種別
feature
難度新しい貢献者にとっての実装難度です。1 は非常に小さな変更、5 は専門的な作業です。
5
推定時間経験ある貢献者が調査、実装、テスト、pull request 準備にかけるおおよその時間範囲です。
over 1 week
活動状況Issue が今どれだけ取り組みやすいかを示します。新しい、活発、古い、ブロック中、またはメンテナー入力待ちなどです。
blocked
明確さ期待される変更、受け入れ条件、次の手順がどれだけ明確かを示します。
mostly clear
前提条件
PythonMachine learningNNIModel training
初心者向け度初回貢献者にどれだけ取り組みやすいかを 1-100 で推定したスコアです。
10
調査方針
The issue requests a new framework for model selection within NNI. To start, review the existing NNI codebase for how hyperparameter tuning and model training are handled (e.g., the `nni` SDK, `tuner` interfaces, and experiment configuration). Examine how different model types like scikit learn, Keras, and XGBoost are currently integrated. Look at the assignee's branch or open threads for any preliminary work. The design should focus on an API that allows users to specify a set of model candidates and automatically compare them using a metric, potentially leveraging existing search algorithms. Consider reading the NNI documentation on general tuning architecture and any related feature requests or discussions in the issue tracker.