microsoft/nni

Framework for Model Selection

Open

#3.376 geöffnet am 9. Feb. 2021

Auf GitHub ansehen
 (2 Kommentare) (0 Reaktionen) (1 zugewiesene Person)Python (13.504 Stars) (1.830 Forks)batch import
Framework Supportfeature requesthelp wantednew featureuser raised

Beschreibung

What I would you like to be added:

It would be great if NNI had an extension/framework for model selection.

Why is this needed:

Model selection is a very relevant decision for an ML project, arguably more important than hyperparameter tuning.

Without this feature, how does current nni work：

Currently, NNI seems to require you to decide on a model type in advance and then you optimise this model type. For example, you decide to use XGBoost and then you optimise hyperparameters. But you cannot compare / select from all the different possible model types that NNI supports in principle (e.g. models from scikit-learn, Keras, XGBoost etc.). In summary, it seems that NNI currently covers a little bit of feature selection and a lot of hyperparameter tuning; but it does not cover model selection at all.

I could not find any examples where NNI is used for model selection. If I missed something, I'd be thankful for your reply.

Contributor Guide

Tech Stack
pythonscikit learntensorflow
Domain
machine learning
Issue Type
feature
SchwierigkeitGeschätzte Implementierungsschwierigkeit für neue Contributors, von 1 für sehr kleine Änderungen bis 5 für Expertenarbeit.
5
Geschätzte ZeitEin grober Zeitrahmen, um zu recherchieren, zu implementieren, zu testen und einen Pull Request vorzubereiten.
over 1 week
AktivitätsstatusWie verfügbar das Issue gerade wirkt: frisch, aktiv, stale, blockiert oder wartet auf Maintainer-Input.
blocked
KlarheitWie klar das Issue die erwartete Änderung, Akzeptanzkriterien und den nächsten Schritt erklärt.
mostly clear
Voraussetzungen
PythonMachine learningNNIModel training
EinsteigerfreundlichkeitEin Score von 1 bis 100, der schätzt, wie zugänglich dieses Issue für First-Time Contributors ist.
10
Research-Richtung
The issue requests a new framework for model selection within NNI. To start, review the existing NNI codebase for how hyperparameter tuning and model training are handled (e.g., the `nni` SDK, `tuner` interfaces, and experiment configuration). Examine how different model types like scikit learn, Keras, and XGBoost are currently integrated. Look at the assignee's branch or open threads for any preliminary work. The design should focus on an API that allows users to specify a set of model candidates and automatically compare them using a metric, potentially leveraging existing search algorithms. Consider reading the NNI documentation on general tuning architecture and any related feature requests or discussions in the issue tracker.