extend active learning with multiple annotators to multi-label datasets
#930 aberto em 21 de dez. de 2023
Métricas do repositório
- Stars
- (7.479 estrelas)
- Métricas de merge de PR
- (Nenhuma PRs mesclada em 30d)
Description
Requested in the Slack community
Objectives:
-
Make a multi-label version of the
get_active_learning_scoresfunction: https://docs.cleanlab.ai/master/cleanlab/multiannotator.html#cleanlab.multiannotator.get_active_learning_scores -
Extend other functions in the
cleanlab.multiannotatormodule as necessary for this to work for multi-label datasets. All of this new functionality should live in a new module:cleanlab.multilabel_classification.multiannotator -
Make a multi-label version of this tutorial notebook: https://github.com/cleanlab/examples/tree/master/active_learning_multiannotator
Simple implementation:
For a multi-label dataset with K nondisjoint classes (aka tags), we compute the multilabel active learning scores like this:
multi_annotator_active_learning_scores = np.zeros(len(dataset),)
for class in 1:K:
y_onevsrest = create a one-vs-rest binary dataset labeling each example in as class 1 if it has class k in its given label, otherwise as class 0.
pred_probs_class = pred_probs[:,k]
class_k_active_learning_scores = cleanlab.multiannotator.get_active_learning_scores(y_onevsrest, pred_probs_class)
multilabel_active_learning_scores += class_k_active_learning_scores
return (multilabel_active_learning_scores / K)