Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

BertScorer early-stopping default ([email protected]) collapses on sparse multilabel

Open
  • Is something wrong?
#323 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
machine-learning

Research direction

Start in autointent/modules/scoring/_bert.py, reading the default EarlyStoppingConfig, _get_compute_metrics(), and _train(). Reproduce the GoEmotions multilabel behavior described in the issue, then compare the stopping metric with the available threshold-free or disabled-early-stopping options. Done means the default no longer stops immediately or restores a near-random checkpoint on sparse multilabel data while early stopping remains useful for multiclass tasks.

Written by the indexing model from the issue text.

Description

Summary

BertScorer's default early stopping uses scoring_f1 (F1 thresholded at 0.5) as metric_for_best_model. On sparse multilabel data, calibrated sigmoid probabilities sit well below 0.5, so this metric reads ≈0 from the very first eval, the EarlyStoppingCallback (patience 3) fires almost immediately, and load_best_model_at_end=True restores a near-random early checkpoint. The fitted model then outputs near-constant low probabilities → after thresholding the pipeline predicts essentially all-positive (degenerate).

Where

autointent/modules/scoring/_bert.py:

  • Default EarlyStoppingConfig → {'val_fraction': 0.2, 'patience': 3, 'threshold': 0.0, 'metric': 'scoring_f1'}.
  • _get_compute_metrics() computes scoring_f1 on the raw eval predictions (0.5-thresholded for multilabel).
  • _train() sets metric_for_best_model=self.early_stopping_config.metric and load_best_model_at_end=(metric is not None), plus an EarlyStoppingCallback.
Evidence

Fine-tuning bert-base-uncased on GoEmotions (28 classes, ~4% positive rate, ~2.5k balanced rows):

  • With the default early stopping, eval_scoring_f1 is stuck at ~0.034 every epoch and the pipeline scores macro decision_f1 ≈ 0.107 (degenerate all-positive; recall→1.0, precision→base rate).
  • With early stopping disabled (train full epochs) at the same LR, the same model learns and reaches macro decision_f1 ≈ 0.17–0.41 depending on data size.

So the model is trainable; the threshold-0.5 early-stopping metric is what breaks it.

Suggested fix

Use a threshold-free signal for metric_for_best_model on multilabel:

  • eval_loss, or a ranking metric (e.g. neg_coverage / MAP), or compute F1 at a tuned/optimal threshold rather than a fixed 0.5; or
  • disable early stopping by default for multilabel tasks.

Any of these would prevent the immediate-stop collapse while keeping early stopping useful for multiclass.

Environment

AutoIntent 0.3.1, MPS, transformers-no-hpo preset, GoEmotions multilabel.

Dominant language
Python
Stars
53
Forks
16
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deeppavlov/AutoIntent

All issues in deeppavlov/AutoIntent

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.