Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Classification lessons apply SMOTE before train_test_split, so the test set contains synthetic samples

Open
  • Is something wrong?
#1,055 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
jupyter-notebook, python, scikit-learn
Domain
data, machine-learning

Research direction

Read the Lesson 4.1 README and the Lesson 4.2–4.4 READMEs and notebooks, focusing on where cleaned_cuisines.csv is saved, loaded, and passed to train_test_split. Check the logistic regression workflow in Lesson 4.2 first; done means SMOTE is applied only to training data across the affected lessons, with the lesson explaining why the split must come first.

Written by the indexing model from the issue text.

Description

Hi, and thanks for this curriculum!

What happens

Lesson 4.1 balances the whole dataset with SMOTE and saves the result:

oversample = SMOTE()
transformed_feature_df, transformed_label_df = oversample.fit_resample(feature_df, labels_df)
...
transformed_df.to_csv("../data/cleaned_cuisines.csv")

Lesson 4.2 (and 4.3, 4.4) then load cleaned_cuisines.csv and only afterwards call train_test_split.

SMOTE creates new rows by interpolating between existing ones. So the test set ends up containing synthetic rows built from training rows (and the other way round), and its class balance is artificial. The model is partly evaluated on near-copies of what it was trained on.

How much it matters here

I re-ran Lesson 4.2's logistic regression on cuisines.csv with 5 random seeds:

accuracy
as in the lessons (SMOTE, then split) 0.799 ± 0.015
split first, SMOTE only on the training part 0.777 ± 0.012 (balanced accuracy 0.755 ± 0.008)

The effect on this dataset is moderate, about 2–4 points. But the lessons teach the order "resample, then split", which can inflate results much more on other data, e.g. heavily imbalanced datasets.

Suggested fix

  • Lesson 4.1: keep the SMOTE demonstration, but save the cleaned, unbalanced data to cleaned_cuisines.csv.

  • Lessons 4.2–4.4: split first and balance only the training data:

    X_train, X_test, y_train, y_test = train_test_split(
    cuisines_feature_df, cuisines_label_df, test_size=0.3, stratify=cuisines_label_df)
X_train, y_train = SMOTE().fit_resample(X_train, y_train)

    or use imblearn.pipeline.make_pipeline(SMOTE(), model) with cross-validation, so resampling happens inside each training fold.

  • A sentence on why the order matters would make a nice teaching moment.

Happy to open a PR for the English lessons and notebooks if that would help (the translated copies contain the same code).

Found with unleak, a static analyzer for data leakage that follows data across notebooks.

Dominant language
Jupyter Notebook
Stars
91.7k
Forks
22.7k
No merged PRs in 30d

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/ML-For-Beginners

All issues in microsoft/ML-For-Beginners

Similar issues

More Data Engineering issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.