Examples: can remove cross-validation from outlier detection
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- jupyter-notebook
- Domain
- machine-learning
Research direction
Start with the outlier_detection_cifar10 example linked in the issue and inspect how its current train/test OOD settings use cross-validation. Add or evaluate the proposed training setting with outliers, compare whether cross-validation helps, and update the example based on that result.
Written by the indexing model from the issue text.
Description
This example: https://github.com/cleanlab/examples/tree/master/outlier_detection_cifar10
could become much more straightforward without cross-validation now that we've seen it doesn't help too much in the train/test OOD settings.
But to truly assess the utility of cross-validation, we should first include an additional benchmark setting where we evaluate OOD detection on a training dataset with some outliers included (during training not just during testing). Cross-validation may still help in this setting.
- Dominant language
- Jupyter Notebook
- Stars
- 136
- Forks
- 28
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cleanlab/examples
-
help wanted
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
help wanted
Difficulty 3/5 1-2 days Newbie friendliness 42/100
-
good first issue help wanted
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
help wanted
Difficulty 4/5 3-5 days Newbie friendliness 35/100
All issues in cleanlab/examples
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
huggingface/trl#8296 ·
Maintainers usually reply within 2 days
-
bot-triaged module: correctness (silent) module: inductor oncall: pt2 release triage triage review
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apple/coreai-models#523 ·
Maintainers usually reply within 1 day
-
bug low
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
anubhavaanand/project-cyan-phishNet#150 ·
Maintainers usually reply within 1 day