Training Validation - When to stop?
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- audio-video-rtc, machine-learning, observability
Research direction
Start by locating the PL training pipeline and its existing checkpoint and logging hooks. Review how validation metrics such as CLAPscore, FDopenl3, and KLpasst could be evaluated and sent to WandB. Done should include repeatable validation during training, useful metric logging, and guidance for assessing overfitting and selecting checkpoints.
Written by the indexing model from the issue text.
Description
Hello all, and congratulations on the great repository!
I have sucessfully finetuned the stable audio open model on a couple of different tasks already. But so far I've had to define the stopping of training and choosing checkpoints by listen and evaluating samples manually.
It would be nice to have those on the PL training pipeline and log to WandB.
It is hard to assess overfitting, tune hyper-parameters and decide when to stop training by listening to samples and analyzing the train/loss curve.
Currently I am implementing a custom validation pipeline using the metrics described on the paper [and equivalents]: CLAPscore, FDopenl3, KLpasst.
- Dominant language
- Python
- Stars
- 3.9k
- Forks
- 488
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Stability-AI/stable-audio-tools
-
Possible incorrect use of `and` with multiple context managersPossibly taken @rodriguescarson claimed this 116 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Stability-AI/stable-audio-tools#238 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
run_gradio.py fails out of the box on Apple Silicon (MPS): float64 crash, slice TypeError, missing ui depsMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 4/5 3-5 days Newbie friendliness 72/100
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
Difficulty 3/5 1-2 days Newbie friendliness 25/100
All issues in Stability-AI/stable-audio-tools
Similar issues
-
link-rot
Difficulty 1/5 Under an hour Newbie friendliness 67/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
developmentseed/titiler-cmr#265 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Insecure Brotli version <1.2.1Possibly taken @jamescw19 claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 85/100
stac-utils/stac-fastapi-pgstac#438 · 1 reaction ·
-
bug triage
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day