New Version is 14x slower then the one with the torchao error
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 15/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Active
- Domain
- machine-learning, performance
Research direction
The report compares two ai-toolkit versions on identical Windows/5070ti hardware, claiming the only differences are the torchao error and the toolkit version; it names no file, log or reproduction command. Start by finding where torchao and the training loop's optimization path are configured, then diff the reporter's version against current main to bisect the slowdown. Done means an identified commit or config change plus repeated step-time measurements on the same setup confirming the regression.
Written by the indexing model from the issue text.
Description
This is for bugs only
Did you already ask in the discord?
No and never will (I am boycotting discord, and you should too)
You verified that this is a bug and not a feature request or question by asking in the discord?
Yes
Describe the bug
I mean how?
Version With Torchao Error
HR_Krea2: 19%|#9 | 501/2600 [00:21<12:24:06, 21.27s/it, lr: 9.2e-05 loss: 6.988e-02]
Latest version:
HR_Krea2: 19%|#9 | 501/2600 [05:03<177:04:17, 303.70s/it, lr: 9.3e-05 loss: 1.196e-01]
Beside the torchao and AI-Toolkit version, the environment is exactly the same. Windows/5070ti.
Name: torch
Version: 2.13.0+cu130
Name: torchvision
Version: 0.28.0+cu130
- Dominant language
- Python
- Stars
- 12.2k
- Forks
- 1.6k
- Avg merge
- 1m
- Merged PRs (30d)
- 1
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ostris/ai-toolkit
-
Bug in Web UI: Unknown model arch 'sd15' when selecting SD 1.5Possibly taken @ngo-binh0108 claimed this 1 day ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 86/100
ostris/ai-toolkit#1092 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ostris/ai-toolkit#1086 ·
Maintainers usually reply within 1 day
-
Bug: Prodigy 8-bit optimizer state fails to restore due to Auto8bitTensor deserialization errorOpen
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ostris/ai-toolkit#1076 ·
Maintainers usually reply within 1 day
-
Bug: Zero-gradient deadlock when expanding LoRA/DoRA rank (New capacity remains completely unused)Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ostris/ai-toolkit#1056 · 1 reaction ·
Maintainers usually reply within 1 day
-
Windows: stale cron worker survives UI shutdown, breaks next startup with EPERM on prisma generateOpen
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ostris/ai-toolkit#1033 ·
Maintainers usually reply within 1 day
All issues in ostris/ai-toolkit
Similar issues
-
test-failure
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
teramoby/speedle-plus#129 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
btclib-org/btclib-ecc#404 ·
-
verify: just verify runs tools/verify_agents_imports.py (the recipe line was only read, not run)Openneeds-verification
Difficulty 1/5 Under an hour Newbie friendliness 76/100
EchoTools/nevr-runtime#1322 ·
Maintainers usually reply within 1 day
-
changelog investigate
Difficulty 1/5 Under an hour Newbie friendliness 78/100
ramnes/notion-sdk-py#430 ·
-
bug good first issue
Difficulty 1/5 Under an hour Newbie friendliness 91/100
hyeonjoong/debussy#56 ·