Reproduce fine tuning but score poorly on the evaluation dataset
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- bash, python
- Domain
- machine-learning
Research direction
Start with training/train.py and the train_marigold_e2e_ft_depth.sh configuration, then compare the reported arguments with the project's reproduction guidance. Re-run fine-tuning and evaluation on NYUv2, KITTI, ETH3D, ScanNet, and DIODE, and determine why the reproduced scores differ from the expected results.
Written by the indexing model from the issue text.
Description
Hi,
Thanks to the author for the contribution, but I had some problems reproducing it.
Why do I get bad scores on the evaluation dataset when reproducing fine-tuning results on my RTX3090?
The scores are as follows:
| Dataset | AbsRel ↓ | Delta_1 ↑ |
|---|---|---|
| NYUv2 | 0.056 | 0.963 |
| KITTI | 0.092 | 0.928 |
| ETH3D | 0.064 | 0.961 |
| ScanNet | 0.062 | 0.956 |
| DIODE | 0.299 | 0.780 |
The following is the train script configuration I use, refer to train_marigold_e2e_ft_depth.sh:
Note.
The following are the modified parts:
--checkpointing_steps 500 => to store the best checkpoint
--dataloader_num_workers 4 => speed up training time
--mixed_precision "bf16" => reduce memory usage
--seed 1234 => fixed seed
The complete script is as follows:
#!/bin/bash
accelerate launch training/train.py \
--pretrained_model_name_or_path "prs-eth/marigold-v1-0" \
--modality "depth" \
--noise_type "zeros" \
--max_train_steps 20000 \
--checkpointing_steps 500 \
--train_batch_size 2 \
--gradient_accumulation_steps 16 \
--gradient_checkpointing \
--learning_rate 3e-05 \
--lr_total_iter_length 20000 \
--lr_exp_warmup_steps 100 \
--dataloader_num_workers 4 \
--mixed_precision "bf16" \
--output_dir "model-finetuned/marigold_e2e_ft_depth_bf16" \
--enable_xformers_memory_efficient_attention \
--seed 1234 \
"$@"
- Dominant language
- Python
- Stars
- 521
- Forks
- 22
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from VisualComputingInstitute/diffusion-e2e-ft
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
All issues in VisualComputingInstitute/diffusion-e2e-ft
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100