Training script stuck (using eval only)
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning
Research direction
Reproduce the commands from codegen_sources/model/train.py and codegen_sources.model.translate, starting with the reported missing test_dataset/*.pth files and the TransCoder_model_2.pth.1 reload log. Trace the dataset and model-loading entry points to identify where execution stops; done means the cause is isolated and either the command proceeds or the failure is reported with a concrete diagnostic.
Written by the indexing model from the issue text.
Description
I'm trying to run
python codegen_sources/model/train.py --eval_only True --reload_model 'TransCoder_model_2.pth,TransCoder_model_2.pth' --data_path "test_dataset" --exp_name transcoder --dump_path 'dump' --lgs 'java_sa-python_sa' --bt_steps 'python_sa-java_sa-python_sa,java_sa-python_sa-java_sa' --ae_steps 'python_sa,java_sa' --mt_steps 'java_sa-python_sa,python_sa-java_sa' --encoder_only False --emb_dim 1024 --n_heads 8 --n_layers 0 --n_layers_encoder 6 --n_layers_decoder 6 --eval_bleu true --eval_computation true --has_sentences_ids true
but it is unable to find the following files in the test_dataset (downloaded from the transcoder doc)
test_dataset/train.java_sa.pth not found
test_dataset/valid.java_sa.pth not found
test_dataset/test.java_sa.pth not found
test_dataset/train.python_sa.pth not found
test_dataset/valid.python_sa.pth not found
test_dataset/test.python_sa.pth not found
test_dataset/train.java_sa-python_sa.java_sa.pth not found
test_dataset/train.java_sa-python_sa.python_sa.pth not found
and gets stuck after this log message:
SLURM job: False
0 - Number of nodes: 1
0 - Node ID : 0
0 - Local rank : 0
0 - Global rank : 0
0 - World size : 1
0 - GPUs per node : 1
0 - Master : True
0 - Multi-node : False
0 - Multi-GPU : False
0 - Hostname : <host>
Any ideas why this might be happening?
I also tried running this translation script, which similarly seems to get stuck:
python -m codegen_sources.model.translate --src_lang python --tgt_lang java --model_path TransCoder_model_2.pth.1 --beam_size 1 < hello.py
adding to path /srv/home/akshit/CodeGen
INFO - 09/27/21 05:56:56 - 0:00:06 - ============ Model Reloading
INFO - 09/27/21 05:56:56 - 0:00:06 - Reloading encoder from TransCoder_model_2.pth.1 ...
- Dominant language
- Python
- Stars
- 777
- Forks
- 144
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from facebookresearch/CodeGen
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
facebookresearch/CodeGen#109 ·
-
Hash function Open
Difficulty 3/5 1-2 days Newbie friendliness 42/100
facebookresearch/CodeGen#108 ·
-
Dataset issues Open
Difficulty 3/5 1-2 days Newbie friendliness 45/100
facebookresearch/CodeGen#107 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
facebookresearch/CodeGen#106 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
facebookresearch/CodeGen#105 ·
All issues in facebookresearch/CodeGen
Similar issues
-
bug ci good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
documentation
Difficulty 2/5 Half a day Newbie friendliness 62/100
inmanta/inmanta-core#10835 ·
-
sponsored
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Diaoul/subliminal#1382 ·