Training script stuck (using eval only)

Open
#26 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python

Research direction

Reproduce the commands from codegen_sources/model/train.py and codegen_sources.model.translate, starting with the reported missing test_dataset/*.pth files and the TransCoder_model_2.pth.1 reload log. Trace the dataset and model-loading entry points to identify where execution stops; done means the cause is isolated and either the command proceeds or the failure is reported with a concrete diagnostic.

Written by the indexing model from the issue text.

Description

I'm trying to run

python codegen_sources/model/train.py --eval_only True --reload_model 'TransCoder_model_2.pth,TransCoder_model_2.pth' --data_path "test_dataset" --exp_name transcoder --dump_path 'dump' --lgs 'java_sa-python_sa'  --bt_steps 'python_sa-java_sa-python_sa,java_sa-python_sa-java_sa'  --ae_steps 'python_sa,java_sa'  --mt_steps 'java_sa-python_sa,python_sa-java_sa' --encoder_only False --emb_dim 1024 --n_heads 8 --n_layers 0 --n_layers_encoder 6  --n_layers_decoder 6 --eval_bleu true --eval_computation true --has_sentences_ids true

but it is unable to find the following files in the test_dataset (downloaded from the transcoder doc)

test_dataset/train.java_sa.pth not found
test_dataset/valid.java_sa.pth not found
test_dataset/test.java_sa.pth not found
test_dataset/train.python_sa.pth not found
test_dataset/valid.python_sa.pth not found
test_dataset/test.python_sa.pth not found
test_dataset/train.java_sa-python_sa.java_sa.pth not found
test_dataset/train.java_sa-python_sa.python_sa.pth not found

and gets stuck after this log message:

SLURM job: False
0 - Number of nodes: 1
0 - Node ID        : 0
0 - Local rank     : 0
0 - Global rank    : 0
0 - World size     : 1
0 - GPUs per node  : 1
0 - Master         : True
0 - Multi-node     : False
0 - Multi-GPU      : False
0 - Hostname       : <host>

Any ideas why this might be happening?

I also tried running this translation script, which similarly seems to get stuck:

python -m codegen_sources.model.translate --src_lang python --tgt_lang java --model_path TransCoder_model_2.pth.1 --beam_size 1 < hello.py
adding to path /srv/home/akshit/CodeGen
INFO - 09/27/21 05:56:56 - 0:00:06 - ============ Model Reloading
INFO - 09/27/21 05:56:56 - 0:00:06 - Reloading encoder from TransCoder_model_2.pth.1 ...
Dominant language
Python
Stars
777
Forks
144
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from facebookresearch/CodeGen

All issues in facebookresearch/CodeGen

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.