How to use new BPE codes and vocab to generate parallel data
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Documentation
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- data, machine-learning
Research direction
Start with data/test_dataset and the transcoder_test_set.zip archive, then trace the Monolingual and Monolingual Functions preprocessing entry points mentioned in the questions. Compare the .tok inputs with the test.cpp_sa-java_sa.cpp_sa.pth and test.cpp_sa-java_sa.java_sa.pth files. Done means documenting the .pth contents, how the learned codes and vocabulary are used, which vocabulary trains CodeGen, and why two BPE code sets exist.
Written by the indexing model from the issue text.
Description
I have a few questions.
-
I have used the data in the folder https://github.com/facebookresearch/CodeGen/tree/main/data/test_dataset learned BPE codes and vocab using Monolingual Functions mode. I want to know how to use the .tok files in the zip file https://dl.fbaipublicfiles.com/transcoder/test_set/transcoder_test_set.zip and generate files like test.cpp_sa-java_sa.cpp_sa.pth using my bpe codes and vocab?
-
What is the contents of file test.cpp_sa-java_sa.cpp_sa.pth ? Also what is the difference between files test.cpp_sa-java_sa.cpp_sa.pth and test.cpp_sa-java_sa.java_sa.pth ?
-
I first preprocessed data in Monolingual mode, learned BPE codes, and then did my MLM training. Then I preprocessed the data in Monolingual Functions mode and learned new BPE codes and vocab. My question is, which vocab have you used to train CodeGen? Also, why two different BPE codes are learned?
- Dominant language
- Python
- Stars
- 777
- Forks
- 144
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from facebookresearch/CodeGen
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
facebookresearch/CodeGen#109 ·
-
Hash function Open
Difficulty 3/5 1-2 days Newbie friendliness 42/100
facebookresearch/CodeGen#108 ·
-
Dataset issues Open
Difficulty 3/5 1-2 days Newbie friendliness 45/100
facebookresearch/CodeGen#107 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
facebookresearch/CodeGen#106 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
facebookresearch/CodeGen#105 ·
All issues in facebookresearch/CodeGen
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
syfoud/Simulated_Scepter#172 ·
-
A cancelled tests run makes the coverage comment workflow fail and reports it as a red check on main Openarea: ci bug perceived difficulty: 3
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Nitjsefnie-Harness-Commons/daedalus#921 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ClickHouse/clickhouse-connect#1057 ·