Question for validation and test sets
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Documentation
- Clarity
- Needs clarification
- Activity status
- Stale
- Domain
- data, machine-learning
Research direction
Start by comparing the files in data/transcoder_evaluation_gfg with the validation and test counts reported in Tables 4 and 5 of the referenced paper. Trace how the parallel functions are paired and split across Python, Java, and C++, then document the reason for the differing directional counts. Done means the repository data and paper figures are reconciled or the discrepancy is clearly explained.
Written by the indexing model from the issue text.
Description
Hi,
as the paper "Unsupervised Translation of programming languages" mentioned, there are 852 parallel functions. So I checked the data in this repo fold and found (each file contain one function code with unit test cases, actually there are 852 union filenames in python/java/cpp folder):
- 698 cpp functions
- 717 java functions
- 702 python functions
the number of the test/validate dataset sizes is different as mentioned in Table 4 of the raw paper :
- c++ 466/231
- java 234/481
- python 237/463
And another question is function pairs number in Table 5 of the raw paper. I'm wander why the C++ -> java hava 481 tests functions while the java -> c++ only hava 466 test functions. If my understanding is right, there should have a same number of tests for giving parallel functions (java to c++ or c++ to java). Why the count of test functions if different for c++->java and java->c++ ? (same for other language pairs)
really thanks
- Dominant language
- Python
- Stars
- 777
- Forks
- 144
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from facebookresearch/CodeGen
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
facebookresearch/CodeGen#109 ·
-
Hash function Open
Difficulty 3/5 1-2 days Newbie friendliness 42/100
facebookresearch/CodeGen#108 ·
-
Dataset issues Open
Difficulty 3/5 1-2 days Newbie friendliness 45/100
facebookresearch/CodeGen#107 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
facebookresearch/CodeGen#106 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
facebookresearch/CodeGen#105 ·
All issues in facebookresearch/CodeGen
Similar issues
-
essnmx good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
[Feature] 奇物选择添加优先级 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
syfoud/Simulated_Scepter#174 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Giskard-AI/giskard-oss#2840 · 1 comment ·
-
A claim comment carrying the issue number is silently declined while the workflow reports success Openarea: repo bug perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
yeti-platform/yeti#1380 ·