Why tokenizer.pad_token == args.pad (i.e., 50256)??
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 20/100
- Issue 类型
- 缺陷
- 描述清晰度
- 需要澄清
- 活跃度
- 停滞
- 技术栈
- python
调研方向
Start with jaxformer/hf/sample.py around lines 234 and 240 to inspect how tokenizer.pad_token and pad_token_id are passed during sampling. Then compare that procedure with the single-turn HumanEval setup used for the paper's Table 1 results; done means the padding behavior and benchmark discrepancy are explained.
由索引模型根据 Issue 内容生成。
描述
Hi,
For my project, I'm trying to fine-tune CodeGen models on my dataset and evaluate the resulting fine-tuned model on the HumanEval benchmark dataset. I have a few questions that I would appreciate if you could address.
-
First, why in the sampling code, at line 234, we have
tokenizer.pad_token == args.pad, which is 50256. Shouldn't we set the pad_token to eos_token, not 50256 (which is the eos_token_id)? I'm confused by this. At line 240, you set the parameter pad_token_id=args.pad. So in your sampling code, both pad_token and pad_token_id are set to 50256. Can you please elaborate on this? That would be super helpful. -
As a baseline, I need to replicate your single-turn HumanEval benchmark results, but unfortunately, I'm getting surprisingly lower results compared to what is reported in the paper. And, I'm 99% positive that I'm probably missing a point. To produce Table 1 results in the paper, did you use the exact same sampling procedure as sample.py?
Thanks a lot for your time.
- 主要语言
- Python
- 星标
- 5.2k
- 派生
- 420
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
salesforce/CodeGen 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 45/100
salesforce/CodeGen#107 ·
-
难度 5/5 一周以上 新手友好度 20/100
salesforce/CodeGen#104 ·
-
难度 5/5 一周以上 新手友好度 25/100
salesforce/CodeGen#101 ·
-
难度 1/5 1 小时以内 新手友好度 48/100
salesforce/CodeGen#95 ·
-
难度 3/5 1-2 天 新手友好度 35/100
salesforce/CodeGen#94 · 8 条评论 ·
查看 salesforce/CodeGen 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 86/100
维护者通常 1 天内回复
-
难度 2/5 1-2 天 新手友好度 70/100
-
难度 2/5 1-3 小时 新手友好度 88/100
维护者通常 7 天内回复
-
难度 2/5 1-3 小时 新手友好度 70/100
lmstudio-ai/mlx-engine#376 ·
-
难度 2/5 1-3 小时 新手友好度 72/100
pyiron/bagofholding#166 ·