`fastBPE` fix path

Open

#86 opened on Nov 1, 2022 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 1/5
Estimated time: Under an hour
Newbie friendliness: 35/100

Issue type: Bug
Clarity: Clearly specified
Activity status: Stale
Tech stack: python
Domain: data

Research direction

Start in codegen_sources/preprocessing/bpe_modes/fast_bpe_mode.py and compare the FAST path with the fastBPE location specified by install_env.sh. Verify the resulting command uses codegen_sources/model/tools/fastBPE/fast and no longer raises the reported failed-to-learn-BPE assertion.

Written by the indexing model from the issue text.

Description

I against assertion AssertionError: failed to learn bpe on /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb, command: /home/dungnm/CodeGen/fastBPE/fast learnbpe 50000 /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.tok.shuf.50gb > /media/Z/dungnm31/transcoder/cpp-java-python.monolingual.codes

It turn out the command itself was not right. The fastBPE path will be located at "codegen_sources/model/tools/fastBPE/fast" according to install_env.sh instead of "fastBPE/fast"

Suggest file codegen_sources/preprocessing/bpe_modes/fast_bpe_mode.py change

FAST = str(Path(__file__).parents[3].joinpath("fastBPE/fast"))

FAST = str(Path(__file__).parents[3].joinpath("codegen_sources/model/tools/fastBPE/fast"))

Dominant language: Python
Stars: 777
Forks: 144
PR merge metrics: No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

[Bug] reef-hermes tells me to resume with hermes --resume, which does not work from my shell Open

area: harness bug status: needs-triage

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

Human-Agent-Society/reef#625 · Opened Sep 24, 2026
Bump transitive axios to >=1.18.0 to address GHSA-hcpx-6fm6-wx23 Open

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

coinbase/cdp-sdk#828 · Opened Sep 24, 2026
AccessibleResourceCard/AccessibleChannelCard crash when a resource has more than one learning activity (getActivityTags returns an object instead of an array) Open

Difficulty 1/5 Under an hour Newbie friendliness 80/100

learningequality/kolibri#15351 · 2 comments · Opened Sep 24, 2026
audeer.path(): optionally expand environment variables Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

audeering/audeer#212 · 1 comment · Opened Sep 24, 2026
Name consistency Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

eellak/triplestore#65 · 1 comment · Opened Sep 24, 2026

More Python issues