Build 1-gram / 2-gram / 3-gram tables
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning
Research direction
No source files or tests are named. Start by locating how training data enters the Python project, then use the stated table format and verify the serialization, memory-footprint documentation, and lookup-time benchmark against the success criteria.
Written by the indexing model from the issue text.
Description
Summary
Build n-gram frequency tables from the training data for lookup-based generation.
Success Criteria
- 1-gram table: token → count
- 2-gram table: (token₁) → [(token₂, count), ...]
- 3-gram table: (token₁, token₂) → [(token₃, count), ...]
- Tables serialized efficiently (consider trie structure)
- Memory footprint documented
- Lookup time benchmarked (<1μs per lookup target)
Implementation Notes
- Consider smoothing for unseen n-grams
- Store both raw counts and probabilities
- Prune low-frequency entries to control size
Table Format
```python
{
"unigram": {"feat": 15000, "fix": 12000, ...},
"bigram": {"feat": {"add": 5000, "implement": 3000, ...}, ...},
"trigram": {("feat", "add"): {"new": 2000, "support": 1500, ...}, ...}
}
```
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aRustyDev/ccgram
-
docs experiment
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
docs
Difficulty 4/5 3-5 days Newbie friendliness 55/100
-
Freeze architecture Openinfra model
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Document findings Opendocs experiment
Difficulty 3/5 3-5 days Newbie friendliness 45/100
-
ablation evaluation
Difficulty 4/5 3-5 days Newbie friendliness 30/100
All issues in aRustyDev/ccgram
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
anthropics/skills#1811 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
speaches-ai/speaches#678 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
datalayer/mcp-compose#42 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
conda-forge/spacy-feedstock#177 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
UKGovernmentBEIS/inspect_evals#2523 ·