bpe-openai: allow lazy loading BPE and feature flag each model

Open
#82 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
rust
Domain
performance

Research direction

The issue names no files, tests, or entry points; begin by locating the bpe-openai generated .dict models and the code that translates or loads them. Done means models can be loaded lazily, individual models can be selected with feature flags, and binary and memory savings are demonstrated without losing supported behavior.

Written by the indexing model from the issue text.

Description

The fully generated .dict models take 70mb, which bloats binaries substantially. This is compared to 3.4mb for the targz.

It would be nice to support lazily doing this translation, which takes a modest hit to runtime performance in return for dramatically smaller binary, and the ability to only load certain models reducing memory consumption.

It would also be nice to have feature flags to turn off certain models; this is less important if we add lazy loading, though, as it would just allow you to save 1-2mb off the binary.

Dominant language
Rust
Stars
134
Forks
24
Avg merge
16h 27m
Merged PRs (30d)
11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from github/rust-gems

All issues in github/rust-gems

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.