Tokenise a FineWeb-Edu shard set and publish its manifest

Open
  • Is something wrong?
#22 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
62/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
huggingface, numpy, python
Domain
data, machine-learning

Research direction

Read docs/run0.md, manifests/datasets.toml, and the Run 0 config to identify the tokenizer and expected OLMo-core format; check OLMo-core's data docs for the tokenisation workflow. Tokenise a small FineWeb-Edu shard set and record its source, licence, revision, command, output names, sizes, and SHA-256 hashes in the manifest. Document the command, verify a repeat run produces the same hashes, then run make check and make test.

Written by the indexing model from the issue text.

Description

agent-authored area/data good first issue help wanted layer/5-storage type/feature

Context

Run 0 needs tokenised data in the format OLMo-core reads, and every dataset we use must be recorded with source, licence, revision and hash (README, What is open).

Task

Tokenise a small FineWeb-Edu shard set (for example 1B tokens or less) with the tokenizer the Run 0 config names, in OLMo-core's numpy format. Record it in manifests/datasets.toml: HF dataset id and revision, licence, the exact tokenisation command, output file names, sizes and SHA-256 hashes. Publishing the shards to Hugging Face is done by a maintainer; attach the manifest and hashes.

Definition of done

  • Manifest entry passes make check
  • Hashes are reproducible: a second run of your command gives the same SHA-256
  • Command documented in docs/run0.md
  • make check and make test pass
  • Commits signed off (git commit -s)

Pointers

Skills: Data, Python

Agent: omp / claude-opus-5-5 (issue drafted by an agent; a maintainer reviews the PR)

Dominant language
Python
Stars
1
Forks
2
Avg merge
2h 4m
Merged PRs (30d)
23

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenFrontierLabs/openfrontier

All issues in OpenFrontierLabs/openfrontier

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.