[doc] Document the data machinery
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 22/100
- Issue 类型
- 文档
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 技术栈
- python
- 领域
- data, documentation
调研方向
The payload names no files, only the Data class and the changes in #37, #40 and #104, so start by reading those pull requests and the existing docs to find where the dataset machinery is described. Work through the checklist in the issue: dataset types, the GPT and generic dataset classes, how Data assembles them, and the existing prepare recipe. Done means a doc section covers each item, with a working example for the gpt_custom extension case.
由索引模型根据 Issue 内容生成。
描述
We need to document the new dataa mechanism, introduced in #37, #40, #104, etc.
The doc should at least describe:
- The different types of datasets (samplable, sampled, indexed, etc.)
- The different dataset classes available for GPT, and generic ones.
- The dynamic dataset class instantiation mechanism (to move elsewhere if/once we generalize)
- How datasets are put together in
Data - Maybe also the dataset preparation mechanism.
It should let the user know how to:
- Prepare a dataset with the prepare command (recipe is already in the doc, might just need to point to it)
- Configure Fast-LLM for typical dataset use cases (ex. dataset in multiple files, train/valid/test split, blending multiple datasets, etc.)
- Extend GPT with a custom configurable dataset class
- Extend GPT data and/or datasets for a custom model (ex.
gpt_custom) - (Advanced, can be postponed) Create a data and dataset for a custom model that doesn't extend GPT, reusing the existing generic machinery. (Need to generalize dynamic configuration first)
- (Can be postponed) Create a custom data that doesn't reuse the dataset machinery. Easy but not sure how much we need it.
- 主要语言
- Python
- 星标
- 337
- 派生
- 52
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
ServiceNow/Fast-LLM 的其他 Issue
-
难度 4/5 3-5 天 新手友好度 25/100
ServiceNow/Fast-LLM#556 ·
-
难度 4/5 3-5 天 新手友好度 30/100
ServiceNow/Fast-LLM#538 ·
-
enhancement
难度 5/5 一周以上 新手友好度 12/100
ServiceNow/Fast-LLM#533 ·
-
难度 5/5 一周以上 新手友好度 15/100
ServiceNow/Fast-LLM#510 · 1 条评论 ·
-
难度 5/5 一周以上 新手友好度 12/100
ServiceNow/Fast-LLM#507 ·
查看 ServiceNow/Fast-LLM 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 76/100
pytest-dev/pytest-html#1085 ·
-
难度 2/5 1-3 小时 新手友好度 90/100
pollen-robotics/reachy_mini#1527 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 66/100
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 66/100
维护者通常 1 天内回复