Request to Modify Code to Enable TEXT_SPLITTER_EMBEDDING_MODEL Customization through Configuration File
@sumitkbh is already working on this.
Since Jan 18, 2024.
Assessment
This issue has not been assessed yet.
Description
I am looking to create a Chinese RAG demo service using RetrievalAugmentedGeneration.
However, I encountered an issue where the default SentenceTransformersTokenTextSplitter model used in the RetrievalAugmentedGeneration/common/utils.py file is hardcoded as 'intfloat/e5-large-v2'. This model generates a significant number of [UNK] tokens when processing Chinese text.
I would like the ability to specify a specific model for the text splitter, similar to how the embedding model can be specified through the config.yaml file.
Thank you for your assistance and support.
- Dominant language
- Jupyter Notebook
- Stars
- 4.2k
- Forks
- 1.1k
- Avg merge
- 10h 15m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/GenerativeAIExamples
-
Difficulty 1/5 Under an hour Newbie friendliness 70/100
NVIDIA/GenerativeAIExamples#361 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NVIDIA/GenerativeAIExamples#299 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
NVIDIA/GenerativeAIExamples#437 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 45/100
NVIDIA/GenerativeAIExamples#400 ·
-
Difficulty 5/5 Over a week Newbie friendliness 10/100
NVIDIA/GenerativeAIExamples#399 ·