arc53/DocsGPT

🚀 Feature Extract and process images from source uploads

Open

#1,451 建立於 2024年11月23日

在 GitHub 查看
 (8 留言) (3 反應) (3 負責人)Python (13,691 star) (1,359 fork)batch import
applicationbackendenhancementhacktoberfesthelp wantedpython

描述

🔖 Feature description

@dartpain Basically allow our AI's to eat images. good place to start is just open ai llms, even docsgpt free service should work

🎤 Why is this feature needed ?

Accept image and encoded base64

  • Maybe user wants to send a message with an image when chatting with AI

✌️ How do you aim to achieve this?

  • When a source has an image, for example there is a link (publicly accessible) in markdown file that was ingested, we can pass it to llm inside classic rag retriever when its generating an answer.

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

  • I checked and didn't find similar issue

Are you willing to submit PR?

Yes I am willing to submit a PR!

貢獻者指南

技術棧
python
領域
backendai
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
3-5 days
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
blocked
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
PythonLLM conceptsRAG pipelineDocsGPT codebase
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
35
研究方向
Investigate the current document ingestion and retrieval pipeline in the DocsGPT codebase, focusing on how markdown files are processed and passed to the RAG retriever. Look for modules handling source uploads and text extraction. Then, design a method to detect image links within markdown content, download the images, encode them as base64, and incorporate them into the prompt for the OpenAI LLM. Review any existing comments or related issues for additional context on implementation preferences.