arc53/DocsGPT

🚀 Feature Extract and process images from source uploads

Open

Aperta il 23 nov 2024

Vedi su GitHub
 (8 commenti) (3 reazioni) (3 assegnatari)Python (13.691 star) (1359 fork)batch import
applicationbackendenhancementhacktoberfesthelp wantedpython

Descrizione

🔖 Feature description

@dartpain Basically allow our AI's to eat images. good place to start is just open ai llms, even docsgpt free service should work

🎤 Why is this feature needed ?

Accept image and encoded base64

  • Maybe user wants to send a message with an image when chatting with AI

✌️ How do you aim to achieve this?

  • When a source has an image, for example there is a link (publicly accessible) in markdown file that was ingested, we can pass it to llm inside classic rag retriever when its generating an answer.

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

  • I checked and didn't find similar issue

Are you willing to submit PR?

Yes I am willing to submit a PR!

Guida contributor

Tech stack
python
Dominio
backendai
Tipo issue
feature
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
3
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
3-5 days
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
blocked
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
mostly clear
Prerequisiti
PythonLLM conceptsRAG pipelineDocsGPT codebase
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
35
Direzione di ricerca
Investigate the current document ingestion and retrieval pipeline in the DocsGPT codebase, focusing on how markdown files are processed and passed to the RAG retriever. Look for modules handling source uploads and text extraction. Then, design a method to detect image links within markdown content, download the images, encode them as base64, and incorporate them into the prompt for the OpenAI LLM. Review any existing comments or related issues for additional context on implementation preferences.