arc53/DocsGPT

🚀 Feature Extract and process images from source uploads

Open

#1451 opened on Nov 23, 2024

View on GitHub
 (8 comments) (3 reactions) (3 assignees)Python (13,691 stars) (1,359 forks)batch import
applicationbackendenhancementhacktoberfesthelp wantedpython

Description

🔖 Feature description

@dartpain Basically allow our AI's to eat images. good place to start is just open ai llms, even docsgpt free service should work

🎤 Why is this feature needed ?

Accept image and encoded base64

  • Maybe user wants to send a message with an image when chatting with AI

✌️ How do you aim to achieve this?

  • When a source has an image, for example there is a link (publicly accessible) in markdown file that was ingested, we can pass it to llm inside classic rag retriever when its generating an answer.

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

  • I checked and didn't find similar issue

Are you willing to submit PR?

Yes I am willing to submit a PR!

Contributor guide

Tech stack
python
Domain
backendai
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
PythonLLM conceptsRAG pipelineDocsGPT codebase
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Investigate the current document ingestion and retrieval pipeline in the DocsGPT codebase, focusing on how markdown files are processed and passed to the RAG retriever. Look for modules handling source uploads and text extraction. Then, design a method to detect image links within markdown content, download the images, encode them as base64, and incorporate them into the prompt for the OpenAI LLM. Review any existing comments or related issues for additional context on implementation preferences.