Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Feature Request]: Add Chroma vector database ingestion connector for RAG pipelines

Aperta
  • Qualcosa non va?
#40,047 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@nitinware ci sta già lavorando.

Dal 7/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

io new feature P2 python
What would you like to happen?

Motivation

Beam's Python ml/rag ingestion module currently supports Milvus and Qdrant (sdks/python/apache_beam/ml/rag/ingestion/). Chroma is one of the most widely-adopted open-source vector databases, especially for local and embedded RAG workloads, and is the missing piece in Beam's OSS vector-DB coverage.

Proposal

Add sdks/python/apache_beam/ml/rag/ingestion/chroma.py implementing a WriteToChroma PTransform matching the shape of the existing Milvus and Qdrant connectors:

  • Accepts Chunk records from the existing RAG chunking transforms
  • Supports both chromadb.HttpClient (remote) and chromadb.PersistentClient
    (embedded / local) client modes
  • Configurable collection name, embedding + metadata mapping, batch size
  • Standard retry/backoff

Benefits

  • Completes Beam's coverage of the OSS local-first vector-DB tier
  • Chroma is Apache-2.0 licensed — no dependency-license concerns
  • PersistentClient runs in-process, so integration tests do not need
    Docker or a remote service — a meaningful advantage over Milvus/Qdrant
    which require running servers in CI

Testing plan

  • Unit tests mirroring qdrant_test.py
  • Integration test using chromadb.PersistentClient against a tmpdir
    (no external service required)
  • Docs + example consistent with existing Milvus / Qdrant docs
Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components
  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
Lingua principale
Java
Stelle
8.7k
Fork
4.7k
Merge medio
2g 4h
PR unite (30g)
278

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/beam

Tutte le issue di apache/beam

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.