Completing Protobuf / gRPC support (JENA-2167): message-level StreamRDF/RowSet adapters, Protobuf RDFPatch, and RDFLink over gRPC

Aperta
  • Qualcosa non va?
#4,364 14 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java
Ambito
backend-api-design

Direzione di ricerca

Start by reading the attached reviewed approach and examining Jena's existing Protobuf converters and Thrift schema, which the issue identifies as the relevant starting points. The proposed work spans message-level adapters, RDF Patch support, and a gRPC transport, so confirm the boundaries and expected APIs with maintainers before starting. Done means the agreed work is implemented with the promised test coverage; the issue proposes splitting it into three PRs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement
Version

6.x (main)

Feature
TLDR - the full roadmap

I would like to contribute to the Jena project and help improve it's support for Protobuf and gRPC integration. As an apache contributor, I am trying to expand gRPC support in the ecosystem as part of greater efforts for Apache licensed software.

Digging through your tickets, I found some interest in gRPC support. I'd like to propose 3 PRs to get this done.

What I'd like to contribute
  • Direct Protobuf message streaming without byte-stream overhead: Allow applications to convert RDF triples, quads, and SPARQL result rows directly to and from Jena's existing Protobuf objects in memory, rather than having to serialize and re-parse them through Java byte streams when using frameworks like gRPC or Kafka.
  • Protobuf support for RDF Patch: Bring Jena's Protobuf schema up to parity with its Thrift schema by adding support for RDF Patches, so transactional dataset additions, deletions, and prefix changes can be streamed and recorded in Protobuf format.
  • A gRPC server and client for Jena datasets: Provide an optional gRPC transport that plugs into Jena's standard connection APIs and storage engines (such as TDB2) without pulling in an embedded HTTP/servlet server like Jetty.
  • End-to-end streaming for graph reads, writes, and SPARQL queries: Support streaming quad pattern lookups, bulk RDF loading, transactional patch replication, and SPARQL query results natively over gRPC streams.

I'd make 3 PRs to get this done.

  1. Expose the in-memory Protobuf message converters
  2. RDF Patch support
  3. gRPC transport module

As far as approach goes, it's pretty straightforward and I attached an AI-assisted approach to it. I've fully vetted and edited it, but because it has massive references to the codebase that a normal human wouldn't write, I have attached it as an ai-generated plan and won't call it my own (but think it's exactly right and would seek feedback).

ai-generated-approach-reviewed.md

I'd recommend we get an implementation going first then we can figure out the server infrastructure for serving gRPC if you're interested.

Prior experience to get this quickly with quality

I've contributed gRPC improvements to a multitude of projects

  • Quarkus - fixed critical bugs + massive speed improvements
  • Micronaut - created gRPC <--> REST automation
  • Apicurio - better serialization of protobufs and better performance enhancements
  • OpenNLP - I'm a committer to this project. We're going to push an initial gRPC server that'll support all of OpenNLPs rich feature set (probably early 2027)
  • Tika - working with the team to come up with a parser design with their v2 of their grpc service

and many more.

I'll make sure that there's plenty of test coverage, I'll attribute all code that is AI written and review it carefully before opening a PR, and am totally OK if the patch is too large to break it up into smaller tickets that are easy for a maintainer to understand. This is an important feature to me; and I'll start working on it now. If you make a branch for this, push my changes there as a PR from my branch or directly. I'd only recommend this so we can approve the code incrementally and push to a branch before merging a full branch and allowing for contributions and collaboration from maintainers and committers within jena.

Are you interested in contributing a solution yourself?

Yes

Lingua principale
Java
Stelle
1.5k
Fork
729
Merge medio
1g 8h
PR unite (30g)
56

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/jena

Tutte le issue di apache/jena

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.