Self-hosted: list and create Redpanda Connect pipelines backed by the operator's Pipeline CRD

Aperta
  • Qualcosa non va?
#2,697 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
32/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, kubernetes, typescript
Ambito
api, backend, infrastructure

Direzione di ricerca

Start with backend/pkg/console/endpoint_compatibility.go, where PipelineService is marked unsupported for self-hosted Console, and trace the existing PipelineService handlers and UI calls. The issue describes the Kubernetes Pipeline CRD operations and fields to map, but leaves configuration, Secret handling, cluster binding, and RBAC decisions open. Done means the configured self-hosted Console can list, view, create, update, delete, start, and stop Pipeline CRs through the existing UI.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

feature

Self-hosted users on Kubernetes now run Redpanda Connect through the Redpanda Operator's Pipeline CRD (pipelines.cluster.redpanda.com, cluster.redpanda.com/v1alpha2, operator v26.2.x with connectController.enabled). Console can't see or manage those pipelines today, so they have to use kubectl or GitOps for everything. We'd like Console to read Pipeline CRs from the cluster it runs in and to create and manage them from the existing Redpanda Connect UI.

What happens today

Tested on EKS 1.37 with operator v26.2.5 and Console v3.9.0 and v3.13.1:

  • The Redpanda Connect pages (/rp-connect, /rp-connect/create) render, but every redpanda.api.console.v1alpha1.PipelineService call returns HTTP 501: 
    {"code":"unimplemented","message":"redpanda connect is not enabled", ... "reason":"REASON_FEATURE_NOT_CONFIGURED"}
  • Pipelines created as CRs never show up in the list, even though their topics and consumer groups are visible in Console.
  • Filling in the "Create Pipeline" form and clicking Create sends CreatePipeline, which also returns 501. No Pipeline CR is created.
  • PipelineService is hard-coded as unsupported for self-hosted in backend/pkg/console/endpoint_compatibility.go ("OSS defaults").

Ask

A Kubernetes-backed PipelineService for self-hosted Console:

  • Read: list, get and watch Pipeline CRs in the configured namespace(s). Map displayName, description, tags, configYaml, replicas and resources into the existing list and details views, and status.phase / status.conditions into pipeline state.
  • Write: create, update and delete from the UI write the CR. Start and stop map to spec.paused, and compute units map to spec.resources. The existing Connect config linting could validate before submit.
  • Config: opt-in, e.g. redpandaConnect.kubernetes.enabled plus a namespace list, using the in-cluster ServiceAccount. When enabled, /api/console/endpoints should report PipelineService as supported.
  • RBAC: get/list/watch/create/update/patch/delete on pipelines.cluster.redpanda.com. On the operator side, the Console CR and console chart could grant this when the feature is on. I can open the operator follow-up.

Open questions:

  • How the Secrets UI should map to the CRD's valueSources (Secret, ConfigMap or ExternalSecret refs).
  • Whether pipelines should bind to the same cluster Console is connected to via cluster.clusterRef.

The operator-side test harness and evidence are available if useful. Curious if this fits the roadmap, and let me know if a different API shape works better.

Lingua principale
TypeScript
Stelle
4.3k
Fork
433
Merge medio
2g 9h
PR unite (30g)
29

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di redpanda-data/console

Tutte le issue di redpanda-data/console

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.