Self-hosted: list and create Redpanda Connect pipelines backed by the operator's Pipeline CRD

Abierto
  • ¿Algo no está bien?
#2,697 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
32/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go, kubernetes, typescript
Área
api, backend, infrastructure

Línea de trabajo

Start with backend/pkg/console/endpoint_compatibility.go, where PipelineService is marked unsupported for self-hosted Console, and trace the existing PipelineService handlers and UI calls. The issue describes the Kubernetes Pipeline CRD operations and fields to map, but leaves configuration, Secret handling, cluster binding, and RBAC decisions open. Done means the configured self-hosted Console can list, view, create, update, delete, start, and stop Pipeline CRs through the existing UI.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

feature

Self-hosted users on Kubernetes now run Redpanda Connect through the Redpanda Operator's Pipeline CRD (pipelines.cluster.redpanda.com, cluster.redpanda.com/v1alpha2, operator v26.2.x with connectController.enabled). Console can't see or manage those pipelines today, so they have to use kubectl or GitOps for everything. We'd like Console to read Pipeline CRs from the cluster it runs in and to create and manage them from the existing Redpanda Connect UI.

What happens today

Tested on EKS 1.37 with operator v26.2.5 and Console v3.9.0 and v3.13.1:

  • The Redpanda Connect pages (/rp-connect, /rp-connect/create) render, but every redpanda.api.console.v1alpha1.PipelineService call returns HTTP 501: 
    {"code":"unimplemented","message":"redpanda connect is not enabled", ... "reason":"REASON_FEATURE_NOT_CONFIGURED"}
  • Pipelines created as CRs never show up in the list, even though their topics and consumer groups are visible in Console.
  • Filling in the "Create Pipeline" form and clicking Create sends CreatePipeline, which also returns 501. No Pipeline CR is created.
  • PipelineService is hard-coded as unsupported for self-hosted in backend/pkg/console/endpoint_compatibility.go ("OSS defaults").

Ask

A Kubernetes-backed PipelineService for self-hosted Console:

  • Read: list, get and watch Pipeline CRs in the configured namespace(s). Map displayName, description, tags, configYaml, replicas and resources into the existing list and details views, and status.phase / status.conditions into pipeline state.
  • Write: create, update and delete from the UI write the CR. Start and stop map to spec.paused, and compute units map to spec.resources. The existing Connect config linting could validate before submit.
  • Config: opt-in, e.g. redpandaConnect.kubernetes.enabled plus a namespace list, using the in-cluster ServiceAccount. When enabled, /api/console/endpoints should report PipelineService as supported.
  • RBAC: get/list/watch/create/update/patch/delete on pipelines.cluster.redpanda.com. On the operator side, the Console CR and console chart could grant this when the feature is on. I can open the operator follow-up.

Open questions:

  • How the Secrets UI should map to the CRD's valueSources (Secret, ConfigMap or ExternalSecret refs).
  • Whether pipelines should bind to the same cluster Console is connected to via cluster.clusterRef.

The operator-side test harness and evidence are available if useful. Curious if this fits the roadmap, and let me know if a different API shape works better.

Lenguaje dominante
TypeScript
Estrellas
4.4k
Forks
435
Merge medio
2 d 8 h
PR fusionados (30 d)
30

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de redpanda-data/console

Todos los issues de redpanda-data/console

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.