nsqio/nsq

*: Support NSQ in ArgoLabs Dataflow

Open

#1381 opened on Oct 19, 2021

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (25,788 stars) (2,898 forks)batch import
help wantedrequest

Description

In Dataflow, we currently have support for Kafka, Jetstream, NATS Streaming and other sources and sinks. It would be amazing to add support for NSQ too. We know that we plenty of use cases in the OSS community and this would get a lot of love.

The Dataflow team are not NSQ experts. My ask is - would anyone be interested in implementing the changes needed?

Here is a pull request for Jetstream, breaking it down:

  1. Add manifests to install a dev NSQ into Kubernetes cluster.
  2. Add APIs to specify a NSQ source and sink (e.g. the URL, authentication).
  3. Implement the source and sink.
  4. Write test infrastructure (e.g. check the right number of messages get written to a sink).
  5. Write tests (e.g. does the right number of message get written, even if the pod is deleted).
  6. Write examples.
  7. Write docs.

I'll raise a ticket against the main repo, and the Go client repo (there might be engineers there with the exact skills needed).

Contributor guide

Tech stack
gokubernetes
Domain
backenddatacloud
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Go programmingKubernetes basicsArgo DataflowNSQ concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Start by studying the referenced Jetstream pull request (https://github.com/argoproj labs/argo dataflow/commit/11251f5806ed8bfbd9e6b5017259c19b006118ca) to understand the pattern for adding a new source/sink. Explore the argo dataflow repository for existing source/sink implementations and the NSQ Go client library for API details. Identify the required Kubernetes manifests and test infrastructure from similar implementations.