connection refused for kafka transport

Open
#427 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
docker-compose, go, kafka
Domain
backend

Research direction

Start with the Kafka transport configuration and broker connection handling in the repository, then check how the provided Docker Compose setup configures Kafka listeners and startup readiness. Reproduce the connection refused error using the reported compose configuration and identify whether the broker address or startup sequence causes it. Done means the transport connects successfully or the issue includes a confirmed cause and a clear configuration fix.

Written by the indexing model from the issue text.

Description

Hi,

I am having this issue with the code while trying to use : confluentinc/cp-kafka

Error :

time=2025-10-01T05:35:12.167Z level=ERROR msg="error transporter" error="kafka: client has run out of available brokers to talk to: dial tcp 172.20.0.3:9092: connect: connection refused for kafka transport"

docker-compose. File:


`services:
  kafka:
    image: confluentinc/cp-kafka:latest
    ports:
      - 9092:9092
    environment:

      KAFKA_KRAFT_MODE: "true"  # Enables KRaft mode.
      KAFKA_PROCESS_ROLES: controller,broker  # Kafka acts as both controller and broker.
      KAFKA_NODE_ID: 1  # Unique ID for the Kafka instance.
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_ADVERTISED_LISTENERS: "BROKER://kafka:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,BROKER:PLAINTEXT,IB:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: BROKER
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LOG_DIRS: /var/lib/kafka/data  # Log storage location.
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"  # Enables automatic topic creation.
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1  # Single replica for simplicity.
      KAFKA_LOG_RETENTION_HOURS: 168  # Log retention period (7 days).
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0  # No rebalance delay.
      CLUSTER_ID: AAAAAAAAAAAAAAAAAAAAAA
      KAFKA_ENABLE_KRAFT: yes


        # dd if=/dev/zero of=/dev/stdout count=16 bs=1 | openssl enc -A -a


        # head -c 20 /dev/zero | base64
    restart: always
  grafana:
    image: grafana/grafana:12.1
    environment:
      - GF_INSTALL_PLUGINS=vertamedia-clickhouse-datasource
    #  - GF_INSTALL_PLUGINS=grafana-clickhouse-datasource
    #  - GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=vertamedia-clickhouse-datasource
    ports:
      - 3000:3000
    restart: always
    volumes:
      - ./grafana/datasources-ch.yml:/etc/grafana/provisioning/datasources/datasources-ch.yml
      - ./grafana/dashboards.yml:/etc/grafana/provisioning/dashboards/dashboards.yml
      - ./grafana/dashboards:/var/lib/grafana/dashboards
  prometheus:
    image: prom/prometheus:v3.6.0
    ports:
      - 9090:9090
    restart: always
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
  goflow2:
    build:
      context: ../../
      dockerfile: Dockerfile
      args:
        VERSION: compose
        LDFLAGS: -X main.version=compose
    image: netsampler/goflow2
    depends_on:
      - kafka
    ports:
      - 8080:8080
      - 6343:6343/udp
      - 2055:2055/udp
    restart: always
    command:
      - -transport.kafka.brokers=kafka:9092
      - -transport=kafka
      - -transport.kafka.topic=flows
      - -format=bin
  db:
    image: clickhouse/clickhouse-server:24.11.1.2557-alpine
    ports:
      - 8123:8123
    volumes:
      - ./clickhouse:/docker-entrypoint-initdb.d/
      #- ../../pb/flow.proto:/var/lib/clickhouse/format_schemas/flow.proto
      # the following files have 101:101 as owner
      - ./clickhouse/flow.proto:/var/lib/clickhouse/format_schemas/flow.proto
      - ./clickhouse/protocols.csv:/var/lib/clickhouse/user_files/protocols.csv
    depends_on:
      - kafka`


Dominant language
Go
Stars
817
Forks
182
Avg merge
4h 18m
Merged PRs (30d)
6

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from netsampler/goflow2

All issues in netsampler/goflow2

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.