Linux: published ports unreachable from remote hosts — SYN arrives, LISTEN socket held by kiri, no SYN-ACK (loopback works)

Open
#649 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
docker, linux, rust
Domain
networking

Research direction

Start by reproducing on one Linux host: publish a port through the Docker API on /run/kiri/docker.sock, then compare ss -ltnp and tcpdump output for a loopback versus a remote connect. The report says the kiri process already holds the LISTEN socket in the init netns, so read the port-publishing and netns/iptables setup paths and check what the process does with accepted connections versus the kernel handshake. Also reproduce the restart and leaked-listener cases, and treat kiri diag report plus the manual MASQUERADE/FORWARD rules as the baseline for what done looks like.

Written by the indexing model from the issue text.

Description

Summary

On Linux, containers with published ports are reachable from the host itself (loopback and the host's LAN IP), but not from any remote machine. The SYN arrives on the host's interface, kiri holds a LISTEN socket on 0.0.0.0:<port>, yet no SYN-ACK is ever sent and the client times out. Kernel sockets (sshd, docker-proxy) on the same hosts accept remote connections normally — only sockets held by the kiri process are affected.

This made kiri 0.2.2 unusable as a dockerd replacement on our servers; we rolled the fleet back to dockerd. Reporting with full diagnostics in case it helps.

Environment

  • kiri 0.2.2 (kiri-lite deb) on multiple hosts:
    • Debian 13, kernel 6.12.117+deb13-cloud-amd64 (VM)
    • Debian 13, kernel 6.12.107+deb13-cloud-amd64 (VM)
    • Proxmox host, kernel 7.0.14-14-pve, Debian 13 trixie
    • Arch Linux (rolling), Hyprland laptop
  • Docker CE installed but disabled during testing (systemctl disable --now docker.socket docker.service); its iptables rules flushed where present (iptables -t nat -F DOCKER) to rule out stale DNAT.
  • Firewalls checked: iptables INPUT policy ACCEPT, no drops on the test ports; nftables (single inet backend) shows only tailscale chains and our own test rules; no XDP/eBPF attached to interfaces; conntrack clean.

Reproduction

  1. Install kiri-lite on a Linux host with no other engine running.
  2. Run any container with a published port, e.g. the equivalent of docker run -d -p 8888:80 nginx (via the Docker API on /run/kiri/docker.sock).
  3. From the host itself: curl http://127.0.0.1:8888/ → 200 OK. curl http://<host-LAN-IP>:8888/ from the host also works.
  4. From another machine on the same LAN: curl http://<host-LAN-IP>:8888/ → connection times out.

Evidence

tcpdump on the receiving interface during a remote attempt (three curls from 192.168.1.188 to 192.168.1.55:8888):

21:34:44.167713 eth0  In  IP 192.168.1.188.57454 > 192.168.1.55.8888: Flags [S], seq 2847638399, win 64240, options [mss 1460,sackOK,TS ...], length 0
21:34:45.199470 eth0  In  IP 192.168.1.188.57454 > 192.168.1.55.8888: Flags [S], ...   (retransmits, 4 total)

Only SYNs in, zero replies of any kind out.

At the same moment, the socket state on the host:

LISTEN 0  1024  0.0.0.0:8888  0.0.0.0:*  users:(("kiri",pid=3128755,fd=84))
LISTEN 0  1024  [::]:8888     [::]:*     users:(("kiri",pid=3128755,fd=85))

The kiri process (pid=3128755) is in the init network namespace (verified: /proc/1/ns/net and /proc/<pid>/ns/net both net:[4026531840]). So the kernel should complete the handshake for any matching LISTEN socket. It does not.

Outbound NAT is also absent: kiri installs no MASQUERADE/FORWARD rules for its bridge subnets, so containers have no outbound connectivity at all until rules are added manually:

iptables -t nat -A POSTROUTING -s <kiri-subnet> ! -o <kiri-bridge> -j MASQUERADE
iptables -A FORWARD -i <kiri-bridge> -j ACCEPT
iptables -A FORWARD -o <kiri-bridge> -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Secondary issues found in the same rollout

  1. Restarting a container wedges its published ports. docker restart <container> leaves the host ports dead afterwards (forwarding entry appears stale). Only compose up -d --force-recreate restores them.
  2. Leaked listeners. After a container publishing 127.0.0.1:443 was removed, kiri kept holding 127.0.0.1:443 and [::1]:443 listeners until the daemon was stopped — which then blocked other engines from binding those ports.

Status

Rolled back to dockerd everywhere; kiri 0.2.2 installed but disabled. Happy to re-enable kiri on a test host and run any diagnostics you need (kiri diag report, packet captures, experimental builds).

Dominant language
Rust
Stars
7.5k
Forks
329
Avg merge
14h 18m
Merged PRs (30d)
204

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from libracoder/kiri

All issues in libracoder/kiri

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.