portlist: TestPortlistBackendChurn is flaky when other processes open sockets

Open
  • Is something wrong?
#22,348 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
70/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, linux
Domain
networking, testing

Research direction

Start with TestPortlistBackendChurn in portlist_linux_bench_test.go and inspect how it exercises the /proc and inet_diag listing paths. The issue proposes using a dedicated tcp4 listener on a port absent from the listing, then checking that the port appears before closing and disappears afterward. Run the churn test under concurrent socket activity; done means the assertion tracks the fixture port reliably across both paths.

Written by the indexing model from the issue text.

Description

bug testing

TestPortlistBackendChurn fails when other sockets are opened in the same network namespace while it runs:

portlist_linux_bench_test.go:341: churn did not remove listener: before=11 after=11

This happens in nixpkgs' package tests. There, one go test -p $NIX_BUILD_CORES run tests many packages in parallel inside a single build sandbox, and the sandbox has one network namespace shared by all of them.

The test closes one fixture listener and asserts that len(AppendListeningPorts) shrank. That listing covers every socket in the namespace, so any other process or test that opens a socket between the two listings breaks the assertion. As a reproducer, I ran four processes that bind and close loopback TCP/UDP sockets in a tight loop alongside go test -count=500 -run '^TestPortlistBackendChurn$' ./portlist/. It failed 354 of 500 runs.

The obvious fix is to check that the closed listener's own port disappears, rather than comparing counts. That approach has a second, rarer problem, which the count check also has. The listing is deduplicated by (Proto, Port) regardless of address. The fixture's tcp6 listener on [::1]:0 sometimes gets the same port number as the tcp4 listener on 127.0.0.1:0. When it does, closing the tcp4 listener leaves the (tcp, port) entry in the listing. I saw this once in 2000 runs.

Here's a possible way forward.

  • open a dedicated tcp4 listener for the churn step, on a port that is not already in the listing;
  • assert that this port is listed before the close and absent after it, for both the /proc and inet_diag paths.

I prototyped that. Under the same socket noise, that failed 5 of 5773 runs. Each failure was the kernel giving the freed port to one of the noise processes right after the close, which is unlikely at any realistic socket churn rate.

I have a patch ready if that approach sounds right.

Dominant language
Go
Stars
38.8k
Forks
3.4k
Avg merge
2d 7h
Merged PRs (30d)
194

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tailscale/tailscale

All issues in tailscale/tailscale

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.