fork-thread hangs under short-lived connection load: all threads stuck waiting for collection

Open
#1,055 5 comments 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
24/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
c
Domain
backend

Research direction

Start with the thread deactivation and collection rendezvous paths named in the report: s/arm64.ss, c/types.h, and s/7.ss. Trace how a pending collection request is signaled when threads enter and leave foreign calls, then reproduce the TCP server hang with the supplied minimal example. Done means identifying and fixing the lost wakeup so the server continues serving requests under the reported load, with relevant regression coverage.

Written by the indexing model from the issue text.

Description

A minimal TCP server using fork-thread per connection plus a single foreign
write per thread reliably hangs after ~60 connections: new connections are
accepted but never served, and every thread is stuck in S_condition_wait
waiting for collection while no collection is running. Reproduced on 10.0.0
and 10.4.1 (macOS arm64). A single-threaded continuation-based server with
the same protocol is stable over hundreds of requests.

Repro

C helper (AF_INET socket wrappers) — tcp.c:

#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
int tcp_socket(void) { return socket(AF_INET, SOCK_STREAM, 0); }
int tcp_bind(int s, int port) { struct sockaddr_in a = {0}; a.sin_family = AF_INET; a.sin_port = htons(port); return bind(s, (void*)&a, sizeof a); }
int tcp_listen(int s, int n) { return listen(s, n); }
int tcp_accept(int s) { return accept(s, 0, 0); }
int tcp_write(int s, char *buf, long start, long n) { long m = n; buf += start; while (m > 0) { long i = write(s, buf, m); if (i < 0) return i; m -= i; buf += i; } return n; }

Build: cc tcp.c -dynamiclib -o tcp.so (macOS) / cc -shared -fPIC tcp.c -o tcp.so (Linux).

Scheme server:

(load-shared-object "libSystem.B.dylib") ; Linux: "libc.so.6"
(load-shared-object "./tcp.so")
(define c-socket (foreign-procedure "tcp_socket" () int))
(define c-bind   (foreign-procedure "tcp_bind" (int int) int))
(define c-listen (foreign-procedure "tcp_listen" (int int) int))
(define c-accept (foreign-procedure "tcp_accept" (int) int))
(define c-write  (foreign-procedure "tcp_write" (int u8* size_t ssize_t) ssize_t))
(define c-close  (foreign-procedure "close" (int) int))

(define (handle fd)
  (let ([resp (string->utf8 "HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")])
    (c-write fd resp 0 (bytevector-length resp)))
  (c-close fd))

(let ([sock (c-socket)])
  (c-bind sock 8080)
  (c-listen sock 128)
  (let loop ()
    (let ([fd (c-accept sock)])
      (fork-thread (lambda () (handle fd)))
      (loop))))

Client loop:

for i in $(seq 1 200); do curl -m 3 -s -o /dev/null http://localhost:8080/; done

Hangs at request 60. Variations of the handler change the hang point:
per-byte read loop hangs at 48; a single 1024-byte read hangs at 50.

Observations

  • After the hang, connections are still accepted (lsof shows CLOSE_WAIT)
    but never served; curl times out; the process never recovers.
  • sample of the stuck process: main thread blocked in __accept;
    all other threads in S_condition_wait → _pthread_cond_wait
    (the is_collect path). No thread is running a collection.
  • Timing-sensitive: adding diagnostic prints shifts the hang point.
    Higher FFI frequency → earlier hang.
  • A workaround attempt makes it worse: calling (collect) in the accept
    loop every 5 connections hangs at request 6.

Environment

  • macOS arm64 (tarm64osx), Chez Scheme 10.0.0 (release) and 10.4.1
    (built from source, default configure) — both reproduce
  • Linux not tested

Analysis

Each foreign call is wrapped in deactivate-thread / activate-thread
(s/arm64.ss), and deactivate signals $collect-cond /
$collect-thread0-cond only when collect_request_pending is set and
the active thread count reaches zero (c/types.h). The rendezvous in
$collect-rendezvous (s/7.ss) then expects some thread to run the
collect-request handler. The hang is consistent with a lost signal:
a connection thread's deactivate check runs before
collect_request_pending is set (no signal), and the triggering thread
then waits in the rendezvous — its own is_collect deactivate
deliberately does not signal. The normal fallback (a thread returning
to Scheme pushes the rendezvous at an allocation point) can be skipped
when connection threads exit or immediately re-enter another foreign
call without allocating.

Separately, a pure (fork-thread (lambda () (getpid))) tight loop
fails with fork-thread: failed: Resource temporarily unavailable
after a few seconds — likely an unrelated pthread resource-limit issue,
noted in case it is not known.

Dominant language
Scheme
Stars
7.4k
Forks
1k
Avg merge
1d 20h
Merged PRs (30d)
19

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cisco/ChezScheme

All issues in cisco/ChezScheme

Similar issues

More Backend & API Design issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.