🐛 --output json writes invalid JSON lines when events are logged concurrently
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 2/5
- Tiempo estimado
- 1-3 horas
- Aptitud para principiantes
- 78/100
- Tipo de issue
- Error
- Claridad
- Bien especificado
- Estado de actividad
- Activo
- Stack tecnológico
- go
- Área
- observability
Línea de trabajo
El error está en consoleWriter.Write en logger/console.go, que codifica cada evento directamente en c.out y, por tanto, realiza una escritura por cada campo JSON. Lee esa función y luego reproduce el problema con el repro independiente del issue o con la prueba del writer contador llamada TestConsoleLoggerSingleWritePerEvent. Está terminado cuando el codificador escribe en un bytes.Buffer, ese buffer se escribe una sola vez y go test ./logger/ pasa.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Describe the bug
With --output json, cloudflared sometimes writes invalid JSON log lines: one event is cut in pieces and another event, logged at the same time from another goroutine, is written in the middle of it.
The cause is in logger/console.go. consoleWriter.Write re-encodes the event with jsoniter.ConfigCompatibleWithStandardLibrary.NewEncoder(c.out) directly into os.Stderr. That config sorts map keys, and sortKeysMapEncoder calls stream.Write for each key. In json-iterator v1.1.12, Stream.Write writes straight through to the underlying writer. So one log event becomes one write(2) per field, plus a final one for }\n, with no lock between them. Events logged concurrently interleave at field boundaries.
strace of a single event:
write(2, "{\"connIndex\":7", 14) = 14
write(2, ",\"dest\":\"https://example.com/api"..., 51) = 51
write(2, ",\"error\":\"Incoming request ended"..., 60) = 60
write(2, ",\"event\":0", 10) = 10
write(2, ",\"level\":\"error\"", 16) = 16
write(2, ",\"message\":\"Request failed\"", 27) = 27
write(2, ",\"time\":\"2026-10-09T07:19:55Z\"", 35) = 35
write(2, ",\"type\":\"http\"", 14) = 14
write(2, "}\n", 2) = 2
The text output (zerolog.ConsoleWriter) is not affected: it builds the whole line in a buffer and writes it once.
To Reproduce
- Run
cloudflared tunnel --no-autoupdate --loglevel info --output json run(remotely managed tunnel, token auth). - Generate bursts of concurrent errors, e.g. many proxied requests cancelled by clients at the same time (
Incoming request ended abruptly: context canceled). - Parse stderr line by line as JSON: some lines are invalid.
If it's an issue with Cloudflare Tunnel:
4. Tunnel ID: not relevant, the bug is in the logger and reproduces without a tunnel (see below)
5. cloudflared config: no config file, only the command-line flags above
Standalone repro: https://github.com/Sirz3chs/cloudflared-json-log-interleave (runnable with ./check.sh, results also visible in its CI). It uses a verbatim copy of consoleWriter, with json-iterator v1.1.12 and zerolog v1.20.0 as in go.mod:
package main
import (
"bytes"
"fmt"
"io"
"os"
"sync"
jsoniter "github.com/json-iterator/go"
"github.com/rs/zerolog"
)
var json = jsoniter.ConfigCompatibleWithStandardLibrary
// Verbatim copy of logger/console.go consoleWriter.
type consoleWriter struct{ out io.Writer }
func (c *consoleWriter) Write(p []byte) (n int, err error) {
var evt map[string]any
d := json.NewDecoder(bytes.NewReader(p))
d.UseNumber()
if err = d.Decode(&evt); err != nil {
return n, fmt.Errorf("cannot decode event: %s", err)
}
e := json.NewEncoder(c.out)
return len(p), e.Encode(evt)
}
func main() {
log := zerolog.New(&consoleWriter{out: os.Stderr}).With().Timestamp().Logger()
var wg sync.WaitGroup
for g := 0; g < 8; g++ {
wg.Add(1)
go func(g int) {
defer wg.Done()
for i := 0; i < 5000; i++ {
log.Error().Int("connIndex", g).Int("event", 0).Str("dest", "https://example.com/api/x").
Str("error", "Incoming request ended abruptly: context canceled").Str("type", "http").Msg("Request failed")
}
}(g)
}
wg.Wait()
}
$ ./repro 2>&1 >/dev/null | jq -R 'fromjson? // "BAD"' | grep -c '^"BAD"$'
10169 # out of 40000 lines
Expected behavior
Each log event is written as one complete JSON line, whatever the concurrency.
Proposed fix
Encode into a buffer and write once. With this change the repro gives 0 invalid lines and 1 write(2) per event instead of 9. go test ./logger/ passes.
--- a/logger/console.go
+++ b/logger/console.go
@@ -32,6 +36,10 @@ func (c *consoleWriter) Write(p []byte) (n int, err error) {
return n, fmt.Errorf("cannot decode event: %s", err)
}
- e := json.NewEncoder(c.out)
- return len(p), e.Encode(evt)
+ var buf bytes.Buffer
+ if err = json.NewEncoder(&buf).Encode(evt); err != nil {
+ return n, err
+ }
+ _, err = c.out.Write(buf.Bytes())
+ return len(p), err
}
A regression test that asserts a single Write per event (it fails on current master with got 7):
type countingWriter struct {
bytes.Buffer
writes int
}
func (c *countingWriter) Write(p []byte) (int, error) {
c.writes++
return c.Buffer.Write(p)
}
func TestConsoleLoggerSingleWritePerEvent(t *testing.T) {
w := &countingWriter{}
logger := zerolog.New(&consoleWriter{out: w}).With().Timestamp().Logger()
logger.Error().Int("connIndex", 3).Str("error", "context canceled").Str("type", "http").Msg("Request failed")
if w.writes != 1 {
t.Errorf("expected 1 write for the log event, got %d: %s", w.writes, w.String())
}
}
Environment and versions
- OS: Linux, official
cloudflare/cloudflaredcontainer image on Kubernetes (EKS, containerd) - Architecture: not architecture specific (reproduced on amd64 with the repro above)
- Version: 2026.10.0, and the code is unchanged on master since TUN-9371 (2025.6)
Logs and errors
Raw container log (CRI file read by kubectl logs --timestamps), URLs redacted. Line 1 is cut before its last }, an entire second event is written inside it, and the orphan } comes next:
2026-10-09T03:03:26.001181426Z {"connIndex":3,"error":"Incoming request ended abruptly: context canceled","event":1,"ingressRule":6,"level":"error","originService":"http://origin.internal","time":"2026-10-09T03:03:26Z"{"connIndex":3,"dest":"https://example.com/api/cart/…","error":"Incoming request ended abruptly: context canceled","event":0,"ip":"198.41.192.167","level":"error","message":"Request failed","time":"2026-10-09T03:03:26Z","type":"http"}
2026-10-09T03:03:26.001206366Z }
Another shape, where the remaining fields of the first event come out on their own line:
{"connIndex":2,"error":"context canceled","event":0,"ip":"198.41.192.77","level":"error"{"connIndex":3,…full event…}
,"message":"failed to run the datagram handler","time":"…"}
Additional context
In production we see 93 invalid lines in 72 h across 2 tunnels, always during bursts of concurrent errors. Log pipelines that parse JSON end up with unparseable fragments, and the original events can't be fully reconstructed.
- Lenguaje dominante
- Go
- Estrellas
- 16k
- Forks
- 1.5k
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de cloudflare/cloudflared
-
🐛 cfRay is missing from proxied request logs (newHTTPLogger discards the zerolog context)Posiblemente ocupada @pankajc46 la tomó hace 1 día. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
cloudflare/cloudflared#1756 ·
-
🐛 After SIGTERM, `cloudflared tunnel run` uses 100% of one CPU core for the whole graceful-shutdown periodPosiblemente ocupada @cyphercodes la tomó hace 1 día. AbiertoPriority: Normal Type: Bug
Dificultad 1/5 Menos de una hora Aptitud para principiantes 88/100
cloudflare/cloudflared#1753 · 3 comentarios ·
-
tunnel route ip show: --filter-network-is-subset-of sends the superset filterPosiblemente ocupada @wangyusheng1985 la tomó hace 1 día. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 92/100
cloudflare/cloudflared#1750 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
cloudflare/cloudflared#1748 ·
-
🐛 QUIC Hijack() skips the status-written check that HTTP/2 enforcesPosiblemente ocupada @Asthenia0412 la tomó hace 11 días. AbiertoPriority: Normal Type: Bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
cloudflare/cloudflared#1747 ·
Todos los issues de cloudflare/cloudflared
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
OpenTollGate/tollgate-module-basic-go#833 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
siyuan-note/siyuan#20353 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
Los mantenedores suelen responder en 1 día
-
attributes-natural-language "en-US" is rejected by PAPPL >= 1.4.12 printers (RFC 8011 requires lowercase)Posiblemente ocupada @ChrisEdgington la tomó hoy. Abierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 84/100
OpenPrinting/ipp-usb#140 ·