Title: Panic in trace! call due to unwrap() on non-UTF-8 bytes in protocols/http/v1/client.rs:818
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- backend, networking
Research direction
Start in pingora-core/src/protocols/http/v1/client.rs at the trace! call around line 818, focusing on how raw response headers are converted before logging. Verify behavior with non-UTF-8 upstream response headers under INFO logging; done means the request no longer panics and the raw header remains safely loggable.
Written by the indexing model from the issue text.
Description
Hello,
We experienced an outage across all our production services utilizing Pingora, caused by yet another unhandled .unwrap() within the Pingora codebase. Unfortunately, the library relies heavily on .unwrap(), which poses significant reliability risks and makes it ill-suited for critical production environments. Following our investigation, here is the root cause :
Describe the bug
Under heavy load, when an upstream backend returns response headers or status containing non-UTF-8 bytes (e.g. ISO-8859-1/Latin-1 characters or binary payload residues), the Pingora HTTP proxy service panics and crashes with SIGABRT (status=6/ABRT).
Even when the active logging level is set to INFO, the panic still triggers because Rust evaluates macro arguments eagerly before evaluating log level filters.
Log Output
2026-09-28T08:03:24.279410Z ERROR panic: called Result::unwrap() on an Err value: Utf8Error { valid_up_to: 218, error_len: Some(1) }
2026-09-28T08:03:24.279557Z ERROR location: pingora-core/src/protocols/http/v1/client.rs:818
thread 'Pingora HTTP Proxy Service' panicked at pingora-core/src/protocols/http/v1/client.rs:818:56:
called Result::unwrap() on an Err value: Utf8Error { valid_up_to: 218, error_len: Some(1) }
xxx.service: Main process exited, code=killed, status=6/ABRT
xxx.service: Failed with result 'signal'.
Root Cause
At line 818 of pingora-core/src/protocols/http/v1/client.rs, a trace! statement converts raw response bytes using std::str::from_utf8(...).unwrap().
pub async fn read_response_task(&mut self) -> Result<HttpTask> {
if self.should_read_resp_header() {
let resp_header = self.read_resp_header_parts().await?;
let end_of_body = self.is_body_done();
debug!("Response header: {resp_header:?}");
trace!(
"Raw Response header: {:?}",
str::from_utf8(self.get_headers_raw()).unwrap()
);
Ok(HttpTask::Header(resp_header, end_of_body))
} else if self.is_body_done() {
// no body
debug!("Response is done");
Ok(HttpTask::Done)
} else {
/* need to read body */
let body = self.read_body_bytes().await?;
let end_of_body = self.is_body_done();
debug!(
"Response body: {} bytes, end: {end_of_body}",
body.as_ref().map_or(0, |b| b.len())
);
trace!("Response body: {body:?}, upgraded: {}", self.upgraded);
if self.upgraded {
Ok(HttpTask::UpgradedBody(body, end_of_body))
} else {
Ok(HttpTask::Body(body, end_of_body))
}
}
// TODO: support h1 trailer
}
Unsafe unwrap on external network input: Upstream server responses cannot be guaranteed to be 100% valid UTF-8 (especially under edge cases, error conditions, or legacy ISO-8859-1 encodings).
Eager argument evaluation: Because from_utf8(...).unwrap() is passed as an argument directly into trace!, the expression evaluates before the logger checks whether TRACE level is enabled. As a result, production services running at INFO level still suffer the panic and abort.
Suggested Fix
Use String::from_utf8_lossy instead of .unwrap() in tracing/logging calls, or wrap the log statement behind an explicit log_enabled! guard:
trace!("...", String::from_utf8_lossy(&bytes));
Alternatively:
if log::log_enabled!(log::Level::Trace) {
if let Ok(s) = std::str::from_utf8(&bytes) {
trace!("...", s);
}
}
- Dominant language
- Rust
- Stars
- 27.6k
- Forks
- 1.8k
- Avg merge
- 1h 10m
- Merged PRs (30d)
- 2
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cloudflare/pingora
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 92/100
cloudflare/pingora#1015 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
cloudflare/pingora#1013 · 2 comments ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
cloudflare/pingora#1006 ·
Maintainers usually reply within 1 day
-
Test
Difficulty 1/5 Under an hour Newbie friendliness 92/100
cloudflare/pingora#984 ·
Maintainers usually reply within 1 day
-
dependencies
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
cloudflare/pingora#875 · 1 comment · 5 reactions ·
Maintainers usually reply within 1 day
All issues in cloudflare/pingora
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
trezor/trezor-firmware#7997 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
smol-machines/smolvm#1489 · 1 comment · 1 reaction ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day