Integration test downstream client crashes when a long send is interrupted with an early close
#17,753 opened on Aug 18, 2021
Repository metrics
- Stars
- (27,997 stars)
- PR merge metrics
- (Avg merge 8d) (378 merged PRs in 30d)
Description
Discussed this crash with @htuch before making public
Title: Integration test downstream client crashes when a long send is interrupted with an early close
Description: I encountered the following crash while fuzzing the ext_proc filter.
If the downstream client is making a large POST request in chunks and Envoy responds before the downstream client is finished sending, a seg fault occurs.
Adding the following test case to ext_proc's streaming_integration_test can recreate this scenario:
// Perform a long chunked send while the external process immediately responds to the headers
TEST_P(StreamingIntegrationTest, LongSendCrash) {
const uint32_t num_chunks = 150;
const uint32_t chunk_size = 100;
test_processor_.start(
[](grpc::ServerReaderWriter<ProcessingResponse, ProcessingRequest>* stream) {
ProcessingRequest header_req;
ASSERT_TRUE(stream->Read(&header_req));
ASSERT_TRUE(header_req.has_request_headers());
ProcessingResponse header_resp;
auto* immediate = header_resp.mutable_immediate_response();
immediate->mutable_status()->set_code(envoy::type::v3::StatusCode::Unauthorized);
immediate->set_body("{\"reason\": \"Not authorized\"}");
immediate->set_details("Failed because you are not authorized");
stream->Write(header_resp);
});
initializeConfig();
HttpIntegrationTest::initialize();
sendPostRequest(num_chunks, chunk_size, absl::nullopt);
// TODO skipping checks of response
}
Additional Information: I'm not sure if this is specific to ext_proc.
The test is fairly flaky, you can increase the num_chunks value if the issue isn't being recreated on your machine.
Further, I could only recreate the crash after executing all tests in the test suite.
Logs: I have provided truncated.log which shows the crash, but I removed the output from the other tests in the suite.