wasi-http p3: reused instance leaks handles until it traps when a `content-length` response body is dropped after `content-length` bytes
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 8/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- rust, wasm
- Ambito
- backend, networking
Direzione di ricerca
Start by running ./run.sh in the linked wasi-repro, then read LimitedGuestBodyConsumer and GuestBody::size_hint in src/p3/body.rs, where the host drops a content-length body early. The guest side is http_into_wasi_response and BodyWriter::send_http_body, which spawn a task that may never finish. Done means 300,000 requests succeed on one instance with until-length and 2 requests in flight.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
A wasi:http/[email protected] instance that is reused for many requests leaks handles until it traps, if all three of these are true:
- the guest's response sets
content-length; - the host reads exactly
content-lengthbytes of the response body and then drops the body without polling it to the end. hyper does this when it serves a response withcontent-length, sowasmtime serve-style embeddings will hit it; - at least 2 requests are in flight on the instance at once.
If any one of these is false, 300,000 requests run fine on one instance.
The trap is raised while the guest creates the next response, so it looks like resources from earlier responses are never released:
| wasmtime | Fails after | Error |
|---|---|---|
| 50.0.0-rc.1 | ~100,000 requests | resource table has no free keys, in the guest's Response::new |
| 49.0.3 | ~25,400 requests | cannot allocate another handle: store handle limit exceeded, in the guest's BodyWriter::new |
Test Case
Repro: https://github.com/avkonst/wasi-repro (./run.sh runs the failing case and the three passing ones). In summary:
- guest: a
wasip3::http_compatcomponent with two routes that differ only in whether the response setscontent-length. - host: one instance in one
Store, with up to N calls in flight, each started withhandler::PreparedinsideStore::run_concurrent. Each response body is either read to the end (full) or read untilcontent-lengthbytes have arrived and then dropped (until-length, like hyper).
Steps to Reproduce
git clone https://github.com/avkonst/wasi-repro && cd wasi-repro
./run.sh # rust-toolchain.toml pins rustc 1.99.0 + wasm32-wasip2
# or by hand:
W=guest/target/wasm32-wasip2/release/guest.wasm
H=host/target/release/host
# fails
$H $W /with-content-length 300000 until-length 2 direct
# pass
$H $W /with-content-length 300000 until-length 1 direct
$H $W /with-content-length 300000 full 2 direct
$H $W /without-content-length 300000 until-length 2 direct
Expected Results
All 300,000 requests succeed on one instance, as they do when any one of the three conditions is removed.
Actual Results
instance failed after 99998 requests: error while executing at wasm backtrace:
0: 0x3ee1d - <unknown>!<wasm function 9>
1: 0x8ead - guest.wasm!wasip3::service::wasi::http::types::Response::new
2: 0x53db - guest.wasm!wasip3::http_compat::conversions::http_into_wasi_response
3: 0x5924 - guest.wasm!wasip3::service::exports::wasi::http::handler::_export_handle_cabi::{{closure}}
4: 0x112ca - guest.wasm!<futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
5: 0x14a0c - guest.wasm!wit_bindgen::rt::async_support::spawn::Tasks::poll_next
6: 0x11876 - guest.wasm!wit_bindgen::rt::async_support::TaskState::callback
7: 0x116a2 - guest.wasm!wit_bindgen::rt::async_support::callback
8: 0x55bf - guest.wasm!wit_bindgen::rt::async_support::start_task
9: 0x7e65 - guest.wasm![async-lift]wasi:http/[email protected]#handle
Caused by:
resource table has no free keys
On 49.0.3
instance failed after 25394 requests: error while executing at wasm backtrace:
0: 0x10262 - guest.wasm!wasip3::http_compat::body_writer::BodyWriter::new
1: 0x538a - guest.wasm!wasip3::http_compat::conversions::http_into_wasi_response
...
Caused by:
cannot allocate another handle: store handle limit exceeded
To build the host against 49.0.3, change =50.0.0-rc.1 to =49.0.3 in host/Cargo.toml, and TaskGroupId to GuestTaskId in host/src/main.rs.
Results (/with-content-length unless noted):
| Driver | Body | In flight | 50.0.0-rc.1 | 49.0.3 |
|---|---|---|---|---|
| direct | until-length | 1 | 300,000 ok | 300,000 ok |
| direct | until-length | 2 | traps after 99,998 | traps after 25,394 |
| direct | until-length | 64 | traps after 99,968 | traps after 25,344 |
| direct | full | 64 | 300,000 ok | 300,000 ok |
direct, /without-content-length |
until-length | 64 | 300,000 ok | 300,000 ok |
ProxyHandler (reuse, never expire) |
until-length | 64 | 6,000,000 ok over 25 instances | worker trapped or panicked after 2,263,112 over 27 instances |
ProxyHandler (reuse, never expire) |
full | 64 | 3,000,000 ok over 27 instances | 3,000,000 ok over 26 instances |
The ProxyHandler only starts a concurrent request on an instance whose tasks are blocked on I/O, so each instance sees less overlap. On 49.0.3 that only delays the failure.
Versions and Environment
- Wasmtime version or commit:
wasmtime,wasmtime-wasi,wasmtime-wasi-http50.0.0-rc.1 (component-model-async); also 49.0.3 - Guest:
wasip30.9.0+wasi-0.3.0 (http-compat), wit-bindgen 0.62.0 - rustc 1.99.0, target
wasm32-wasip2 - Operating system: Linux
- Architecture: x86_64
Extra Info
- With
content-length, wasmtime-wasi-http consumes the guest body withLimitedGuestBodyConsumer(src/p3/body.rs), notUnlimitedGuestBodyConsumer.GuestBody::size_hintis then exact, which is what lets hyper stop after the declared length and drop the body. - The guest writes the body and then the trailers from a task that
http_into_wasi_responsespawns (spawn_local, thenBodyWriter::send_http_body). That task, or the stream and future ends it holds, may be what never completes once the host drops the body. If so, it may be partly or wholly awasip3/wit-bindgen issue and not only a host-side one. - Workaround attempt on 49.0.3: polling the dropped body to the end on a spawned host task made the trap less frequent but did not stop it.
Guest
The whole guest, for reference (the host is in the repo):
//! A `wasi:http/[email protected]` component with two routes that differ only in
//! whether the response carries a `content-length` header:
//!
//! - `GET /with-content-length`
//! - `GET /without-content-length`
use bytes::Bytes;
use http::header;
use http_body_util::Full;
use wasip3::http::types::{self, ErrorCode};
use wasip3::http_compat::{http_from_wasi_request, http_into_wasi_response};
wasip3::http::service::export!(Guest);
struct Guest;
impl wasip3::exports::http::handler::Guest for Guest {
async fn handle(request: types::Request) -> Result<types::Response, ErrorCode> {
let request = http_from_wasi_request(request)?;
let body = "hello\n";
let mut response = http::Response::new(Full::new(Bytes::from_static(body.as_bytes())));
if request.uri().path() == "/with-content-length" {
response
.headers_mut()
.insert(header::CONTENT_LENGTH, body.len().into());
}
http_into_wasi_response(response)
}
}
- Lingua principale
- Rust
- Stelle
- 18.7k
- Fork
- 1.9k
- Merge medio
- 21h 36m
- PR unite (30g)
- 316
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di bytecodealliance/wasmtime
-
bug wasm-proposal:exceptions wasmtime:debugging
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
bytecodealliance/wasmtime#14102 ·
I maintainer di solito rispondono entro 1 giorno
-
wasm-proposal:gc
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
bytecodealliance/wasmtime#13808 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
cranelift cranelift:area:clif cranelift:goal:optimize-speed enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
bytecodealliance/wasmtime#1598 · 9 commenti · 1 reazione ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 42/100
bytecodealliance/wasmtime#14790 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
cranelift cranelift:goal:optimize-speed
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
bytecodealliance/wasmtime#14699 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di bytecodealliance/wasmtime
Issue simili
-
build: Update rust versionForse già presa Una pull request collegata a questa issue è aperta o già unita. Apertadependencies
Difficoltà 2/5 1-3 ore Idoneità per principianti 67/100
ibm-granite-community/granite-cli#229 ·
I maintainer di solito rispondono entro 1 giorno
-
setTargetAtTime booked ahead of time starts its curve early (values blow up before start_time)Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
orottier/web-audio-api-rs#658 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
aosama/astronomical#1318 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
shep-pm/shep-kelpie#980 ·
I maintainer di solito rispondono entro 1 giorno