Wrong order of modules in Http.client.stack

Responses are handled by ClientTracingFilter before PayloadSizeFilter, causing unwanted side effects on Zipkin side.

Expected behavior

ClientTracingFilter should be placed before PayloadSizeFilter in Http.client.stack

Actual behavior

When working on zipkin - finagle integration we had an issue with weird "unknown" spans occurring after some delay after finished http request.

I found that the problem is in the order of modules: ClientTracingFilter and PayloadSizeFilter in Http.client.stack.

By default PayloadSizeFilter goes before ClientTracingFilter in the chain. When a response is coming, ClientTracingFilter is working beforehand. The problem is that, it makes collected spans to be pushed the Zipkin server. When finally PayloadSizeFilter is trying to generate another span with sent/received bytes it is already too late. As other spans have be already sent, it cannot be properly matched. When the flushing mechanism pushes it to the server it will be annotated with "unknown" label and destroy Zipkin UI.

I made a temporary workaround modifying Http.client.stack configuration

Http.client.stack
 .remove(ClientTracingFilter.role).insertBefore(Stack.Role("PayloadSize"), 
    ClientTracingFilter.module[http.Request, http.Response])

but I believe it should be fixed in the source code.

I described this situation on forum as well. https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/finaglers/U9pVm_eFiJI

Libs: com.twitter:finagle-http_2.12:19.2.0 com.twitter:finagle-core_2.12:19.2.0

io.zipkin.reporter2:zipkin-sender-okhttp3:2.7.13 io.zipkin.reporter2:zipkin-reporter:2.7.13

io.zipkin.finagle2:zipkin-finagle_2.12: 2.0.11 io.zipkin.finagle2:zipkin-finagle-http_2.12:2.0.11