twitter/finagle

Wrong order of modules in Http.client.stack

Open

#783 opened on Jul 31, 2019

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Scala (8,864 stars) (1,435 forks)batch import
help wantedwaiting

Description

Responses are handled by ClientTracingFilter before PayloadSizeFilter, causing unwanted side effects on Zipkin side.

Expected behavior

ClientTracingFilter should be placed before PayloadSizeFilter in Http.client.stack

Actual behavior

When working on zipkin - finagle integration we had an issue with weird "unknown" spans occurring after some delay after finished http request.

I found that the problem is in the order of modules: ClientTracingFilter and PayloadSizeFilter in Http.client.stack.

By default PayloadSizeFilter goes before ClientTracingFilter in the chain. When a response is coming, ClientTracingFilter is working beforehand. The problem is that, it makes collected spans to be pushed the Zipkin server. When finally PayloadSizeFilter is trying to generate another span with sent/received bytes it is already too late. As other spans have be already sent, it cannot be properly matched. When the flushing mechanism pushes it to the server it will be annotated with "unknown" label and destroy Zipkin UI.

I made a temporary workaround modifying Http.client.stack configuration

Http.client.stack
 .remove(ClientTracingFilter.role).insertBefore(Stack.Role("PayloadSize"), 
    ClientTracingFilter.module[http.Request, http.Response])

but I believe it should be fixed in the source code.

I described this situation on forum as well. https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/finaglers/U9pVm_eFiJI

Libs: com.twitter:finagle-http_2.12:19.2.0 com.twitter:finagle-core_2.12:19.2.0

io.zipkin.reporter2:zipkin-sender-okhttp3:2.7.13 io.zipkin.reporter2:zipkin-reporter:2.7.13

io.zipkin.finagle2:zipkin-finagle_2.12: 2.0.11 io.zipkin.finagle2:zipkin-finagle-http_2.12:2.0.11

Contributor guide

Tech stack
scala
Domain
backendapiperformance
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
under 1 hour
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Scala basicsFinagle HTTP client
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
Fix the order of ClientTracingFilter and PayloadSizeFilter in Http.client.stack. The stack is defined in the finagle http module, likely in a file like Http.scala or ClientBuilder.scala. Swap the modules so that ClientTracingFilter is placed before PayloadSizeFilter. Verify that the fix resolves the Zipkin span issue described in the forum post.