Encoding detection unpredictable

まず java -cp vnu.jar nu.validator.servlet.Main 8888 でバリデーションサーバーを起動し、リンクされた testCase.txt と ApacheBench コマンドを使ってレポートを再現します。断続的に発生する “big5” の検出と “Unmappable byte sequence” メッセージについて、同時実行時のレスポンスを比較します。繰り返し同時実行したバリデーションで、散発的な結果が発生せず、予測可能な charset 検出が行われれば完了です。

When validating multiple files in concurrency, the encoding detection is not always predictable.
The easiest way to explain is probably to reproduce my test case:

• launch the validation server (15.6.29)

java -cp vnu.jar nu.validator.servlet.Main 8888

• get the test case file (this file contains an unbreakable space and doesn't specify a content type in a meta header): https://gist.githubusercontent.com/marob/d8f9cb25a64b3905616e/raw/c521f951a70518815dd380a2d3a038f1ab2f9bbb/testCase.txt
• validate this file with concurrency and grep results on "Unmappable" keyword:

ab -T"text/html; encoding=utf-8" -p testCase.txt -c 100 -n 10000 -v 4 http://localhost:8888/?out=json | grep Unmappable

You should obtain some results of the form (if not, try to restart the server and re-validate):

{"messages":[{"type":"info","message":"The Content-Type was “text/html”. Using the HTML parser."},{"type":"error","message":"The character encoding was not declared. Proceeding using “big5”."},{"type":"error","lastLine":7,"lastColumn":47,"message":"Unmappable byte sequence: “c2”, “a0”."},{"type":"info","message":"Using the schema for HTML5 + SVG 1.1 + MathML 3.0 + RDFa Lite 1.1."}]}

As you can see, the detected encoding is sometimes "big5", but only once or twice among thousands... hence the unpredictability.
When this encoding is detected, the unbreakable space triggers the "Unmappable byte sequence" error.

I think the charset detection should be predictable.
My test demonstrate it is not.

Regards