Encoding detection unpredictable
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- java
- Domain
- backend, testing-qa
Research direction
Start by running the validation server with java -cp vnu.jar nu.validator.servlet.Main 8888, then reproduce the report using the linked testCase.txt and the ApacheBench command. Compare concurrent responses for the intermittent “big5” detection and “Unmappable byte sequence” message. Done means repeated concurrent validation produces predictable charset detection without the sporadic result.
Written by the indexing model from the issue text.
Description
When validating multiple files in concurrency, the encoding detection is not always predictable.
The easiest way to explain is probably to reproduce my test case:
- launch the validation server (15.6.29)
java -cp vnu.jar nu.validator.servlet.Main 8888
- get the test case file (this file contains an unbreakable space and doesn't specify a content type in a meta header): https://gist.githubusercontent.com/marob/d8f9cb25a64b3905616e/raw/c521f951a70518815dd380a2d3a038f1ab2f9bbb/testCase.txt
- validate this file with concurrency and grep results on "Unmappable" keyword:
ab -T"text/html; encoding=utf-8" -p testCase.txt -c 100 -n 10000 -v 4 http://localhost:8888/?out=json | grep Unmappable
You should obtain some results of the form (if not, try to restart the server and re-validate):
{"messages":[{"type":"info","message":"The Content-Type was “text/html”. Using the HTML parser."},{"type":"error","message":"The character encoding was not declared. Proceeding using “big5”."},{"type":"error","lastLine":7,"lastColumn":47,"message":"Unmappable byte sequence: “c2”, “a0”."},{"type":"info","message":"Using the schema for HTML5 + SVG 1.1 + MathML 3.0 + RDFa Lite 1.1."}]}
As you can see, the detected encoding is sometimes "big5", but only once or twice among thousands... hence the unpredictability.
When this encoding is detected, the unbreakable space triggers the "Unmappable byte sequence" error.
I think the charset detection should be predictable.
My test demonstrate it is not.
Regards
- Dominant language
- Java
- Stars
- 65
- Forks
- 29
- Avg merge
- 11d 21h
- Merged PRs (30d)
- 3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from validator/htmlparser
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
validator/htmlparser#98 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
validator/htmlparser#96 · 1 comment · 1 reaction ·
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
validator/htmlparser#95 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
validator/htmlparser#87 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
validator/htmlparser#79 ·
All issues in validator/htmlparser
Similar issues
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 90/100
apache/cloudstack#14222 ·
-
[BUG]茶杯方块在取茶时会引发崩溃 Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
1.0.0-alpha2 Type/Improvement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
wso2/dpdp-accelerator#272 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
area/frontend
Difficulty 2/5 1-3 hours Newbie friendliness 65/100