Allow specifying charset and/or improve charset detection
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Reproduce the supplied HTML case and trace the charset-detection path used with Heuristics.ALL; the issue names no source files or tests. Define how callers specify a known charset and how UTF-8 detection should behave, then verify both cases with regression coverage.
Written by the indexing model from the issue text.
Description
Currently the only way to specify the charset is in the document (with BOM or <meta charset=); if the charset is known but not specified in the document, there is no way to specify it.
Additionally, charset detection even with Heuristics.ALL does not always work well; in particular, it fails to recognize UTF-8 at least if the first non-ASCII byte is late in the document. The WHATWG spec recommends that systems are able to recognize UTF-8 even if they arenʼt good at other charsets (as a non-normative note)
The UTF-8 encoding has a highly detectable bit pattern. Files from the local file system that contain bytes with values greater than 0x7F which match the UTF-8 pattern are very likely to be UTF-8, while documents with byte sequences that do not match it are very likely not. When a user agent can examine the whole file, rather than just the preamble, detecting for UTF-8 specifically can be especially effective. [PPUTF8] [UTF8DET]
(This is reproduced with multiple test documents; the smallest is below but another one output the warning method that the UTF-8 character was invalid in Windows-1252, meaning that went with the default which was a particularly bad guess)
<!DOCTYPE html>
<html lang="en">
<head>
<link rel="stylesheet" href="https://fred-wang.github.io/mathml.css/mathml.css">
<title>Circle equation</title>
<!-- <meta charset="utf-8" /> -->
</head>
<body>
<p>
The equation
<math display=inline>
<mi>y</mi><mo>=</mo><mo>±</mo>
<msqrt>
<msup><mi>r</mi><mn>2</mn></msup>
<mo>-</mo>
<msup><mi>x</mi><mn>2</mn></msup>
</msqrt>
</math>
produces a circle with radius <math display=inline><mi>r</mi></math>:
</p>
<svg width="10em" height="10em" viewBox="0 0 100 100">
<desc>A circle</desc>
<circle cx="50" cy="50" r="40" fill="none" stroke="blue" stroke-width="1" />
</svg>
</body>
</html>
- Dominant language
- Java
- Stars
- 65
- Forks
- 29
- Avg merge
- 11d 21h
- Merged PRs (30d)
- 3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from validator/htmlparser
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
validator/htmlparser#98 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
validator/htmlparser#96 · 1 comment · 1 reaction ·
-
Difficulty 4/5 3-5 days Newbie friendliness 38/100
validator/htmlparser#95 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
validator/htmlparser#87 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
validator/htmlparser#60 ·
All issues in validator/htmlparser
Similar issues
-
bug untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
opensearch-project/ml-commons#5094 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
emitter:client:csharp feature
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
affects/8.10 affects/8.9 component/clients kind/bug likelihood/mid severity/mid
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Two open-case totals on one screen: the Programs tile says 15,858 and the nav badge says 15,868 Openbug frontend maui-pilot
Difficulty 2/5 1-3 hours Newbie friendliness 72/100