Politeness keys do not collapse IDN host spellings (unicode vs punycode)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 74/100
Research direction
Start at URLUtil.getCanonicalHost and compare the canonical keys produced for the Unicode and punycode host examples in the issue. Check how okhttp handles IDN.toASCII and preserve the raw host for invalid labels. Done means both valid spellings produce one key while invalid hosts still retain their raw value; add or update focused tests for these cases.
Written by the indexing model from the issue text.
Description
Follow-up from #2122 (review by rzo1).
okhttp punycodes the host when it parses a URL: IDN.toASCII("ExAmpleé.org") gives xn--exampl-gva.org. URLUtil.getCanonicalHost (introduced in #2122 for politeness queues and the robots.txt cache) lowercases and percent-decodes the host but does not punycode it, so the Unicode and punycode spellings of one server still get separate politeness queues and separate robots.txt cache entries:
http://ExAmpleé.org/→ hostExAmpleé.org→ keyexamplé.orghttp://xn--exampl-gva.org/→ keyxn--exampl-gva.org
Suggested fix: apply IDN.toASCII in getCanonicalHost (falling back to the raw host when the label is not a valid IDN), so both spellings collapse the way okhttp collapses them at connect time.
- Dominant language
- Java
- Stars
- 996
- Forks
- 292
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 72
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/stormcrawler
-
enhancement
Difficulty 3/5 1-2 days Newbie friendliness 72/100
apache/stormcrawler#2166 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 70/100
apache/stormcrawler#2131 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 58/100
apache/stormcrawler#2103 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 58/100
apache/stormcrawler#2098 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 68/100
apache/stormcrawler#2097 ·
All issues in apache/stormcrawler
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
infinispan/infinispan#18150 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
opensearch-project/k-NN#3597 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100