StringScanner#skip returns nil inside an invalid character in non-UTF-8 strings (was an exception in 3.1.6)
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 54/100
Research direction
Start by comparing StringScanner#skip behavior in the reported strscan versions for the Shift_JIS and UTF-16LE examples, then inspect the implementation and tests for skip on non-UTF-8 strings. Decide whether returning nil is intended or should raise consistently with the prior behavior; done means the behavior is specified and covered by regression tests, including the reported encodings.
Written by the indexing model from the issue text.
Description
Thanks for maintaining strscan.
On Ruby master (strscan 3.1.9.dev), StringScanner#skip returns nil when the scan position is inside an invalid or misaligned character in a non-UTF-8 string. strscan 3.1.6 (Ruby 4.0.9) raises there:
require "strscan"
[["/\xFF".dup.force_encoding("Shift_JIS"), "Shift_JIS"], ["/a".encode("UTF-16LE"), "UTF-16LE"]].each do |str, name|
s = StringScanner.new(str)
s.pos = 1
p [name, (s.skip(/./) rescue $!.class)]
end
# master (strscan 3.1.9.dev): ["Shift_JIS", nil] ["UTF-16LE", nil]
# 4.0.9 (strscan 3.1.6): ["Shift_JIS", ArgumentError] ["UTF-16LE", Encoding::CompatibilityError]
The same input as UTF-8 ("/\xFF", pos 1) raises ArgumentError on both.
Is returning nil here intended? We noticed it because Rails' route-path scanner advances with pos += 1 and then relies on skip(/./) either matching or raising. With nil, its loop never advances, so ActionDispatch::Journey::Parser.parse hangs on master for paths such as "/\xFF|bc" in Shift_JIS. Such paths are rare, and Rails could guard its loop. We'd just like to know whether the new result is deliberate before we propose anything there.
Found while comparing Rails' request path on Ruby master and Ruby 4.0.9 (record, part of my research project Lowrail).
- Dominant language
- C
- Stars
- 88
- Forks
- 51
- Avg merge
- 8m
- Merged PRs (30d)
- 1
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ruby/strscan
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Similar issues
-
HTTP Request Smuggling: non-final chunked Transfer-Encoding not rejected (CWE-444)Possibly taken @code1a2024 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
jaredgei/wow-forever-android#97 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
openziti/ziti-sdk-c#1254 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
flatpak/xdg-desktop-portal#2215 ·
Maintainers usually reply within 2 days
-
http_fetch tests: the two no-child rows fail, not skip, where a task's children file does not existPossibly taken A pull request linked to this issue is open or already merged. Openarea:http kind:gate-defect
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
InauguralSystems/EigenScript#2397 ·
Maintainers usually reply within 1 day