StringScanner#skip returns nil inside an invalid character in non-UTF-8 strings (was an exception in 3.1.6)

Open
#223 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
54/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
c, ruby
Domain
backend

Research direction

Start by comparing StringScanner#skip behavior in the reported strscan versions for the Shift_JIS and UTF-16LE examples, then inspect the implementation and tests for skip on non-UTF-8 strings. Decide whether returning nil is intended or should raise consistently with the prior behavior; done means the behavior is specified and covered by regression tests, including the reported encodings.

Written by the indexing model from the issue text.

Description

Thanks for maintaining strscan.

On Ruby master (strscan 3.1.9.dev), StringScanner#skip returns nil when the scan position is inside an invalid or misaligned character in a non-UTF-8 string. strscan 3.1.6 (Ruby 4.0.9) raises there:

require "strscan"
[["/\xFF".dup.force_encoding("Shift_JIS"), "Shift_JIS"], ["/a".encode("UTF-16LE"), "UTF-16LE"]].each do |str, name|
  s = StringScanner.new(str)
  s.pos = 1
  p [name, (s.skip(/./) rescue $!.class)]
end
# master (strscan 3.1.9.dev): ["Shift_JIS", nil]            ["UTF-16LE", nil]
# 4.0.9  (strscan 3.1.6):     ["Shift_JIS", ArgumentError]  ["UTF-16LE", Encoding::CompatibilityError]

The same input as UTF-8 ("/\xFF", pos 1) raises ArgumentError on both.

Is returning nil here intended? We noticed it because Rails' route-path scanner advances with pos += 1 and then relies on skip(/./) either matching or raising. With nil, its loop never advances, so ActionDispatch::Journey::Parser.parse hangs on master for paths such as "/\xFF|bc" in Shift_JIS. Such paths are rare, and Rails could guard its loop. We'd just like to know whether the new result is deliberate before we propose anything there.


Found while comparing Rails' request path on Ruby master and Ruby 4.0.9 (record, part of my research project Lowrail).

Dominant language
C
Stars
88
Forks
51
Avg merge
8m
Merged PRs (30d)
1

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ruby/strscan

All issues in ruby/strscan

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.