Discarding PM_TOKEN_HEREDOC_START in hash pattern key recovery strands PM_LEX_HEREDOC on the lex mode stack

Start in src/prism.c at parse_pattern_primitive, specifically the PM_TOKEN_BRACE_LEFT_HASH recovery branch, and trace how PM_TOKEN_HEREDOC_START is handled compared with parse_heredoc. Reproduce with ./bin/parse -e $'x in {<<0\n#{class' and inspect the parser tests for pattern recovery and heredocs. Done means this input no longer strands a lex mode or produces a node with an underflowed location length, while retaining appropriate parse errors.

Fuzzer find on bdbd482dc.

Simulation at https://stevenjohnstone.github.io/prism-loupe/?version=v2.1.0&code=eCBpbiB7PDwwCiN7Y2xhc3M%3D.

In parse_pattern_primitive (case PM_TOKEN_BRACE_LEFT_HASH:), when the parser encounters an invalid token where a hash pattern key is expected, the default: error-recovery branch calls parser_lex(parser) to discard parser->current.

When parser->current is PM_TOKEN_HEREDOC_START (e.g., x in {<<0), lex_heredoc_start has already pushed PM_LEX_HEREDOC onto parser->lex_modes. Discarding PM_TOKEN_HEREDOC_START with a raw parser_lex(parser) instead of leaving it to be parsed via parse_heredoc leaves PM_LEX_HEREDOC stranded on parser->lex_modes while the parser unwinds back to outer statement parsing. When the stranded PM_LEX_HEREDOC later reaches EOF inside an outer node on line 2, it rewinds parser->next_start back to line 1 mid-node, causing start > end and underflowing uint32_t location.length.

Reproduction

✗ ./bin/parse -e $'x in {<<0\n#{class'
Errors:
[#<Prism::ParseError @type=:pattern_hash_key @message="unexpected heredoc beginning; expected a key in the hash pattern" @location=#<Prism::Location @start_offset=6 @length=3 start_line=1> @level=:syntax>,
 #<Prism::ParseError @type=:pattern_hash_key_label @message="expected a label as the key in the hash pattern" @location=#<Prism::Location @start_offset=6 @length=3 start_line=1> @level=:syntax>,
 #<Prism::ParseError @type=:pattern_term_brace @message="expected a `}` to close the pattern expression" @location=#<Prism::Location @start_offset=5 @length=1 start_line=1> @level=:syntax>,
 #<Prism::ParseError @type=:expect_eol_after_statement @message="unexpected '\#{', expecting end-of-input" @location=#<Prism::Location @start_offset=10 @length=2 start_line=2> @level=:syntax>,
 #<Prism::ParseError @type=:unexpected_token_ignore @message="unexpected '\#{', ignoring it" @location=#<Prism::Location @start_offset=10 @length=2 start_line=2> @level=:syntax>,
 #<Prism::ParseError @type=:heredoc_term @message="unterminated heredoc; can't find string \"0\" anywhere before EOF" @location=#<Prism::Location @start_offset=8 @length=1 start_line=1> @level=:syntax>,
 #<Prism::ParseError @type=:class_name @message="unexpected constant path after `class`; class/module name must be CONSTANT" @location=#<Prism::Location @start_offset=12 @length=5 start_line=2> @level=:syntax>,
 #<Prism::ParseError @type=:class_name @message="unexpected constant path after `class`; class/module name must be CONSTANT" @location=#<Prism::Location @start_offset=12 @length=5 start_line=2> @level=:syntax>,
 #<Prism::ParseError @type=:unexpected_token_ignore @message="unexpected heredoc ending, ignoring it" @location=#<Prism::Location @start_offset=17 @length=0 start_line=2> @level=:syntax>,
 #<Prism::ParseError @type=:unexpected_token_close_context @message="unexpected end-of-input, assuming it is closing the parent top level context" @location=#<Prism::Location @start_offset=17 @length=0 start_line=2> @level=:syntax>,
 #<Prism::ParseError @type=:class_term @message="expected an `end` to close the `class` statement" @location=#<Prism::Location @start_offset=12 @length=5 start_line=2> @level=:syntax>,
 #<Prism::ParseError @type=:class_name @message="unexpected constant path after `class`; class/module name must be CONSTANT" @location=#<Prism::Location @start_offset=12 @length=5 start_line=2> @level=:syntax>]

AST:
@ ProgramNode (location: (1,0)-(2,0))
├── flags: ∅
├── locals: []
└── statements:
    @ StatementsNode (location: (1,0)-(2,0))
    ├── flags: ∅
    └── body: (length: 3)
        ├── @ MatchPredicateNode (location: (1,0)-(1,9))
        │   ├── flags: newline
        │   ├── value:
        │   │   @ CallNode (location: (1,0)-(1,1))
        │   │   ├── flags: variable_call, ignore_visibility
        │   │   ├── receiver: ∅
        │   │   ├── call_operator_loc: ∅
        │   │   ├── name: :x
        │   │   ├── message_loc: (1,0)-(1,1) = "x"
        │   │   ├── opening_loc: ∅
        │   │   ├── arguments: ∅
        │   │   ├── closing_loc: ∅
        │   │   ├── equal_loc: ∅
        │   │   └── block: ∅
        │   ├── pattern:
        │   │   @ HashPatternNode (location: (1,5)-(1,9))
        │   │   ├── flags: ∅
        │   │   ├── constant: ∅
        │   │   ├── elements: (length: 1)
        │   │   │   └── @ AssocNode (location: (1,6)-(1,9))
        │   │   │       ├── flags: ∅
        │   │   │       ├── key:
        │   │   │       │   @ ErrorRecoveryNode (location: (1,6)-(1,9))
        │   │   │       │   ├── flags: ∅
        │   │   │       │   └── unexpected: ∅
        │   │   │       ├── value:
        │   │   │       │   @ ErrorRecoveryNode (location: (1,6)-(1,9))
        │   │   │       │   ├── flags: ∅
        │   │   │       │   └── unexpected: ∅
        │   │   │       └── operator_loc: ∅
        │   │   ├── rest: ∅
        │   │   ├── opening_loc: (1,5)-(1,6) = "{"
        │   │   └── closing_loc: ∅
        │   └── keyword_loc: (1,2)-(1,4) = "in"
        ├── @ ErrorRecoveryNode (location: (1,9)-(1,9))
        │   ├── flags: newline
        │   └── unexpected: ∅
        └── @ ClassNode (location: (2,2)-(2,4294967296))
            ├── flags: newline
            ├── locals: []
            ├── class_keyword_loc: (2,2)-(2,7) = "class"
            ├── constant_path:
            │   @ ErrorRecoveryNode (location: (2,2)-(2,7))
            │   ├── flags: ∅
            │   └── unexpected: ∅
            ├── inheritance_operator_loc: ∅
            ├── superclass: ∅
            ├── body:
            │   @ StatementsNode (location: (1,9)-(2,4))
            │   ├── flags: ∅
            │   └── body: (length: 2)
            │       ├── @ ErrorRecoveryNode (location: (2,2)-(2,7))
            │       │   ├── flags: newline
            │       │   └── unexpected: ∅
            │       └── @ ErrorRecoveryNode (location: (1,9)-(2,0))
            │           ├── flags: newline
            │           └── unexpected: ∅
            ├── end_keyword_loc: (2,0)-(2,0) = ""
            └── name: :class

Byte offsets in "x in {<<0\n\#{class" (17 bytes):

• x in {: 0..6 ((1,0)-(1,6))
• <<0: 6..9 ((1,6)-(1,9))
• \n: 9..10 ((1,9)-(2,0))
• #{: 10..12 ((2,0)-(2,2))
• class: 12..17 ((2,2)-(2,7))

In Prism.parse("x in {<<0\n\#{class"), the ClassNode on line 2 starts at byte 12 (class at (2,2)) and ends at byte 10 ((2,0), at the end of the rewound \n 9..10), wrapping uint32_t location.length (U32(10 - 12)) to 4294967294 (0xFFFFFFFE, printed as (2,2)-(2,4294967296) by ./bin/parse).

Root Cause

1. On line 1 (x in {<<0\n), in begins a pattern matching predicate (parse_pattern), and { begins a hash pattern (case PM_TOKEN_BRACE_LEFT_HASH: in parse_pattern_primitive).
2. Inside {, the next token is <<0 (PM_TOKEN_HEREDOC_START, offsets 6..9). When parser_lex lexes <<0, it pushes PM_LEX_HEREDOC onto parser->lex_modes (saving lex_mode.as.heredoc.next_start = 9, pointing at \n on line 1) and sets parser->next_start = 10 (#{class on line 2).
3. A heredoc is not a valid hash pattern key. In parse_pattern_primitive (src/prism.c), the default: branch currently does:

default: {
    PM_PARSER_ERR_TOKEN_FORMAT(parser, &parser->current, PM_ERR_PATTERN_HASH_KEY, pm_token_str(parser->current.type));
    parser_lex(parser);

    first_node = UP(pm_error_recovery_node_create(parser, PM_TOKEN_START(parser, &parser->previous), PM_TOKEN_LENGTH(&parser->previous)));
    break;
}

4. Calling parser_lex(parser) discards PM_TOKEN_HEREDOC_START without calling parse_heredoc, and immediately lexes the first token of the heredoc body on line 2 (#{, PM_TOKEN_EMBEXPR_BEGIN[10..12], which pushes PM_LEX_EMBEXPR on top of the stranded PM_LEX_HEREDOC without parse_string_part ever pushing PM_CONTEXT_EMBEXPR onto parser->contexts).
5. parse_pattern_hash records PM_ERR_PATTERN_HASH_KEY_LABEL, expect1_opening records a missing } (PM_ERR_PATTERN_TERM_BRACE), and MatchPredicateNode returns to the top-level parse_statements loop with parser->current = PM_TOKEN_EMBEXPR_BEGIN[10..12].
6. The top-level parse_statements loop skips the unexpected #{ token via parser_lex(parser) (advancing to class at 12..17) and enters parse_class while PM_LEX_EMBEXPR and the stranded PM_LEX_HEREDOC are still on parser->lex_modes.
7. Consuming class (12..17) hits EOF at byte 17, popping PM_LEX_EMBEXPR and running the stranded PM_LEX_HEREDOC at EOF, which emits PM_TOKEN_HEREDOC_END[17..17] and sets parser->next_start = 9 (\n on line 1) and parser->heredoc_end = 17.
8. Inside parse_class, parse_statements(parser, PM_CONTEXT_CLASS, ...) skips PM_TOKEN_HEREDOC_END, rewinds to lex and accept PM_TOKEN_NEWLINE[9..10] (flushing next_start = 17 back to EOF[17..17]), sets parser->recovering = true on EOF, and synthesizes a missing end token at parser->previous.end = 10 ((2,0)), giving ClassNode start = 12 and end = 10 (U32(10 - 12) = 4294967294).