Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

uninitialized token indexes in some fragments

Open
#91 3 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
sql
Domain
compilers

Research direction

Start in SqlScriptDom/Parser/TSql/TSql80.g around lines 6276-6294 and reproduce the two SQL examples from the issue. Inspect how token indexes are assigned to OptimizerHint and StatementList; the work is done when those fragments report the correct first and last token indexes instead of -1.

Written by the indexing model from the issue text.

Description

Using Microsoft.SqlServer.TransactSql.ScriptDom version 161.9135.0.

For example:

select 1
option(recompile)

gets parsed as:

TOKENS:
0: Select "select"
1: WhiteSpace " "
2: Integer "1"
3: WhiteSpace "\n"
4: Option "option"
5: LeftParenthesis "("
6: Identifier "recompile"
7: RightParenthesis ")"
8: EndOfFile 

FRAGMENTS:
TSqlScript {
  FirstTokenIndex = 0,
  LastTokenIndex = 8,
  Batches = [
    TSqlBatch {
      FirstTokenIndex = 0,
      LastTokenIndex = 7,
      Statements = [
        SelectStatement {
          FirstTokenIndex = 0,
          LastTokenIndex = 7,
          QueryExpression = QuerySpecification {
            FirstTokenIndex = 0,
            LastTokenIndex = 2,
            UniqueRowFilter = NotSpecified,
            SelectElements = [
              SelectScalarExpression {
                FirstTokenIndex = 2,
                LastTokenIndex = 2,
                Expression = IntegerLiteral {
                  FirstTokenIndex = 2,
                  LastTokenIndex = 2,
                  LiteralType = Integer,
                  Value = 1,
                },
              },
            ],
          },
          OptimizerHints = [
            OptimizerHint {
              FirstTokenIndex = -1,
              LastTokenIndex = -1,
              HintKind = Recompile,
            },
          ],
        },
      ],
    },
  ],
}

where OptimizerHint ends up with -1 in FirstTokenIndex and LastTokenIndex.

I think the issue here is that here

https://github.com/microsoft/SqlScriptDOM/blob/d84cc30809b29cc5497809c2b0432bf23c412c69/SqlScriptDom/Parser/TSql/TSql80.g#L6276-L6294

tokens get assigned to vParent and should be assigned to vHint.

Another example:

if 1=1
begin
    select 1
end

gets parsed as:

TOKENS:
0: If "if"
1: WhiteSpace " "
2: Integer "1"
3: EqualsSign "="
4: Integer "1"
5: WhiteSpace "\n"
6: Begin "begin"
7: WhiteSpace "\n"
8: WhiteSpace "    "
9: Select "select"
10: WhiteSpace " "
11: Integer "1"
12: WhiteSpace "\n"
13: End "end"
14: EndOfFile 

FRAGMENTS:
TSqlScript {
  FirstTokenIndex = 0,
  LastTokenIndex = 14,
  Batches = [
    TSqlBatch {
      FirstTokenIndex = 0,
      LastTokenIndex = 13,
      Statements = [
        IfStatement {
          FirstTokenIndex = 0,
          LastTokenIndex = 13,
          Predicate = BooleanComparisonExpression {
            FirstTokenIndex = 2,
            LastTokenIndex = 4,
            ComparisonType = Equals,
            FirstExpression = IntegerLiteral {
              FirstTokenIndex = 2,
              LastTokenIndex = 2,
              LiteralType = Integer,
              Value = 1,
            },
            SecondExpression = IntegerLiteral {
              FirstTokenIndex = 4,
              LastTokenIndex = 4,
              LiteralType = Integer,
              Value = 1,
            },
          },
          ThenStatement = BeginEndBlockStatement {
            FirstTokenIndex = 6,
            LastTokenIndex = 13,
            StatementList = StatementList {
              FirstTokenIndex = -1,
              LastTokenIndex = -1,
              Statements = [
                SelectStatement {
                  FirstTokenIndex = 9,
                  LastTokenIndex = 11,
                  QueryExpression = QuerySpecification {
                    FirstTokenIndex = 9,
                    LastTokenIndex = 11,
                    UniqueRowFilter = NotSpecified,
                    SelectElements = [
                      SelectScalarExpression {
                        FirstTokenIndex = 11,
                        LastTokenIndex = 11,
                        Expression = IntegerLiteral {
                          FirstTokenIndex = 11,
                          LastTokenIndex = 11,
                          LiteralType = Integer,
                          Value = 1,
                        },
                      },
                    ],
                  },
                },
              ],
            },
          },
        },
      ],
    },
  ],
}

here StatementList is missing token indexes.

Dominant language
GAP
Stars
277
Forks
43
Avg merge
6d 17h
Merged PRs (30d)
3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/SqlScriptDOM

All issues in microsoft/SqlScriptDOM

Similar issues

More Compilers issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.