dotnet/roslyn

Compiler does not correctly interpret surrogate pairs when used in an identifier

Open

#9,731 opened on Mar 14, 2016

View on GitHub
 (13 comments) (7 reactions) (1 assignee)C# (20,414 stars) (4,257 forks)batch import
Area-CompilersBugLanguage-C#Tenet-Localizationhelp wanted

Description

The C# specification states that an identifier can start with or contain anything matching letter-character, which is defined as:

letter-character::
A Unicode character of classes Lu, Ll, Lt, Lm, Lo, or Nl
A unicode-escape-sequence representing a character of classes Lu, Ll, Lt, Lm, Lo, or Nl

However, the compiler does not appear to correctly interpret some characters which match the above categories if they are part of a surrogate pair.

For example the sumerian character 𒅴 is categorized as 'OtherLetter' (matching 'Lo' above) when processed through char.GetUnicodeCategory("𒅴", 0).

However, the compiler is interpreting this character as two separate characters (and reporting CS1056 for both). It is likely checking each character individually, rather than checking if the first character is part of a surrogate pair and interpreting the character appropriately if it is.

Contributor guide

Tech stack
csharp
Domain
tooling
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
C# language specificationUnicode surrogate pairscompiler internals
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
Search the Roslyn lexer source code (e.g., Lexer.cs) for identifier parsing logic. The bug is that the compiler checks each char individually instead of handling surrogate pairs. Look for calls to CharUnicodeInfo or similar and modify to recognize surrogate pairs. See comments in the issue for discussion on potential fixes.