JuliaLang/julia

Base.isidentifier(::Symbol) should check normalization

Open

#52,641 opened on Dec 27, 2023

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Julia (5,773 forks)batch import
bugdisplay and printinggood first issueparser

Repository metrics

Stars
 (48,709 stars)
PR merge metrics
 (Avg merge 20d 6h) (157 merged PRs in 30d)

Description

As discussed on discourse, while the Symbol("...") foo constructor allows one to construct invalid Julia identifiers and in particular identifiers lacking normalization (an intentional decision way back in #5462), the Base.isidentifier function doesn't check for normalization of Symbol arguments.

This also has the consequence that show is incorrect for non-normalized Symbols, since it checks isidentifier to see whether they can be printed as :foo rather than as Symbol("foo"):

julia> "e\u0301" # e with acute accent, not NFC normalized
"é"

julia> Symbol("e\u0301") == :é   # correct: :é gets normalized
false

julia> length(String(Symbol("e\u0301"))), length(String(:é))
(2, 1)

julia> Base.isidentifier(Symbol("e\u0301"))   # incorrect: should check normalization
true

julia> Symbol("e\u0301") # incorrect display: not the same as :é
:é

My suggestion is that https://github.com/JuliaLang/julia/blob/66e941073f400b92a6d8cf4d6bce5a6cb2d00b6c/base/show.jl#L1538 should be changed to something like:

function isidentifier(s::Symbol)
    str = string(s)
    ct(codepoint::UInt32) = get(Unicode._julia_charmap, codepoint, codepoint)
    isidentifier(str) && str == Unicode.normalize(str; stable=true, compose=true, chartransform=ct)
end

(Note that isidentifier should continue to not check normalization for ::String arguments, since in that case it has a slightly different meaning — it checks whether the string can be parsed into a valid identifier.)

Contributor guide