JuliaLang/julia

Base.isidentifier(::Symbol) should check normalization

Open

#52,641 opened on 2023年12月27日

GitHub で見る
 (5 comments) (0 reactions) (0 assignees)Julia (5,773 forks)batch import
bugdisplay and printinggood first issueparser

Repository metrics

Stars
 (48,709 stars)
PR merge metrics
 (平均マージ 20d 6h) (30d で 157 merged PRs)

説明

As discussed on discourse, while the Symbol("...") foo constructor allows one to construct invalid Julia identifiers and in particular identifiers lacking normalization (an intentional decision way back in #5462), the Base.isidentifier function doesn't check for normalization of Symbol arguments.

This also has the consequence that show is incorrect for non-normalized Symbols, since it checks isidentifier to see whether they can be printed as :foo rather than as Symbol("foo"):

julia> "e\u0301" # e with acute accent, not NFC normalized
"é"

julia> Symbol("e\u0301") == :é   # correct: :é gets normalized
false

julia> length(String(Symbol("e\u0301"))), length(String(:é))
(2, 1)

julia> Base.isidentifier(Symbol("e\u0301"))   # incorrect: should check normalization
true

julia> Symbol("e\u0301") # incorrect display: not the same as :é
:é

My suggestion is that https://github.com/JuliaLang/julia/blob/66e941073f400b92a6d8cf4d6bce5a6cb2d00b6c/base/show.jl#L1538 should be changed to something like:

function isidentifier(s::Symbol)
    str = string(s)
    ct(codepoint::UInt32) = get(Unicode._julia_charmap, codepoint, codepoint)
    isidentifier(str) && str == Unicode.normalize(str; stable=true, compose=true, chartransform=ct)
end

(Note that isidentifier should continue to not check normalization for ::String arguments, since in that case it has a slightly different meaning — it checks whether the string can be parsed into a valid identifier.)

コントリビューターガイド