JuliaLang/julia

Base.isidentifier(::Symbol) should check normalization

Open

#52.641 aperta il 27 dic 2023

Vedi su GitHub
 (5 commenti) (0 reazioni) (0 assegnatari)Julia (5773 fork)batch import
bugdisplay and printinggood first issueparser

Metriche repository

Star
 (48.709 star)
Metriche merge PR
 (Merge medio 20g 6h) (157 PR mergiate in 30 g)

Descrizione

As discussed on discourse, while the Symbol("...") foo constructor allows one to construct invalid Julia identifiers and in particular identifiers lacking normalization (an intentional decision way back in #5462), the Base.isidentifier function doesn't check for normalization of Symbol arguments.

This also has the consequence that show is incorrect for non-normalized Symbols, since it checks isidentifier to see whether they can be printed as :foo rather than as Symbol("foo"):

julia> "e\u0301" # e with acute accent, not NFC normalized
"é"

julia> Symbol("e\u0301") == :é   # correct: :é gets normalized
false

julia> length(String(Symbol("e\u0301"))), length(String(:é))
(2, 1)

julia> Base.isidentifier(Symbol("e\u0301"))   # incorrect: should check normalization
true

julia> Symbol("e\u0301") # incorrect display: not the same as :é
:é

My suggestion is that https://github.com/JuliaLang/julia/blob/66e941073f400b92a6d8cf4d6bce5a6cb2d00b6c/base/show.jl#L1538 should be changed to something like:

function isidentifier(s::Symbol)
    str = string(s)
    ct(codepoint::UInt32) = get(Unicode._julia_charmap, codepoint, codepoint)
    isidentifier(str) && str == Unicode.normalize(str; stable=true, compose=true, chartransform=ct)
end

(Note that isidentifier should continue to not check normalization for ::String arguments, since in that case it has a slightly different meaning — it checks whether the string can be parsed into a valid identifier.)

Guida contributor