Base.isidentifier(::Symbol) should check normalization
#52.641 aperta il 27 dic 2023
Metriche repository
- Star
- (48.709 star)
- Metriche merge PR
- (Merge medio 20g 6h) (157 PR mergiate in 30 g)
Descrizione
As discussed on discourse, while the Symbol("...") foo constructor allows one to construct invalid Julia identifiers and in particular identifiers lacking normalization (an intentional decision way back in #5462), the Base.isidentifier function doesn't check for normalization of Symbol arguments.
This also has the consequence that show is incorrect for non-normalized Symbols, since it checks isidentifier to see whether they can be printed as :foo rather than as Symbol("foo"):
julia> "e\u0301" # e with acute accent, not NFC normalized
"é"
julia> Symbol("e\u0301") == :é # correct: :é gets normalized
false
julia> length(String(Symbol("e\u0301"))), length(String(:é))
(2, 1)
julia> Base.isidentifier(Symbol("e\u0301")) # incorrect: should check normalization
true
julia> Symbol("e\u0301") # incorrect display: not the same as :é
:é
My suggestion is that https://github.com/JuliaLang/julia/blob/66e941073f400b92a6d8cf4d6bce5a6cb2d00b6c/base/show.jl#L1538 should be changed to something like:
function isidentifier(s::Symbol)
str = string(s)
ct(codepoint::UInt32) = get(Unicode._julia_charmap, codepoint, codepoint)
isidentifier(str) && str == Unicode.normalize(str; stable=true, compose=true, chartransform=ct)
end
(Note that isidentifier should continue to not check normalization for ::String arguments, since in that case it has a slightly different meaning — it checks whether the string can be parsed into a valid identifier.)