JuliaLang/julia

Base.isidentifier(::Symbol) should check normalization

Open

#52,641 创建于 2023年12月27日

在 GitHub 查看
 (5 评论) (0 反应) (0 负责人)Julia (5,773 fork)batch import
bugdisplay and printinggood first issueparser

仓库指标

Star
 (48,709 star)
PR 合并指标
 (平均合并 20天 6小时) (30 天内合并 157 个 PR)

描述

As discussed on discourse, while the Symbol("...") foo constructor allows one to construct invalid Julia identifiers and in particular identifiers lacking normalization (an intentional decision way back in #5462), the Base.isidentifier function doesn't check for normalization of Symbol arguments.

This also has the consequence that show is incorrect for non-normalized Symbols, since it checks isidentifier to see whether they can be printed as :foo rather than as Symbol("foo"):

julia> "e\u0301" # e with acute accent, not NFC normalized
"é"

julia> Symbol("e\u0301") == :é   # correct: :é gets normalized
false

julia> length(String(Symbol("e\u0301"))), length(String(:é))
(2, 1)

julia> Base.isidentifier(Symbol("e\u0301"))   # incorrect: should check normalization
true

julia> Symbol("e\u0301") # incorrect display: not the same as :é
:é

My suggestion is that https://github.com/JuliaLang/julia/blob/66e941073f400b92a6d8cf4d6bce5a6cb2d00b6c/base/show.jl#L1538 should be changed to something like:

function isidentifier(s::Symbol)
    str = string(s)
    ct(codepoint::UInt32) = get(Unicode._julia_charmap, codepoint, codepoint)
    isidentifier(str) && str == Unicode.normalize(str; stable=true, compose=true, chartransform=ct)
end

(Note that isidentifier should continue to not check normalization for ::String arguments, since in that case it has a slightly different meaning — it checks whether the string can be parsed into a valid identifier.)

贡献者指南