`String::from_utf8_lossy` is 6–19x slower than str::from_utf8 on valid ASCII

未关闭
#165,973 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
3/5
预计耗时
1-2 天
新手友好度
45/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
rust
领域
backend

调研方向

Start with Utf8Chunks::next in library/core/src/str/lossy.rs, which checks bytes one at a time, and compare it with the word-at-a-time ASCII skip in library/core/src/str/validations.rs. Run the repro in the issue to confirm the ratio on 1 MiB of ASCII versus German and Russian text. Done means the ASCII case gets close to str::from_utf8 speed while the German and Russian timings and the existing lossy tests stay unchanged.

由索引模型根据 Issue 内容生成。

描述

A-str C-optimization T-libs

On 1 MiB of valid ASCII, String::from_utf8_lossy takes about 260 µs while str::from_utf8 takes about 17 µs, although both validate the whole input and return it borrowed. from_utf8 skips ASCII a word at a time (library/core/src/str/validations.rs), whereas Utf8Chunks::next in library/core/src/str/lossy.rs checks byte by byte. On German or Russian text the two are equally fast, so only the ASCII fast path is missing, and everything built on utf8_chunks() pays the same cost.

Repro:

use std::hint::black_box;
use std::time::Instant;

fn micros<R>(mut f: impl FnMut() -> R) -> f64 {
    let mut best = f64::MAX;
    for _ in 0..5 {
        let t = Instant::now();
        for _ in 0..50 {
            black_box(f());
        }
        best = best.min(t.elapsed().as_secs_f64() / 50.0);
    }
    best * 1e6
}

fn main() {
    const LEN: usize = 1 << 20;
    let inputs = [
        ("ascii", "lorem ipsum dolor sit amet ".repeat(LEN / 27)),
        ("german", "Größe über Straße, schön für Käufer ".repeat(LEN / 42)),
        ("russian", "Привет мир, это проверка строки ".repeat(LEN / 58)),
    ];

    println!("{:<10} {:>14} {:>22} {:>8}", "1 MiB", "from_utf8 (us)", "from_utf8_lossy (us)", "ratio");
    for (name, text) in &inputs {
        let bytes = text.as_bytes();
        let strict = micros(|| std::str::from_utf8(black_box(bytes)).unwrap().len());
        let lossy = micros(|| String::from_utf8_lossy(black_box(bytes)).len());
        println!("{name:<10} {strict:>14.1} {lossy:>22.1} {:>7.1}x", lossy / strict);
    }
}
主要语言
Rust
星标
122k
派生
16.7k
平均合并
2 天 16 小时
30 天内合并 PR
515

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

rust-lang/rust 的其他 Issue

查看 rust-lang/rust 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。