`String::from_utf16` would be faster for ascii inputs with a dedicated ascii check

Abierto
#165,974 4 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Refactorización
Claridad
Bien especificado
Estado de actividad
Activo
Área
performance

Línea de trabajo

The slow path is the decode_utf16 loop in from_utf16_units in library/alloc/src/string.rs, and the encode side is EncodeUtf16 layered over Chars in the str iterator code. Start by running the reproduction in the issue to confirm the timings, then read how those paths handle input that is all ASCII. Done means a measured speedup on the ASCII benchmark with the existing UTF-16 tests still passing.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

A-str C-optimization T-libs

String::from_utf16 on 1M ASCII code units takes 590 to 690 us. A version that first checks that every unit is ASCII and then narrows them to bytes takes about 135 us (roughly 5x).

The other direction looks the same: str::encode_utf16().collect() takes about 530 us, checking is_ascii() and widening the bytes about 34 us (15x).

Both go through one char at a time. For decoding that is the decode_utf16 loop in from_utf16_units (library/alloc/src/string.rs), for encoding it is EncodeUtf16 on top of Chars. Neither has a path for input that is all ASCII.

Repro:

use std::hint::black_box;
use std::time::Instant;

fn micros<R>(mut f: impl FnMut() -> R) -> f64 {
    let mut best = f64::MAX;
    for _ in 0..5 {
        let t = Instant::now();
        for _ in 0..50 {
            black_box(f());
        }
        best = best.min(t.elapsed().as_secs_f64() / 50.0);
    }
    best * 1e6
}

fn from_utf16_narrowing(v: &[u16]) -> String {
    if v.iter().all(|&u| u < 0x80) {
        let bytes: Vec<u8> = v.iter().map(|&u| u as u8).collect();
        // ASCII is valid UTF-8
        return unsafe { String::from_utf8_unchecked(bytes) };
    }
    String::from_utf16(v).unwrap()
}

fn encode_utf16_widening(s: &str) -> Vec<u16> {
    if s.is_ascii() {
        return s.bytes().map(u16::from).collect();
    }
    s.encode_utf16().collect()
}

fn main() {
    let text = "lorem ipsum dolor sit amet ".repeat((1 << 20) / 27);
    let units: Vec<u16> = text.encode_utf16().collect();
    assert_eq!(from_utf16_narrowing(&units), text);
    assert_eq!(encode_utf16_widening(&text), units);

    println!("{} ASCII code units", units.len());

    let std_dec = micros(|| String::from_utf16(black_box(&units)).unwrap().len());
    let own_dec = micros(|| from_utf16_narrowing(black_box(&units)).len());
    println!("String::from_utf16        {std_dec:>8.1} us");
    println!("  check + narrow          {own_dec:>8.1} us   ({:.1}x)", std_dec / own_dec);

    let std_enc = micros(|| black_box(text.as_str()).encode_utf16().collect::<Vec<u16>>().len());
    let own_enc = micros(|| encode_utf16_widening(black_box(&text)).len());
    println!("encode_utf16().collect()  {std_enc:>8.1} us");
    println!("  check + widen           {own_enc:>8.1} us   ({:.1}x)", std_enc / own_enc);
}
Lenguaje dominante
Rust
Estrellas
122k
Forks
16.7k
Merge medio
2 d 16 h
PR fusionados (30 d)
515

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de rust-lang/rust

Todos los issues de rust-lang/rust

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.