Native approx_top_k functions: audit the frequent-items sketch against Java and implement

Abierto
  • ¿Algo no está bien?
#7,400 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
12/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
java, rust, scala, spark
Área
data-engineering

Línea de trabajo

This is an audit-first design task spanning the Rust frequencies feature of the sketch crate and Spark-side serde and expression wiring. Start by scoring the family with suggest-native-expression, then compare sketch bytes against DataSketches-Java 6.2.0 fixtures for each item type Spark accepts. Done means byte parity for every supported item type, opt-in native functions, and SQL file tests that read Spark-produced sketch columns and the reverse.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:aggregation area:expressions enhancement

What is the problem the feature request solves?

Part of #7397.

Spark 4.1+ provides approx_top_k, approx_top_k_accumulate, approx_top_k_combine and approx_top_k_estimate, built on the DataSketches frequent-items sketch. None of them run natively in Comet. The crate's frequencies feature provides the sketch.

Describe the potential solution

  1. Score the family with suggest-native-expression first.
  2. Audit the crate's frequencies feature against DataSketches-Java 6.2.0 using committed fixtures. The item type matters here more than in the other families: Spark supports several item types, and each has its own item serialization in the sketch bytes, so check each one Spark accepts rather than only strings. Also check tie-breaking and the order of returned items, the error bounds, and how Spark derives the maximum map size from its arguments.
  3. Implement the serde and native functions once the audit shows byte parity for every supported item type, opt-in until then. Fall back for any item type that is not covered.
  4. Add SQL file tests, including a Spark-produced sketch column read by Comet and the reverse.
Lenguaje dominante
Scala
Estrellas
1.3k
Forks
414
Merge medio
3 d 5 h
PR fusionados (30 d)
210

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-comet

Todos los issues de apache/datafusion-comet

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.