[FLINK] [VL] Support binary (VARBINARY/BYTES) arguments for the hash expressions (MD5/SHA1/SHA2 family)

Abierto
#13,635 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp, java

Línea de trabajo

Start with the LogicalTypeConverter in gluten-flink, which has no VarBinaryType mapping and fails with 'Unsupported logical type: BYTES'. Then find the velox flinksql hash registrations, which only declare Varchar signatures for md5, sha1, sha2 and the sha224 to sha512 variants, and add Varbinary overloads with gtest cases. Done when MD5(X'6162') returns 187ef4436122d1cc2f40dc2b92f0eba0, the SHA-2 family matches native Flink, and empty bytes and NULL are covered.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug triage
Backend

VL (Velox)

Bug description
Background

Flink's hash expressions (MD5, SHA1, SHA2 / SHA224 / SHA256 / SHA384 / SHA512)
accept the STRING family, which includes binary types, and hash the raw bytes of the input,
returning a lowercase hex string. Native reference (Flink 1.19.x):

SELECT MD5(X'6162') -> 187ef4436122d1cc2f40dc2b92f0eba0, the same as MD5('ab').

gluten-flink currently supports VARCHAR only:

  • velox's flinksql hash functions register Varchar input signatures only;
  • LogicalTypeConverter has no VarBinaryType mapping, so any BYTES column or expression
    fails at planning with java.lang.RuntimeException: Unsupported logical type: BYTES NOT NULL
    before the hash converter is reached.

The SHA-2 family (#13484) has exactly the same limitation, and end-to-end runs confirm it
(gluten fails at planning, native succeeds). This is not specific to MD5/SHA1 (#13420).

Scope
  1. velox (flinksql namespace): register (Varbinary) -> Varchar for md5, sha1, sha2,
    sha224, sha256, sha384, sha512, hashing the raw bytes and rendering lowercase hex;
    add gtest coverage for binary input (including empty bytes and NULL).
  2. gluten-flink LogicalTypeConverter: map VarBinaryType to the velox VarbinaryType.
  3. Tests: gluten UT with a BYTES column (X'6162', empty bytes, NULL) plus an end-to-end
    dual-run against native Flink.
  4. Covers MD5/SHA1 (#13420) and the SHA-2 family (#13484).
Expected behavior

MD5(bytes), SHA1(bytes) and SHA2(bytes, n) return the same digests as native Flink.

Gluten version

main branch

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

Lenguaje dominante
Scala
Estrellas
1.6k
Forks
685
Merge medio
4 d 16 h
PR fusionados (30 d)
84

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/gluten

Todos los issues de apache/gluten

Issues similares

Más issues de Scala

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.