[FLINK] [VL] Support binary (VARBINARY/BYTES) arguments for the hash expressions (MD5/SHA1/SHA2 family)

Open
#13,635 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, java

Research direction

Start with the LogicalTypeConverter in gluten-flink, which has no VarBinaryType mapping and fails with 'Unsupported logical type: BYTES'. Then find the velox flinksql hash registrations, which only declare Varchar signatures for md5, sha1, sha2 and the sha224 to sha512 variants, and add Varbinary overloads with gtest cases. Done when MD5(X'6162') returns 187ef4436122d1cc2f40dc2b92f0eba0, the SHA-2 family matches native Flink, and empty bytes and NULL are covered.

Written by the indexing model from the issue text.

Description

bug triage
Backend

VL (Velox)

Bug description
Background

Flink's hash expressions (MD5, SHA1, SHA2 / SHA224 / SHA256 / SHA384 / SHA512)
accept the STRING family, which includes binary types, and hash the raw bytes of the input,
returning a lowercase hex string. Native reference (Flink 1.19.x):

SELECT MD5(X'6162') -> 187ef4436122d1cc2f40dc2b92f0eba0, the same as MD5('ab').

gluten-flink currently supports VARCHAR only:

  • velox's flinksql hash functions register Varchar input signatures only;
  • LogicalTypeConverter has no VarBinaryType mapping, so any BYTES column or expression
    fails at planning with java.lang.RuntimeException: Unsupported logical type: BYTES NOT NULL
    before the hash converter is reached.

The SHA-2 family (#13484) has exactly the same limitation, and end-to-end runs confirm it
(gluten fails at planning, native succeeds). This is not specific to MD5/SHA1 (#13420).

Scope
  1. velox (flinksql namespace): register (Varbinary) -> Varchar for md5, sha1, sha2,
    sha224, sha256, sha384, sha512, hashing the raw bytes and rendering lowercase hex;
    add gtest coverage for binary input (including empty bytes and NULL).
  2. gluten-flink LogicalTypeConverter: map VarBinaryType to the velox VarbinaryType.
  3. Tests: gluten UT with a BYTES column (X'6162', empty bytes, NULL) plus an end-to-end
    dual-run against native Flink.
  4. Covers MD5/SHA1 (#13420) and the SHA-2 family (#13484).
Expected behavior

MD5(bytes), SHA1(bytes) and SHA2(bytes, n) return the same digests as native Flink.

Gluten version

main branch

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

Dominant language
Scala
Stars
1.6k
Forks
679
Avg merge
4d 16h
Merged PRs (30d)
84

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/gluten

All issues in apache/gluten

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.