[FLINK] [VL] Support binary (VARBINARY/BYTES) arguments for the hash expressions (MD5/SHA1/SHA2 family)
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Domain
- data-engineering
Research direction
Start with the LogicalTypeConverter in gluten-flink, which has no VarBinaryType mapping and fails with 'Unsupported logical type: BYTES'. Then find the velox flinksql hash registrations, which only declare Varchar signatures for md5, sha1, sha2 and the sha224 to sha512 variants, and add Varbinary overloads with gtest cases. Done when MD5(X'6162') returns 187ef4436122d1cc2f40dc2b92f0eba0, the SHA-2 family matches native Flink, and empty bytes and NULL are covered.
Written by the indexing model from the issue text.
Description
Backend
VL (Velox)
Bug description
Background
Flink's hash expressions (MD5, SHA1, SHA2 / SHA224 / SHA256 / SHA384 / SHA512)
accept the STRING family, which includes binary types, and hash the raw bytes of the input,
returning a lowercase hex string. Native reference (Flink 1.19.x):
SELECT MD5(X'6162') -> 187ef4436122d1cc2f40dc2b92f0eba0, the same as MD5('ab').
gluten-flink currently supports VARCHAR only:
- velox's flinksql hash functions register
Varcharinput signatures only; LogicalTypeConverterhas noVarBinaryTypemapping, so anyBYTEScolumn or expression
fails at planning withjava.lang.RuntimeException: Unsupported logical type: BYTES NOT NULL
before the hash converter is reached.
The SHA-2 family (#13484) has exactly the same limitation, and end-to-end runs confirm it
(gluten fails at planning, native succeeds). This is not specific to MD5/SHA1 (#13420).
Scope
- velox (flinksql namespace): register
(Varbinary) -> Varcharformd5,sha1,sha2,
sha224,sha256,sha384,sha512, hashing the raw bytes and rendering lowercase hex;
add gtest coverage for binary input (including empty bytes and NULL). - gluten-flink
LogicalTypeConverter: mapVarBinaryTypeto the veloxVarbinaryType. - Tests: gluten UT with a
BYTEScolumn (X'6162', empty bytes, NULL) plus an end-to-end
dual-run against native Flink. - Covers MD5/SHA1 (#13420) and the SHA-2 family (#13484).
Expected behavior
MD5(bytes), SHA1(bytes) and SHA2(bytes, n) return the same digests as native Flink.
Gluten version
main branch
Spark version
None
Spark configurations
No response
System information
No response
Relevant logs
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 679
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 84
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/gluten
-
[CORE] Selecting a cost model the backend has no coster for fails with ClassNotFoundExceptionPossibly taken @LuciferYang claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
bug unit testing velox backend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/gluten#13463 · 1 comment ·
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
[VL] madvise(WILLNEED) call fails in MmapFileStream because of wrong calculation of fetching lengthMay be free again A pull request for this issue was closed without being merged. Openbug triage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 3 days
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
rchillyard/TableParser#206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
? - Needs Triage bug
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
NVIDIA/cudf-spark#17063 ·
Maintainers usually reply within 2 days
-
bug 🐞 User issue 🚨
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
valkey-io/valkey-glide#7813 · 2 comments ·
Maintainers usually reply within 1 day