Implement Spark-compatible `aes_decrypt`

Aperta
  • Qualcosa non va?
#296 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust, spark
Ambito
cryptography, databases

Direzione di ricerca

The work belongs in crates/spark/src/function/misc/, beside the existing AES code; first read Spark's AesDecrypt in misc.scala and aesDecrypt in ExpressionImplUtils.java for the exact argument layout and error mapping. Reuse a helper shared with aes_encrypt/try_aes_decrypt, wiring RustCrypto aes, aes-gcm, cbc and PKCS7 padding. Add crates/sqllogictest/test_files/misc/aes_decrypt.slt covering the listed examples, NULLs, key lengths and mode/padding errors; done when those queries return Spark's bytes and sqlStates.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Part of #272 and apache/datafusion-comet#7300.

What the function does

aes_decrypt(expr, key[, mode[, padding[, aad]]]) decrypts a value produced by aes_encrypt and returns BINARY.
expr, key and aad are BINARY; mode and padding are STRING (collated strings accepted from 4.0).
ImplicitCastInputTypes casts string arguments for expr, key and aad to their UTF-8 bytes. Defaults:
mode = 'GCM', padding = 'DEFAULT', aad = ''. Since 3.3.0 (aad since 3.5.0). NULL in any argument gives NULL.
There is no iv argument: the IV is read from the input.

Supported (mode, padding) pairs (case-insensitive): ECB + PKCS/DEFAULT, CBC + PKCS/DEFAULT,
GCM + NONE/DEFAULT. The key must be 16, 24 or 32 bytes. Expected input layout: ECB is the ciphertext only;
CBC is iv (16) || ciphertext; GCM is iv (12) || ciphertext || tag (16), with the 128-bit tag verified.
The result is the raw plaintext bytes. Use CAST(... AS STRING) to get text (the ExpressionDescription shows the
bytes as text).

Examples

From the ExpressionDescription, DataFrameFunctionsSuite and MiscFunctionsSuite:

SELECT cast(aes_decrypt(unhex('83F16B2AA704794132802D248E6BFD4E380078182D1544813898AC97E709B28A94'), '0000111122223333') AS STRING);
-- Spark
SELECT cast(aes_decrypt(unhex('6E7CA17BBB468D3084B5744BCA729FB7B2B7BCB8E4472847D02670489D95FA97DBBA7D3210'), '0000111122223333', 'GCM') AS STRING);
-- Spark SQL
SELECT cast(aes_decrypt(unbase64('3lmwu+Mw0H3fi5NDvcu9lg=='), '1234567890abcdef', 'ECB', 'PKCS') AS STRING);
-- Spark SQL
SELECT cast(aes_decrypt(unbase64('2NYmDCjgXTbbxGA3/SnJEfFC/JQ7olk2VQWReIAAFKo='), '1234567890abcdef', 'CBC') AS STRING);
-- Apache Spark
SELECT cast(aes_decrypt(unbase64('AAAAAAAAAAAAAAAAAAAAAPSd4mWyMZ5mhvjiAPQJnfg='), 'abcdefghijklmnop12345678ABCDEFGH', 'CBC', 'DEFAULT') AS STRING);
-- Spark
SELECT cast(aes_decrypt(unbase64('AAAAAAAAAAAAAAAAQiYi+sTLm7KD9UcZ2nlRdYDe/PX4'), 'abcdefghijklmnop12345678ABCDEFGH', 'GCM', 'DEFAULT', 'This is an AAD mixed into the input') AS STRING);
-- Spark
SELECT cast(aes_decrypt(unbase64('jmTOhz8XTbskI/zYFFgOFQ=='), 'abcdefghijklmnop', 'ECB') AS STRING);
-- (empty string)
SELECT cast(aes_decrypt(unbase64('9J3iZbIxnmaG+OIA9Amd+A=='), binary('abcdefghijklmnop12345678ABCDEFGH'), 'ECB') AS STRING);
-- Spark
SELECT aes_decrypt(cast(null as binary), 'abcdefghijklmnop');
-- NULL
Edge cases to test
  • NULL in each argument gives NULL. String vs binary key give the same result.
  • Key length not 16/24/32 (0, 15, 17): INVALID_PARAMETER_VALUE.AES_KEY_LENGTH (sqlState 22023), parameters
    `key`, `aes_encrypt`/`aes_decrypt`, actualLength.
  • Unsupported pairs ('GSM' with the default padding reports padding = DEFAULT; 'GCM','PKCS'; 'ECB','None';
    'CBC','NoPadding'): UNSUPPORTED_FEATURE.AES_MODE (sqlState 0A000).
  • Wrong key for ECB: INVALID_PARAMETER_VALUE.AES_CRYPTO_ERROR, parameter `expr`, `key`, detailMessage
    "Given final block not properly padded. Such issues can arise if a bad key is used during decryption."
  • Corrupted GCM ciphertext or wrong AAD: AES_CRYPTO_ERROR with "Tag mismatch" (JDK may append !).
  • Truncated CBC input: AES_CRYPTO_ERROR with a message starting "Input length". Empty GCM input:
    AES_CRYPTO_ERROR with "Invalid buffer arguments" (input shorter than the IV). The JCE messages are not
    reproducible in Rust; match the error condition and parameters, not detailMessage.
  • AAD with CBC: UNSUPPORTED_FEATURE.AES_MODE_AAD, and the message names `aes_encrypt` even when decrypting.
    AAD with ECB is silently ignored on decrypt (the ECB branch never looks at aad, by code reading).
  • Round trips through aes_encrypt for all modes and key sizes, empty plaintext, plaintext longer than one block,
    and multi-byte UTF-8 (results are bytes, not validated as UTF-8).
  • Literal and column for each argument, including per-row mode / key.
  • Deterministic.
Differences across Spark versions
  • 3.4: only ECB and GCM, no aad. A too-short GCM input throws an unmapped IllegalArgumentException, because
    only GeneralSecurityException maps to AES_CRYPTO_ERROR.
  • 3.5: adds CBC, aad, UNSUPPORTED_FEATURE.AES_MODE_AAD, and maps IllegalArgumentException too.
  • 4.0: collated mode / padding accepted. No behavior change through 4.2.
Implementation notes
  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala (AesDecrypt,
    RuntimeReplaceable over StaticInvoke) and
    sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java (aesDecrypt,
    aesInternal). Errors in sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala.
  • DataFusion core has no AES functions.
  • Suggested crates: RustCrypto aes, aes-gcm, cbc, ecb + block-padding (PKCS7).
  • Put it in crates/spark/src/function/misc/ with a helper module shared with aes_encrypt and try_aes_decrypt.
    The helper should return a typed error so try_aes_decrypt can turn every failure into NULL.
  • Tests: add crates/sqllogictest/test_files/misc/aes_decrypt.slt; document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Lingua principale
Rust
Stelle
7
Fork
15
Merge medio
16h 43m
PR unite (30g)
26

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-spark

Tutte le issue di apache/datafusion-spark

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.