Implement Spark-compatible `aes_decrypt`

Đang mở
  • Có gì không ổn?
#296 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
45/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
rust, spark
Lĩnh vực
cryptography, databases

Hướng nghiên cứu

The work belongs in crates/spark/src/function/misc/, beside the existing AES code; first read Spark's AesDecrypt in misc.scala and aesDecrypt in ExpressionImplUtils.java for the exact argument layout and error mapping. Reuse a helper shared with aes_encrypt/try_aes_decrypt, wiring RustCrypto aes, aes-gcm, cbc and PKCS7 padding. Add crates/sqllogictest/test_files/misc/aes_decrypt.slt covering the listed examples, NULLs, key lengths and mode/padding errors; done when those queries return Spark's bytes and sqlStates.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

enhancement

Part of #272 and apache/datafusion-comet#7300.

What the function does

aes_decrypt(expr, key[, mode[, padding[, aad]]]) decrypts a value produced by aes_encrypt and returns BINARY.
expr, key and aad are BINARY; mode and padding are STRING (collated strings accepted from 4.0).
ImplicitCastInputTypes casts string arguments for expr, key and aad to their UTF-8 bytes. Defaults:
mode = 'GCM', padding = 'DEFAULT', aad = ''. Since 3.3.0 (aad since 3.5.0). NULL in any argument gives NULL.
There is no iv argument: the IV is read from the input.

Supported (mode, padding) pairs (case-insensitive): ECB + PKCS/DEFAULT, CBC + PKCS/DEFAULT,
GCM + NONE/DEFAULT. The key must be 16, 24 or 32 bytes. Expected input layout: ECB is the ciphertext only;
CBC is iv (16) || ciphertext; GCM is iv (12) || ciphertext || tag (16), with the 128-bit tag verified.
The result is the raw plaintext bytes. Use CAST(... AS STRING) to get text (the ExpressionDescription shows the
bytes as text).

Examples

From the ExpressionDescription, DataFrameFunctionsSuite and MiscFunctionsSuite:

SELECT cast(aes_decrypt(unhex('83F16B2AA704794132802D248E6BFD4E380078182D1544813898AC97E709B28A94'), '0000111122223333') AS STRING);
-- Spark
SELECT cast(aes_decrypt(unhex('6E7CA17BBB468D3084B5744BCA729FB7B2B7BCB8E4472847D02670489D95FA97DBBA7D3210'), '0000111122223333', 'GCM') AS STRING);
-- Spark SQL
SELECT cast(aes_decrypt(unbase64('3lmwu+Mw0H3fi5NDvcu9lg=='), '1234567890abcdef', 'ECB', 'PKCS') AS STRING);
-- Spark SQL
SELECT cast(aes_decrypt(unbase64('2NYmDCjgXTbbxGA3/SnJEfFC/JQ7olk2VQWReIAAFKo='), '1234567890abcdef', 'CBC') AS STRING);
-- Apache Spark
SELECT cast(aes_decrypt(unbase64('AAAAAAAAAAAAAAAAAAAAAPSd4mWyMZ5mhvjiAPQJnfg='), 'abcdefghijklmnop12345678ABCDEFGH', 'CBC', 'DEFAULT') AS STRING);
-- Spark
SELECT cast(aes_decrypt(unbase64('AAAAAAAAAAAAAAAAQiYi+sTLm7KD9UcZ2nlRdYDe/PX4'), 'abcdefghijklmnop12345678ABCDEFGH', 'GCM', 'DEFAULT', 'This is an AAD mixed into the input') AS STRING);
-- Spark
SELECT cast(aes_decrypt(unbase64('jmTOhz8XTbskI/zYFFgOFQ=='), 'abcdefghijklmnop', 'ECB') AS STRING);
-- (empty string)
SELECT cast(aes_decrypt(unbase64('9J3iZbIxnmaG+OIA9Amd+A=='), binary('abcdefghijklmnop12345678ABCDEFGH'), 'ECB') AS STRING);
-- Spark
SELECT aes_decrypt(cast(null as binary), 'abcdefghijklmnop');
-- NULL
Edge cases to test
  • NULL in each argument gives NULL. String vs binary key give the same result.
  • Key length not 16/24/32 (0, 15, 17): INVALID_PARAMETER_VALUE.AES_KEY_LENGTH (sqlState 22023), parameters
    `key`, `aes_encrypt`/`aes_decrypt`, actualLength.
  • Unsupported pairs ('GSM' with the default padding reports padding = DEFAULT; 'GCM','PKCS'; 'ECB','None';
    'CBC','NoPadding'): UNSUPPORTED_FEATURE.AES_MODE (sqlState 0A000).
  • Wrong key for ECB: INVALID_PARAMETER_VALUE.AES_CRYPTO_ERROR, parameter `expr`, `key`, detailMessage
    "Given final block not properly padded. Such issues can arise if a bad key is used during decryption."
  • Corrupted GCM ciphertext or wrong AAD: AES_CRYPTO_ERROR with "Tag mismatch" (JDK may append !).
  • Truncated CBC input: AES_CRYPTO_ERROR with a message starting "Input length". Empty GCM input:
    AES_CRYPTO_ERROR with "Invalid buffer arguments" (input shorter than the IV). The JCE messages are not
    reproducible in Rust; match the error condition and parameters, not detailMessage.
  • AAD with CBC: UNSUPPORTED_FEATURE.AES_MODE_AAD, and the message names `aes_encrypt` even when decrypting.
    AAD with ECB is silently ignored on decrypt (the ECB branch never looks at aad, by code reading).
  • Round trips through aes_encrypt for all modes and key sizes, empty plaintext, plaintext longer than one block,
    and multi-byte UTF-8 (results are bytes, not validated as UTF-8).
  • Literal and column for each argument, including per-row mode / key.
  • Deterministic.
Differences across Spark versions
  • 3.4: only ECB and GCM, no aad. A too-short GCM input throws an unmapped IllegalArgumentException, because
    only GeneralSecurityException maps to AES_CRYPTO_ERROR.
  • 3.5: adds CBC, aad, UNSUPPORTED_FEATURE.AES_MODE_AAD, and maps IllegalArgumentException too.
  • 4.0: collated mode / padding accepted. No behavior change through 4.2.
Implementation notes
  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala (AesDecrypt,
    RuntimeReplaceable over StaticInvoke) and
    sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java (aesDecrypt,
    aesInternal). Errors in sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala.
  • DataFusion core has no AES functions.
  • Suggested crates: RustCrypto aes, aes-gcm, cbc, ecb + block-padding (PKCS7).
  • Put it in crates/spark/src/function/misc/ with a helper module shared with aes_encrypt and try_aes_decrypt.
    The helper should return a typed error so try_aes_decrypt can turn every failure into NULL.
  • Tests: add crates/sqllogictest/test_files/misc/aes_decrypt.slt; document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Ngôn ngữ chính
Rust
Star
7
Fork
15
Merge trung bình
16 giờ 43 phút
Pull request đã merge (30 ngày)
26

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/datafusion-spark

Tất cả issue của apache/datafusion-spark

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.