Implement Spark-compatible `aes_encrypt`
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 45/100
Línea de trabajo
Start by reading how existing Spark functions are registered in crates/spark/src/function/misc/ (group misc_funcs) and whether an aes_decrypt helper module already exists, since this issue asks for a shared mode/padding/key-check module. Cross-check the vectors against Spark's ExpressionImplUtils.java (aesEncrypt, aesInternal) and misc.scala before trusting the issue's examples, which are LLM-generated. Done looks like crates/sqllogictest/test_files/misc/aes_encrypt.slt covering the fixed-IV vectors, NULL/key-length/unsupported-pair errors, and round trips through aes_decrypt, plus #[user_doc] docs.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Part of #272 and apache/datafusion-comet#7300.
What the function does
aes_encrypt(expr, key[, mode[, padding[, iv[, aad]]]]) encrypts expr with AES and returns BINARY.
expr, key, iv and aad are BINARY; mode and padding are STRING (collated strings accepted from 4.0).
Through ImplicitCastInputTypes, string arguments for expr, key, iv and aad are cast to binary (their UTF-8
bytes). Defaults: mode = 'GCM', padding = 'DEFAULT', iv = '', aad = ''. Since 3.3.0 (iv and aad since
3.5.0). The result is NULL if any argument is NULL (StaticInvoke with null propagation).
Supported (mode, padding) pairs, both case-insensitive: ECB with PKCS/DEFAULT (AES/ECB/PKCS5Padding),
CBC with PKCS/DEFAULT (AES/CBC/PKCS5Padding), GCM with NONE/DEFAULT (AES/GCM/NoPadding, 128-bit tag).
The key must be 16, 24 or 32 bytes (AES-128/192/256). It is used as is, with no key derivation.
Output layout: ECB is the ciphertext only. CBC is iv (16 bytes) || ciphertext. GCM is
iv (12 bytes) || ciphertext || tag (16 bytes). If iv is empty, a random IV is drawn from SecureRandom, so CBC and
GCM output is nondeterministic. Only ECB, or CBC/GCM with an explicit IV, give fixed results.
Examples
Deterministic ones (from the ExpressionDescription, ExpressionImplUtilsSuite and DataFrameFunctionsSuite):
SELECT base64(aes_encrypt('Spark SQL', '1234567890abcdef', 'ECB', 'PKCS'));
-- 3lmwu+Mw0H3fi5NDvcu9lg==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop', 'ECB'));
-- 4Hv0UKCx6nfUeAoPZo1z+w==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678', 'ECB'));
-- NeTYNgA+PCQBN50DA//O2w==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'ECB'));
-- 9J3iZbIxnmaG+OIA9Amd+A==
SELECT base64(aes_encrypt('', 'abcdefghijklmnop', 'ECB'));
-- jmTOhz8XTbskI/zYFFgOFQ==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'CBC', 'DEFAULT', unhex('00000000000000000000000000000000')));
-- AAAAAAAAAAAAAAAAAAAAAPSd4mWyMZ5mhvjiAPQJnfg=
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'GCM', 'DEFAULT', unhex('000000000000000000000000')));
-- AAAAAAAAAAAAAAAAQiYi+sRNYDAOTjdSEcYBFsAWPL1f
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'GCM', 'DEFAULT', unhex('000000000000000000000000'), 'This is an AAD mixed into the input'));
-- AAAAAAAAAAAAAAAAQiYi+sTLm7KD9UcZ2nlRdYDe/PX4
SELECT aes_encrypt(cast(null as string), 'abcdefghijklmnop');
-- NULL
Random-IV examples can only be checked by round trip, or by length:
SELECT cast(aes_decrypt(aes_encrypt('Spark', 'abcdefghijklmnop'), 'abcdefghijklmnop') AS STRING);
-- Spark
SELECT length(aes_encrypt('Spark', '0000111122223333')); -- 33 (12 IV + 5 + 16 tag), by code reading
SELECT length(aes_encrypt('Apache Spark', '1234567890abcdef', 'CBC')); -- 32 (16 IV + 16), by code reading
Edge cases to test
- NULL in each of the six arguments gives NULL. String vs binary
exprandkeygive the same bytes. - Key length other than 16/24/32 (0, 15, 17 bytes):
INVALID_PARAMETER_VALUE.AES_KEY_LENGTH, sqlState 22023,
"The value of parameter(s)keyinaes_encrypt/aes_decryptis invalid: expects a binary value with 16, 24 or
32 bytes, but got 17 bytes." The key is checked before the mode. - Unsupported pair (
'CBC','None','ECB','NoPadding','GCM','PKCS','GSM'):UNSUPPORTED_FEATURE.AES_MODE,
sqlState 0A000, "The feature is not supported: AES-CBC with the padding None by theaes_encrypt/aes_decrypt
function." The parameters echo the user's original casing. - Non-empty IV with ECB:
UNSUPPORTED_FEATURE.AES_MODE_IV(aes_encryptwith AES-ECB does not support
initialization vectors (IVs).). - Wrong IV length (5 bytes for CBC or GCM):
INVALID_PARAMETER_VALUE.AES_IV_LENGTHwith parameter`iv`and
"supports 16-byte CBC IVs and 12-byte GCM IVs, but got 5 bytes for CBC." - Non-empty AAD with ECB or CBC:
UNSUPPORTED_FEATURE.AES_MODE_AAD. Emptyiv/aad('') behave exactly like
the defaults, so ECB withiv = ''andaad = ''is accepted. - Any JCE failure becomes
INVALID_PARAMETER_VALUE.AES_CRYPTO_ERRORwith parameter`expr`, `key`and the JCE
message asdetailMessage. A Rust implementation can't reproduce those messages exactly; match the condition. - Empty plaintext: ECB and CBC produce one full padding block, GCM produces IV + tag only (28 bytes).
- Plaintext longer than one block (
This message is longer than a single AES block and should work fine.) and
multi-byte UTF-8 strings. - Literal and column for every argument, including per-row
mode,padding,keyandivcolumns. - Determinism: Spark does not mark
aes_encryptnondeterministic. A call where every argument is a literal is
constant-folded (StaticInvokeis foldable), so every row gets the same IV. With a column argument each row gets a
fresh IV. In DataFusion the UDF should beVolatile. Tests must round-trip or pass a fixed IV.
Differences across Spark versions
- 3.4: only ECB and GCM, signature
(expr, key[, mode[, padding]]). OnlyGeneralSecurityExceptionis mapped to
AES_CRYPTO_ERROR. - 3.5: adds CBC (SPARK-43038),
ivandaad,UNSUPPORTED_FEATURE.AES_MODE_IV,UNSUPPORTED_FEATURE.AES_MODE_AAD,
INVALID_PARAMETER_VALUE.AES_IV_LENGTH.IllegalArgumentExceptionfrom the JCE is also mapped to
AES_CRYPTO_ERROR. - 4.0:
mode/paddingaccept collated strings. No behavior change through 4.2.
Implementation notes
- Spark:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala(AesEncrypt, a
RuntimeReplaceableoverStaticInvoke) and
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
(aesEncrypt,aesInternal,CipherMode). Errors inQueryExecutionErrors.scala. - DataFusion core has no AES functions.
- Suggested crates: RustCrypto
aes,aes-gcm,cbc,ecb(withblock-paddingPKCS7, which matches JCE
PKCS5Paddingfor 16-byte blocks). Random IVs fromrand(already a dependency) with an OS RNG. - Put it in
crates/spark/src/function/misc/(Spark groupmisc_funcs), sharing a helper module with
aes_decryptandtry_aes_decryptfor mode/padding parsing, key checks and error construction. - Tests: add
crates/sqllogictest/test_files/misc/aes_encrypt.sltwith the fixed-IV and ECB vectors above and
round-trip tests throughaes_decrypt; document with#[user_doc].
This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.
- Lenguaje dominante
- Rust
- Estrellas
- 7
- Forks
- 15
- Merge medio
- 16 h 43 min
- PR fusionados (30 d)
- 26
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de apache/datafusion-spark
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
apache/datafusion-spark#287 ·
Los mantenedores suelen responder en 1 día
-
enhancement good first issue
Dificultad 2/5 1-3 horas Aptitud para principiantes 80/100
apache/datafusion-spark#266 ·
Los mantenedores suelen responder en 1 día
-
[Bug] make_dt_interval returns NULL for more than about 1.49 million days, Spark returns the intervalPosiblemente ocupada @Subodh-17 la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
apache/datafusion-spark#109 ·
Los mantenedores suelen responder en 1 día
-
enhancement
Dificultad 5/5 Más de una semana Aptitud para principiantes 18/100
apache/datafusion-spark#299 ·
Los mantenedores suelen responder en 1 día
-
enhancement
Dificultad 4/5 3-5 días Aptitud para principiantes 35/100
apache/datafusion-spark#298 ·
Los mantenedores suelen responder en 1 día
Todos los issues de apache/datafusion-spark
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
storytold/photocraft#15232 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
suhina-daw/suhina-daw#1477 ·
Los mantenedores suelen responder en 1 día
-
parity priority: low ui
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
storytold/pdfcraft#7483 · 1 comentario ·
Los mantenedores suelen responder en 2 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
IngvarConsulting/unica#3097 ·
Los mantenedores suelen responder en 1 día
-
enhancement in-progress
Dificultad 1/5 Menos de una hora Aptitud para principiantes 64/100