[C++][Parquet] Plaintext-footer files written with AES_GCM_CTR_V1 record AES_GCM_V1 as the encryption algorithm and cannot be read
维护者通常 1 天内回复
评估
调研方向
从 cpp/src/parquet/metadata.cc 中的 FileMetaDataBuilder::FileMetaDataBuilderImpl::Finish 开始,然后检查 cpp/src/parquet/encryption/write_configurations_test.cc 中的加密配置。为使用明文 footer 的 AES_GCM_CTR_V1 添加覆盖测试,并验证加密文件能够成功完成往返。完成的标准是元数据记录文件的实际算法,同时 footer 签名仍能通过验证。
由索引模型根据 Issue 内容生成。
描述
Describe the bug, including details regarding any error messages, version, and platform.
A Parquet file written with modular encryption in plaintext-footer mode and the AES_GCM_CTR_V1 algorithm cannot be read back, by Arrow or by any other reader. The writer encrypts the pages with AES-CTR, as it should, but records AES_GCM_V1 in FileMetaData.encryption_algorithm. A reader takes the algorithm from that field, tries to open the CTR pages as GCM modules, and fails authentication.
Nothing is reported at write time, so the file looks fine until someone tries to read an encrypted column.
Reproduction
import pyarrow as pa, pyarrow.parquet as pq, pyarrow.parquet.encryption as pe
key = b"0123456789012345"
table = pa.table({"a": [1, 2, 3]})
for algo in ("AES_GCM_V1", "AES_GCM_CTR_V1"):
props = pe.create_encryption_properties(
key, plaintext_footer=True, encryption_algorithm=algo)
pq.write_table(table, "t.parquet", encryption_properties=props)
try:
pq.read_table(
"t.parquet",
decryption_properties=pe.create_decryption_properties(key))
print(pa.__version__, algo, "ok")
except OSError as e:
print(pa.__version__, algo, "FAILED:", e)
Output:
25.0.1 AES_GCM_V1 ok
25.0.1 AES_GCM_CTR_V1 FAILED: Failed decryption finalization
The same two algorithms with an encrypted footer (plaintext_footer=False) both round-trip, as does AES_GCM_V1 with a plaintext footer. Only the combination of a plaintext footer and AES_GCM_CTR_V1 fails.
Cause
FileMetaDataBuilder::FileMetaDataBuilderImpl::Finish in cpp/src/parquet/metadata.cc builds the footer's algorithm for plaintext-footer mode like this (current main):
// if plaintext footer, set footer signing algorithm
auto file_encryption_properties = properties_->file_encryption_properties();
if (file_encryption_properties && !file_encryption_properties->encrypted_footer()) {
EncryptionAlgorithm signing_algorithm;
EncryptionAlgorithm algo = file_encryption_properties->algorithm();
signing_algorithm.aad.aad_file_unique = algo.aad.aad_file_unique;
signing_algorithm.aad.supply_aad_prefix = algo.aad.supply_aad_prefix;
if (!algo.aad.supply_aad_prefix) {
signing_algorithm.aad.aad_prefix = algo.aad.aad_prefix;
}
signing_algorithm.algorithm = ParquetCipher::AES_GCM_V1;
metadata_->__set_encryption_algorithm(ToThrift(signing_algorithm));
The hard-coded AES_GCM_V1 treats FileMetaData.encryption_algorithm as the algorithm of the footer signature, which is indeed always GCM. But the field is the file's encryption algorithm. parquet.thrift describes it as:
/**
* Encryption algorithm. This field is set only in encrypted files
* with plaintext footer. Files with encrypted footer store algorithm id
* in FileCryptoMetaData structure.
*/
8: optional EncryptionAlgorithm encryption_algorithm
and the reader uses it that way: SerializedFile::ParseMetaDataOfEncryptedFileWithPlaintextFooter in cpp/src/parquet/file_reader.cc passes file_metadata_->encryption_algorithm().algorithm to the InternalFileDecryptor, which then builds the page decryptors from it.
parquet-java writes the file's actual algorithm in this field (ParquetFileWriter.serializeFooter: parquetMetadata.setEncryption_algorithm(fileEncryptor.getEncryptionAlgorithm())).
Evidence that the pages are fine and only the field is wrong
Taking a file produced by the reproduction above, changing the one byte that selects the EncryptionAlgorithm union member in the footer from AES_GCM_V1 to AES_GCM_CTR_V1, and recomputing the footer signature for the changed bytes, gives a file that pyarrow 25.0.1 reads correctly, with footer verification enabled and the data equal to what was written. So the reader needs no change, and the pages in affected files are valid CTR modules.
Suggested fix
Write the file's algorithm instead of the constant:
signing_algorithm.algorithm = algo.algorithm;
The signature itself does not depend on this value: both the writer and FileMetaData::VerifySignature use the GCM cipher for the footer regardless of the file's algorithm (metadata = true).
The existing encryption configurations in cpp/src/parquet/encryption/write_configurations_test.cc use AES_GCM_CTR_V1 only with an encrypted footer and a plaintext footer only with AES_GCM_V1, which is why the round-trip tests do not catch this. A configuration combining the two would cover it.
Files already written with this combination stay unreadable after the fix, since their footers name the wrong algorithm.
Version and platform
- pyarrow 25.0.1 (PyPI wheel, manylinux_2_28 x86_64), Python 3.14.4, Linux x86_64 (Fedora 44).
- The hard-coded line is present on
mainas of commit fcac3dfd8e (2026-09-30).
Component(s)
C++, Parquet
- 主要语言
- C++
- 星标
- 17.2k
- 派生
- 4.4k
- 平均合并
- 3 天 22 小时
- 30 天内合并 PR
- 80
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
apache/arrow 的其他 Issue
-
[C++][Python] IPC reader rejects a DictionaryEncoding without indexType, which the format allows可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭Component: C++ Component: Python
难度 2/5 1-3 小时 新手友好度 68/100
维护者通常 1 天内回复
-
[R] Silent loss of metadata integrity for date-time and numeric attributes in write_parquet()/read_parquet()可能重新可做 @james-finn-travers 于 7 天前认领,目前没有进行中的 PR。 未关闭Component: R Type: bug
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
[R] Expose ignore_extra_columns and pad_short_rows CSV parse options可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭Component: R good-first-issue Type: enhancement
难度 2/5 1-3 小时 新手友好度 76/100
维护者通常 1 天内回复
-
Component: C++
难度 2/5 1-3 小时 新手友好度 86/100
维护者通常 1 天内回复
-
Component: C++ Type: enhancement
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 72/100
-
upstream update
难度 2/5 1-3 小时 新手友好度 65/100
conan-io/conan-center-index#31142 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 85/100
FISCO-BCOS/FISCO-BCOS#5684 ·
维护者通常 4 天内回复