Audit third-party code bundled in published JARs
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 48/100
- Tipo di issue
- Documentazione
- Chiarezza
- Da chiarire
- Stato di attività
- Tranquilla
- Stack tecnologico
- bash, java
- Ambito
- build-system, documentation, release
Direzione di ricerca
Parti dalla radice del repository con Java 17, quindi esegui i comandi di configurazione Maven elencati per compilare i JAR pubblicati e gli artefatti di supporto. Esegui separatamente lo script di verifica Bash fornito per esaminare le classi, i sorgenti, i test e gli asset Javadoc inclusi. Il lavoro è completato quando i risultati sono documentati, la verifica copre i 59 JAR e il controllo dei file legali ha esito positivo.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Background
We need to identify which published JARs bundle third-party code. ASF release policy requires JARs to place LICENSE and NOTICE in META-INF and requires the LICENSE file to cover every bundled non-Apache component. Any required third-party notices also need to be included.
This issue documents the findings and provides a script to verify them.
Findings
Cross-checking the final JAR contents against the Maven Shade configuration found:
- 7 binary JARs with bundled third-party classes (these need to be reflected in their respective LICENSE and NOTICE files)
- 13 Javadoc JARs with JDK-generated assets
- 39 JARs with no bundled third-party code
| Artifact | Maven configuration | Final JAR contents |
|---|---|---|
parquet-avro-*.jar |
Explicitly shades FastUtil | FastUtil |
parquet-column-*.jar |
Inherits FastUtil/OpenHFT Shade rules | FastUtil, OpenHFT |
parquet-format-structures-*.jar |
Explicitly includes libthrift |
Apache Thrift |
parquet-hadoop-*.jar |
Shade processes Jackson and FastUtil; filters/minimization remove Jackson | FastUtil only |
parquet-jackson-*.jar |
Explicitly includes Jackson artifacts | Jackson |
parquet-hadoop-bundle-*.jar |
Includes other org.apache.parquet:parquet-* JARs |
Carries their already-shaded third-party classes |
parquet-cli-*-runtime.jar |
shadedArtifactAttached=true, classifier runtime, includes all dependencies |
Full runtime dependency set |
A few edge cases:
parquet-variant,parquet-protobuf, andparquet-thriftrun Shade, but filtering and minimization leave no third-party classes in the final JARs.parquet-benchmarks.jarbundles the full dependency tree, but it is a local-only uber-JAR. The published versionedparquet-benchmarks-*.jarcontains only project classes.- The 15 source and 15 test JARs contain project output rather than dependency code.
- The 13 Javadoc JARs contain JDK-generated JavaScript and CSS. ASF policy explicitly permits these assets in Maven Javadoc JARs.
Setup and verification commands
Setup
Run the setup commands from the repository root with Java 17. Maven output is left visible so build progress and failures are easy to follow.
java -version
./mvnw clean package --batch-mode -DskipTests
./mvnw --batch-mode process-resources source:jar-no-fork javadoc:jar
Verification
After setup completes, run the verification separately:
bash <<'EOF'
set -euo pipefail
export LC_ALL=C
tmp_dir=$(mktemp -d)
trap 'rm -rf "$tmp_dir"' EXIT
candidate_jars="$tmp_dir/candidates"
third_party_jars="$tmp_dir/third-party"
javadoc_jars="$tmp_dir/javadocs"
clean_jars="$tmp_dir/clean"
: >"$third_party_jars"
: >"$javadoc_jars"
: >"$clean_jars"
find . -mindepth 3 -maxdepth 3 -type f \
-path './*/target/*.jar' \
! -name 'original-*.jar' \
! -name 'parquet-benchmarks.jar' \
-print0 >"$candidate_jars"
count_non_parquet_classes() {
jar tf "$1" | awk '
/\.class$/ {
path=$0
sub(/^META-INF\/versions\/[0-9]+\//, "", path)
if (path !~ /^org\/apache\/parquet\//) count++
}
END { print count+0 }
'
}
count_external_sources() {
jar tf "$1" | awk '
/\.java$/ && $0 !~ /^org\/apache\/parquet\// { count++ }
END { print count+0 }
'
}
count_extra_test_classes() {
local jar_file=$1
local module_dir=$2
local jar_classes="$tmp_dir/jar-classes"
local project_classes="$tmp_dir/project-classes"
jar tf "$jar_file" | awk '/\.class$/' | sort -u >"$jar_classes"
: >"$project_classes"
if [[ -d "$module_dir/target/test-classes" ]]; then
(
cd "$module_dir/target/test-classes"
find . -type f -name '*.class' -print |
sed 's#^\./##' |
sort -u
) >"$project_classes"
fi
comm -13 "$project_classes" "$jar_classes" | awk 'END { print NR+0 }'
}
printf 'Inspecting existing JARs...\n'
jar_count=0
while IFS= read -r -d '' jar_file; do
jar_count=$((jar_count + 1))
jar_name=$(basename "$jar_file")
module_dir=${jar_file%/target/*}
case "$jar_name" in
*-javadoc.jar)
unzip -p "$jar_file" script.js >"$tmp_dir/script.js"
if grep -Fq 'Oracle and/or its affiliates' "$tmp_dir/script.js"; then
printf '%s\tJDK-generated JavaScript/CSS\n' "$jar_file" >>"$javadoc_jars"
else
printf '%s\n' "$jar_file" >>"$clean_jars"
fi
continue
;;
*-sources.jar)
bundled_count=$(count_external_sources "$jar_file")
;;
*-tests.jar)
bundled_count=$(count_extra_test_classes "$jar_file" "$module_dir")
;;
*)
bundled_count=$(count_non_parquet_classes "$jar_file")
;;
esac
if [[ $bundled_count -gt 0 ]]; then
printf '%s\t%d bundled classes or sources\n' \
"$jar_file" "$bundled_count" >>"$third_party_jars"
else
printf '%s\n' "$jar_file" >>"$clean_jars"
fi
done <"$candidate_jars"
if [[ $jar_count -eq 0 ]]; then
printf 'No JARs found. Run the setup commands first.\n' >&2
exit 1
fi
printf '\n=== JARs containing bundled third-party code ===\n\n'
sort "$third_party_jars"
printf '\n=== Javadoc JARs containing JDK-provided assets ===\n\n'
sort "$javadoc_jars"
printf '\n=== JARs with no bundled third-party code found ===\n\n'
sort "$clean_jars"
printf '\nInspected %d published JARs.\n' "$jar_count"
EOF
The Java 17 build completed successfully, and the legal-file check verified all 59 JARs.
- Lingua principale
- Java
- Stelle
- 3.1k
- Fork
- 1.6k
- Merge medio
- 6g 16h
- PR unite (30g)
- 36
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di apache/parquet-java
-
Type: bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
apache/parquet-java#3792 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
apache/parquet-java#3767 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
apache/parquet-java#3695 · 1 commento ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
apache/parquet-java#3667 ·
-
Type: bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
apache/parquet-java#3574 · 1 commento ·
Tutte le issue di apache/parquet-java
Issue simili
-
area-deployment area-integrations triage:bot-seen
Difficoltà 2/5 Mezza giornata Idoneità per principianti 86/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
apache/flink-agents#1156 ·
-
[source-shopify] FAILED bulk operation without partialDataUrl is silently treated as successful Apertaarea/connectors autoteam community connectors/source/shopify needs-triage team/use type/bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 85/100