arm64 and x86_64 linux: TF java full native builds are failing to find the native headers

レポートに記載されている tensorflow-core のネイティブファイルと、v1.0.0-rc.1 のネイティブビルド用 Maven/Bazel コマンドから始めます。Linux arm64 でヘッダーが見つからないエラーを再現し、その後、報告されている x86_64 のケースとビルドの挙動を比較します。ネイティブ Java ビルドが依存関係を解決し、正常に完了すれば完了です。

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04 x86_64): Linux Ubuntu 22.04, aarch64
• TensorFlow installed from (source or binary): Source
• TensorFlow version: TF version is v2.16.1, TF java version is tag: v1.0.0-rc.1
• Java version (i.e., the output of java -version): openjdk 11.0.23 2024-04-16
• Java command line flags (e.g., GC parameters): mvn install -P native-build -Dbazel.build.flags='--verbose_failures -s --config=mkl_aarch64_threadpool' -X
• Installed from Maven Central?: No
• Bazel version (if compiling from source): bazel 6.5.0
• GCC/Compiler version (if compiling from source):gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
• CUDA/cuDNN version: Not applicable
• GPU model and memory: Not applicable

Describe the problem
TensorFlow java source builds are failing on aarch64 linux system with the missing native headers. please let me know how it's built for x86_64 linux platform.

based on my debugging so far it looks like the dependency comes from this commit which added C API extension for custom gradient functions, and introduced these headers and .cc which requires several third_party libraries from tensorflow native but none of those bazel workspaces are cloned.

tensorflow-core/tensorflow-core-native/src/main/native/org/tensorflow/internal/c_api$ 
tfj_gradients.h  tfj_gradients_impl.cc  tfj_graph.h  tfj_graph_impl.cc  tfj_scope.h  tfj_scope_impl.cc

I tried to manually clone the missing workspaces into bazel cache, but the cycle is never ending, it's missing tsl, eigen, ml_dtype, absl, protobuf, and now compiled headers for protobuf....

Provide the exact sequence of commands / steps that you executed before running into the problem

sudo apt-get install pkg-config ccache clang ant python3-pip swig git file wget unzip tar bzip2 gzip patch autoconf-archive autogen automake make cmake libtool bison flex perl nasm curl gfortran libasound2-dev freeglut3-dev libgtk2.0-dev libusb-dev zlib1g libffi-dev libbz2-dev zlib1g-dev

sudo apt install maven default-jdk

cd $HOME
mkdir bazel
cd bazel
wget https://github.com/bazelbuild/bazel/releases/download/6.5.0/bazel-6.5.0-linux-arm64
mv bazel-6.5.0-linux-arm64 bazel
chmod a+x bazel
export PATH=/home/ubuntu/bazel/:$PATH

# Build and install javacpp-presets.
# Clone the following forked repo to exclude the libraries that are not supported and not required
git clone https://github.com/snadampal/javacpp-presets.git
cd javacpp-presets
git checkout tfjava_aarch64
mvn install -Djavacpp.platform=linux-arm64 -Dmaven.javadoc.skip=true -X -T 16

# Build and install tensorflow java bindings
git clone https://github.com/tensorflow/java.git
cd java
git checkout v1.0.0-rc.1
mvn install -P native-build -Dbazel.build.flags='--verbose_failures -s --config=mkl_aarch64_threadpool' -X

Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.