qdrant/qdrant

Support for PowerPC `ppc64le` architecture

Open

#3940 aperta il 30 mar 2024

Vedi su GitHub
 (1 commento) (0 reazioni) (0 assegnatari)Rust (2263 fork)batch import
help wanted

Metriche repository

Star
 (31.321 star)
Metriche merge PR
 (Merge medio 2g 20h) (276 PR mergiate in 30 g)

Descrizione

Is your feature request related to a problem? Please describe. I tried building the qdrant for ppc64le architecutre using the Dockerfile provided in the repo. But i was getting error at the last step.

# Cross-compiling using Docker multi-platform builds/images and `xx`.
#
# https://docs.docker.com/build/building/multi-platform/
# https://github.com/tonistiigi/xx
FROM --platform=${BUILDPLATFORM:-linux/ppc64le} tonistiigi/xx AS xx

# Utilizing Docker layer caching with `cargo-chef`.
#
# https://www.lpalmieri.com/posts/fast-rust-docker-builds/
FROM --platform=${BUILDPLATFORM:-linux/ppc64le} rust AS chef 
# We only pay the installation cost once, 
# it will be cached from the second build onwards
RUN cargo install cargo-chef 
WORKDIR app


FROM chef AS planner
WORKDIR /qdrant
COPY . .
RUN cargo chef prepare --recipe-path recipe.json


FROM chef AS builder
WORKDIR /qdrant

COPY --from=xx / /

# Relative order of `ARG` and `RUN` commands in the Dockerfile matters.
#
# If you pass a different `ARG` to `docker build`, it would invalidate Docker layer cache
# for the next steps. (E.g., the following steps may depend on a new `ARG` value, so Docker would
# have to re-execute them instead of using a cached layer from a previous run.)
#
# Steps in this stage are ordered in a way that should maximize Docker layer cache utilization,
# so, please, don't reorder them without prior consideration. 🥲

RUN apt-get update \
    && apt-get install -y clang lld cmake protobuf-compiler jq \
    && rustup component add rustfmt

# `ARG`/`ENV` pair is a workaround for `docker build` backward-compatibility.
#
# https://github.com/docker/buildx/issues/510
ARG BUILDPLATFORM
ENV BUILDPLATFORM=${BUILDPLATFORM:-linux/ppc64le}

ARG MOLD_VERSION=2.3.2

RUN case "$BUILDPLATFORM" in \
        */amd64 ) PLATFORM=x86_64 ;; \
        */arm64 | */arm64/* ) PLATFORM=aarch64 ;; \
        */ppc64le ) PLATFORM=ppc64le ;; \   
        * ) echo "Unexpected BUILDPLATFORM '$BUILDPLATFORM'" >&2; exit 1 ;; \
    esac; \
    \
    mkdir -p /opt/mold; \
    cd /opt/mold; \
    \
    # Assuming mold has a ppc64le pre-built binary:
    TARBALL="mold-$MOLD_VERSION-$PLATFORM-linux.tar.gz"; \
    curl -sSLO "https://github.com/rui314/mold/releases/download/v$MOLD_VERSION/$TARBALL"; \
    tar -xf "$TARBALL" --strip-components 1; \
    rm "$TARBALL"


# `ARG`/`ENV` pair is a workaround for `docker build` backward-compatibility.
#
# https://github.com/docker/buildx/issues/510
ARG TARGETPLATFORM
ENV TARGETPLATFORM=${TARGETPLATFORM:-linux/ppc64le}

ARG GIT_COMMIT_ID
ENV GIT_COMMIT_ID=${GIT_COMMIT_ID}

RUN xx-apt-get install -y pkg-config gcc g++ libc6-dev libunwind-dev

# Select Cargo profile (e.g., `release`, `dev` or `ci`)
ARG PROFILE=release

# Enable crate features
ARG FEATURES

# Pass custom `RUSTFLAGS` (e.g., `--cfg tokio_unstable` to enable Tokio tracing/`tokio-console`)
ARG RUSTFLAGS

# Select linker (e.g., `mold`, `lld` or an empty string for the default linker)
ARG LINKER=mold

COPY --from=planner /qdrant/recipe.json recipe.json

RUN apt-get update && apt-get install -y libunwind-dev
RUN case "$(xx-info)" in \
        amd64 ) PLATFORM=x86_64 ;; \
        arm64 ) PLATFORM=aarch64 ;; \
        powerpc64le-linux-gnu ) PLATFORM=ppc64le ;; \
        * ) echo "Unexpected platform '$(xx-info)'" >&2; exit 1 ;; \
    esac; \
    \
    PKG_CONFIG="/usr/bin/$PLATFORM-pkg-config" \
    PATH="$PATH:/opt/mold/bin" \
    RUSTFLAGS="${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS" \
    xx-cargo chef cook --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace --recipe-path recipe.json

COPY . .
# `PKG_CONFIG=...` is a workaround for `xx-cargo` bug for crates based on `pkg-config`!
#
# https://github.com/tonistiigi/xx/issues/107
# https://github.com/tonistiigi/xx/pull/108
RUN PKG_CONFIG="/usr/bin/$(xx-info)-pkg-config" \
    PATH="$PATH:/opt/mold/bin" \
    RUSTFLAGS="${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS" \
    xx-cargo build --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace --bin qdrant \
    && PROFILE_DIR=$(if [ "$PROFILE" = dev ]; then echo debug; else echo $PROFILE; fi) \
    && mv target/$(xx-cargo --print-target-triple)/$PROFILE_DIR/qdrant /qdrant/qdrant

# Download and extract web UI
RUN mkdir /static ; STATIC_DIR='/static' ./tools/sync-web-ui.sh


FROM debian:12-slim AS qdrant

RUN apt-get update \
    && apt-get install --no-install-recommends -y ca-certificates tzdata libunwind8 \
    && rm -rf /var/lib/apt/lists/*

ARG APP=/qdrant

RUN mkdir -p "$APP"

COPY --from=builder /qdrant/qdrant "$APP"/qdrant
COPY --from=builder /qdrant/config "$APP"/config
COPY --from=builder /qdrant/tools/entrypoint.sh "$APP"/entrypoint.sh
COPY --from=builder /static "$APP"/static

WORKDIR "$APP"

ARG USER_ID=0

RUN if [ "$USER_ID" != 0 ]; then \
        groupadd --gid "$USER_ID" qdrant; \
        useradd --uid "$USER_ID" --gid "$USER_ID" -m qdrant; \
        mkdir -p "$APP"/storage "$APP"/snapshots; \
        chown -R "$USER_ID:$USER_ID" "$APP"; \
    fi

USER "$USER_ID:$USER_ID"

ENV TZ=Etc/UTC \
    RUN_MODE=production

EXPOSE 6333
EXPOSE 6334

LABEL org.opencontainers.image.title="Qdrant"
LABEL org.opencontainers.image.description="Official Qdrant image"
LABEL org.opencontainers.image.url="https://qdrant.com/"
LABEL org.opencontainers.image.documentation="https://qdrant.com/docs"
LABEL org.opencontainers.image.source="https://github.com/qdrant/qdrant"
LABEL org.opencontainers.image.vendor="Qdrant"

CMD ["./entrypoint.sh"]

The above is my changed Dockerfile for linux/ppc64le

39.80    Compiling filetime v0.2.19
39.80    Compiling nix v0.25.1
39.81    Compiling terminal_size v0.1.17
39.89 error[E0583]: file not found for module `native`
39.89   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:15:1
39.89    |
39.89 15 | mod native;
39.89    | ^^^^^^^^^^^
39.89    |
39.89    = help: to create the module `native`, create file "/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/native.rs" or "/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/native/mod.rs"
39.89    = note: if there is a `mod native` elsewhere in the crate already, import it with `use crate::...` instead
39.89 
39.90    Compiling regex v1.8.4
39.94    Compiling console v0.15.2
39.96    Compiling cexpr v0.6.0
40.02    Compiling const-random v0.1.17
40.04    Compiling rstar v0.12.0
40.04 error[E0412]: cannot find type `unw_word_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:54:18
40.04    |
40.04 54 |     pub opaque: [unw_word_t; UNW_TDEP_CURSOR_LEN as usize],
40.04    |                  ^^^^^^^^^^ not found in this scope
40.04 
40.04    Compiling approx v0.5.1
40.04    Compiling earcutr v0.4.2
40.04    Compiling ordered-float v2.10.1
40.04    Compiling num-integer v0.1.46
40.04    Compiling rmp v0.8.11
40.04 error[E0425]: cannot find value `UNW_TDEP_CURSOR_LEN` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:54:30
40.04    |
40.04 54 |     pub opaque: [unw_word_t; UNW_TDEP_CURSOR_LEN as usize],
40.04    |                              ^^^^^^^^^^^^^^^^^^^ not found in this scope
40.04 
40.04 error[E0412]: cannot find type `unw_word_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:54:18
40.04    |
40.04 54 |     pub opaque: [unw_word_t; UNW_TDEP_CURSOR_LEN as usize],
40.04    |                  ^^^^^^^^^^ not found in this scope
40.04    |
40.04 help: you might be missing a type parameter
40.04    |
40.04 53 | pub struct unw_cursor_t<unw_word_t> {
40.04    |                        ++++++++++++
40.04 
40.04 error[E0425]: cannot find value `UNW_TDEP_CURSOR_LEN` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:54:30
40.04    |
40.04 54 |     pub opaque: [unw_word_t; UNW_TDEP_CURSOR_LEN as usize],
40.04    |                              ^^^^^^^^^^^^^^^^^^^ not found in this scope
40.04    |
40.04 help: you might be missing a const parameter
40.04    |
40.04 53 | pub struct unw_cursor_t<const UNW_TDEP_CURSOR_LEN: /* Type */> {
40.04    |                        +++++++++++++++++++++++++++++++++++++++
40.04 
40.04 error[E0412]: cannot find type `unw_tdep_context_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:57:26
40.04    |
40.04 57 | pub type unw_context_t = unw_tdep_context_t;
40.04    |                          ^^^^^^^^^^^^^^^^^^ not found in this scope
40.04 
40.04 error[E0412]: cannot find type `unw_word_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:62:19
40.04    |
40.04 62 |     pub start_ip: unw_word_t,
40.04    |                   ^^^^^^^^^^ not found in this scope
40.04 
40.04 error[E0412]: cannot find type `unw_word_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:63:17
40.04    |
40.04 63 |     pub end_ip: unw_word_t,
40.04    |                 ^^^^^^^^^^ not found in this scope
40.04 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:64:15
40.05    |
40.05 64 |     pub lsda: unw_word_t,
40.05    |               ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:65:18
40.05    |
40.05 65 |     pub handler: unw_word_t,
40.05    |                  ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:66:13
40.05    |
40.05 66 |     pub gp: unw_word_t,
40.05    |             ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:67:16
40.05    |
40.05 67 |     pub flags: unw_word_t,
40.05    |                ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_tdep_proc_info_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:71:16
40.05    |
40.05 71 |     pub extra: unw_tdep_proc_info_t,
40.05    |                ^^^^^^^^^^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:62:19
40.05    |
40.05 62 |     pub start_ip: unw_word_t,
40.05    |                   ^^^^^^^^^^ not found in this scope
40.05    |
40.05 help: you might be missing a type parameter
40.05    |
40.05 61 | pub struct unw_proc_info_t<unw_word_t> {
40.05    |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_tdep_proc_info_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:71:16
40.05    |
40.05 71 |     pub extra: unw_tdep_proc_info_t,
40.05    |                ^^^^^^^^^^^^^^^^^^^^ not found in this scope
40.05    |
40.05 help: you might be missing a type parameter
40.05    |
40.05 61 | pub struct unw_proc_info_t<unw_tdep_proc_info_t> {
40.05    |                           ++++++++++++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:79:17
40.05    |
40.05 79 |             ip: unw_word_t,
40.05    |                 ^^^^^^^^^^ not found in this scope
40.05    |
40.05 help: you might be missing a type parameter
40.05    |
40.05 75 | pub struct unw_accessors_t<unw_word_t> {
40.05    |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:95:25
40.05    |
40.05 95 |             dilap: *mut unw_word_t,
40.05    |                         ^^^^^^^^^^ not found in this scope
40.05    |
40.05 help: you might be missing a type parameter
40.05    |
40.05 75 | pub struct unw_accessors_t<unw_word_t> {
40.05    |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:102:19
40.05     |
40.05 102 |             addr: unw_word_t,
40.05     |                   ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:103:24
40.05     |
40.05 103 |             valp: *mut unw_word_t,
40.05     |                        ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:112:24
40.05     |
40.05 112 |             valp: *mut unw_word_t,
40.05     |                        ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:129:19
40.05     |
40.05 129 |             addr: unw_word_t,
40.05     |                   ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:132:24
40.05     |
40.05 132 |             offp: *mut unw_word_t,
40.05     |                        ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:140:15
40.05     |
40.05 140 |     pub addr: unw_word_t,
40.05     |               ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_tdep_save_loc_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:148:16
40.05     |
40.05 148 |     pub extra: unw_tdep_save_loc_t,
40.05     |                ^^^^^^^^^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0425]: cannot find value `UNW_TDEP_IP` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:151:31
40.05     |
40.05 151 | pub const UNW_REG_IP: c_int = UNW_TDEP_IP;
40.05     |                               ^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0425]: cannot find value `UNW_TDEP_SP` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:152:31
40.05     |
40.05 152 | pub const UNW_REG_SP: c_int = UNW_TDEP_SP;
40.05     |                               ^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0425]: cannot find value `UNW_TDEP_EH` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:153:31
40.05     |
40.05 153 | pub const UNW_REG_EH: c_int = UNW_TDEP_EH;
40.05     |                               ^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0425]: cannot find value `UNW_TDEP_LAST_REG` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:154:33
40.05     |
40.05 154 | pub const UNW_REG_LAST: c_int = UNW_TDEP_LAST_REG;
40.05     |                                 ^^^^^^^^^^^^^^^^^ not found in this scope
40.05 
40.06    Compiling dlv-list v0.5.2
40.07    Compiling rand_core v0.5.1
40.08 Some errors have detailed explanations: E0412, E0425, E0583.
40.08 For more information about an error, try `rustc --explain E0412`.
40.08    Compiling tar v0.4.40
40.08 error: could not compile `unwind-sys` (lib) due to 28 previous errors
40.08 warning: build failed, waiting for other jobs to finish...
55.35 thread 'main' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cargo-chef-0.1.66/src/recipe.rs:218:27:
55.35 Exited with status code: 101
55.35 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
------
Dockerfile:92
--------------------
  91 |     RUN apt-get update && apt-get install -y libunwind-dev
  92 | >>> RUN case "$(xx-info)" in \
  93 | >>>         amd64 ) PLATFORM=x86_64 ;; \
  94 | >>>         arm64 ) PLATFORM=aarch64 ;; \
  95 | >>>         powerpc64le-linux-gnu ) PLATFORM=ppc64le ;; \
  96 | >>>         * ) echo "Unexpected platform '$(xx-info)'" >&2; exit 1 ;; \
  97 | >>>     esac; \
  98 | >>>     \
  99 | >>>     PKG_CONFIG="/usr/bin/$PLATFORM-pkg-config" \
 100 | >>>     PATH="$PATH:/opt/mold/bin" \
 101 | >>>     RUSTFLAGS="${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS" \
 102 | >>>     xx-cargo chef cook --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace --recipe-path recipe.json
 103 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c case \"$(xx-info)\" in         amd64 ) PLATFORM=x86_64 ;;         arm64 ) PLATFORM=aarch64 ;;         powerpc64le-linux-gnu ) PLATFORM=ppc64le ;;         * ) echo \"Unexpected platform '$(xx-info)'\" >&2; exit 1 ;;     esac;         PKG_CONFIG=\"/usr/bin/$PLATFORM-pkg-config\"     PATH=\"$PATH:/opt/mold/bin\"     RUSTFLAGS=\"${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS\"     xx-cargo chef cook --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace --recipe-path recipe.json" did not complete successfully: exit code: 101

Can you help me with doing necessary changes so qdrant will be supported in ppc64le arch

Guida contributor