qdrant/qdrant

Support for PowerPC `ppc64le` architecture

Open

#3.940 geöffnet am 30. März 2024

Auf GitHub ansehen
 (1 Kommentar) (0 Reaktionen) (0 zugewiesene Personen)Rust (2.263 Forks)batch import
help wanted

Repository-Metriken

Stars
 (31.321 Stars)
PR-Merge-Metriken
 (Durchschn. Merge 2T 20h) (276 gemergte PRs in 30 T)

Beschreibung

Is your feature request related to a problem? Please describe. I tried building the qdrant for ppc64le architecutre using the Dockerfile provided in the repo. But i was getting error at the last step.

# Cross-compiling using Docker multi-platform builds/images and `xx`.
#
# https://docs.docker.com/build/building/multi-platform/
# https://github.com/tonistiigi/xx
FROM --platform=${BUILDPLATFORM:-linux/ppc64le} tonistiigi/xx AS xx

# Utilizing Docker layer caching with `cargo-chef`.
#
# https://www.lpalmieri.com/posts/fast-rust-docker-builds/
FROM --platform=${BUILDPLATFORM:-linux/ppc64le} rust AS chef 
# We only pay the installation cost once, 
# it will be cached from the second build onwards
RUN cargo install cargo-chef 
WORKDIR app


FROM chef AS planner
WORKDIR /qdrant
COPY . .
RUN cargo chef prepare --recipe-path recipe.json


FROM chef AS builder
WORKDIR /qdrant

COPY --from=xx / /

# Relative order of `ARG` and `RUN` commands in the Dockerfile matters.
#
# If you pass a different `ARG` to `docker build`, it would invalidate Docker layer cache
# for the next steps. (E.g., the following steps may depend on a new `ARG` value, so Docker would
# have to re-execute them instead of using a cached layer from a previous run.)
#
# Steps in this stage are ordered in a way that should maximize Docker layer cache utilization,
# so, please, don't reorder them without prior consideration. 🥲

RUN apt-get update \
    && apt-get install -y clang lld cmake protobuf-compiler jq \
    && rustup component add rustfmt

# `ARG`/`ENV` pair is a workaround for `docker build` backward-compatibility.
#
# https://github.com/docker/buildx/issues/510
ARG BUILDPLATFORM
ENV BUILDPLATFORM=${BUILDPLATFORM:-linux/ppc64le}

ARG MOLD_VERSION=2.3.2

RUN case "$BUILDPLATFORM" in \
        */amd64 ) PLATFORM=x86_64 ;; \
        */arm64 | */arm64/* ) PLATFORM=aarch64 ;; \
        */ppc64le ) PLATFORM=ppc64le ;; \   
        * ) echo "Unexpected BUILDPLATFORM '$BUILDPLATFORM'" >&2; exit 1 ;; \
    esac; \
    \
    mkdir -p /opt/mold; \
    cd /opt/mold; \
    \
    # Assuming mold has a ppc64le pre-built binary:
    TARBALL="mold-$MOLD_VERSION-$PLATFORM-linux.tar.gz"; \
    curl -sSLO "https://github.com/rui314/mold/releases/download/v$MOLD_VERSION/$TARBALL"; \
    tar -xf "$TARBALL" --strip-components 1; \
    rm "$TARBALL"


# `ARG`/`ENV` pair is a workaround for `docker build` backward-compatibility.
#
# https://github.com/docker/buildx/issues/510
ARG TARGETPLATFORM
ENV TARGETPLATFORM=${TARGETPLATFORM:-linux/ppc64le}

ARG GIT_COMMIT_ID
ENV GIT_COMMIT_ID=${GIT_COMMIT_ID}

RUN xx-apt-get install -y pkg-config gcc g++ libc6-dev libunwind-dev

# Select Cargo profile (e.g., `release`, `dev` or `ci`)
ARG PROFILE=release

# Enable crate features
ARG FEATURES

# Pass custom `RUSTFLAGS` (e.g., `--cfg tokio_unstable` to enable Tokio tracing/`tokio-console`)
ARG RUSTFLAGS

# Select linker (e.g., `mold`, `lld` or an empty string for the default linker)
ARG LINKER=mold

COPY --from=planner /qdrant/recipe.json recipe.json

RUN apt-get update && apt-get install -y libunwind-dev
RUN case "$(xx-info)" in \
        amd64 ) PLATFORM=x86_64 ;; \
        arm64 ) PLATFORM=aarch64 ;; \
        powerpc64le-linux-gnu ) PLATFORM=ppc64le ;; \
        * ) echo "Unexpected platform '$(xx-info)'" >&2; exit 1 ;; \
    esac; \
    \
    PKG_CONFIG="/usr/bin/$PLATFORM-pkg-config" \
    PATH="$PATH:/opt/mold/bin" \
    RUSTFLAGS="${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS" \
    xx-cargo chef cook --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace --recipe-path recipe.json

COPY . .
# `PKG_CONFIG=...` is a workaround for `xx-cargo` bug for crates based on `pkg-config`!
#
# https://github.com/tonistiigi/xx/issues/107
# https://github.com/tonistiigi/xx/pull/108
RUN PKG_CONFIG="/usr/bin/$(xx-info)-pkg-config" \
    PATH="$PATH:/opt/mold/bin" \
    RUSTFLAGS="${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS" \
    xx-cargo build --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace --bin qdrant \
    && PROFILE_DIR=$(if [ "$PROFILE" = dev ]; then echo debug; else echo $PROFILE; fi) \
    && mv target/$(xx-cargo --print-target-triple)/$PROFILE_DIR/qdrant /qdrant/qdrant

# Download and extract web UI
RUN mkdir /static ; STATIC_DIR='/static' ./tools/sync-web-ui.sh


FROM debian:12-slim AS qdrant

RUN apt-get update \
    && apt-get install --no-install-recommends -y ca-certificates tzdata libunwind8 \
    && rm -rf /var/lib/apt/lists/*

ARG APP=/qdrant

RUN mkdir -p "$APP"

COPY --from=builder /qdrant/qdrant "$APP"/qdrant
COPY --from=builder /qdrant/config "$APP"/config
COPY --from=builder /qdrant/tools/entrypoint.sh "$APP"/entrypoint.sh
COPY --from=builder /static "$APP"/static

WORKDIR "$APP"

ARG USER_ID=0

RUN if [ "$USER_ID" != 0 ]; then \
        groupadd --gid "$USER_ID" qdrant; \
        useradd --uid "$USER_ID" --gid "$USER_ID" -m qdrant; \
        mkdir -p "$APP"/storage "$APP"/snapshots; \
        chown -R "$USER_ID:$USER_ID" "$APP"; \
    fi

USER "$USER_ID:$USER_ID"

ENV TZ=Etc/UTC \
    RUN_MODE=production

EXPOSE 6333
EXPOSE 6334

LABEL org.opencontainers.image.title="Qdrant"
LABEL org.opencontainers.image.description="Official Qdrant image"
LABEL org.opencontainers.image.url="https://qdrant.com/"
LABEL org.opencontainers.image.documentation="https://qdrant.com/docs"
LABEL org.opencontainers.image.source="https://github.com/qdrant/qdrant"
LABEL org.opencontainers.image.vendor="Qdrant"

CMD ["./entrypoint.sh"]

The above is my changed Dockerfile for linux/ppc64le

39.80    Compiling filetime v0.2.19
39.80    Compiling nix v0.25.1
39.81    Compiling terminal_size v0.1.17
39.89 error[E0583]: file not found for module `native`
39.89   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:15:1
39.89    |
39.89 15 | mod native;
39.89    | ^^^^^^^^^^^
39.89    |
39.89    = help: to create the module `native`, create file "/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/native.rs" or "/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/native/mod.rs"
39.89    = note: if there is a `mod native` elsewhere in the crate already, import it with `use crate::...` instead
39.89 
39.90    Compiling regex v1.8.4
39.94    Compiling console v0.15.2
39.96    Compiling cexpr v0.6.0
40.02    Compiling const-random v0.1.17
40.04    Compiling rstar v0.12.0
40.04 error[E0412]: cannot find type `unw_word_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:54:18
40.04    |
40.04 54 |     pub opaque: [unw_word_t; UNW_TDEP_CURSOR_LEN as usize],
40.04    |                  ^^^^^^^^^^ not found in this scope
40.04 
40.04    Compiling approx v0.5.1
40.04    Compiling earcutr v0.4.2
40.04    Compiling ordered-float v2.10.1
40.04    Compiling num-integer v0.1.46
40.04    Compiling rmp v0.8.11
40.04 error[E0425]: cannot find value `UNW_TDEP_CURSOR_LEN` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:54:30
40.04    |
40.04 54 |     pub opaque: [unw_word_t; UNW_TDEP_CURSOR_LEN as usize],
40.04    |                              ^^^^^^^^^^^^^^^^^^^ not found in this scope
40.04 
40.04 error[E0412]: cannot find type `unw_word_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:54:18
40.04    |
40.04 54 |     pub opaque: [unw_word_t; UNW_TDEP_CURSOR_LEN as usize],
40.04    |                  ^^^^^^^^^^ not found in this scope
40.04    |
40.04 help: you might be missing a type parameter
40.04    |
40.04 53 | pub struct unw_cursor_t<unw_word_t> {
40.04    |                        ++++++++++++
40.04 
40.04 error[E0425]: cannot find value `UNW_TDEP_CURSOR_LEN` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:54:30
40.04    |
40.04 54 |     pub opaque: [unw_word_t; UNW_TDEP_CURSOR_LEN as usize],
40.04    |                              ^^^^^^^^^^^^^^^^^^^ not found in this scope
40.04    |
40.04 help: you might be missing a const parameter
40.04    |
40.04 53 | pub struct unw_cursor_t<const UNW_TDEP_CURSOR_LEN: /* Type */> {
40.04    |                        +++++++++++++++++++++++++++++++++++++++
40.04 
40.04 error[E0412]: cannot find type `unw_tdep_context_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:57:26
40.04    |
40.04 57 | pub type unw_context_t = unw_tdep_context_t;
40.04    |                          ^^^^^^^^^^^^^^^^^^ not found in this scope
40.04 
40.04 error[E0412]: cannot find type `unw_word_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:62:19
40.04    |
40.04 62 |     pub start_ip: unw_word_t,
40.04    |                   ^^^^^^^^^^ not found in this scope
40.04 
40.04 error[E0412]: cannot find type `unw_word_t` in this scope
40.04   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:63:17
40.04    |
40.04 63 |     pub end_ip: unw_word_t,
40.04    |                 ^^^^^^^^^^ not found in this scope
40.04 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:64:15
40.05    |
40.05 64 |     pub lsda: unw_word_t,
40.05    |               ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:65:18
40.05    |
40.05 65 |     pub handler: unw_word_t,
40.05    |                  ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:66:13
40.05    |
40.05 66 |     pub gp: unw_word_t,
40.05    |             ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:67:16
40.05    |
40.05 67 |     pub flags: unw_word_t,
40.05    |                ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_tdep_proc_info_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:71:16
40.05    |
40.05 71 |     pub extra: unw_tdep_proc_info_t,
40.05    |                ^^^^^^^^^^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:62:19
40.05    |
40.05 62 |     pub start_ip: unw_word_t,
40.05    |                   ^^^^^^^^^^ not found in this scope
40.05    |
40.05 help: you might be missing a type parameter
40.05    |
40.05 61 | pub struct unw_proc_info_t<unw_word_t> {
40.05    |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_tdep_proc_info_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:71:16
40.05    |
40.05 71 |     pub extra: unw_tdep_proc_info_t,
40.05    |                ^^^^^^^^^^^^^^^^^^^^ not found in this scope
40.05    |
40.05 help: you might be missing a type parameter
40.05    |
40.05 61 | pub struct unw_proc_info_t<unw_tdep_proc_info_t> {
40.05    |                           ++++++++++++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:79:17
40.05    |
40.05 79 |             ip: unw_word_t,
40.05    |                 ^^^^^^^^^^ not found in this scope
40.05    |
40.05 help: you might be missing a type parameter
40.05    |
40.05 75 | pub struct unw_accessors_t<unw_word_t> {
40.05    |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05   --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:95:25
40.05    |
40.05 95 |             dilap: *mut unw_word_t,
40.05    |                         ^^^^^^^^^^ not found in this scope
40.05    |
40.05 help: you might be missing a type parameter
40.05    |
40.05 75 | pub struct unw_accessors_t<unw_word_t> {
40.05    |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:102:19
40.05     |
40.05 102 |             addr: unw_word_t,
40.05     |                   ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:103:24
40.05     |
40.05 103 |             valp: *mut unw_word_t,
40.05     |                        ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:112:24
40.05     |
40.05 112 |             valp: *mut unw_word_t,
40.05     |                        ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:129:19
40.05     |
40.05 129 |             addr: unw_word_t,
40.05     |                   ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:132:24
40.05     |
40.05 132 |             offp: *mut unw_word_t,
40.05     |                        ^^^^^^^^^^ not found in this scope
40.05     |
40.05 help: you might be missing a type parameter
40.05     |
40.05 75  | pub struct unw_accessors_t<unw_word_t> {
40.05     |                           ++++++++++++
40.05 
40.05 error[E0412]: cannot find type `unw_word_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:140:15
40.05     |
40.05 140 |     pub addr: unw_word_t,
40.05     |               ^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0412]: cannot find type `unw_tdep_save_loc_t` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:148:16
40.05     |
40.05 148 |     pub extra: unw_tdep_save_loc_t,
40.05     |                ^^^^^^^^^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0425]: cannot find value `UNW_TDEP_IP` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:151:31
40.05     |
40.05 151 | pub const UNW_REG_IP: c_int = UNW_TDEP_IP;
40.05     |                               ^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0425]: cannot find value `UNW_TDEP_SP` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:152:31
40.05     |
40.05 152 | pub const UNW_REG_SP: c_int = UNW_TDEP_SP;
40.05     |                               ^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0425]: cannot find value `UNW_TDEP_EH` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:153:31
40.05     |
40.05 153 | pub const UNW_REG_EH: c_int = UNW_TDEP_EH;
40.05     |                               ^^^^^^^^^^^ not found in this scope
40.05 
40.05 error[E0425]: cannot find value `UNW_TDEP_LAST_REG` in this scope
40.05    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/unwind-sys-0.1.3/src/lib.rs:154:33
40.05     |
40.05 154 | pub const UNW_REG_LAST: c_int = UNW_TDEP_LAST_REG;
40.05     |                                 ^^^^^^^^^^^^^^^^^ not found in this scope
40.05 
40.06    Compiling dlv-list v0.5.2
40.07    Compiling rand_core v0.5.1
40.08 Some errors have detailed explanations: E0412, E0425, E0583.
40.08 For more information about an error, try `rustc --explain E0412`.
40.08    Compiling tar v0.4.40
40.08 error: could not compile `unwind-sys` (lib) due to 28 previous errors
40.08 warning: build failed, waiting for other jobs to finish...
55.35 thread 'main' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cargo-chef-0.1.66/src/recipe.rs:218:27:
55.35 Exited with status code: 101
55.35 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
------
Dockerfile:92
--------------------
  91 |     RUN apt-get update && apt-get install -y libunwind-dev
  92 | >>> RUN case "$(xx-info)" in \
  93 | >>>         amd64 ) PLATFORM=x86_64 ;; \
  94 | >>>         arm64 ) PLATFORM=aarch64 ;; \
  95 | >>>         powerpc64le-linux-gnu ) PLATFORM=ppc64le ;; \
  96 | >>>         * ) echo "Unexpected platform '$(xx-info)'" >&2; exit 1 ;; \
  97 | >>>     esac; \
  98 | >>>     \
  99 | >>>     PKG_CONFIG="/usr/bin/$PLATFORM-pkg-config" \
 100 | >>>     PATH="$PATH:/opt/mold/bin" \
 101 | >>>     RUSTFLAGS="${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS" \
 102 | >>>     xx-cargo chef cook --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace --recipe-path recipe.json
 103 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c case \"$(xx-info)\" in         amd64 ) PLATFORM=x86_64 ;;         arm64 ) PLATFORM=aarch64 ;;         powerpc64le-linux-gnu ) PLATFORM=ppc64le ;;         * ) echo \"Unexpected platform '$(xx-info)'\" >&2; exit 1 ;;     esac;         PKG_CONFIG=\"/usr/bin/$PLATFORM-pkg-config\"     PATH=\"$PATH:/opt/mold/bin\"     RUSTFLAGS=\"${LINKER:+-C link-arg=-fuse-ld=}$LINKER $RUSTFLAGS\"     xx-cargo chef cook --profile $PROFILE ${FEATURES:+--features} $FEATURES --features=stacktrace --recipe-path recipe.json" did not complete successfully: exit code: 101

Can you help me with doing necessary changes so qdrant will be supported in ppc64le arch

Contributor Guide