Failed to download from nim/nvidia/nv-embedqa-e5-v5:5_tokenizer

Bắt đầu với docker-compose.yaml, ../../local_deploy/docker-compose-nim-ms.yaml và cấu hình image nv-embedqa-e5-v5:1.0.1. Kiểm tra lý do khi khởi động lại yêu cầu nim/nvidia/nv-embedqa-e5-v5:5_tokenizer và liệu tài nguyên NGC được tham chiếu cùng phiên bản image có còn tương thích hay không. Được xem là hoàn tất khi dịch vụ khởi động thành công với các tài nguyên model được hỗ trợ hoặc cấu hình bắt buộc và phiên bản được hỗ trợ đã được ghi lại.

I am experiencing an issue where the nemo-retriever-embedding-microservice container fails to download the required model files during startup. Here are the details:

Command:
$ docker compose -f docker-compose.yaml -f ../../local_deploy/docker-compose-nim-ms.yaml -f ../../local_deploy/docker-compose-vectordb.yaml --profile local-nim up

Error Logs:

nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | =========================================
nemo-retriever-embedding-microservice | == NVIDIA Retriever Text Embedding NIM ==
nemo-retriever-embedding-microservice | =========================================
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | NVIDIA Release 1.0.1 (build 0cf1c7ad8e51bdffc4e4d4226735dbb7f59d70d4)
nemo-retriever-embedding-microservice | Model: nvidia/nv-embedqa-e5-v5
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | Container image Copyright (c) 2016-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
nemo-retriever-embedding-microservice | This NIM container is governed by the NVIDIA AI Product Agreement here:
nemo-retriever-embedding-microservice | https://www.nvidia.com/en-us/data-center/products/nvidia-ai-enterprise/eula/.
nemo-retriever-embedding-microservice | A copy of this license can be found under /opt/nim/LICENSE.
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | The use of this model is governed by the AI Foundation Models Community License
nemo-retriever-embedding-microservice | here: https://docs.nvidia.com/ai-foundation-models-community-license.pdf.
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | OTEL Logging handler requested, but Python logging auto-instrumentation not set up. Set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true to enable logging auto-instrumentation.
nemo-retriever-embedding-microservice | downloading nim/nvidia/nv-embedqa-e5-v5:5_tokenizer
nemo-retriever-embedding-microservice | This could take a while.
nemo-retriever-embedding-microservice | --2025-01-22 23:34:03-- https://api.ngc.nvidia.com/v2/org/nim/team/nvidia/models/nv-embedqa-e5-v5/versions/5_tokenizer/zip
nemo-retriever-embedding-microservice | Resolving api.ngc.nvidia.com (api.ngc.nvidia.com)... 54.68.27.61, 44.227.231.63
nemo-retriever-embedding-microservice | Connecting to api.ngc.nvidia.com (api.ngc.nvidia.com)|54.68.27.61|:443... connected.
nemo-retriever-embedding-microservice | HTTP request sent, awaiting response... 410 Gone
nemo-retriever-embedding-microservice | 2025-01-22 23:34:04 ERROR 410: Gone.
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice | 2025-01-22T23:34:04Z WARNING: tools.nim.ngc_models - Failed to download from nim/nvidia/nv-embedqa-e5-v5:5_tokenizer: Failure from NGC CLI -
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice |
nemo-retriever-embedding-microservice exited with code 1

Environment:

• Docker Compose Version: v2.5.0

Issue:
The nemo-retriever-embedding-microservice service uses the image nvcr.io/nim/nvidia/nv-embedqa-e5-v5:1.0.1. However, during startup, it attempts to download the resource nim/nvidia/nv-embedqa-e5-v5:5_tokenizer from NVIDIA NGC. The request fails with a 410 Gone error, indicating that the resource might no longer be available.

Questions:
1.Why is the nvcr.io/nim/nvidia/nv-embedqa-e5-v5:1.0.1 image trying to download the 5_tokenizer resource?
2.Has this resource (nim/nvidia/nv-embedqa-e5-v5:5_tokenizer) been deprecated or removed from NGC?
3.Should I update the image version or modify the service configuration to resolve this issue?
4.Is there an alternative approach to pre-download the required models and avoid runtime dependency issues?
Any guidance on resolving this issue would be greatly appreciated.

Thank you!