Kubernetes instance pass own request resource and ingnore InstantceType configuration

未關閉

#1,900 建立於 2023年4月19日 0 則留言 1 個 reaction 已指派 0 人在 GitHub 檢視

還沒有人認領這個 Issue。

評估

難度: 4/5
預估耗時: 3-5 天
新手友好度: 25/100

Issue 類型: 缺陷
描述清晰度: 基本清楚
活躍度: 停滯
技術堆疊: azure, kubernetes
領域: cloud, infrastructure, machine-learning

研究方向

首先，使用提供的 InstanceType YAML 重現 AKS 部署，並檢查產生的 Pod manifest，尤其是 inference-server 的資源。比較已設定的 requests 與最後產生的 100m CPU 和 512Mi 記憶體值；當部署 Pod 使用 InstanceType 的 request 值，同時保留所示的 limits 和 node selector 時，即表示完成。

由索引模型根據 Issue 內容生成。

描述

I am creating instance type configuration in AKS cluster.
Enable autoscale for nodepools gpuproc2 and cpuproc2.
When deployment is creating configuration spec.resources.requests from defined Instance type is ignored and pod are created with other values.
Values are seeing like hardcoded values cpu 100m and memory 512Mi.
Until this will working properly I can't allocate pods on different nodes from same nodepool group.

---
apiVersion: amlarc.azureml.com/v1alpha1
kind: InstanceType
metadata:
  name: standard-nc4
spec:
  nodeSelector:
    agentpool: gpuproc2
  resources:
    limits:
      cpu: "1"
      memory: "20Gi"
    requests:
      cpu: "700m"
      memory: "1900Mi"
---
apiVersion: amlarc.azureml.com/v1alpha1
kind: InstanceType
metadata:
  name: standard-cpuproc
spec:
  nodeSelector:
    agentpool: cpuproc2
  resources:
    limits:
      cpu: "3"
      memory: "15Gi"
    requests:
      cpu: "1100m"
      memory: "1500Mi"

Pod created by azure workspace deployment:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2023-04-19T04:06:00Z"
  generateName: deployname-134205-model-name-79c786f5c-
  labels:
    azuremlappname: deployname-134205-model-name
    isazuremlapp: "true"
    ml.azure.com/compute: aksnemlflowprod
    ml.azure.com/deployment-name: deployname-134205
    ml.azure.com/endpoint-name: model-name
    ml.azure.com/identity: deployname-134205-model-name
    ml.azure.com/resource-group: smarteye-ml-ne-prod
    ml.azure.com/scrape-metrics: "true"
    ml.azure.com/subscription-id: 89f996fd-5276-46d1-87c6-64544e672483
    ml.azure.com/workspace: ne-mlflow-prod
    pod-template-hash: 79c786f5c
  name: deployname-134205-model-name-79c786f5c-sd2ch
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: deployname-134205-model-name-79c786f5c
    uid: b72dae92-2086-44fa-899b-9aadedc0a1ec
  resourceVersion: "29057046"
  uid: c6ff4522-57a9-4cb9-963e-f8064a9daac8
spec:
  automountServiceAccountToken: true
  containers:
  - command:
    - runsvdir
    - /var/runit
    env:
    - name: AML_APP_ROOT
      value: /var/azureml-app/product-detection
    - name: AZUREML_ENTRY_SCRIPT
      value: azureml_score.py
    - name: AZUREML_MODEL_DIR
      value: /var/azureml-app/azureml-models/model_name/5
    - name: AZURE_STORAGE_CONNECTION_STRING
      value: SharedAccessSignature=;BlobEndpoint=https://account.blob.core.windows.net/;
    - name: CELERY_WORKER_QUEUE
      value: queue_name
    - name: CLASSIFICATION_MODEL_NAME
      value: model_name
    - name: CLASSIFICATION_MODEL_ROOT
      value: /opt/project/models
    - name: DEFAULT_CONSUMER
      value: InferenceDetector
    - name: DEFAULT_CONSUMER_MODULE
      value: model_extractor.queue_consumer
    - name: DETECTOR_MODEL_NAME
      value: name_of_package
    - name: DETECTOR_MODEL_PACKAGE
      value: https://url.tld/file
    - name: DETECTOR_MODEL_ROOT
      value: /opt/project/models
    - name: GPU_MEMORY_LIMIT
      value: "4096"
    - name: MLFLOW_MODEL_FOLDER
      value: /var/azureml-app/azureml-models
    - name: PREDICT_BATCH_SIZE
      value: "4"
    - name: SERVICE_NAME
      value: model-name
    - name: SERVICE_PATH_PREFIX
      value: api/v1/endpoint/model-name
    - name: TF_GPU_ALLOCATOR
      value: cuda_malloc_async
    - name: MSI_SECRET
      valueFrom:
        secretKeyRef:
          key: secret
          name: deployname-134205-model-name-sidecar
    - name: IDENTITY_HEADER
      valueFrom:
        secretKeyRef:
          key: secret
          name: deployname-134205-model-name-sidecar
    - name: MSI_ENDPOINT
      value: http://localhost:9999/token
    - name: IDENTITY_ENDPOINT
      value: http://localhost:9999/token
    image: registry.azurecr.io/image/path:v3.6.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 30
      httpGet:
        path: /
        port: 5001
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 180
    name: inference-server
    ports:
    - containerPort: 5001
      protocol: TCP
    readinessProbe:
      failureThreshold: 30
      httpGet:
        path: /
        port: 5001
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 2
    resources:
      limits:
        cpu: "1"
        memory: 20Gi
      requests:
        cpu: 100m
        memory: 512Mi
    securityContext:
      privileged: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/azureml-app
      name: model-mount-0
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wjf8j
      readOnly: true
  - args:
    - --identity
    - /identity/identity_secret.json
    - --token
    - /identity/sidecar_token
    - --ca
    - /identity/ca.crt
    - --remote-host
    - https://amlarc-identity-proxy-service.azureml.svc.cluster.local
    - --port
    - "9999"
    - --path
    - /token
    - --remote-path
    - /token
    env:
    - name: MSI_SECRET
      valueFrom:
        secretKeyRef:
          key: secret
          name: deployname-134205-model-name-sidecar
    image: mcr.microsoft.com/azureml/amlarc/docker/identity-sidecar:1.1.25
    imagePullPolicy: IfNotPresent
    name: identity-sidecar
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
      requests:
        cpu: 100m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /identity
      name: identity-secret-volume
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wjf8j
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: deployname-134205-model-name-registry.azurecr.io
  initContainers:
  - env:
    - name: STORAGE_MANIFEST_URL
      value: https://account.blob.core.windows.net/azureml/model-name_deployname-134205-model-name_model_config_map.json_f455169bb7e34736bc0c618741641388
    - name: STORAGE_DOWNLOAD_PATH
      value: /var/azureml-app
    - name: STORAGE_CREDENTIAL_CLIENTID
      value: uuid
    - name: STORAGE_CREDENTIAL_TYPE
      value: Amlarc
    - name: STORAGE_CREDENTIAL_ENDPOINT
      value: https://amlarc-identity-proxy-service.azureml.svc.cluster.local/token
    - name: STORAGE_CREDENTIAL_TOKENFILE
      value: /identity/sidecar_token
    image: mcr.microsoft.com/mir/mir-storageinitializer:46571814.1631244300887
    imagePullPolicy: IfNotPresent
    name: storageinitializer-modeldata
    resources:
      limits:
        cpu: 100m
        memory: 500Mi
      requests:
        cpu: 100m
        memory: 500Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/azureml-app
      name: model-mount-0
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wjf8j
      readOnly: true
    - mountPath: /identity
      name: identity-secret-volume
      readOnly: true
  nodeName: aks-gpuproc2-28077253-vmss00000n
  nodeSelector:
    agentpool: gpuproc2
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: deployname-134205-model-name
  serviceAccountName: deployname-134205-model-name
  terminationGracePeriodSeconds: 30
  tolerations:
  - key: ml.azure.com/amlarc
    operator: Equal
    value: "true"
  - key: ml.azure.com/amlarc-workload
    operator: Equal
    value: "true"
  - key: ml.azure.com/resource-group
    operator: Equal
    value: smarteye-ml-ne-prod
  - key: ml.azure.com/workspace
    operator: Equal
    value: ne-mlflow-prod
  - key: ml.azure.com/compute
    operator: Equal
    value: aksnemlflowprod
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  volumes:
  - emptyDir: {}
    name: model-mount-0
  - name: kube-api-access-wjf8j
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
  - name: identity-secret-volume
    secret:
      defaultMode: 420
      items:
      - key: identityStore
        path: identity_secret.json
      - key: token
        path: sidecar_token
      - key: ca
        path: ca.crt
      secretName: deployname-134205-model-name-sidecar
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-04-19T04:06:16Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-04-19T04:06:41Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-04-19T04:06:41Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-04-19T04:06:00Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://c6cc35cc482f7bdab5a6a5c1e5a2fb46ac14e82eac9a4e2684626cc05e175139
    image: mcr.microsoft.com/azureml/amlarc/docker/identity-sidecar:1.1.25
    imageID: mcr.microsoft.com/azureml/amlarc/docker/identity-sidecar@sha256:2aaae7af6e0bd3e5847c55feca123cf50d968fee52616f7e412b0a76b51ce035
    lastState: {}
    name: identity-sidecar
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-04-19T04:06:16Z"
  - containerID: containerd://8dc55c77556ee15f64321dd0c2967eef6002fdbbb92eccf1c0c9d76fc168f5a9
    image: registry.azurecr.io/image/path:v3.6.2
    imageID: registry.azurecr.io/image/path@sha256:8736a082e2f7698e6c6e5a055e9a17585d15bac3f57b833d889977d5dd2142e3
    lastState: {}
    name: inference-server
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-04-19T04:06:16Z"
  hostIP: 10.224.0.4
  initContainerStatuses:
  - containerID: containerd://4bfced7cb46a4f6e855cd613dbe87f54d226bff8871f6a3295b76dd49b116afb
    image: mcr.microsoft.com/mir/mir-storageinitializer:46571814.1631244300887
    imageID: mcr.microsoft.com/mir/mir-storageinitializer@sha256:c5dc758e64d7cf4a571c3877f9195eabd97e752f23ab01965e7c40abac95b83e
    lastState: {}
    name: storageinitializer-modeldata
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://4bfced7cb46a4f6e855cd613dbe87f54d226bff8871f6a3295b76dd49b116afb
        exitCode: 0
        finishedAt: "2023-04-19T04:06:16Z"
        reason: Completed
        startedAt: "2023-04-19T04:06:01Z"
  phase: Running
  podIP: 10.244.0.17
  podIPs:
  - ip: 10.244.0.17
  qosClass: Burstable
  startTime: "2023-04-19T04:06:00Z"

主要語言: Jupyter Notebook
星號: 4.4k
分支: 2.6k
PR 合併指標: 30 天內沒有已合併 PR

貢獻指南

這個儲存庫沒有索引到貢獻指南

從這裡開始

先讀完整個 Issue，再讀專案的貢獻指南。
在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
Fork 儲存庫，在一個分支上完成修改。
送出 Pull Request，並在描述裡引用這個 Issue 編號。

Azure/MachineLearningNotebooks 的其他 Issue

Incorrect type hint for azure.ai.ml.sweep.Choice 未關閉

難度 2/5 1-3 小時新手友好度 72/100

Azure/MachineLearningNotebooks#1975 · 1 則留言 · 2024年11月15日建立
duplicates 未關閉

難度 1/5 1 小時以內新手友好度 68/100

Azure/MachineLearningNotebooks#1960 · 2024年2月22日建立
machine 未關閉

難度 5/5 一週以上新手友好度 10/100

Azure/MachineLearningNotebooks#1987 · 2026年7月17日建立
Using Workspace Connections for Docker Authentication in Serverless Builds 未關閉

難度 5/5 一週以上新手友好度 25/100

Azure/MachineLearningNotebooks#1985 · 1 則留言 · 2025年11月7日建立
Dependencies to azure-storage-blob should be updated 未關閉

難度 3/5 1-2 天新手友好度 35/100

Azure/MachineLearningNotebooks#1981 · 1 個 reaction · 2025年3月19日建立

查看 Azure/MachineLearningNotebooks 的全部 Issue

相似的 Issue

Azure Sandboxes: Small and ExtraSmall tiers always fail to deploy (disk request exceeds tier maximum) 未關閉

area-deployment area-integrations triage:bot-seen

難度 2/5 半天新手友好度 86/100

microsoft/aspire#20459 · 1 則留言 · 2026年9月24日建立
Helm chart on main defaults to unpublished image rustfs/rustfs:1.0.1 (ImagePullBackOff) 未關閉

難度 2/5 1-3 小時新手友好度 74/100

rustfs/rustfs#8110 · 2026年9月24日建立
No icon on https://charmhub.io/postgresql-watcher 未關閉

bug

難度 2/5 1-3 小時新手友好度 75/100

canonical/postgresql-watcher-operator#111 · 2 則留言 · 2026年9月24日建立
[receiver/azureblob] Malformed Event Hub message panics and crashes the collector 未關閉

bug needs triage receiver/azureblob

難度 2/5 1-3 小時新手友好度 78/100

open-telemetry/opentelemetry-collector-contrib#51474 · 1 則留言 · 2026年9月24日建立
Dependency Dashboard 未關閉

難度 1/5 1 小時以內新手友好度 60/100

IBM/paas-starter#5 · 2026年9月24日建立

更多 Cloud Issue