jobs not resubmitted in `SLURMCluster` after graceful closure via `--lifetime`

Đang mở

#691 mở ngày 10/6/2025 7 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó: 4/5
Thời gian dự kiến: 3-5 ngày
Mức phù hợp với người mới: 35/100

Loại issue: Lỗi
Độ rõ ràng: Khá rõ ràng
Mức độ hoạt động: Đình trệ
Công nghệ: python
Lĩnh vực: distributed-systems, infrastructure

Hướng nghiên cứu

Bắt đầu với ví dụ SLURMCluster và các thiết lập thời gian tồn tại của worker được nêu trong issue, sau đó tái hiện hành vi bằng --lifetime và --lifetime-stagger trong khi quan sát việc scaling thích ứng. Theo dõi cách các lần shutdown graceful của worker đã đạt đến thời gian tồn tại (worker-lifetime-reached) được báo cáo cho cluster và scheduler. Hoàn tất khi các job SLURM thay thế được gửi sau khi các worker đóng một cách graceful, kèm theo một regression test bao quát hành vi này.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug

Hi there,

I set up a SLURMCluster following the example below, including --lifetime and --lifetime-stagger to ensure jobs are closed gracefully by the dask scheduler.

import time

from dask.distributed import Client
from dask_jobqueue import SLURMCluster

GPU_CONFIG = {
    'queue': 'batch_gpu',
    'cores': 8,
    'memory': '8GB',
    'job_extra_directives': [
        '--gres=gpu:1',
    ],
    'walltime': '00:05:00',
    'worker_extra_args': ["--lifetime", "10s", "--lifetime-stagger", "10s"],
}

cluster = SLURMCluster(**GPU_CONFIG)
client = Client(cluster)
cluster.adapt(minimum_jobs=1, maximum_jobs=5)

while True:
    print(client)
    time.sleep(5)

My expectation was that this would submit new jobs as the original ones were closed I am not seeing this behavior. I am fairly sure that when I used this exact setup a few years ago it worked correctly...

The stdout below clearly shows that the initial two jobs submit and connect to the cluster before being killed and not respawning.

<Client: 'tcp://10.164.24.30:39231' processes=0 threads=0, memory=0 B>
<Client: 'tcp://10.164.24.30:39231' processes=8 threads=16, memory=14.88 GiB>
<Client: 'tcp://10.164.24.30:39231' processes=7 threads=14, memory=13.02 GiB>
<Client: 'tcp://10.164.24.30:39231' processes=4 threads=8, memory=7.44 GiB>
<Client: 'tcp://10.164.24.30:39231' processes=4 threads=8, memory=7.44 GiB>
<Client: 'tcp://10.164.24.30:39231' processes=0 threads=0, memory=0 B>
<Client: 'tcp://10.164.24.30:39231' processes=0 threads=0, memory=0 B>
<Client: 'tcp://10.164.24.30:39231' processes=0 threads=0, memory=0 B>
<Client: 'tcp://10.164.24.30:39231' processes=0 threads=0, memory=0 B>

The following worker log shows the worker processes die as they reach their lifetime and close gracefully as expected.

2025-06-09 17:06:59,476 - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.164.25.82:35379'
2025-06-09 17:06:59,480 - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.164.25.82:33079'
2025-06-09 17:06:59,481 - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.164.25.82:41377'
2025-06-09 17:06:59,483 - distributed.nanny - INFO -         Start Nanny at: 'tcp://10.164.25.82:35255'
2025-06-09 17:06:59,900 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-scratch-space/worker-p715og9m', purging
2025-06-09 17:06:59,901 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-scratch-space/worker-9_cdptn2', purging
2025-06-09 17:07:00,230 - distributed.worker - INFO -       Start worker at:   tcp://10.164.25.82:40503
2025-06-09 17:07:00,230 - distributed.worker - INFO -          Listening to:   tcp://10.164.25.82:40503
2025-06-09 17:07:00,230 - distributed.worker - INFO -           Worker name:           SLURMCluster-0-0
2025-06-09 17:07:00,230 - distributed.worker - INFO -          dashboard at:         10.164.25.82:44353
2025-06-09 17:07:00,230 - distributed.worker - INFO - Waiting to connect to:   tcp://10.164.24.30:39231
2025-06-09 17:07:00,231 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,231 - distributed.worker - INFO -               Threads:                          2
2025-06-09 17:07:00,231 - distributed.worker - INFO -                Memory:                   1.86 GiB
2025-06-09 17:07:00,231 - distributed.worker - INFO -       Local Directory: /tmp/dask-scratch-space/worker-guwx53y4
2025-06-09 17:07:00,231 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,240 - distributed.worker - INFO -       Start worker at:   tcp://10.164.25.82:45191
2025-06-09 17:07:00,240 - distributed.worker - INFO -          Listening to:   tcp://10.164.25.82:45191
2025-06-09 17:07:00,240 - distributed.worker - INFO -           Worker name:           SLURMCluster-0-3
2025-06-09 17:07:00,240 - distributed.worker - INFO -          dashboard at:         10.164.25.82:46333
2025-06-09 17:07:00,240 - distributed.worker - INFO - Waiting to connect to:   tcp://10.164.24.30:39231
2025-06-09 17:07:00,240 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,240 - distributed.worker - INFO -               Threads:                          2
2025-06-09 17:07:00,240 - distributed.worker - INFO -                Memory:                   1.86 GiB
2025-06-09 17:07:00,240 - distributed.worker - INFO -       Local Directory: /tmp/dask-scratch-space/worker-0j9vxv6a
2025-06-09 17:07:00,240 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,242 - distributed.worker - INFO -       Start worker at:   tcp://10.164.25.82:42045
2025-06-09 17:07:00,242 - distributed.worker - INFO -          Listening to:   tcp://10.164.25.82:42045
2025-06-09 17:07:00,242 - distributed.worker - INFO -       Start worker at:   tcp://10.164.25.82:36879
2025-06-09 17:07:00,242 - distributed.worker - INFO -           Worker name:           SLURMCluster-0-1
2025-06-09 17:07:00,242 - distributed.worker - INFO -          dashboard at:         10.164.25.82:43743
2025-06-09 17:07:00,242 - distributed.worker - INFO -          Listening to:   tcp://10.164.25.82:36879
2025-06-09 17:07:00,242 - distributed.worker - INFO - Waiting to connect to:   tcp://10.164.24.30:39231
2025-06-09 17:07:00,242 - distributed.worker - INFO -           Worker name:           SLURMCluster-0-2
2025-06-09 17:07:00,242 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,242 - distributed.worker - INFO -          dashboard at:         10.164.25.82:39849
2025-06-09 17:07:00,242 - distributed.worker - INFO - Waiting to connect to:   tcp://10.164.24.30:39231
2025-06-09 17:07:00,242 - distributed.worker - INFO -               Threads:                          2
2025-06-09 17:07:00,242 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,243 - distributed.worker - INFO -                Memory:                   1.86 GiB
2025-06-09 17:07:00,243 - distributed.worker - INFO -       Local Directory: /tmp/dask-scratch-space/worker-n29701zb
2025-06-09 17:07:00,243 - distributed.worker - INFO -               Threads:                          2
2025-06-09 17:07:00,243 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,243 - distributed.worker - INFO -                Memory:                   1.86 GiB
2025-06-09 17:07:00,243 - distributed.worker - INFO -       Local Directory: /tmp/dask-scratch-space/worker-cxvq1qyq
2025-06-09 17:07:00,243 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,245 - distributed.worker - INFO - Starting Worker plugin shuffle
2025-06-09 17:07:00,246 - distributed.worker - INFO -         Registered to:   tcp://10.164.24.30:39231
2025-06-09 17:07:00,246 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,246 - distributed.core - INFO - Starting established connection to tcp://10.164.24.30:39231
2025-06-09 17:07:00,253 - distributed.worker - INFO - Starting Worker plugin shuffle
2025-06-09 17:07:00,253 - distributed.worker - INFO -         Registered to:   tcp://10.164.24.30:39231
2025-06-09 17:07:00,253 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,254 - distributed.core - INFO - Starting established connection to tcp://10.164.24.30:39231
2025-06-09 17:07:00,255 - distributed.worker - INFO - Starting Worker plugin shuffle
2025-06-09 17:07:00,256 - distributed.worker - INFO -         Registered to:   tcp://10.164.24.30:39231
2025-06-09 17:07:00,256 - distributed.worker - INFO - Starting Worker plugin shuffle
2025-06-09 17:07:00,256 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,256 - distributed.worker - INFO -         Registered to:   tcp://10.164.24.30:39231
2025-06-09 17:07:00,256 - distributed.worker - INFO - -------------------------------------------------
2025-06-09 17:07:00,256 - distributed.core - INFO - Starting established connection to tcp://10.164.24.30:39231
2025-06-09 17:07:00,256 - distributed.core - INFO - Starting established connection to tcp://10.164.24.30:39231
2025-06-09 17:07:02,099 - distributed.worker - INFO - Closing worker gracefully: tcp://10.164.25.82:45191. Reason: worker-lifetime-reached
2025-06-09 17:07:02,100 - distributed.worker - INFO - Stopping worker at tcp://10.164.25.82:45191. Reason: worker-lifetime-reached
2025-06-09 17:07:02,102 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://10.164.25.82:33079'. Reason: worker-lifetime-reached
2025-06-09 17:07:02,102 - distributed.worker - INFO - Removing Worker plugin shuffle
2025-06-09 17:07:02,103 - distributed.core - INFO - Connection to tcp://10.164.24.30:39231 has been closed.
2025-06-09 17:07:02,103 - distributed.nanny - INFO - Worker closed
2025-06-09 17:07:04,104 - distributed.nanny - ERROR - Worker process died unexpectedly
2025-06-09 17:07:04,202 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.164.25.82:33079'. Reason: nanny-close-gracefully
2025-06-09 17:07:04,203 - distributed.nanny - INFO - Nanny at 'tcp://10.164.25.82:33079' closed.
2025-06-09 17:07:04,590 - distributed.worker - INFO - Closing worker gracefully: tcp://10.164.25.82:36879. Reason: worker-lifetime-reached
2025-06-09 17:07:04,592 - distributed.worker - INFO - Stopping worker at tcp://10.164.25.82:36879. Reason: worker-lifetime-reached
2025-06-09 17:07:04,593 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://10.164.25.82:35255'. Reason: worker-lifetime-reached
2025-06-09 17:07:04,593 - distributed.worker - INFO - Removing Worker plugin shuffle
2025-06-09 17:07:04,594 - distributed.core - INFO - Connection to tcp://10.164.24.30:39231 has been closed.
2025-06-09 17:07:04,594 - distributed.nanny - INFO - Worker closed
2025-06-09 17:07:06,693 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.164.25.82:35255'. Reason: nanny-close-gracefully
2025-06-09 17:07:06,693 - distributed.nanny - INFO - Nanny at 'tcp://10.164.25.82:35255' closed.
2025-06-09 17:07:15,959 - distributed.worker - INFO - Closing worker gracefully: tcp://10.164.25.82:42045. Reason: worker-lifetime-reached
2025-06-09 17:07:15,961 - distributed.worker - INFO - Stopping worker at tcp://10.164.25.82:42045. Reason: worker-lifetime-reached
2025-06-09 17:07:15,962 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://10.164.25.82:35379'. Reason: worker-lifetime-reached
2025-06-09 17:07:15,962 - distributed.worker - INFO - Removing Worker plugin shuffle
2025-06-09 17:07:15,963 - distributed.core - INFO - Connection to tcp://10.164.24.30:39231 has been closed.
2025-06-09 17:07:15,964 - distributed.nanny - INFO - Worker closed
2025-06-09 17:07:16,803 - distributed.worker - INFO - Closing worker gracefully: tcp://10.164.25.82:40503. Reason: worker-lifetime-reached
2025-06-09 17:07:16,805 - distributed.worker - INFO - Stopping worker at tcp://10.164.25.82:40503. Reason: worker-lifetime-reached
2025-06-09 17:07:16,806 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://10.164.25.82:41377'. Reason: worker-lifetime-reached
2025-06-09 17:07:16,806 - distributed.worker - INFO - Removing Worker plugin shuffle
2025-06-09 17:07:16,807 - distributed.core - INFO - Connection to tcp://10.164.24.30:39231 has been closed.
2025-06-09 17:07:16,808 - distributed.nanny - INFO - Worker closed
2025-06-09 17:07:17,965 - distributed.nanny - ERROR - Worker process died unexpectedly
2025-06-09 17:07:18,063 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.164.25.82:35379'. Reason: nanny-close-gracefully
2025-06-09 17:07:18,063 - distributed.nanny - INFO - Nanny at 'tcp://10.164.25.82:35379' closed.
2025-06-09 17:07:18,993 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.164.25.82:41377'. Reason: nanny-close-gracefully
2025-06-09 17:07:18,993 - distributed.nanny - INFO - Nanny at 'tcp://10.164.25.82:41377' closed.
2025-06-09 17:07:18,993 - distributed.dask_worker - INFO - End worker

I also confirmed that I see no new workers in the SLURM job queue with squeue.

Is this a bug or am I doing something obviously wrong here? Many thanks in advance for any help with this

Ngôn ngữ chính: Python
Star: 256
Fork: 149
Chỉ số merge pull request: Không có pull request nào được merge trong 30 ngày

Chuẩn bị môi trường

Không có Dockerfile hay tệp Docker Compose
Không có mẫu pull request
Đọc hướng dẫn đóng góp

Bắt đầu từ đâu

Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
Fork repository và làm thay đổi trên một nhánh.
Mở pull request có tham chiếu số hiệu của issue.

Issue khác của dask/dask-jobqueue

LSF broken for runtimes given in HH:MM:SS format
Đang mở
- Có gì không ổn?
bug LSF

Độ khó 3/5 1-2 ngày Mức phù hợp với người mới 65/100

dask/dask-jobqueue#703 · 1 bình luận · Mở ngày 28/7/2026
Control idle time on SLURMCluster.adapt() to control when workers are released.
Đang mở
- Có gì không ổn?
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 35/100

dask/dask-jobqueue#701 · Mở ngày 17/3/2026
core.py's Job class default regex not selecting correct job id
Đang mở
- Có gì không ổn?
Độ khó 3/5 1-2 ngày Mức phù hợp với người mới 45/100

dask/dask-jobqueue#699 · 2 bình luận · Mở ngày 29/1/2026
Potential for SLURMCluster log directory creation failing
Đang mở
- Có gì không ổn?
bug

Độ khó 3/5 1-2 ngày Mức phù hợp với người mới 38/100

dask/dask-jobqueue#692 · 1 bình luận · Mở ngày 28/7/2025
Improve documentation for `SLURMRunner`
Đang mở
- Có gì không ổn?
Độ khó 2/5 Nửa ngày Mức phù hợp với người mới 45/100

dask/dask-jobqueue#686 · 3 bình luận · Mở ngày 10/4/2025

Tất cả issue của dask/dask-jobqueue

Issue tương tự

Tighten language in 3.2.2 about MPIC
Đang mở
- Có gì không ổn?
Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 72/100

letsencrypt/cp-cps#353 · Mở ngày 30/9/2026
Marble Madness II is missing
Đang mở
- Có gì không ổn?
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100

MiSTer-devel/Downloader_MiSTer#65 · Mở ngày 30/9/2026
Journey-mode exits ignore capacity_agents_per_s
Đang mở
- Có gì không ổn?
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 84/100

PedestrianDynamics/pyFDS-Evac#394 · Mở ngày 30/9/2026

Maintainer thường phản hồi trong vòng 1 ngày
NetCdfWrite drops string coordinates that share a char length
Đang mở
- Có gì không ổn?
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 78/100

DOI-USGS/pywatershed#421 · Mở ngày 30/9/2026
XBM loader is too strict on variable name
Đang mở
- Có gì không ổn?
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 78/100

python-pillow/Pillow#10087 · 1 bình luận · Mở ngày 30/9/2026

Maintainer thường phản hồi trong vòng 1 ngày

Thêm issue về Python

jobs not resubmitted in `SLURMCluster` after graceful closure via `--lifetime`

Có gì không ổn?

Đánh giá

Hướng nghiên cứu

Mô tả

Chuẩn bị môi trường

Bắt đầu từ đâu

Issue khác của dask/dask-jobqueue

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Issue tương tự

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Có gì không ổn?

Nhận issue mới trong hộp thư của bạn