Bug description

I have trained a model using multiple gpu . after that i am trying to resume training from checkpoint on the same multi-gpu instance but i am getting memory related error. but same when i am trying on single gpu it started training but its not started from last epoch i guess becuase loss if high this time . also does it use the same learning rate that was found in previous training usnig lr_finder callback.?

What version are you seeing the problem on?

1.9.x

How to reproduce the bug

trainer = pl.Trainer(logger=wandb_logger,resume_from_checkpoint="./580sdysk/checkpoints/model-epoch=09-val_loss=0.01.ckpt",
        accelerator=config.ACCELERATOR,
        devices=config.DEVICES,
        min_epochs=1,
        max_epochs=config.NUM_EPOCHS,
        callbacks=[checkpoint_callback,lr_findr]
    )

 
    trainer.fit(model, dm)#,ckpt_path="./580sdysk/checkpoints/model-epoch=09-val_loss=0.01.ckpt")
    trainer.validate(model, dm)
    trainer.test(model, dm)

Error messages and logs

Error messages and logs here please

Traceback (most recent call last):
  File "train.py", line 70, in <module>
    trainer.fit(model, dm)#,ckpt_path="./emb_training/580sdysk/checkpoints/Content_Emb-epoch=09-val_loss=0.01.ckpt")
  File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 113, in launch
    mp.start_processes(
  File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes
    while not context.join():
  File "/opt/conda/envs/emb_poc_v1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 140, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with signal SIGKILL

Environment

• CUDA: - GPU: - Tesla T4 - Tesla T4 - Tesla T4 - Tesla T4 - available: True - version: 11.7
• Lightning: - lightning-utilities: 0.8.0 - pytorch-lightning: 1.9.4 - torch: 2.0.0+cu117 - torchaudio: 2.0.1+cu117 - torchmetrics: 0.11.4 - torchvision: 0.15.1+cu117
• Packages: - absl-py: 1.4.0 - aiohttp: 3.8.4 - aiosignal: 1.3.1 - anyio: 3.6.2 - appdirs: 1.4.4 - argon2-cffi: 21.3.0 - argon2-cffi-bindings: 21.2.0 - arrow: 1.2.3 - asttokens: 2.2.1 - async-timeout: 4.0.2 - attrs: 23.1.0 - backcall: 0.2.0 - beautifulsoup4: 4.12.2 - bleach: 6.0.0 - cachetools: 5.3.0 - certifi: 2022.12.7 - cffi: 1.15.1 - charset-normalizer: 2.1.1 - click: 8.1.3 - cmake: 3.25.0 - comm: 0.1.3 - debugpy: 1.6.7 - decorator: 5.1.1 - defusedxml: 0.7.1 - docker-pycreds: 0.4.0 - executing: 1.2.0 - fastjsonschema: 2.16.3 - filelock: 3.9.0 - fqdn: 1.5.1 - frozenlist: 1.3.3 - fsspec: 2023.3.0 - gcsfs: 2023.3.0 - gitdb: 4.0.10 - gitpython: 3.1.31 - google-api-core: 2.11.0 - google-auth: 2.17.1 - google-auth-oauthlib: 1.0.0 - google-cloud-core: 2.3.2 - google-cloud-storage: 2.8.0 - google-crc32c: 1.5.0 - google-resumable-media: 2.4.1 - googleapis-common-protos: 1.59.0 - grpcio: 1.53.0 - huggingface-hub: 0.13.3 - idna: 3.4 - importlib-metadata: 6.5.0 - importlib-resources: 5.12.0 - ipykernel: 6.22.0 - ipython: 8.12.0 - ipython-genutils: 0.2.0 - ipywidgets: 8.0.6 - isoduration: 20.11.0 - jedi: 0.18.2 - jinja2: 3.1.2 - joblib: 1.2.0 - jsonpointer: 2.3 - jsonschema: 4.17.3 - jupyter: 1.0.0 - jupyter-client: 8.2.0 - jupyter-console: 6.6.3 - jupyter-core: 5.3.0 - jupyter-events: 0.6.3 - jupyter-server: 2.5.0 - jupyter-server-terminals: 0.4.4 - jupyterlab-pygments: 0.2.2 - jupyterlab-widgets: 3.0.7 - lightning-utilities: 0.8.0 - lit: 15.0.7 - markdown: 3.4.3 - markupsafe: 2.1.2 - matplotlib-inline: 0.1.6 - mistune: 2.0.5 - mpmath: 1.2.1 - multidict: 6.0.4 - nbclassic: 0.5.5 - nbclient: 0.7.3 - nbconvert: 7.3.1 - nbformat: 5.8.0 - nest-asyncio: 1.5.6 - networkx: 3.0 - notebook: 6.5.4 - notebook-shim: 0.2.2 - numpy: 1.24.1 - oauthlib: 3.2.2 - openai: 0.27.4 - packaging: 23.1 - pandas: 2.0.0 - pandocfilters: 1.5.0 - parso: 0.8.3 - pathtools: 0.1.2 - pexpect: 4.8.0 - pickleshare: 0.7.5 - pillow: 9.3.0 - pip: 23.0.1 - pkgutil-resolve-name: 1.3.10 - platformdirs: 3.2.0 - prometheus-client: 0.16.0 - prompt-toolkit: 3.0.38 - protobuf: 4.22.1 - psutil: 5.9.5 - ptyprocess: 0.7.0 - pure-eval: 0.2.2 - pyasn1: 0.4.8 - pyasn1-modules: 0.2.8 - pycparser: 2.21 - pygments: 2.15.1 - pyrsistent: 0.19.3 - python-dateutil: 2.8.2 - python-dotenv: 1.0.0 - python-json-logger: 2.0.7 - pytorch-lightning: 1.9.4 - pytz: 2023.3 - pyyaml: 6.0 - pyzmq: 25.0.2 - qtconsole: 5.4.2 - qtpy: 2.3.1 - regex: 2023.3.23 - requests: 2.28.1 - requests-oauthlib: 1.3.1 - rfc3339-validator: 0.1.4 - rfc3986-validator: 0.1.1 - rsa: 4.9 - scikit-learn: 0.23.0 - scipy: 1.10.1 - send2trash: 1.8.0 - sentry-sdk: 1.19.0 - setproctitle: 1.3.2 - setuptools: 67.6.1 - six: 1.16.0 - smmap: 5.0.0 - sniffio: 1.3.0 - soupsieve: 2.4.1 - stack-data: 0.6.2 - sympy: 1.11.1 - tensorboard: 2.12.2 - tensorboard-data-server: 0.7.0 - tensorboard-plugin-wit: 1.8.1 - terminado: 0.17.1 - threadpoolctl: 3.1.0 - tiktoken: 0.3.3 - tinycss2: 1.2.1 - tokenizers: 0.13.2 - torch: 2.0.0+cu117 - torchaudio: 2.0.1+cu117 - torchmetrics: 0.11.4 - torchvision: 0.15.1+cu117 - tornado: 6.3 - tqdm: 4.65.0 - traitlets: 5.9.0 - transformers: 4.27.4 - triton: 2.0.0 - typing-extensions: 4.5.0 - tzdata: 2023.3 - uri-template: 1.2.0 - urllib3: 1.26.13 - wandb: 0.14.0 - wcwidth: 0.2.6 - webcolors: 1.13 - webencodings: 0.5.1 - websocket-client: 1.5.1 - werkzeug: 2.2.3 - wheel: 0.40.0 - widgetsnbextension: 4.0.7 - yarl: 1.8.2 - zipp: 3.15.0
• System: - OS: Linux - architecture: - 64bit - ELF - processor:
  - python: 3.8.13 - version: #1 SMP Debian 4.19.269-1 (2022-12-20)

More info

how can i get the values of learning rate found using lr_finder callback