Lightning-AI/pytorch-lightning

Incompatibility with MLFlow 1.30

Open

#17,084 创建于 2023年3月14日

在 GitHub 查看
 (4 评论) (0 反应) (1 负责人)Python (3,233 fork)batch import
3rd partybughelp wantedlogger: mlflow

仓库指标

Star
 (26,687 star)
PR 合并指标
 (平均合并 9天 15小时) (30 天内合并 3 个 PR)

描述

Bug description

This is just to let you guys know that there's an incompatibility bug with MLFlow 1.30.

PL creates runs through this line without specifying the run_name, which will make MLFlow generate a random run_name here and then this will be added to the tags here. But if a run_name has already been passed through PL with the tags, this will create a duplication error.

This is fixed in MLFlow 2.0. So could you perhaps change the version requirement, or add run_name to the create_run line in src/pytorch_lightning/loggers/mlflow.py? Or if someone can investigate further it would be great too. Thanks!

How to reproduce the bug

- start an experiment with run_name specified in the config, and log with MLFlow.

Error messages and logs

Traceback (most recent call last):
  File "run_finetune.py", line 104, in main
    logger.log_hyperparams({'trainer': OmegaConf.to_object(cfg.trainer)})
  File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
    return fn(*args, **kwargs)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 231, in log_hyperparams
    self.experiment.log_param(self.run_id, k, v)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 53, in experiment
    return get_experiment() or DummyExperiment()
  File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
    return fn(*args, **kwargs)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 51, in get_experiment
    return fn(self)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 195, in experiment
    run = self._mlflow_client.create_run(experiment_id=self._experiment_id, tags=resolve_tags(self.tags))
  File "***/lib/python3.8/site-packages/mlflow/tracking/client.py", line 270, in create_run
    return self._tracking_client.create_run(experiment_id, start_time, tags, run_name)
  File "***/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in create_run
    return self.store.create_run(
  File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 204, in create_run
    response_proto = self._call_endpoint(CreateRun, req_body)
  File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 57, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
  File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 280, in call_endpoint
    response = verify_rest_response(response, endpoint)
  File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 206, in verify_rest_response
    raise RestException(json.loads(response.text))
mlflow.exceptions.RestException: BAD_REQUEST: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "tag_pk"
DETAIL:  Key (key, run_uuid)=(mlflow.runName, 8****) already exists.

[SQL: INSERT INTO tags (key, value, run_uuid) VALUES (%(key)s, %(value)s, %(run_uuid)s)]
[parameters: (****{'key': 'mlflow.runName', 'value': 'finetune_trial', 'run_uuid': '8****'}, {'key': 'mlflow.runName', 'value': 'bouncy-squid-39', 'run_uuid': '8****'})]

Environment

PyTorch Lightning 1.8.6
MLFlow 1.30.0

More info

No response

贡献者指南