Lightning-AI/pytorch-lightning

Incompatibility with MLFlow 1.30

Open

#17,084 opened on 2023年3月14日

GitHub で見る
 (4 comments) (0 reactions) (1 assignee)Python (3,233 forks)batch import
3rd partybughelp wantedlogger: mlflow

Repository metrics

Stars
 (26,687 stars)
PR merge metrics
 (平均マージ 9d 15h) (30d で 3 merged PRs)

説明

Bug description

This is just to let you guys know that there's an incompatibility bug with MLFlow 1.30.

PL creates runs through this line without specifying the run_name, which will make MLFlow generate a random run_name here and then this will be added to the tags here. But if a run_name has already been passed through PL with the tags, this will create a duplication error.

This is fixed in MLFlow 2.0. So could you perhaps change the version requirement, or add run_name to the create_run line in src/pytorch_lightning/loggers/mlflow.py? Or if someone can investigate further it would be great too. Thanks!

How to reproduce the bug

- start an experiment with run_name specified in the config, and log with MLFlow.

Error messages and logs

Traceback (most recent call last):
  File "run_finetune.py", line 104, in main
    logger.log_hyperparams({'trainer': OmegaConf.to_object(cfg.trainer)})
  File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
    return fn(*args, **kwargs)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 231, in log_hyperparams
    self.experiment.log_param(self.run_id, k, v)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 53, in experiment
    return get_experiment() or DummyExperiment()
  File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
    return fn(*args, **kwargs)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 51, in get_experiment
    return fn(self)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 195, in experiment
    run = self._mlflow_client.create_run(experiment_id=self._experiment_id, tags=resolve_tags(self.tags))
  File "***/lib/python3.8/site-packages/mlflow/tracking/client.py", line 270, in create_run
    return self._tracking_client.create_run(experiment_id, start_time, tags, run_name)
  File "***/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in create_run
    return self.store.create_run(
  File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 204, in create_run
    response_proto = self._call_endpoint(CreateRun, req_body)
  File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 57, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
  File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 280, in call_endpoint
    response = verify_rest_response(response, endpoint)
  File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 206, in verify_rest_response
    raise RestException(json.loads(response.text))
mlflow.exceptions.RestException: BAD_REQUEST: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "tag_pk"
DETAIL:  Key (key, run_uuid)=(mlflow.runName, 8****) already exists.

[SQL: INSERT INTO tags (key, value, run_uuid) VALUES (%(key)s, %(value)s, %(run_uuid)s)]
[parameters: (****{'key': 'mlflow.runName', 'value': 'finetune_trial', 'run_uuid': '8****'}, {'key': 'mlflow.runName', 'value': 'bouncy-squid-39', 'run_uuid': '8****'})]

Environment

PyTorch Lightning 1.8.6
MLFlow 1.30.0

More info

No response

コントリビューターガイド