Lightning-AI/pytorch-lightning

Incompatibility with MLFlow 1.30

Open

#17.084 aperta il 14 mar 2023

Vedi su GitHub
 (4 commenti) (0 reazioni) (1 assegnatario)Python (3233 fork)batch import
3rd partybughelp wantedlogger: mlflow

Metriche repository

Star
 (26.687 star)
Metriche merge PR
 (Merge medio 9g 15h) (3 PR mergiate in 30 g)

Descrizione

Bug description

This is just to let you guys know that there's an incompatibility bug with MLFlow 1.30.

PL creates runs through this line without specifying the run_name, which will make MLFlow generate a random run_name here and then this will be added to the tags here. But if a run_name has already been passed through PL with the tags, this will create a duplication error.

This is fixed in MLFlow 2.0. So could you perhaps change the version requirement, or add run_name to the create_run line in src/pytorch_lightning/loggers/mlflow.py? Or if someone can investigate further it would be great too. Thanks!

How to reproduce the bug

- start an experiment with run_name specified in the config, and log with MLFlow.

Error messages and logs

Traceback (most recent call last):
  File "run_finetune.py", line 104, in main
    logger.log_hyperparams({'trainer': OmegaConf.to_object(cfg.trainer)})
  File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
    return fn(*args, **kwargs)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 231, in log_hyperparams
    self.experiment.log_param(self.run_id, k, v)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 53, in experiment
    return get_experiment() or DummyExperiment()
  File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
    return fn(*args, **kwargs)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 51, in get_experiment
    return fn(self)
  File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 195, in experiment
    run = self._mlflow_client.create_run(experiment_id=self._experiment_id, tags=resolve_tags(self.tags))
  File "***/lib/python3.8/site-packages/mlflow/tracking/client.py", line 270, in create_run
    return self._tracking_client.create_run(experiment_id, start_time, tags, run_name)
  File "***/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in create_run
    return self.store.create_run(
  File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 204, in create_run
    response_proto = self._call_endpoint(CreateRun, req_body)
  File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 57, in _call_endpoint
    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
  File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 280, in call_endpoint
    response = verify_rest_response(response, endpoint)
  File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 206, in verify_rest_response
    raise RestException(json.loads(response.text))
mlflow.exceptions.RestException: BAD_REQUEST: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "tag_pk"
DETAIL:  Key (key, run_uuid)=(mlflow.runName, 8****) already exists.

[SQL: INSERT INTO tags (key, value, run_uuid) VALUES (%(key)s, %(value)s, %(run_uuid)s)]
[parameters: (****{'key': 'mlflow.runName', 'value': 'finetune_trial', 'run_uuid': '8****'}, {'key': 'mlflow.runName', 'value': 'bouncy-squid-39', 'run_uuid': '8****'})]

Environment

PyTorch Lightning 1.8.6
MLFlow 1.30.0

More info

No response

Guida contributor