Métriques du dépôt
- Stars
- (26 687 stars)
- Métriques de merge PR
- (Merge moyen 9j 15h) (3 PRs mergées en 30 j)
Description
Bug description
This is just to let you guys know that there's an incompatibility bug with MLFlow 1.30.
PL creates runs through this line without specifying the run_name, which will make MLFlow generate a random run_name here and then this will be added to the tags here. But if a run_name has already been passed through PL with the tags, this will create a duplication error.
This is fixed in MLFlow 2.0. So could you perhaps change the version requirement, or add run_name to the create_run line in src/pytorch_lightning/loggers/mlflow.py? Or if someone can investigate further it would be great too. Thanks!
How to reproduce the bug
- start an experiment with run_name specified in the config, and log with MLFlow.
Error messages and logs
Traceback (most recent call last):
File "run_finetune.py", line 104, in main
logger.log_hyperparams({'trainer': OmegaConf.to_object(cfg.trainer)})
File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
return fn(*args, **kwargs)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 231, in log_hyperparams
self.experiment.log_param(self.run_id, k, v)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 53, in experiment
return get_experiment() or DummyExperiment()
File "***/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 27, in wrapped_fn
return fn(*args, **kwargs)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/logger.py", line 51, in get_experiment
return fn(self)
File "***/lib/python3.8/site-packages/pytorch_lightning/loggers/mlflow.py", line 195, in experiment
run = self._mlflow_client.create_run(experiment_id=self._experiment_id, tags=resolve_tags(self.tags))
File "***/lib/python3.8/site-packages/mlflow/tracking/client.py", line 270, in create_run
return self._tracking_client.create_run(experiment_id, start_time, tags, run_name)
File "***/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 108, in create_run
return self.store.create_run(
File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 204, in create_run
response_proto = self._call_endpoint(CreateRun, req_body)
File "***/lib/python3.8/site-packages/mlflow/store/tracking/rest_store.py", line 57, in _call_endpoint
return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 280, in call_endpoint
response = verify_rest_response(response, endpoint)
File "***/lib/python3.8/site-packages/mlflow/utils/rest_utils.py", line 206, in verify_rest_response
raise RestException(json.loads(response.text))
mlflow.exceptions.RestException: BAD_REQUEST: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "tag_pk"
DETAIL: Key (key, run_uuid)=(mlflow.runName, 8****) already exists.
[SQL: INSERT INTO tags (key, value, run_uuid) VALUES (%(key)s, %(value)s, %(run_uuid)s)]
[parameters: (****{'key': 'mlflow.runName', 'value': 'finetune_trial', 'run_uuid': '8****'}, {'key': 'mlflow.runName', 'value': 'bouncy-squid-39', 'run_uuid': '8****'})]
Environment
PyTorch Lightning 1.8.6
MLFlow 1.30.0
More info
No response