Append revision to filepath in `--output_dir`?

Currently, lighteval stores results/details in a path that is determined by the model name, e.g.

scratch/evals
├── details
│   └── Qwen
│       └── Qwen1.5-0.5B-Chat
│           ├── 2024-02-26T15-36-31.681219
│           │   └── details_lighteval|truthfulqa:mc|0_2024-02-26T15-36-31.681219.parquet
│           └── results_2024-02-26T15-36-31.681219.json
└── results
    └── Qwen
        └── Qwen1.5-0.5B-Chat
            └── results_2024-02-26T15-36-31.681219.json

However, I am quite often evaluating models with different revisions and the current save logic groups these all together in the same subfolder which makes it hard to determine which result corresponds to which run.

Would it make sense to append the model revision parameter to the filepaths, e.g. something like this for the main revision (or whatever is passed to the revision arg in the script):

scratch/evals
├── details
│   └── Qwen
│       └── Qwen1.5-0.5B-Chat
│           └── main
│               ├── 2024-02-26T15-36-31.681219
│               │   └── details_lighteval|truthfulqa:mc|0_2024-02-26T15-36-31.681219.parquet
│               └── results_2024-02-26T15-36-31.681219.json
└── results
    └── Qwen
        └── Qwen1.5-0.5B-Chat
            └── main
                └── results_2024-02-26T15-36-31.681219.json

My current workaround is to manually specify the model path in --output_dir={ORG}/{MODEL_ID}/{REVISION} and then glob the files. This is fine, but a bit clunky because one ends up with a long nested path like {ORG}/{MODEL_ID}/{REVISION}/results/{ORG}/{MODEL_ID}