[Roadmap] Benchmark results and Model Zoo

🚀 The feature, motivation and pitch

It would be nice to have pretrained models available for download that reproduce the original papers' results, along with an easy way to compare performance results on some benchmark tasks in a Model Zoo. This helps validate that the implementations are working properly, and aggregates this information in one place.

I know benchmarks/ has started this, but these scripts seem to be primarily for quick runtime benchmarking. While examples/ seems to more closely reproduce models based on the original papers, I don't think performance results / outputs of these scripts are available anywhere? Additionally, graphgym/ seems to only contain results/example_node_grid_example/agg.

---

As a precursor, it'd be nice to have the stdout logs for all the examples/ available, along with the best & last model checkpoint, repo commit hash, and system information.

---

Alternatives

torch-points3d also has a benchmark/ dir, but it only has the script stdout output. Their PretrainedRegistry class does link to a wandb project which serves as a model zoo.

There is also the new examples GH action that seems to log results to wandb.

Additional context

mmdetection3d does this well, with model cards that include results tables, trained models, experiment logs, and documentation on implementation details.

Can also look to the vision community for inspiration on how to present performance results. Examples such as torchvision, timm, and detectron2.