Repository metrics
- Stars
- (1,244 stars)
- PR merge metrics
- (PR metrics pending)
Description
‼️ If you want to work on this issue, please comment below and wait until a maintainer assigns this issue to you before opening a PR to avoid several contributions on the same issue. Thanks! 😊
✨ What You’ll Do
Pruna already provides several evaluation metrics. Your task is to create a clear, practical tutorial showing how to prepare, run, and analyze a benchmark, for example, a sustainability benchmark using metrics such as CO2_emissions, total_time, and energy_consumed.
The tutorial should help users understand not only how to run a benchmark, but also why benchmarks matter, which metrics to look at, and how to interpret and visualize the results.
You can do this by following this overview:
- Select 1 or 2 small models to compare, and define a clear benchmarking goal, such as comparing two compression methods
- Define a baseline and apply Pruna optimizations to evaluate efficiency tradeoffs
- Introduce and explain the metrics used in the benchmark
- Run the evaluation agent
- Present the results with clear visualizations
- Interpret the results and highlight the main takeaways
Add the final tutorial notebook under docs/tutorials
🤖 Useful Resources
- The pruna documentation: https://docs.pruna.ai/
- Existing metrics: https://docs.pruna.ai/en/stable/reference/evaluation.html#total-time
- How to evaluate with pruna: https://docs.pruna.ai/en/stable/docs_pruna/user_manual/evaluate.html
- Other tutorials: https://docs.pruna.ai/en/stable/docs_pruna/tutorials/image_generation.html
✅ Acceptance Criteria
- It follows the style guidelines.
- It contains all the information in a clear and coherent way.
And don’t forget to give us a ⭐️!
❓ Questions?
Feel free to jump into the #contributing Discord channel if you hit any roadblocks. Can’t wait to see your contribution! 🚀
