✨ What You’ll Do

Adding more metrics makes Pruna’s evaluations even sharper — so we’re thrilled to welcome T2I-CompBench++ (paper, project)! 🎉

With 8 000 prompts spanning attributes, relationships, numeracy, and complex compositions — plus fresh detection-based metrics and multimodal LLM analyses — this benchmark is a huge step forward for testing compositionality, spatial reasoning, and numeracy in generative models.

---

📦 Data & Dependency

• Download the T2I-CompBench(++) dataset and include it in our evaluation assets.
• Install dependencies from the benchmark’s evaluation toolkit (BLIP-VQA, UniDet, 3-in-1 metrics).
• Ensure detection models for 3D spatial relationships and numeracy are available.

---

📐 Metric Class

For the full deep-dive on Pruna metrics (class structure, state handling, tests, registration), check out our guide: Customize a Metric.

Quick Overview

1. File & Class – Create pruna/evaluation/metrics/metric_t2i_compbench.py (or src/pruna/... if your repo is under src/) and subclass StatefulMetric.
2. Configuration – Pick a sensible call_type (likely per_batch for efficiency) and set metric_name = "t2i_compbench".
3. State Handling – Define and register any internal state via self.add_state(...), e.g.:

   self.add_state("scores", default=[], dist_reduce_fx="cat")
   

4. Core Methods - update(x, gt, outputs) → Feed generated images + prompts into the benchmark’s evaluation routines (BLIP-VQA for attribute binding, UniDet for spatial reasoning, 3-in-1 metric for complex prompts). Accumulate scores. compute() → Aggregate category scores and return a result dictionary.
5. Registration – Use @MetricRegistry.register("t2i_compbench") so it’s callable by name.
6. Testing & Docs – Write unit tests (tests/evaluation/metrics/test_t2i_compbench.py) and update user_manual/evaluation.rst.

---

✅ Acceptance Criteria

• Efficient Loading → Dataset + models (detection nets, MLLMs) are loaded once in the constructor.
• Correct Output → compute() returns meaningful scores capturing compositionality, 3D spatial reasoning, and numeracy (per the paper).
• Tests & Docs Updated → All tests pass; docs explain how to use the metric.

---

❓ Questions?

Hop into our Discord channel if you get stuck. We’re excited to see this comprehensive benchmark join Pruna’s evaluation lineup! 🚀