Fix Doc init and missing metadata in Summarization tasks

Open Beginner friendly
#1,134 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
python
Domain
machine-learning

Research direction

Start in lighteval/tasks/tasks/summarization.py with the cnn_dm_prompt and xsum_prompt functions, then reproduce the constructor error using the CNN/DM example from the issue. Check that both summarization documents satisfy the Doc API and preserve the source article in Doc.specific. Done means the prompts construct successfully and extractiveness and faithfulness can access the source text.

Written by the indexing model from the issue text.

Description

Describe the bug

The cnn_dm_prompt and xsum_prompt functions in the summarization task fail with a TypeError because they do not provide the required choices and gold_index arguments to the Doc constructor. Furthermore, metrics such as extractiveness and faithfulness fail because the source text is not passed in the Doc.specific field.

To Reproduce

from lighteval.tasks.tasks.summarization import cnn_dm_prompt
# Using a standard record from CNN/DM
# doc = cnn_dm_prompt({"article": "...", "highlights": "..."}, "cnn_dm")
# TypeError: Doc.__init__() missing 2 required positional arguments: 'choices' and 'gold_index'

Expected behavior

  • The Doc object should be initialized with placeholder choices=[""] and gold_index=0 to satisfy the API for generative tasks.
  • The specific field should include {"text": line["article"]} (or document) for metric internal access.

Version info

  • OS: mac
  • Lighteval version: main (local development)
Dominant language
Python
Stars
2.5k
Forks
557
Avg merge
1d 6h
Merged PRs (30d)
1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/lighteval

All issues in huggingface/lighteval

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.