Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Confusion about evaluation datasets

Open
#3 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
php
Domain
testing-qa

Research direction

Inspect evaluation dataset entries 297 and 298, which are the two examples named in the issue, and compare their PHP conversion prompts with the embedded reference code. Done means establishing whether the prompt/reference mismatch is intentional or correcting the affected dataset entries accordingly.

Written by the indexing model from the issue text.

Description

When I was looking at the evaluation questions, I found that some PHP questions required the code to be converted into PHP, but the reference code provided was already PHP. Is this intentionally designed to test the model copying ability or is it a design error?

Two examples are as follows:

"content": "Please convert the following code to PHP, with the function signature `function order_by_points($nums)`\nNote:\n1. Only implement the target function and import necessary packages, do not generate other code\n\n```php\n\nfunction order_by_points($nums) {\n    \n    function digits_sum($n) {\n        $neg = 1;\n        if ($n < 0) {\n            $n = -1 * $n;\n            $neg = -1;\n        }\n        $n = array_map('intval', str_split($n));\n        $n[0] = $n[0] * $neg;\n        return array_sum($n);\n    }\n    usort($nums, function($a, $b) {\n        return digits_sum($a) <=> digits_sum($b);\n    });\n    return $nums;\n}\n\n```",
"id": 297,

"content": "Please convert the following code to PHP, with the function signature `function do_algebra($operator, $operand)`. Note: Only implement the target function and import necessary packages, do not generate other code.\n\n```php\n\nfunction do_algebra($operator, $operand) {\n\n    $expression = strval($operand[0]);\n    foreach (array_combine($operator, array_slice($operand, 1)) as $oprt => $oprn) {\n        $expression .= $oprt . strval($oprn);\n    }\n    return eval('return '.$expression.';');\n}\n\n```",
"id": 298,
Dominant language
Python
Stars
122
Forks
10
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.