Industrial Automation Challenge Track 1 grader format clarification
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 38/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- github, python
- Domain
- documentation, testing-qa
Research direction
Start by reading the listed reports under docs/research/industrial-automation-challenge/, especially LEADERBOARD_SUBMISSION_REPORT.md and the live-status README. Reproduce the documented Track 1 submission attempts or inspect the hosted grader behavior. Done means the required schema and answer encoding are clarified, or the metric failure is documented and escalated with reproducible evidence.
Written by the indexing model from the issue text.
Description
Ready-to-send note: Track 1 hosted grader issue
Subject: Industrial Automation Challenge Track 1 grader format clarification
Hello,
We have an AIF KG-governed Industrial Automation Challenge submission package.
Track 2 has been accepted and scored successfully:
- competition:
industrial-automation-challenge-track-2 - submission ref:
55368732 - status:
SubmissionStatus.COMPLETE - public score:
1.00000
Track 1 appears blocked by the hosted grader rather than by local generation.
We tried the public starter-kit format, the visible Kaggle metadata-style
format, and numeric answer encodings. All variants reach Kaggle upload, but
the hosted evaluation fails.
Observed Track 1 refs:
| Ref | Submission shape | Result |
|---|---|---|
55368741 |
3,048-row full test file | Submission has 3048 rows but 2758 were expected |
55368816 |
290-row multi_choice_question_answering tail only |
Submission has 290 rows but 2758 were expected |
55368833 |
2,758-row id,answer with option letters |
Submission column answer must be a number |
55368957 |
2,758-row numeric answer, A=0 | Evaluation metric raised an unexpected error |
55368978 |
2,758-row numeric answer, A=1 | Evaluation metric raised an unexpected error |
55408937 |
2,758-row metadata-preserving numeric answer, A=0 |
Submission column option_desc must be a number |
55408938 |
2,758-row metadata-preserving numeric answer, A=1 |
Submission column option_desc must be a number |
55488733 |
2,758-row public eight-column metadata schema with letter option |
Submission column option must be a number |
55488747 |
2,758-row all-numeric eight-column schema, A=0 | Evaluation metric raised an unexpected error |
55488774 |
2,758-row sequential-ID all-numeric eight-column schema, A=0 | Solution and submission values for id do not match |
55488794 |
2,758-row all-numeric eight-column schema, A=1 | Evaluation metric raised an unexpected error |
55488832 |
2,758-row minimal id,option numeric schema, A=0 |
Evaluation metric raised an unexpected error |
55488849 |
2,758-row minimal id,option numeric schema, A=1 |
Evaluation metric raised an unexpected error |
Could you please clarify the required Track 1 submission schema and answer
encoding, or check whether the hosted Track 1 metric has an internal error?
For reproducibility, a private GitHub handoff repository is available:
https://github.com/Harshul-24/aif-kg-industrial-automation-challenge
The private GitHub handoff repository is shared with biswasprateek. As of the
latest refresh on 2026-08-13 12:35 CDT, pending invitations were empty and
biswasprateek had write permission.
The main local documentation is:
docs/research/industrial-automation-challenge/LEADERBOARD_SUBMISSION_REPORT.mddocs/research/industrial-automation-challenge/COMPLETION_AUDIT.mddocs/research/industrial-automation-challenge/AIF_KG_ASSETOPS_TRACK2_RESEARCH_SUMMARY.mddocs/research/industrial-automation-challenge/runs/kaggle-live-status-20260813/README.md
Thank you.
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 332
- Avg merge
- 8h 40m
- Merged PRs (30d)
- 33
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from IBM/AssetOpsBench
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
IBM/AssetOpsBench#612 ·
Maintainers usually reply within 1 day
-
Add few scenarios in lite configuration for meaPossibly taken @ShuxinLin claimed this 11 days ago. Open
IBM/AssetOpsBench#582 · 1 assignee ·
Maintainers usually reply within 1 day
-
security
Difficulty 4/5 3-5 days Newbie friendliness 25/100
IBM/AssetOpsBench#580 · 9 comments ·
Maintainers usually reply within 1 day
-
experiment harbour design for AssetOpsBenchPossibly taken @ShuxinLin claimed this 14 days ago. Open
IBM/AssetOpsBench#557 · 2 assignees ·
Maintainers usually reply within 1 day
-
Design discussion on FMEAPossibly taken @DhavalRepo18 claimed this 19 days ago. Open
IBM/AssetOpsBench#550 · 2 assignees ·
Maintainers usually reply within 1 day
All issues in IBM/AssetOpsBench
Similar issues
-
enhancement good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
hatchet-dev/hatchet#5179 ·
Maintainers usually reply within 1 day
-
python-version
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
bug javascript P2-medium python release:v3.1
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
adrirubio/claude-deck#546 ·
Maintainers usually reply within 1 day