Measure performance and accuracy on customized dataset

Open

#109 opened on Sep 22, 2021 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 5/5
Estimated time: Over a week
Newbie friendliness: 25/100

Issue type: Feature
Clarity: Needs clarification
Activity status: Stale
Tech stack: c
Domain: machine-learning, performance

Research direction

Start by reproducing the medium-performance benchmark with EE_MODEL_VERSION set to the custom dataset and the dataset under /ulp-mlperf/datasets/. Trace why benchmark mode reports “Need at least 5 inputs” and determine the requirements for measuring performance and accuracy on datasets beyond ad, vww, kws, and ic. Done means custom-dataset evaluation works or its supported limitations are documented.

Written by the indexing model from the issue text.

Description

Hi,
Is there any way to measure performance and accuracy on customized dataset (other than ad, vww, kws, ic) ?
I have tried setting EE_MODEL_VERSION to my own dataset and put the dataset under /ulp-mlperf/datasets/ directory, but I got the following error when running medium performance:

Need at least 5 inputs to run benchmark mode

Dominant language: C
Stars: 479
Forks: 116
PR merge metrics: No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

if(cond,a,b) treats any nonzero cond as true in every bngsim backend (ExprTk interpreter, C codegen, SSA), which contradicts docs/reference/expressions.md ('true when > 0.5') and BNG's run_network. A non-boolean condition in (-inf,0) or... Open

bug

Difficulty 1/5 Under an hour Newbie friendliness 60/100

lanl/bngsim#774 · Opened Sep 23, 2026
Signed left-shift undefined behavior in `nbase/nbase_addrset.c` (`(1<<31)`, 4 sites) Open

Nmap

Difficulty 1/5 Under an hour Newbie friendliness 85/100

nmap/nmap#3514 · Opened Sep 23, 2026
[BUG] libcuml fails to build with -DCUML_ALGORITHMS=SVM (cuVS not linked) Open

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

NVIDIA/cuml#8705 · Opened Sep 23, 2026
Vulkan int8-coopmat matmul on RDNA3 (#27952): RX 7900 XTX 27B benchmark data + a small boundary-safety fix Open

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

ggml-org/llama.cpp#29342 · Opened Sep 23, 2026
[flang][cuda] CUFPredefinedVarToGPU crashes on a fir.copy whose destination is the predefined-variable declare Open

flang:fir-hlfir

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

llvm/llvm-project#225935 · Opened Sep 23, 2026

More C issues