Make reference auditors honor `audit_scoring_policy` in the manifest

Open
  • Is something wrong?
#330 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@0xzoz is already working on this.

Since Nov 12, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
54/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, solidity
Domain
machine-learning, testing

Research direction

Read both reference implementations of load_scoring_policy, including cache_model_1/services/scoring.py, and compare them with the GI-start consumer at dincli/cli/modelownerd/gi.py and the shipped manifest. Trace both auditors' service tests and run focused cases with two different min_accuracy thresholds. Done means both auditors honor the documented canonical field, edge cases for missing, legacy, conflicting, and malformed policies are covered, and the example manifests and documentation explain the median rule.

Written by the indexing model from the issue text.

Description

Approved

Observed against ed175d0deceddfa2e63175ec7752c800b31c1fbf during the November 10 Fashion-MNIST local test investigation. The subsequent full round with three notebooks and a Raspberry Pi used recorded workarounds; its success does not establish that this upstream defect is fixed. Related: #276 and #39.

Problem and evidence

The shipped manifests put thresholds under audit_scoring_policy, and GI-start code reads that field. Both reference load_scoring_policy implementations read scoring_policy. With the shipped layout they silently use defaults instead of the configured audit thresholds.

A focused reproduction extracted the actual upstream loader and default dictionary, wrote the shipped manifest with min_accuracy changed to 99, and called the loader. Both models returned an effective threshold of 50.0. This is a loader-level reproduction, not a second live rejection/acceptance experiment.

The numeric defaults matched our intended gates during the completed round, so this does not show that a model was wrongly accepted in that run. It shows that a model owner's non-default policy can be ignored.

Proposed scope

Choose and document one canonical field across manifests and consumers. If supporting the old name, define explicit precedence or reject conflicting values. Validate policy shape and test non-default values through the auditor service. Align the stale legacy_mean_until_contract_v2 metadata with the median already used on-chain; do not change the contract aggregation rule as part of this fix.

Acceptance criteria
  • Changing audit_scoring_policy.metrics.thresholds.min_accuracy changes the policy used by both reference auditors.
  • A known model passes/fails as expected under two deliberately different thresholds.
  • Missing fields, legacy aliases, conflicting fields and malformed policies have documented behavior.
  • GI pass-score configuration and auditor eligibility policy remain distinct, intentional controls.
  • Example manifests and documentation describe the effective median rule.
Full-round test report

See Discussion #331 — Fashion-MNIST with three notebooks and a Raspberry Pi for the setup diagram, completed protocol roles, results and experiment adaptations.

Dominant language
Python
Stars
4
Forks
7
Avg merge
5d 15h
Merged PRs (30d)
43

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from InfiniteZeroFoundation/DevNet

All issues in InfiniteZeroFoundation/DevNet

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.