DP: default noise_multiplier (σ = 5.0 at clipping_norm 10) reduces the reference model to chance accuracy once DP is enabled

Open
  • Is something wrong?
#299 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
machine-learning, security

Research direction

Start with tests/test_cache_client_dp.py and the test_manifest_dp_full_noise_impact test, then read cache_model_0/manifest.json and Documentation/public/manifest.md for the shipped DP settings. The issue presents a documentation-only interim option and an accountant-dependent option; the latter needs the DP specialist's input on issue No. 279. Done means documenting the measured utility impact without making an ε claim, or agreeing on an accountant-backed setting and updating the test.

Written by the indexing model from the issue text.

Description

Summary

With the reference manifest's DP block switched on, the default noise wipes out the model. cache_model_0/manifest.json ships noise_multiplier: 0.5 with clipping_norm: 10.0 (raised from 1.0 by PR No. 263 / issue No. 247). The noise scale is noise_multiplier × clipping_norm, so σ = 5.0 per coordinate, and accuracy falls to chance:

Source Baseline Clip only Clip + default noise
PR No. 263 tests (seed 42, 2,000 samples from dataset/test/test_dataset.pt, reproducible) 0.860 0.860 0.120
PR No. 293 write-up (separate experiment, script not committed) 85.25% 85–86% ~13%, at every clipping_norm from 1 to 20, and for update_gaussian too

DP is "enabled": false by default, so nothing breaks today. But a model owner who turns DP on with the shipped parameters gets a useless model and no warning.

Why

Noise is added independently to every floating coordinate. For a model with D parameters, the total noise norm grows as σ·√D. cache_model_0's MLP has ~109K parameters (√D ≈ 330), so the noise swamps the signal regardless of how well clipping_norm is tuned. Raising clipping_norm to stop over-clipping (PR No. 263) also raised σ by the same factor. PR No. 293's sweep (not reproduced here) only recovers utility at noise_multiplier ≈ 0.001.

The current test pins this behaviour rather than flagging it: test_manifest_dp_full_noise_impact (tests/test_cache_client_dp.py) asserts dp_acc < baseline_acc - 0.3. Its docstring already suggests ~0.05 "would restore σ≈0.5" but leaves the choice open.

What needs deciding

Picking noise_multiplier is a privacy-budget decision, not a tuning knob. Documentation/technical/audits/dp-mechanism-review.md puts accountant integration (item 3) and DP-SGD (item 4) ahead of any ε claim. So the options are:

  1. Interim (docs/manifest only): keep DP off by default, but warn clearly in Documentation/public/manifest.md and the manifest example that the shipped noise_multiplier gives no utility for cache_model_0, and give a measured utility-vs-noise table. No ε claim.
  2. With the accountant (issue No. 279 / review item 3): choose a default noise_multiplier together with a reported (ε, δ) and a measured accuracy, and update test_manifest_dp_full_noise_impact to assert acceptable utility at that setting.

Option 1 can land now. Option 2 needs the DP specialist's input on issue No. 279.

References

  • Issue No. 247 (closed), PR No. 263: clipping_norm retune and the real-weight accuracy tests
  • PR No. 293: noise_multiplier sweep (Finding 2), changes requested
  • Issue No. 279: DP mechanism fixes, DP specialist tagged
  • cache_model_0/manifest.json dp.parameters; Documentation/public/manifest.md "Choosing clipping_norm"
Dominant language
Python
Stars
4
Forks
7
Avg merge
3d 7h
Merged PRs (30d)
33

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from InfiniteZeroFoundation/DevNet

All issues in InfiniteZeroFoundation/DevNet

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.