Optional [perplexity] extra: reference windowed scorer for the statistical OOD channel (OT.4)

Open
  • Is something wrong?
#235 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python, pytorch
Domain
ai, security

Research direction

Read src/doberman/perplexity.py, the proxy’s default subjective-stack wiring, and pyproject.toml; first inspect how TokenChannelDetector and extra_detectors are used. The issue is blocked by #234, so check that calibration helper before starting. Done means the opt-in scorer and optional dependencies are wired as specified, with tests for the listed guarantees, including host-hook import isolation.

Written by the indexing model from the issue text.

Description

enhancement good first challenge level-6 roadmap

The deterministic Unicode scanner cannot see a GCG-style adversarial suffix: the red-team corpus proves those fixtures pass it clean, and the statistical channel that would catch them has stayed inert because core ships no scorer. Ship the reference scorer as an optional extra.

Build

  • src/doberman/perplexity.py: build_perplexity_fn(model_name="gpt2", window, stride) with lazy transformers/torch imports, scoring the max over sliding token windows so a local suffix inside long benign text still scores high; maybe_build_detector() returns a TokenChannelDetector(perplexity_fn=..., perplexity_threshold=...) or None.
  • pyproject.toml: a new perplexity extra (transformers, torch). Never a default or dev dependency; default CI runs no model.
  • Wiring: the proxy's default subjective stack appends the detector via extra_detectors. The host-hook path never imports this module (tested via a clean-interpreter sys.modules check).

Double opt-in: the extra must be installed AND DOBERMAN_PERPLEXITY=1 set. Without either, behavior is byte-for-byte unchanged.

Guarantees (each one is a test)

  • Raise-only: soft channel, AUTH-capped by the subjective layer; it can never lower a verdict or hard-block.
  • Model-load failure or a raising scorer means the statistical channel is simply absent; every deterministic protection is untouched.
  • Scores only; scored text is never logged or persisted.
  • The shipped default threshold is conservative; operators recalibrate with calibrate_perplexity_threshold on their own benign corpus.

Honest scope: catches high-perplexity adversarial text within a calibrated FPR budget. Defense-in-depth, not a jailbreak detector.

Blocked by #234 (the calibration helper lands first).

Dominant language
Python
Stars
277
Forks
42
Avg merge
1d 19h
Merged PRs (30d)
16

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DobermanCore/Doberman-Core

All issues in DobermanCore/Doberman-Core

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.