Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Prompt router A/B has no control arm by default; MCP <channel> messages are routed as tasks

Open
  • Is something wrong?
#381 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, python, shell
Domain
ai-infra-agents, cli, testing

Research direction

Start with hooks/ripwire-claude-route.sh and resolve_arm to trace the default arm, then inspect bench/routing_ab_report.py and INSTALL.md for the control-arm behavior. Check the prompt filtering for the <channel wrapper alongside the existing skipped prefixes. Run the routing tests and verify that channel events are skipped and the registered A/B can produce a control arm; the optional readout-date suggestion is separate work.

Written by the indexing model from the issue text.

Description

Summary

Field data from one machine running hooks/ripwire-claude-route.sh (installed from 0.5.0, built from bacfa3b7b) between 2026-09-10 and 2026-10-07 shows two problems for the prompt-router A/B pre-registered in docs/EVALS.md ("Claude Code prompt router — PRE-REGISTERED 2026-09-02").

  1. The control arm never existed.

    • No ~/.ripwire/meter.conf was present, so resolve_arm fell through to the treatment default for every session.

    • bench/routing_ab_report.py on the real log:

      arm         prompts  recommended    adopted     rate  sessions
treatment     18968         1959          0     0.0%       427
control           0            0          0      n/a         0
UNDERPOWERED -- ... control has 0 (needs 40 more)

    • The registration defines the verdict as a treatment − control difference with at least 40 recommended prompts per arm. With the shipped default it can never be computed.

    • On current main, INSTALL.md only ever deletes meter.conf; nothing writes arm=auto.

  2. Non-user prompts drive almost all recommendations. Claude Code delivers MCP channel events (a <channel source="…"> wrapper; here from a crew-messaging MCP plugin) through the same UserPromptSubmit path.

    • Current main skips prompts that start with <task-notification> or <system-reminder>, but not <channel.
    • On this machine, over 14 days, 1926 distinct injected recommendations (2540 transcript records, because resumed sessions copy history) broke down by trigger:
      • 926 (48.1%) started with <channel .
      • 845 (43.9%) started with <task-notification (already skipped on main).
    • Typical recommendations for channel messages: a --grep= whose pattern is the channel's source tag, or a --pack-task= whose argument is the channel tag itself.

Adoption

  • Registered window (the next two ripwire calls in the same turn, since the hook deletes its pending file on the next prompt): 3 of 1926 were followed by any ripwire call, and 0 adopted the recommended verb. This agrees with the meter above.
  • Open-ended window (the next two ripwire calls at any later point): 317 followed, 29 adopted (1.5%).
  • Instrument check: a positive control in an isolated RIPWIRE_HOME (a recommend for --grep, then ripwire . --grep=… through ripwire-nudge.sh) logged "outcome":"adopted", so the --observe path can record an adoption.
  • Unexplained: the meter holds 6 RouteObservation rows, while transcripts show 3 same-turn follow-ups.
  • Relevance: we hand-checked 5 recommendations against their prompts; all 5 were unrelated to the task. That is a small sample.

Suggestions

  • Treat a leading <channel like <task-notification> and <system-reminder>: skip the classifier and log skip-system. Channel events are not user input.
  • Either default the router to arm=auto while its registration is open, or have the hook or report state loudly that no control arm exists.
  • Optionally, have routing_ab_report.py print the registered readout date and flag when it has passed.

We have removed the route hook from our settings. The PreToolUse and SessionStart hooks stay installed.

Dominant language
C++
Stars
2.5k
Forks
158
Avg merge
2d 9h
Merged PRs (30d)
55

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from redhat-et/ripwire

All issues in redhat-et/ripwire

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.