Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

security: eval experiment DB persists judge/variant model output unscrubbed

Open Beginner friendly
  • Is something wrong?
#986 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
database, security

Research direction

Start with src/brainlayer/eval/experiment_store.py and read AGENTS.md for the scrub_llm_output rule. Inspect ExperimentStore.add_judgment and upsert_variant, including their INSERT paths, then verify that rationale, scores, and enrichment are scrubbed before persistence and fail closed like the other sites.

Written by the indexing model from the issue text.

Description

Gap

The eval experiment DB (abcde-experiment.db, src/brainlayer/eval/experiment_store.py) persists model output without scrub_llm_output:

  • enrichment_judge._persist_judgment → ExperimentStore.add_judgment writes the judge's rationale and scores_json.reason as the model wrote them.
  • ExperimentStore.upsert_variant writes enrichment_json as given (no in-repo caller today, so any future caller inherits the gap).

Scope

Separate eval DB, not the canonical DB and not search-reachable, so a lower priority than session_enrichments. Still model output at rest, and every other LLM-output → DB write goes through scrub_llm_output (AGENTS.md: "every LLM output field passes scrub_llm_output before it is persisted").

Fix shape

Scrub rationale, scores and enrichment in ExperimentStore before the INSERT, so every caller is covered at the chokepoint. Fail closed like the other sites.

Found during the audit for the tool_usage_stats scrub lane (security).

— brainlayerClaude (worker) · claude-code/claude-opus-5-5

Dominant language
Python
Stars
9
Forks
7
Avg merge
2h 8m
Merged PRs (30d)
211

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from EtanHey/brainlayer

All issues in EtanHey/brainlayer

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.