Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

eval gate is a source reader: intercept spawns in the eval kernels instead

Open
  • Is something wrong?
#13 6 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
bun, javascript, python, typescript
Domain
security

Research direction

Start in packages/coding-agent/src/eval/py and packages/coding-agent/src/eval/js, then read eval/agent-bridge.ts and eval/tool-bridge.ts to understand the existing host bridge. Implement execution-time interception for the listed Python and JavaScript spawn APIs through that bridge, then perform the required binary rebuild and verify the fork remains compatible with upstream merges.

Written by the indexing model from the issue text.

Description

enhancement ready-for-human

PR #12 gates eval's subprocess spawns by reading the submitted source and extracting every spawn whose command is written as a literal. That closes the case that shows up in transcripts, where an agent blocked at bash rewrites the same command in Python. It does not close the case where the code hides the spawn from a source reader.

A command assembled at runtime is handled: it reads as an opaque site and raises a permission request. The gap is a spawn the reader never sees at all, which fails open:

getattr(os, "sys" + "tem")(cmd)
__import__("os").system(cmd)
exec(compile(payload, "<s>", "exec"))

globalThis[["child", "_process"].join("")]

None of those match a callee name, so no site is produced and nothing is judged.

The fix

Intercept spawns inside the eval kernels rather than in the source. Every spawn then calls back to the host classifier at execution time, with the real argv, and there is nothing to hide from.

Touch points in the OMP fork:

  • packages/coding-agent/src/eval/py — patch subprocess and the os.system/os.exec* family in the Python kernel.
  • packages/coding-agent/src/eval/js — patch child_process and Bun.spawn in the worker.

The runtimes already have a host bridge (eval/agent-bridge.ts, eval/tool-bridge.ts); route the classifier call over it rather than opening a second channel.

Cost

A fork edit plus a binary rebuild, and the patch has to survive upstream merges. That is why PR #12 took the plugin-sized version first.

When to do this

When session logs show agents building command strings to slip past the parser. Static extraction is the cheap version and it is enough while the observed behavior is agents taking the shortest path rather than evading a gate. A run that shows reflection or string arithmetic around a spawn is the signal that this is worth the fork.

Dominant language
TypeScript
Stars
0
Forks
1
Avg merge
2h 10m
Merged PRs (30d)
64

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from STRML/omp-classifier

All issues in STRML/omp-classifier

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.