Implement automated pipelines for evaluations

Open
  • Is something wrong?
#26 0 comments 0 reactions 1 assignee View on GitHub

@akh64bit is already working on this.

Since Oct 8, 2025.

Assessment

This issue has not been assessed yet.

Description

Goal

  • Each one of our datasets will have a vetted ground truth
    • Vulnerabilities it should find
    • Vulnerabilities it could find (optional)
  • We will have a tool that automatically runs evals on each test case in the dataset
    • Output results
    • Metadata such as latencies, token usage, tool calls, etc.
  • We will have develop an auto-rater that accurately judges the outcome of the run e.g. TP, FP, FN

Challenges

  • The "locality" of the vulnerability is not exact. Oftentimes there are many valid code locations which represent the vulnerability. In addition, each test case may have multiple vulnerabilities, sometimes closely related.
  • We would like to parallelize test case runs i.e. multiple instances of GCLI, but need to know the practical limit.
  • The auto-rater itself needs to be evaluated and improved against its own ground truth.

Phases

  • A simple runner that automates runs of a dataset
  • An auto-rater that accurately evaluates a run
  • Generalize to all datasets
Dominant language
TypeScript
Stars
793
Forks
60
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from gemini-cli-extensions/security

All issues in gemini-cli-extensions/security

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.