Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[External Plugin]: hermes-jailbench

Open
#3,307 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
github
Domain
ai, devtools, security

Research direction

Start by reviewing the pinned commit ec933be5e4c9f8c07caa0c9f19b45a925067d655 in hermes-labs-ai/hermes-jailbench and compare its submission details with the repository's contribution, security, and responsible AI policies. Check the referenced .claude-plugin/marketplace.json entry and verify that the plugin is suitable for inclusion; done means the review is complete and any required marketplace update is identified.

Written by the indexing model from the issue text.

Description

external-plugin needs-review:HIGH ready-for-review
Plugin name

hermes-jailbench

Short description

Jailbreak regression benchmark for LLM endpoints with repeatable known-pattern attacks and deterministic scoring.

GitHub repository

hermes-labs-ai/hermes-jailbench

Plugin path inside the repository

No response

Ref to review

No response

Commit SHA to review

ec933be5e4c9f8c07caa0c9f19b45a925067d655

Version

0.2.1

License identifier

MIT

Author name

Hermes Labs

Author URL

https://github.com/hermes-labs-ai

Homepage URL

https://github.com/hermes-labs-ai/hermes-jailbench

Keywords

security, red-teaming, llm-security, benchmark, evaluation, developer-tools

Additional notes for reviewers

This is a defensive regression-testing tool: it runs a fixed corpus of publicly known jailbreak patterns against an LLM endpoint the user already controls or is authorized to test, and produces deterministic pass/fail scoring so a team can track whether their own safety mitigations regress between releases. It does not discover novel attacks, target third-party services, or exfiltrate data. Listed in the Hermes Labs Copilot marketplace manifest (https://github.com/hermes-labs-ai/copilot-plugins/blob/main/.claude-plugin/marketplace.json) as a root url source, tracking the main branch. The commit SHA above is main's HEAD at submission time, pinned here for immutable review. Submitted by the plugin's own maintainer (Hermes Labs).

Submission checklist
  • The plugin lives in a public GitHub repository.
  • The ref and/or sha I provided is immutable (release tag and/or full 40-character commit SHA), not a branch.
  • This submission follows this repository's contribution, security, and responsible AI policies.
  • This plugin is not already listed in the Awesome Copilot marketplace.
Dominant language
JavaScript
Stars
39.2k
Forks
5k
Avg merge
1d 18h
Merged PRs (30d)
114

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from github/awesome-copilot

All issues in github/awesome-copilot

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.