Bug: _check_skill_structure validates `body` (no frontmatter) instead of full skill text, causing false failures

Open Beginner friendly
#74 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python
Domain
tooling

Research direction

In evolution/skills/evolve_skill.py, inspect the baseline and evolved validate_all call sites around lines 122 and 189, then compare them with skill_module.py's raw/body definitions. Reproduce with --eval-source sessiondb or synthetic and confirm valid skills no longer produce evolved_FAILED.md and the constraint log passes skill_structure.

Written by the indexing model from the issue text.

Description

Summary

ConstraintValidator._check_skill_structure() always receives skill text without YAML frontmatter, so it perpetually fails with "Skill missing: YAML frontmatter (---)" even when the skill file is perfectly valid.

Root Cause

In evolution/skills/evolve_skill.py, two call sites pass the wrong slice of the parsed skill dict to validate_all():

Baseline check (line ~122):

# BUG: skill["body"] is the markdown body only — no frontmatter
baseline_constraints = validator.validate_all(skill["body"], "skill")

Evolved skill check (line ~189):

# BUG: evolved_body is also frontmatter-stripped; evolved_full has it
evolved_constraints = validator.validate_all(evolved_body, "skill", baseline_text=skill["body"])

skill_module.py documents the dict clearly:

  • "raw" — full file content (frontmatter + body)
  • "frontmatter" — YAML between --- markers
  • "body" — markdown after frontmatter

_check_skill_structure looks for text.strip().startswith("---"), which can never match when called with "body".

Impact

  • Every evolved skill is marked FAILED regardless of actual content quality
  • Evolved output is saved as evolved_FAILED.md and never deployed
  • The constraint gate is entirely non-functional for the skill_structure check
  • baseline_text passed to _check_growth is also wrong ("body" vs "raw"), making the growth-limit comparison slightly incorrect

Fix

# Line ~122 — baseline check
baseline_constraints = validator.validate_all(skill["raw"], "skill")

# Line ~189 — evolved check  
evolved_constraints = validator.validate_all(evolved_full, "skill", baseline_text=skill["raw"])

evolved_full is already computed just above via reassemble_skill(skill["frontmatter"], evolved_body).

Steps to Reproduce

  1. Run any skill evolution with --eval-source sessiondb or synthetic
  2. Observe that evolved output always lands in output/<skill>/evolved_FAILED.md
  3. Check the constraint log — skill_structure will always show ✗

Environment

  • hermes-agent-self-evolution cloned from main (tested 2026-05-10)
  • DSPy fallback to MIPROv2 (GEPA unavailable in installed version)
  • Both video-transcoding and audio-transcription skills affected
Dominant language
Python
Stars
5.4k
Forks
638
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NousResearch/hermes-agent-self-evolution

All issues in NousResearch/hermes-agent-self-evolution

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.