Skill Creator quick validator rejects standard compatibility frontmatter

Open Beginner friendly
  • Is something wrong?
#58,538 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@akhil9tiet is already working on this.

Since Oct 31, 2026.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, yaml
Domain
devtools

Research direction

The bug is in codex-rs/skills/src/assets/samples/skill-creator/scripts/quick_validate.py, where the explicit allowed_properties set omits compatibility. Start there and run the reproduction command on a SKILL.md with a compatibility line to see the unknown-key error. Done means compatibility is accepted as a string of 1-500 characters, non-string, empty or over-length values are rejected, and the existing name, description and unknown-key checks still pass.

Written by the indexing model from the issue text.

Description

bug skills
What issue are you seeing?

The bundled skill-creator/scripts/quick_validate.py rejects the standard optional compatibility frontmatter field as an unknown key, even when PyYAML is already available.

Observed diagnostic (exit code 1):

Unexpected key(s) in SKILL.md frontmatter: compatibility. Allowed properties are: allowed-tools, description, license, metadata, name

This is a defect in the helper's validation rules. It does not establish that the runtime skill loader rejects the field or that metadata has been removed.

What steps can reproduce the bug?

At upstream revision 67f0994b532b8740f14b5b738af7b7b2a915b1d6, create example-skill/SKILL.md:

---
name: example-skill
description: An example skill for validating environment requirements.
compatibility: Requires Python and an MCP connection.
---
Follow the requested workflow.

Using an existing Python environment with PyYAML:

python -B codex-rs/skills/src/assets/samples/skill-creator/scripts/quick_validate.py example-skill

The validator reports the unknown-key error above. The same failure was reproduced on four unchanged real skill files containing environment-requirement strings of 110 or 365 characters; two control skills without this field passed.

What is the expected behavior?

The Agent Skills specification permits optional compatibility metadata with 1–500 characters when supplied. OpenAI's skill documentation identifies the open Agent Skills standard.

The helper should accept a valid string, reject non-string/empty/over-length values, and preserve the existing validation rules. Useful MCP, plugin and other environment requirements should not need to be deleted to pass validation.

Additional information

Source identity: Pinned validator source.

  • Git blob: e27023ece4bd259ef36560e19995eec7b6a345bf
  • SHA256: ee6dba90f44d37171c5a6edb8095979c54919ff6822c1a907afca2e78c48738c
  • Local validation environment: macOS, Python 3.13.18, existing PyYAML 6.0.3. The helper has no independent semantic version.

Root cause: the explicit allowed_properties set omits compatibility. A narrow correction is to include that key and validate its supplied value as a string of 1–500 characters, without trimming, truncation or coercion. This does not require changing the runtime loader.

Local verification: an isolated, unpublished source correction passed 12 focused regression tests and validation of all six unchanged real skill inputs. Coverage includes omission, empty/1/500/501-character values, Unicode lengths, wrong YAML types, malformed/unsafe YAML, unknown keys and existing name/description/placeholder checks. The managed validator and all six original skill files remained unchanged; only the two files in the isolated source checkout were modified.

Changed-file formatting passed with local Ruff 0.16.1, which satisfies the declared range but differs from frozen CI 0.15.13. Supplemental lint retains the same two baseline findings (EXE001 and ISC004); the new test file has none. Full repository/hosted CI was not run. These are helper-level results, not runtime qualification or a released fix.

Related reports checked: #36245 and #24195 concern missing PyYAML; #43509 concerns Python-environment conventions; #44187 concerns empty required metadata and other preflight behavior. Their reviewed discussions do not cover this specific optional-field rejection. The empty-name/description behavior remains outside this report.

This is a bug report and root-cause analysis under the repository's contribution policy. No code was pushed, no external code PR was opened, and no patch was installed.

Dominant language
Rust
Stars
131k
Forks
20.7k
Avg merge
1m
Merged PRs (30d)
997

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/codex

All issues in openai/codex

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.