[AI] Contextual skill loading — progressive disclosure for agent abilities

Open
#37 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
javascript
Domain
ai, tooling

Research direction

Start with the eager tool-loading path in src/extensions/services/react-agent.js around lines 787-790, then inspect tool-registry.js and message-router.js. Review the proposed new src/extensions/services/skill-registry.js and src/extensions/skills/ directory requirements. Done means lite and full discovery modes work, skills load during ReAct, existing registered abilities remain compatible, and third-party skill extension is supported.

Written by the indexing model from the issue text.

Description

Goal

Implement a skill-based progressive disclosure system for the ReAct agent. Instead of sending all tool descriptions to the LLM on every request, the agent starts with a lightweight skill index and loads full instructions + tools on demand when it detects a relevant task.

This is critical for our 4096-token context window (Qwen 3 1.7B) — as the ability count grows beyond the current 13, the system prompt will consume an unsustainable share of available tokens.

Why: The current architecture eagerly loads all abilities into every LLM request (react-agent.js:787-790). The system prompt grows linearly with tool count. Progressive disclosure solves this — the agent sees a compact index of what it can do, and only loads the full details when it needs them. This is the same pattern used by Claude Code (deferred tools + skill loading), MCP tool discovery, and the laravel-ai-sdk-skills package.

How It Works

The Pattern (Progressive Disclosure)

The core idea: don't tell the agent everything upfront — let it discover and load capabilities on demand.

This pattern is well-established in AI tooling:

  • Claude Code uses deferred tools (agent sees tool names only, fetches full schemas on demand) and skills (markdown files with instructions loaded contextually)
  • MCP separates tool discovery (list available tools) from tool invocation (call a specific tool)
  • Laravel AI SDK Skills splits capabilities into markdown files with YAML frontmatter; agent gets lite tags and loads full content via a skill tool call

For our small local model with a 4096-token budget, this matters even more than for cloud LLMs.

  1. Lite mode (default): Agent sees only skill names + one-line descriptions as compact tags:

    <skill name="site-health" description="Check plugins, themes, updates, and disk usage" />
    <skill name="content-management" description="List, search, and manage posts and comments" />
    <skill name="security" description="Security scans, error logs, and user auditing" />
    
  2. On-demand loading: When the agent detects a task matches a skill, it calls a load_skill tool to get the full instructions and tool definitions for that skill only.

  3. Skill definition files: Each skill is a Markdown file with YAML frontmatter:

    ---
    name: site-health
    description: Check plugins, themes, updates, and disk usage
    abilities:
      - plugin-list
      - theme-list
      - update-check
      - disk-usage
    ---
    
    # Site Health
    You are checking the health of a WordPress site. Start with an overview...
    
New Tools for the Agent
  • list_skills — returns the lite index of all available skills
  • load_skill — loads full instructions + tool definitions for a specific skill
  • skill_read (stretch) — read reference files bundled with a skill (e.g., checklists, templates)

Requirements

  • Skills are defined as Markdown files (with YAML frontmatter) in a skills directory
  • Skill definitions group existing abilities by domain and include contextual instructions
  • Agent system prompt includes only skill tags in lite mode, not full tool definitions
  • Agent can load a skill mid-conversation via a load_skill tool call
  • Loaded skill tools become available for subsequent ReAct iterations
  • Support both "lite" (default) and "full" discovery modes
  • Skills should be extensible by third-party plugins (like abilities are today via wp.agenticAdmin.registerAbility())

Key Files

File Changes
src/extensions/services/react-agent.js System prompt refactor — skill tags instead of flat tool list
src/extensions/services/tool-registry.js Add skill-aware getBySkill() / getSkillIndex() methods
src/extensions/services/message-router.js Optionally route to skill before ReAct
src/extensions/skills/ (new) Skill definition Markdown files
src/extensions/services/skill-registry.js (new) Skill discovery, parsing, and loading

Reference Pattern

The progressive disclosure pattern for AI agents follows this general flow:

Agent Init → Register all abilities internally
           → Build lite skill index (name + description only)
           → System prompt includes skill tags, NOT full tool list

User Request → Agent sees skill tags
             → Recognizes relevant skill (e.g., "list plugins" → site-health)
             → Calls load_skill("site-health")
             → Gets full instructions + tool definitions for that skill
             → Calls the actual ability tool (e.g., plugin-list)
             → Returns result to user

Key design principles:

  • Skill = grouping layer on top of existing abilities, not a replacement
  • Markdown + YAML frontmatter for skill definitions (easy for devs to write, digestible for LLMs)
  • Three built-in tools (list_skills, load_skill, skill_read) give the agent self-serve access
  • Discoverable vs active — skills listed in the prompt are discoverable; skills become active only when loaded

Technical Notes

  • Context budget: With 4096 tokens, the current system prompt (~300 tokens for 13 tools) leaves ~3700 for conversation. At 30+ tools this becomes untenable. Skill tags are ~15 tokens each vs ~25 per full tool definition.
  • Relates to #20 (tool selection at scale) — this is the architectural approach to that problem. #20 discussed pre-filtering; skills are the modular unit for that filtering.
  • Relates to #3 (vector database for semantic selection) — skills could be selected semantically as a future enhancement, but keyword matching is sufficient for v1.
  • Backward compatibility: Abilities registered via registerAbility() should continue to work. Skills are a grouping layer on top of abilities, not a replacement.

Skills Needed

  • AI/JS Dev: Skill registry, system prompt refactor, new tools
  • Writers: Define skill groupings and contextual instructions for each skill
  • LLM Tester: Validate that skill loading works reliably with Qwen 3 1.7B
Dominant language
JavaScript
Stars
28
Forks
6
Avg merge
4d 3h
Merged PRs (30d)
2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pluginslab/wp-agentic-admin

All issues in pluginslab/wp-agentic-admin

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.