OpenHands/software-agent-sdk

[Feature] Implement HookType.PROMPT — LLM-based hook evaluation

Open

#2755 aperta il 8 apr 2026

Vedi su GitHub
 (3 commenti) (1 reazione) (0 assegnatari)Python (350 fork)auto 404
Staleenhancementhelp wanted

Metriche repository

Star
 (895 star)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

Description

HookType.PROMPT is declared in openhands-sdk/openhands/sdk/hooks/config.py:43 as # LLM-based evaluation (future) but has no implementation. Users should be able to define hooks that issue a prompt to an LLM and use the
response to decide whether to allow or block an operation, without writing shell scripts.

Motivation

Currently, all hooks are shell commands. This works well for deterministic checks (e.g., blocking dangerous commands via regex), but is limiting for cases that require judgment or natural language reasoning — for example:

  • Stop hooks: "Evaluate whether the agent has completed all items in the task plan before allowing it to stop."
  • Pre-tool-use hooks: "Determine if this terminal command could cause irreversible damage to the production environment."
  • Post-tool-use hooks: "Assess whether the tool output indicates an error the agent should address before continuing."

These are hard to express as shell scripts but natural to express as LLM prompts.

Current State

  • HookType.PROMPT = "prompt" exists as an enum value but nothing else:
  • HookDefinition requires command as a mandatory field, so configuring a prompt hook crashes with a ValidationError (#2749).
  • HookExecutor.execute() never checks hook.type — every hook is unconditionally routed to subprocess.run().

Proposed Behavior

A prompt hook definition would look something like:

{
  "stop": [
    {                                                                                                                                                                                                                                  
      "matcher": "*",                                                                                                                                                                                                                  
      "hooks": [                                                                                                                                                                                                                       
        {                                                                                                                                                                                                                              
          "type": "prompt",                                                                                                                                                                                                            
          "prompt": "Review the conversation so far. Has the agent completed all items in the original task? Respond with ALLOW if done, DENY if not.",                                                                            
          "model": "claude-sonnet-4-6"                                                                                                                                                                                                 
        }                                                                                                                                                                                                                              
      ]                                                                                                                                                                                                                                
    }                                                                                                                                                                                                                                  
  ]                                                                                                                                                                                                                                    
}                                                                                                                                                                                                                                  

When triggered, the executor should:

  1. Send the prompt to the configured LLM (along with relevant context such as the hook event data).
  2. Parse the LLM response to extract an ALLOW/DENY decision.
  3. Return a HookResult with the appropriate decision, reason, and blocked fields.

Implementation Considerations

  • HookDefinition model: Add optional prompt and model fields. Make command optional (required only when type == "command"). Add a model validator to enforce that command hooks have command and prompt hooks have prompt.
  • HookExecutor: Add a branch in execute() that checks hook.type and routes prompt hooks to an LLM call instead of subprocess.run().
  • LLM integration: Decide how the executor obtains an LLM client — via the existing SDK LLM infrastructure or a lightweight standalone client.
  • Context passing: Determine what context is sent alongside the prompt. The HookEvent data is available (event type, tool name, tool input/output, session ID). For stop hooks specifically, it would be valuable to also include
    TASKS.json content (structured, parseable) or conversation history. PLAN.md is free-form markdown so its usefulness as context depends on the LLM's ability to interpret it.
  • Timeout/cost: Prompt hooks add latency and cost. Consider whether timeout should apply, and whether there should be a token limit or model restriction.
  • Response parsing: Define a structured output format (e.g., JSON with decision and reason fields) or use simple keyword detection (ALLOW/DENY).

Relevant Code

  • Enum declaration: openhands-sdk/openhands/sdk/hooks/config.py:43
  • Hook model: openhands-sdk/openhands/sdk/hooks/config.py:46-56
  • Executor (no type check): openhands-sdk/openhands/sdk/hooks/executor.py:114-258
  • Hook manager: openhands-sdk/openhands/sdk/hooks/manager.py
  • Task tracker (structured persistence): openhands-tools/openhands/tools/task_tracker/definition.py
  • Planning file editor (unstructured): openhands-tools/openhands/tools/planning_file_editor/impl.py

Related Issues

  • #2749 — Validation crash when attempting to use HookType.PROMPT

Guida contributor