Add evals for search and execute tools

Start with the existing executor binding for globalOutbound and the eval-tools runTask and eachModel entry points. Add the files under evals/—test-outbound.ts, utils.ts, search.eval.ts, execute.eval.ts, and end-to-end.eval.ts—then configure vitest.config.evals.ts and the package scripts. Run the eval suite across the listed model providers and verify deterministic search and outbound-request assertions.

Summary

Add an evaluation suite to validate that AI models can correctly use the search and execute tools exposed by the MCP server. Multiple models (Workers AI, OpenAI, Anthropic, Google) should be tested to ensure broad compatibility.

The key insight: we can subclass GlobalOutbound to spy on the actual fetch calls made by LLM-generated code. This gives us deterministic assertions on what endpoints were hit, and per-test control over mock responses — all inside workerd, no MSW needed.

Architecture

Spy on GlobalOutbound.fetch

Every fetch from LLM-generated code flows through GlobalOutbound.fetch(). We subclass it with a TestGlobalOutbound that:

1. Records all outbound requests (method, url, body)
2. Matches against registered handlers to return realistic per-endpoint responses
3. Falls back to a generic success response for unhandled routes

This means:

• LLMs still do the hard part — understanding prompts, searching the spec, writing cloudflare.request() code
• Assertions are deterministic — check the actual outbound request method/path/body
• Responses are realistic — per-test handlers return endpoint-appropriate data so LLM code doesn't choke
• No production code changes — just bind TestGlobalOutbound instead of GlobalOutbound in eval config
• Runs inside workerd — no MSW needed, the spy lives where the fetch happens

TestGlobalOutbound

// evals/test-outbound.ts
import { GlobalOutbound } from '../src/index'

type MockHandler = (req: { method: string; url: string; body?: string }) => Response

export class TestGlobalOutbound extends GlobalOutbound {
  static requests: { method: string; url: string; body?: string }[] = []
  static handlers: Map<string, MockHandler> = new Map()

  static reset() {
    this.requests = []
    this.handlers.clear()
  }

  static on(pattern: string, handler: MockHandler) {
    this.handlers.set(pattern, handler)
  }

  async fetch(request: Request): Promise<Response> {
    const url = new URL(request.url)
    const entry = {
      method: request.method,
      url: url.pathname + url.search,
      body: request.body ? await request.clone().text() : undefined,
    }
    TestGlobalOutbound.requests.push(entry)

    // Match against registered handlers
    for (const [pattern, handler] of TestGlobalOutbound.handlers) {
      if (entry.url.includes(pattern)) {
        return handler(entry)
      }
    }

    // Default fallback
    return Response.json({
      success: true,
      result: [],
      errors: [],
      messages: [],
    })
  }
}

Proposed Structure

evals/
├── test-outbound.ts      # TestGlobalOutbound spy
├── utils.ts              # Client setup, model config
├── search.eval.ts        # Search tool evals
├── execute.eval.ts       # Execute tool evals
└── end-to-end.eval.ts    # Search → execute flows

Vitest config (vitest.config.evals.ts)

import { defineWorkersConfig } from '@cloudflare/vitest-pool-workers/config'

export default defineWorkersConfig({
  test: {
    include: ['evals/**/*.eval.?(c|m)[jt]s?(x)'],
    poolOptions: {
      workers: {
        isolatedStorage: true,
        wrangler: { configPath: './wrangler.jsonc' },
        miniflare: {
          bindings: {
            ENVIRONMENT: 'test',
          },
        },
      },
    },
  },
})

Package.json scripts

{
  "eval:dev": "start-server-and-test --expect 404 eval:server http://localhost:8978 'vitest --testTimeout=60000 --config vitest.config.evals.ts'",
  "eval:server": "wrangler dev --var ENVIRONMENT:test --var DEV_DISABLE_OAUTH:true --port 8978",
  "eval:ci": "start-server-and-test --expect 404 eval:server http://localhost:8978 'vitest run --testTimeout=60000 --config vitest.config.evals.ts'"
}

Code Examples

evals/utils.ts

import { MCPClientManager } from 'agents/mcp/client'

export async function initializeClient(): Promise<MCPClientManager> {
  const clientManager = new MCPClientManager('eval-client', '0.0.0')
  await clientManager.connect('http://localhost:8978/sse')
  return clientManager
}

evals/search.eval.ts

import { describe, it, expect } from 'vitest'
import { runTask } from '@repo/eval-tools/src/runTask'
import { eachModel } from '@repo/eval-tools/src/test-models'
import { initializeClient } from './utils'

eachModel('$modelName', ({ model }) => {
  describe('search tool', () => {
    it('finds Workers endpoints', async () => {
      const client = await initializeClient()
      const { toolCalls } = await runTask(
        client,
        model,
        'Find all Cloudflare Workers script endpoints'
      )

      const searchCall = toolCalls.find((call) => call.toolName === 'search')
      expect(searchCall, 'search tool was not called').toBeDefined()
    })

    it('finds KV namespace endpoints', async () => {
      const client = await initializeClient()
      const { toolCalls } = await runTask(
        client,
        model,
        'Find the API endpoint to create a new KV namespace'
      )

      const searchCall = toolCalls.find((call) => call.toolName === 'search')
      expect(searchCall, 'search tool was not called').toBeDefined()
    })
  })
})

evals/execute.eval.ts

import { describe, it, expect } from 'vitest'
import { runTask } from '@repo/eval-tools/src/runTask'
import { eachModel } from '@repo/eval-tools/src/test-models'
import { TestGlobalOutbound } from './test-outbound'
import { initializeClient } from './utils'

eachModel('$modelName', ({ model }) => {
  describe('execute tool', () => {
    it('lists Workers scripts', async () => {
      TestGlobalOutbound.reset()
      TestGlobalOutbound.on('/workers/scripts', () =>
        Response.json({
          success: true,
          result: [{ id: 'my-worker', etag: 'abc123', modified_on: '2025-01-01' }],
          result_info: { page: 1, per_page: 20, count: 1, total_count: 1 },
          errors: [],
          messages: [],
        })
      )

      const client = await initializeClient()
      await runTask(client, model, 'List all my Cloudflare Workers scripts')

      expect(TestGlobalOutbound.requests).toEqual(
        expect.arrayContaining([
          expect.objectContaining({
            method: 'GET',
            url: expect.stringMatching(/\/accounts\/[^/]+\/workers\/scripts/),
          }),
        ])
      )
    })

    it('lists KV namespaces', async () => {
      TestGlobalOutbound.reset()
      TestGlobalOutbound.on('/storage/kv/namespaces', () =>
        Response.json({
          success: true,
          result: [
            { id: 'ns-1', title: 'MY_KV', supports_url_encoding: true },
            { id: 'ns-2', title: 'SESSIONS', supports_url_encoding: true },
          ],
          result_info: { page: 1, per_page: 20, count: 2, total_count: 2 },
          errors: [],
          messages: [],
        })
      )

      const client = await initializeClient()
      await runTask(client, model, 'List all my KV namespaces')

      expect(TestGlobalOutbound.requests).toEqual(
        expect.arrayContaining([
          expect.objectContaining({
            method: 'GET',
            url: expect.stringMatching(/\/accounts\/[^/]+\/storage\/kv\/namespaces/),
          }),
        ])
      )
    })

    it('creates a D1 database with the right name', async () => {
      TestGlobalOutbound.reset()
      TestGlobalOutbound.on('/d1/database', () =>
        Response.json({
          success: true,
          result: { uuid: 'db-123', name: 'my-db', created_at: '2025-01-01' },
          errors: [],
          messages: [],
        })
      )

      const client = await initializeClient()
      await runTask(client, model, 'Create a new D1 database called "my-db"')

      expect(TestGlobalOutbound.requests).toEqual(
        expect.arrayContaining([
          expect.objectContaining({
            method: 'POST',
            url: expect.stringMatching(/\/accounts\/[^/]+\/d1\/database/),
            body: expect.stringContaining('my-db'),
          }),
        ])
      )
    })
  })
})

evals/end-to-end.eval.ts

import { describe, it, expect } from 'vitest'
import { runTask } from '@repo/eval-tools/src/runTask'
import { eachModel } from '@repo/eval-tools/src/test-models'
import { TestGlobalOutbound } from './test-outbound'
import { initializeClient } from './utils'

eachModel('$modelName', ({ model }) => {
  describe('search then execute', () => {
    it('finds and calls the DNS zones endpoint', async () => {
      TestGlobalOutbound.reset()
      TestGlobalOutbound.on('/zones', () =>
        Response.json({
          success: true,
          result: [
            { id: 'zone-1', name: 'example.com', status: 'active' },
            { id: 'zone-2', name: 'test.dev', status: 'active' },
          ],
          result_info: { page: 1, per_page: 20, count: 2, total_count: 2 },
          errors: [],
          messages: [],
        })
      )

      const client = await initializeClient()
      const { toolCalls } = await runTask(
        client,
        model,
        'Find the endpoint to list DNS zones, then call it'
      )

      // Model used search first
      const searchCall = toolCalls.find((call) => call.toolName === 'search')
      expect(searchCall, 'search tool was not called').toBeDefined()

      // Then actually hit the right endpoint
      expect(TestGlobalOutbound.requests).toEqual(
        expect.arrayContaining([
          expect.objectContaining({
            method: 'GET',
            url: expect.stringMatching(/\/zones/),
          }),
        ])
      )
    })
  })
})

Suggested eval coverage

• Search tool: find Workers endpoints, find KV endpoints, find DNS endpoints, search by HTTP method
• Execute tool: list Workers scripts, list KV namespaces, create D1 database, POST with body, query parameters
• End-to-end: search → execute flows, multi-step operations

Models to test

Use eachModel to run across:

• Workers AI models (via workers-ai-provider)
• OpenAI (GPT-4o, GPT-4o-mini)
• Anthropic (Claude Sonnet)
• Google (Gemini Flash)

Route through AI Gateway for unified API management.

Notes

• The eval-tools package from mcp-server-cloudflare (runTask, eachModel) can be extracted into a shared package or vendored. We don't need checkFactuality / describeEval / vitest-evals — the outbound spy gives us deterministic assertions with plain vitest.
• TestGlobalOutbound subclasses GlobalOutbound and binds in via the executor's existing globalOutbound parameter — no changes to production code needed.
• Each test registers its own handlers with TestGlobalOutbound.on() so responses are realistic and endpoint-appropriate.