MCP Apps Conformance Test - Host Interface Requirements

Bắt đầu với đề xuất HostTestInterface trong issue này và so sánh từng phương thức với các điều khoản được liên kết trong đặc tả ngày 2026-01-26 và đặc tả bản nháp. Sau đó, kiểm tra conformance runner hiện có và các shim browser automation cho từng host. Công việc được xem là hoàn tất khi các host có thể cung cấp các phương thức bắt buộc và các suite được liệt kê chạy với trạng thái được cô lập cùng các verdict do hệ sinh thái quản lý.

MCP Apps — Host Test Interface & Conformance Suite

Following up this issue on setting up a public testing platform, we have decided to make a clear definition of which interface is required by host to implement to make the automated testing easier. Here is the proposal of interface, and the thinking behind it, based on the current test we have for the MCP App specification

Why an interface

Today the runner drives each host through browser automation with per-host shims (Playwright selectors, console scraping, tab inspection). That does not scale and couples every test to a host's DOM. Instead, a host that wants to be certified implements one small interface. Each method is the minimum primitive a test needs to observe or drive the host from the outside; the test carries the semantics (assertions, markers, negation).

---

1. Host implementation required

The host (or a thin adapter around it) must implement the following. Signatures are illustrative — the shape matters, not the language.

interface HostTestInterface {
  // Launch the MCP App for the named test suite and return the live iframe
  // element so the runner can inspect it (sandbox attributes) and dispatch real
  // user-gesture events into it. Each suite launches in a FRESH app instance /
  // conversation, which is how state isolation between suites is achieved (see §2).
  displayTestApp(suite: string): Promise<HTMLIFrameElement>;

  // Reset the host to a clean state (WPT-style teardown): dismiss dialogs, clear
  // the launched app, drop any drafted message / model context.
  resetEnvironment(): Promise<void>;

  // The current conversation state (the messages the user/agent see).
  getConversation(): Promise<Conversation>;

  // The context actually handed to the model: the tool list it can call, plus
  // the view-provided context accumulated via ui/update-model-context.
  getConversationContext(): Promise<{ tools: string[]; modelContext: ContentBlock[] }>;

  // Mutate host context — e.g. flip the theme (light/dark) — so the view can be
  // observed reacting to ui/notifications/context-changed.
  changeHostContext(patch: Partial<HostContext>): Promise<void>;

  // Accept the host-native confirmation dialog surfaced by an action that
  // requires user consent (sampling/createMessage and ui/download-file today).
  confirmAction(kind: "create-message" | "download-file"): Promise<void>;

  // Click the chat's send button — commits whatever the app drafted into the
  // composer (e.g. a ui/message). Takes no text; it does not author a message.
  sendMessage(): Promise<void>;

  // Invoke a tool through the host and add its result to the conversation.
  // Deterministic alternative to relying on the model to call a tool: for
  // app-provided tools it exercises the Host→App call path directly.
  callTool(name: string, args?: Record<string, unknown>): Promise<CallToolResult>;
}

Deliberately not in the host interface

These belong to the ecosystem around the host (the browser / OS / test rig),
not to the host under test, so the host does not implement them:

Primitive	Why it's out	Who provides it
checkLinkOpen	Opening an external URL lands in the OS browser, outside the host surface.	ecosystem (browser/tab watcher)
readConsole	Console output is a property of the runtime, not the host protocol.	ecosystem (runtime console)

Tests that depend only on an ecosystem primitive (links/open-external,
security/csp-audit-log) are still run, but their verdict is produced by the
ecosystem harness, not by a host-interface call. They are marked accordingly
below.

---

2. Test suites

Tests are grouped into suites. Each suite is launched with
displayTestApp(suite) in its own fresh app instance / conversation, so state
can never leak between suites.

• core — the stable-spec (2026-01-26) tests that are read-only or self-healing (e.g. they restore any mode they changed via cleanup). They share a single instance and run in sequence.
• draft — the specification/draft tests. Isolated from core so an unstable clause can't fail a stable run.
• one suite per isolated case — a handful of tests cause irreversible mutations to the conversation or host context: you cannot un-send a message, un-seed model context, or cleanly un-poison a display-mode change that another test relies on. Each of these runs alone in a dedicated suite so its side effect can't contaminate the next test.

suite	launch	contents / why isolated
core	displayTestApp("core")	22 stable-spec tests; read-only or self-healing, one shared instance
draft	displayTestApp("draft")	4 draft-spec tests; unstable clauses kept out of the stable run
messages/add-to-conversation	displayTestApp("messages/add-to-conversation")	sends a real user message — pollutes the conversation
model-context/provide-future-turns	displayTestApp("model-context/provide-future-turns")	seeds model context that persists into later turns
model-context/last-wins	displayTestApp("model-context/last-wins")	seeds model context
context/context-changed	displayTestApp("context/context-changed")	flips the host theme — host context leaks
display/return-resulting-mode	displayTestApp("display/return-resulting-mode")	changes the host display mode
display/no-undeclared-mode	displayTestApp("display/no-undeclared-mode")	requests an undeclared mode (pip) — observed to leak into the next test
display/unavailable-returns-current	displayTestApp("display/unavailable-returns-current")	requests an unavailable display mode

Two orthogonal axes describe each test below: its suite (isolation grouping) and its category — Automatic (runs entirely in-view; needs only displayTestApp) vs Require host interaction (needs one or more of the observation/drive methods). The method columns are the discriminating methods; displayTestApp + resetEnvironment are used by every test and omitted. id links to the test's flow description; the line link points into the spec clause.

Spec bases:
2026-01-26 ·
draft
(the draft clause set is unstable and may move).

2a. Automatic

id	suite	clause	spec	line	getConversation	getConversationContext	changeHostContext	confirmAction	sendMessage	callTool
lifecycle/initialize-capabilities	core	MUST	2026-01-26	L624
lifecycle/tool-input	core	MUST	2026-01-26	L1106
lifecycle/tool-input-partial-stop	core	MUST	2026-01-26	L1136
lifecycle/tool-result	core	MUST	2026-01-26	L1155
tools/proxy-call	core	MUST	2026-01-26	L487
visibility/app-tool-call-guard	core	MUST	2026-01-26	L401
dimensions/listen-size-changed	core	MUST	2026-01-26	L718
security/sandbox-distinct-origin	core	MUST	2026-01-26	L474
security/sandbox-permissions	core	MUST	2026-01-26	L475
security/sandbox-proxy-required	core	MUST	2026-01-26	L472
security/csp-construct-from-domains	core	MUST	2026-01-26	L479
security/csp-allow-declared	core	MUST	2026-01-26	L479
security/csp-no-loosening	core	MUST NOT	2026-01-26	L286
context/initialize-hostcontext	core	SHOULD	2026-01-26	L533
context/light-dark	core	SHOULD	2026-01-26	L895
capabilities/server-passthrough	core	SHOULD	2026-01-26	L487
context/theme-variables	core	MAY	2026-01-26	L793
context/theme-fonts	core	MAY	2026-01-26	L922
display/return-resulting-mode	isolated	MUST	2026-01-26	L787
display/no-undeclared-mode	isolated	MUST NOT	2026-01-26	L786
display/unavailable-returns-current	isolated	SHOULD	2026-01-26	L788
capabilities/content-modalities	draft	MAY	draft	L663

2b. Require host interaction

id	suite	clause	spec	line	getConversation	getConversationContext	changeHostContext	confirmAction	sendMessage	callTool
visibility/app-tool-hidden	core	MUST NOT	2026-01-26	L400		✅
security/iframe-sandboxed	core	MUST	2026-01-26	L1698
links/open-external	core	SHOULD	2026-01-26	L996
security/csp-audit-log	core	SHOULD	2026-01-26	L287
messages/add-to-conversation	isolated	SHOULD	2026-01-26	L1033	✅				✅
model-context/provide-future-turns	isolated	SHOULD	2026-01-26	L1097		✅
model-context/last-wins	isolated	SHOULD	2026-01-26	L1101		✅
context/context-changed	isolated	MAY	2026-01-26	L1229			✅
sampling/create-message	draft	SHOULD	draft	L534				✅
download-file/confirm	draft	SHOULD	draft	L1128				✅
app-tools/call	draft	MAY	draft	L1243		✅				✅

security/iframe-sandboxed needs host interaction only in that it reads the
<iframe> returned by displayTestApp — no discriminating method.
links/open-external and security/csp-audit-log are verified by ecosystem
primitives (checkLinkOpen, readConsole), so they carry no host-interface ✅.

---

3. Test descriptions

Notation: (App → Host) = a call the app makes over the mcp-app bridge;
(Runner → Host) = a host-interface call; (Host → App) = a host-initiated
notification; Assertion = the pass condition.

Automatic

lifecycle/initialize-capabilities

• (App → Host) ui/initialize handshake
• Assertion: getHostCapabilities() returns a capabilities object

lifecycle/tool-input

• (Host → App) ui/notifications/tool-input after the view initializes
• Assertion: the view received tool-input carrying the tool arguments

lifecycle/tool-input-partial-stop

• (Host → App) tool-input arrives, then observe a short window
• Assertion: no ui/notifications/tool-input-partial arrives after tool-input

lifecycle/tool-result

• (Host → App) ui/notifications/tool-result on tool completion
• Assertion: the view received tool-result

tools/proxy-call

• (App → Host → Server) tools/call conformance_probe with a payload
• Assertion: the proxied result echoes the payload

visibility/app-tool-call-guard

• (App → Host) tools/call to model_only_probe (a model-only fixture tool)
• Assertion: the host rejects the call (tool lacks app visibility)

dimensions/listen-size-changed

• (App → Host) content grows; autoResize reports the new size
• Assertion: the view's viewport grows (the host resized the iframe) — flexible mode only

security/sandbox-distinct-origin

• (In-view) read window.top.location
• Assertion: it throws cross-origin — host ≠ sandbox origin

security/sandbox-permissions

• (In-view) inspect window.origin and the fact scripts run
• Assertion: origin is non-opaque (allow-same-origin) and scripts execute (allow-scripts)

security/sandbox-proxy-required

• (In-view) compare window.parent vs window.top
• Assertion: an intermediate sandbox-proxy frame sits between the view and the host

security/csp-construct-from-domains

• (In-view) read the applied CSP (meta tag / securitypolicyviolation event)
• Assertion: connect-src includes the declared domain

security/csp-allow-declared

• (App → declared origin) fetch to a connectDomains origin
• Assertion: the fetch is allowed (positive control)

security/csp-no-loosening

• (App → undeclared origin) fetch to an origin not in connectDomains
• Assertion: the fetch stays blocked (no loosening beyond declared domains)

context/initialize-hostcontext

• (App → Host) ui/initialize
• Assertion: getHostContext() returns a context object

context/light-dark

• (In-view) read hostContext.styles.variables
• Assertion: at least one theme-aware value uses CSS light-dark()

capabilities/server-passthrough

• (App → Host → Server) resources/list
• Assertion: the server's ui://conformance/runner resource comes back

context/theme-variables

• (In-view) read hostContext.styles.variables
• Assertion: the host provided ≥1 style variable (signal)

context/theme-fonts

• (In-view) read hostContext.styles.css.fonts
• Assertion: the host provided custom fonts (signal)

display/return-resulting-mode

• (own suite — changes the host display mode)
• (App → Host) ui/request-display-mode { mode: inline }
• Assertion: the response carries a valid resulting mode ∈ {inline, fullscreen, pip}

display/no-undeclared-mode

• (own suite — requesting an undeclared mode can leak into the next test)
• (App → Host) ui/request-display-mode { mode: pip } (pip is undeclared)
• Assertion: the host does not switch to pip

display/unavailable-returns-current

• (own suite — changes/probes the host display mode)
• (App → Host) ui/request-display-mode for a mode absent from availableDisplayModes
• Assertion: the response returns the current mode

capabilities/content-modalities

• (In-view) read hostCapabilities.message / .updateModelContext
• Assertion: the host declares content-block modalities (signal)

Require host interaction

visibility/app-tool-hidden

• (Runner → Host) getConversationContext
• Assertion: the model tool list does not contain conformance_probe (app-only tool must be hidden from the agent)

security/iframe-sandboxed

• (Runner → Host) displayTestApp returns the <iframe> element
• Assertion: the returned iframe has a sandbox attribute set

links/open-external

• (Runner → iframe) click the trigger → (App → Host) ui/open-link
• Ecosystem (checkLinkOpen, outside the host interface): a tab/window opened at the URL
• Assertion: the link opened

security/csp-audit-log

• Ecosystem (readConsole, outside the host interface): scan the host console for the CSP configuration
• Assertion: the host logged its CSP for security review

messages/add-to-conversation

• (own suite — sends a real user message that pollutes the conversation)
• (App → Host) ui/message carrying a marker
• (Runner → Host) sendMessage — click send to commit the message if the host drafted it into the composer
• (Runner → Host) getConversation
• Assertion: the conversation contains the marker

model-context/provide-future-turns

• (own suite — seeds model context that persists into later turns)
• (App → Host) ui/update-model-context carrying a marker
• (Runner → Host) getConversationContext
• Assertion: the model context contains the marker content

model-context/last-wins

• (own suite — seeds model context)
• (App → Host) update-model-context (stale marker), then update-model-context (fresh marker) before the next turn
• (Runner → Host) getConversationContext
• Assertion: the context contains the fresh marker and not the stale one

context/context-changed

• (own suite — flips the host theme, leaking host context)
• (Runner → Host) changeHostContext (toggle theme light↔dark)
• (Host → App) ui/notifications/context-changed
• Assertion: the view observed the context-changed notification

sampling/create-message

• (Runner → iframe) trigger → (App → Host) sampling/createMessage
• (Runner → Host) confirmAction("create-message")
• Assertion: the host surfaced/accepted the request and returned a completion

download-file/confirm

• (Runner → iframe) trigger → (App → Host) ui/download-file
• (Runner → Host) confirmAction("download-file")
• Assertion: the host surfaced/accepted the download

app-tools/call

• (App → Host) register an app tool conformance_ping
• (Runner → Host) getConversationContext — the app-registered tool appears in the model's tool list
• (Runner → Host) callTool("conformance_ping") — invoke it through the host and add its result to the conversation
• (Host → App) tools/call conformance_ping
• Assertion: the tool is exposed to the model and the app's callback fired