feat: add a product analytics plugin, with experimentation inside it (build on demand)

Do not implement yet. The issue is a gap record, not a spec: read #6306 (UX plugin boundary) and #6298 (UI/dashboards) first, then the open interview questions in this ticket. Work starts only after an owner interview confirms vendor tools do not cover the need. Done means a scoped plugin design agreed with the owner, not a first-pass patch.

Problem

The planned user-experience plugin (#6306) chooses what to measure about the experience, but nothing in this marketplace turns those choices into instrumentation, a tracking plan, funnels, cohorts, activation definitions or experiments. Today that work routes to whatever analytics tools a team already has installed. If a team needs vendor-neutral analytics work those tools do not cover, there is no home for it.

This issue records the gap so it is not forgotten. It is not a design. The decision taken during the UX plugin interview is to build this plugin only when repeated use shows a need the installed vendor tools do not meet, and the scope needs a detailed interview with the owner before any work.

Proposed scope

• Tracking plan and event taxonomy, built from the signals the UX plugin says must be captured.
• Activation and retention definitions, funnel and cohort questions, answered through the analytics tools the project already has.
• Online controlled experiments: design, guardrail metrics, sample ratio mismatch checks, and the peeking problem. Experimentation sits inside this plugin, not in a plugin of its own.
• Growth and business questions (activation, conversion) that go beyond user-centered measurement.
• Routing to installed vendor plugins and MCP servers rather than wrapping vendor APIs.

Out of scope (what UX keeps)

• Choosing experience metrics (HEART, Goals-Signals-Metrics).
• Attitudinal measures (standardized usability questionnaires) and task measures (success, time on task, errors).
• The list of HEART signals a tracking plan must capture. UX hands this list over; this plugin implements it.
• How dashboards look, which belongs to the UI plugin (#6298).

Evidence

Confidence labels follow the research behind the UX plugin. Vendor rows are what each vendor says about its own product, so they are single-publisher, MEDIUM at best, and dated 2026-10-04. Naming a vendor here records that a tool exists; it is not a recommendation.

• The boundary is drawn at the question, not the data. HEART turns large-scale behavioral data into user-centered metrics and adds surveys for attitudes; product analytics also answers growth questions; experiments add a causal layer with their own statistics. MEDIUM; the source covers only the HEART half (behavioral data plus surveys for attitudes); the growth and experiment halves and the placement of the line are judgment. Source: Rodden, Hutchinson and Fu, "Measuring the User Experience on a Large Scale" (CHI 2010), https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36299.pdf
• Guardrail metrics alert experimenters to violated assumptions, in a chapter on sample ratio mismatch and other trust-related guardrails. HIGH, single source. Source: Kohavi, Tang and Xu, "Trustworthy Online Controlled Experiments", https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B518F59/listing
• Checking fixed-horizon p-values repeatedly during a test invalidates them; sequential or always-valid methods address this. MEDIUM. Source: Johari et al., https://arxiv.org/abs/1512.04922
• UK central government services must publish four mandatory KPIs. HIGH, single source. That UX can cover the two user-facing ones (satisfaction, completion rate) and analytics the rest is judgment. Source: https://www.gov.uk/service-manual/measuring-success/data-you-must-publish
• No study found tests whether reaching an "activation" event causes retention; activation is a vendor and practitioner construct. MEDIUM (a gap: absence of evidence from one research pass).
• Vendor landscape, single-publisher, MEDIUM, as of 2026-10-04. Each of these vendors states it ships an official MCP server: Mixpanel (https://docs.mixpanel.com/docs/mcp), Amplitude (https://amplitude.com/docs/amplitude-ai/amplitude-mcp), PostHog (https://posthog.com/docs/model-context-protocol), Google Analytics 4, labeled Experimental in its README (https://github.com/googleanalytics/google-analytics-mcp) and read-only per Google's docs (https://developers.google.com/analytics/devguides/MCP), Statsig (https://docs.statsig.com/integrations/mcp/overview), Optimizely (https://docs.optimizely.com/experimentation-strategy/docs/install-optimizely-experimentation-mcp-server), Contentsquare (https://contentsquare.com/blog/introducing-model-context-protocol-mcp/), which states Hotjar is now part of Contentsquare (https://contentsquare.com/hotjar/free-mcp/), and FullStory, labeled beta under Early Access (https://developer.fullstory.com/mcp/faq/). Some also publish Claude Code plugins or skills on top of the server, per the vendors.
• Plan and admin gates, as stated by the vendors: Mixpanel's docs say an org admin must enable MCP except on new Free and Growth accounts created after 2026-08-01; FullStory says its server needs StoryAI and an admin toggle; Amplitude gates read and write by role-based permissions; Optimizely requires Opal. Single-publisher, MEDIUM, and each must be rechecked against the vendor's changelog before use.
• Anthropic's knowledge-work design plugin lists product analytics as a connector placeholder with no bundled server, and its skills fall back to pasted input. Observed in the plugin source. Source: https://github.com/anthropics/knowledge-work-plugins/blob/main/design/CONNECTORS.md

Open questions for a human interview

1. Which analytics and experimentation tools do the teams that would use this run, and are their MCP servers enabled or admin-gated for them?
2. What need would the vendor plugins not meet? Vendor-neutral tracking-plan authoring across several tools is one candidate. What signal would show the need is real?
3. Who owns the tracking plan: product, engineering, analytics, or design?
4. Does experiment design ship in the first version, or only analysis of experiments run elsewhere?
5. What privacy review does event data need, and who signs it off?
6. Should the plugin own instrumentation for apps that have no analytics vendor at all?
7. How do results flow back to the UX plugin (for example, a funnel drop-off that needs a usability study)?

Related

• #6306 (user-experience plugin; this is the analytics sibling its boundary routes to)
• #6298 (UI plugin; owns dashboard presentation)

🤖 Generated with Claude Code