PRODUCT MANAGEMENT TEMPLATE
A/B Test Plan Template
Design an experiment with hypothesis, variants, success metrics, and sample size requirements.
Use this templateWhat's inside
Field | Value |
|---|---|
Name | Descriptive experiment name |
Status | Draft |
Owner | Person responsible for this experiment |
Start Date | |
End Date |
Hypothesis
State your hypothesis using the following structure:
If we [describe the change you are making], then [metric] will [improve/increase/decrease] by [expected magnitude], because [rationale grounded in user behavior, data, or research].
Context / Background
Describe why you are running this experiment. Reference prior data, user research, or business context that motivated the hypothesis.
What user behavior or metric prompted this experiment?
What have you tried before? What were the results?
What is the current baseline performance?
Variants
Variant | Description | % Traffic |
|---|---|---|
Control (A) | Describe the current experience — no changes | e.g., 50% |
Variant B | Describe the change being tested | e.g., 50% |
Variant C (optional) | Describe an additional change, if testing multiple approaches | e.g., 33% |
Primary Metric
Define the single metric that will determine whether this experiment is a success or failure. Be specific about how it is measured and what constitutes a meaningful change.
Metric: Name and definition
Current baseline: Current value
Minimum detectable effect: Smallest change that would be practically meaningful
Direction: Increase / Decrease
Secondary Metrics
Metric | Expected Direction | Why It Matters |
|---|---|---|
Secondary metric 1 | Increase / Decrease / No change | How it relates to the primary metric or user experience |
Secondary metric 2 | Increase / Decrease / No change | How it relates to the primary metric or user experience |
Secondary metric 3 | Increase / Decrease / No change | How it relates to the primary metric or user experience |
Guardrail Metrics
List metrics that must NOT degrade as a result of this experiment. If any guardrail metric moves in the wrong direction, investigate before declaring success.
Page load time must not increase by more than X%
Error rate must remain below X%
Customer support ticket volume must not increase
Revenue per user must not decrease
Sample Size & Duration
Parameter | Value |
|---|---|
Minimum detectable effect (MDE) | e.g., 5% relative improvement |
Statistical significance level (alpha) | e.g., 0.05 (5%) |
Statistical power (1 - beta) | e.g., 0.80 (80%) |
Required sample size per variant | Calculate using your MDE, alpha, and power |
Current daily traffic (eligible users) | Number of users who will enter the experiment per day |
Estimated experiment duration | Required sample / daily traffic |
Targeting / Segmentation
Define who is eligible for this experiment and any segments you plan to analyze separately.
Eligible audience: All users / specific plan tier / specific geography / new users only
Exclusions: Users to exclude (e.g., internal employees, users in other active experiments)
Segments for post-hoc analysis: e.g., new vs. returning users, mobile vs. desktop, plan tier
Implementation Notes
Describe what engineering needs to know to implement this experiment.
Where the experiment code should live (frontend / backend / both)
Feature flag or experimentation platform configuration
Any tracking events that need to be added or modified
Edge cases to handle (e.g., users who switch devices, users who clear cookies)
How to handle users who have seen the experiment before (sticky assignment)
Results
Complete this section after the experiment reaches the required sample size.
Metric | Control | Variant B | Relative Change | p-value | Significant? |
|---|---|---|---|---|---|
Primary metric | Value | Value | % change | p-value | Yes / No |
Secondary metric 1 | Value | Value | % change | p-value | Yes / No |
Secondary metric 2 | Value | Value | % change | p-value | Yes / No |
Guardrail metric 1 | Value | Value | % change | p-value | Pass / Fail |
Summary of findings: Describe what happened and whether the hypothesis was supported.
Decision & Next Steps
Decision | Details |
|---|---|
Ship variant? | Yes — ship to 100% / No — revert to control / Iterate — run a follow-up experiment |
Rationale | Why this decision was made, including any caveats |
Follow-up actions | Next experiment, feature refinement, or broader rollout plan |
Decision maker | Who made the call |
Decision date |
Other Product templates
-
Competitive AnalysisAnalyze competitors across features, pricing, positioning, and market strategy. -
Feature SpecificationDetail a specific feature with user stories, acceptance criteria, edge cases, and dependencies. -
Product BriefPitch a product idea with problem, proposed solution, target audience, and success criteria.