2026-09-02 · Optimization · 4 min read

A/B testing screenshots with Product Page Experiments

TL;DR. Product Page Experiments is Apple's free A/B testing tool for App Store screenshots. It works but it's slow if you run it badly. Run it with a clear hypothesis on one variable at a time, give it the sample size it actually needs, and you get clean directional results in two weeks. Run it with five variables and 50% traffic on each and you get noise for six weeks.

Product Page Experiments (PPE) shipped in iOS 15 and has been the de facto way to A/B test App Store metadata since. You set up a control plus up to three treatment variants, allocate traffic, pick a confidence level, and Apple tells you which won. The infrastructure is solid; the discipline around how to use it isn't always.

What PPE actually measures

PPE measures impressions and conversions on the product page itself. Specifically:

Impressions: users who landed on your product page.
Conversions: users who tapped Get/Buy after landing.

It does NOT measure search-list tap-through (whether users tap your search result to land on the page in the first place). For that you'd need to test the search-result preview, which uses the first 3 panels of your default locale. PPE can test the full carousel but only measures what happens after the user is already on the page.

This matters because changes to panels 4-10 only affect product-page conversion, while changes to panels 1-3 affect both search-list tap-through and product-page conversion. PPE measures the second, not the first.

The "one variable at a time" rule

The temptation is to change the whole carousel between variants. Don't. If variant A has new captions, new device tilt, and a new palette, and variant A wins — you don't know which change caused the lift. You also can't reason about transferring the learning to your next test.

Discipline: change one thing per test. New caption on panel 1, everything else the same. New background color, captions identical. New panel order, content identical. Slow, but each test produces a learning you can build on. Fast iteration on noise isn't progress.

Sample size: the part most teams underestimate

PPE uses standard frequentist hypothesis testing. The sample size you need depends on:

Your baseline conversion rate (typically 1-5% for free apps, lower for paid).
The minimum detectable effect you care about (5% relative lift, 10%, 20%?).
Your traffic to the product page.

Order-of-magnitude: detecting a 10% relative lift on a 3% baseline conversion needs ~20,000 impressions per variant for 95% confidence. If your app gets 2,000 product-page impressions per week, that's 10 weeks per test. Most indie apps don't have the traffic to run statistically clean tests at small effect sizes. Honest options:

Test only big changes — palette swaps, panel order, fundamentally different caption tone. Those tests reach significance fast because the effect size is bigger.
Reduce traffic allocation to one or two variants instead of three.
Accept directional, not significant. If a variant is winning consistently over 4 weeks at 70% confidence, ship it and run a different test next.

The test design that works

Pick one hypothesis. "Bigger captions on panel 1 will improve conversion because thumbnails compress smaller captions to illegible." A specific guess, not "let's see what works".
Design variant A: one change. Same carousel, only the panel 1 caption is changed. Everything else identical.
Allocate 50% to control, 50% to variant. Two-arm tests reach significance faster than three-arm or four-arm.
Set the test duration upfront. "We'll run this for 4 weeks or until 95% significance, whichever comes first." Resist peeking and stopping early.
Document the result. Whether it won, lost, or was inconclusive — write it down. Future you will run the same test six months later if you don't.

What to test, in priority order

Panel 1 caption. Biggest impact because panel 1 dominates both search thumbnails and product-page first impression.
Panel order. Specifically: which feature do you lead with?
Carousel palette / dark vs light. Big effect, fast to test.
Device tilt / orientation. Smaller effect but cheap to set up.
Caption tone (imperative vs descriptive). "Track your runs" vs "Your runs, tracked". Smaller effect, harder to read clean signal.

Stop testing fonts, minor color shifts, or panel 8-10 content. Those tests almost never reach significance and the underlying effects are small enough that the win isn't worth the test budget.

Localization in PPE

PPE supports per-locale variants. If you run a test in your default locale and ship the winner, you still have to manually apply the change to every localized variant of the panel. Pick a locale with enough traffic to reach significance (usually en-US) and treat the result as the global directional guide for other locales. Don't run 39 separate experiments.

Common mistakes

Peeking and stopping early. Significance fluctuates randomly during a test. Stopping the moment a variant looks like it's winning leads to wrong conclusions. Pick a minimum duration and respect it.
Testing too many variables. Wholesale redesigns can't be A/B tested cleanly. You need a separate test for the new design overall (multi-variable, accept some confounding) followed by single-variable iteration tests on the winner.
Running tests through seasonal noise. Black Friday, App Store featuring, marketing campaigns all add traffic and noise. Avoid testing during these unless you control for the spike.
Ignoring locale. A US test result doesn't always generalize to JP. Use it as a starting hypothesis, not a settled answer.

What to do this month

Estimate your sample size needs. Plug your traffic + baseline conversion into a sample-size calculator before setting up the test.
Pick one hypothesis. Write it down in one sentence.
Design A vs B with one variable changed. Spin up in App Store Connect.
Set the run duration and put the end date on your calendar.
Document the result regardless of outcome.

What to read next

Open the editor →