2000 character limit reached
Power Analysis is Essential: High-Powered Tests Suggest Minimal to No Effect of Rounded Shapes on Click-Through Rates
Published 30 Dec 2025 in stat.ME, cs.HC, and stat.AP | (2512.24521v1)
Abstract: Underpowered studies (below 50%) suffer from the winner's curse: a statistically significant result must exaggerate the true treatment effect to meet the significance threshold. A study by Dipayan Biswas, Annika Abell, and Roger Chacko published in the Journal of Consumer Research (2023) reported that in an A/B test simply rounding the corners of square buttons increased the online click-through rate by 55% (p-value 0.037)$\unicode{x2014}$a striking finding with potentially wide-ranging implications for the digital industry that is seeking to enhance consumer engagement. Drawing on our experience with tens of thousands of A/B tests, many involving similar user interface modifications, we found this dramatic claim implausibly large. To evaluate the claim, we conducted three high-powered A/B tests, each involving over two thousand times more users than the original study. All three experiments yielded effect size estimates that were approximately two orders of magnitude smaller than initially reported, with 95% confidence intervals that include zero, that is, not statistically significant at the 0.05 level. Two additional independent replications by Evidoo found similarly small effects. These findings underscore the critical importance of power analysis and experimental design to increase trust and reproducibility of results.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
Plain-language summary of the paper
What is this paper about?
This paper checks a bold claim from another study that said simply rounding the corners of website buttons makes people click 55% more. The authors thought this sounded way too good to be true. So they ran several large, careful A/B tests to see if rounded buttons really boost clicks. They also explain why planning the right sample size (called “power analysis”) is crucial for getting trustworthy results.
What did the researchers want to find out?
In simple terms, they asked:
- Do rounded buttons actually make more people click than square buttons?
- If there is an effect, how big is it?
- Can we trust the earlier study that found a huge 55% increase, or was it likely a fluke caused by a too-small experiment?
- What does this teach us about how to plan better experiments online?
How did they test it?
They used A/B tests, which work like this: imagine flipping a coin for each visitor. Heads = they see one version (say, square corners). Tails = they see another version (rounded corners). Then you compare how many people click in each group.
Key ideas explained in everyday language:
- Click-through rate (CTR): If 100 people see a button and 10 click it, the CTR is 10%.
- Statistical power: Think of power like the brightness of a flashlight in a dark room. A bright flashlight (high power) lets you spot small differences; a dim one (low power) easily misses them. Power depends a lot on how many people are in your test.
- Minimum Detectable Effect (MDE): The smallest change you’re trying to reliably notice. If your MDE is 2%, your test is designed to spot changes at least that big.
- P-value: A way to judge how surprising your result would be if there were actually no real difference. Smaller p-values mean stronger evidence.
- “Winner’s curse”: When you run small or weak tests, any “winning” result you do find often looks much bigger than it really is—like only hearing about lottery winners and thinking everyone must be rich.
- Random assignment and balance: In a fair A/B test, each version should get about the same number of people. If one side gets a lot more (a “sample ratio mismatch”), something is off and you shouldn’t trust the result.
What they did:
- They ran three big A/B tests on real websites: SeaWorld Orlando and two large Norwegian shopping sites (Obs and Obs-BYGG).
- Each test included around two million people per site, which is thousands of times larger than the original study that had only 919 visits.
- They kept the change simple: square corners versus rounded corners on buttons.
- They measured whether more people clicked with one style versus the other.
- Two additional independent tests by another company also tried the same idea.
What did they discover?
Short answer: Rounded corners did not meaningfully increase clicks.
- Across their three large tests, the click differences were tiny: about 0.16%, 0.29%, and 0.73%—and none were strong enough to count as clear evidence of a real difference.
- Two other independent tests also found very small, non-clear differences (about 0.68% and 0.07%).
- This is nowhere near the claimed 55% boost.
They also found problems with the original research:
- One of the original “field” studies wasn’t truly randomized, so you can’t trust it to show cause and effect.
- The only proper A/B test in the original work had very few participants, making it “underpowered.” That means it was like using a tiny telescope: if you think you see a planet with a small telescope, a big telescope should also see it. The big tests here did not.
- In industry, simple design tweaks usually change clicks by less than 1–3%, not 55%. So the big number was suspicious from the start.
Why does this matter?
- For teams building websites and apps: Don’t expect a huge win from rounding corners. If there’s any effect, it’s so small that most sites won’t be able to detect it, and it’s probably not worth prioritizing over bigger ideas.
- For good science and trustworthy results: Always plan your sample size and power before you run a test. Small, underpowered studies can produce eye-catching but misleading results. Replication with larger, well-designed tests builds real confidence.
- For decision-making: Extraordinary claims need strong evidence. When a result seems too good to be true—especially from a small study—test it again properly before acting on it.
Bottom line
Rounding button corners likely has little to no real effect on click-through rates. The earlier 55% claim almost certainly came from a study that was too small and overestimated the effect. Careful planning, enough participants, and proper randomization are essential for running reliable A/B tests and making good choices based on data.
Practical Applications
Immediate Applications
Below are concrete, deployable uses of the paper’s findings and methods across industry, academia, policy, and daily life.
- Industry (software, e‑commerce, product): Experiment power and MDE planning baked into workflow — Require a pre‑experiment power analysis with an explicit MDE informed by priors before green‑lighting UI tests. Tools/Workflow: sample‑size calculators, an “MDE gate” in experiment creation, pre‑analysis plans. Dependencies: traffic estimates, historical variance, a decision threshold for what counts as a practically meaningful lift.
- Industry (experimentation platforms, analytics): Guardrails for data quality and validity — Automatically flag Sample Ratio Mismatch (SRM), unit‑of‑analysis mismatches (user vs visit), and time‑bucket anomalies; block launches until resolved. Tools/Workflow: SRM monitor, randomization integrity checks, unit-consistency linting, dashboards. Dependencies: access to assignment logs, platform support for guardrails.
- Industry (e‑commerce UX, CRO): Backlog reprioritization away from rounded-corner tweaks — Treat corner rounding as low-ROI unless it’s free to implement and brand‑motivated; prioritize changes with historically larger lifts (e.g., contrast/visibility, USPs near CTA, payment option placement). Tools/Workflow: impact vs. cost matrices tied to prior lift distributions (GoodUI, Evidoo, internal repos). Dependencies: availability of prior benchmarks and PM alignment on “minimum worthwhile effect.”
- Industry (marketing/ads): Correct treatment of platform “studies” — Do not interpret Google Ads delivery differences as randomized A/B tests; use true randomization (e.g., ad‑group level random assignment, geo holdouts, ghost ads) for causal claims. Tools/Workflow: causal experiment templates for ad platforms, “non‑randomized” label in reporting. Dependencies: ad platform controls, budget to run holdouts.
- Industry (data science): Meta‑analysis of repeated experiments — Aggregate multiple replications of small‑effect UI changes to inform future MDEs and backlog decisions. Tools/Workflow: meta‑analysis notebooks/templates (fixed/random effects), internal registry of experiments with summary stats. Dependencies: consistent metrics, accessible summary results, tagging of comparable tests.
- Industry (leadership/finance): Decision gates that combine power with expected value — Require proposals to show power at the stated MDE and an expected ROI; deprioritize ideas that are underpowered given traffic. Tools/Workflow: portfolio dashboards that surface power, MDE, runtime, and expected gain side‑by‑side. Dependencies: traffic forecasts, KPI->revenue linkage.
- Academia (statistics, HCI, marketing): Case‑based instruction on power, winner’s curse, and replication — Integrate this paper as a teaching module; assign “small telescope” exercises and power planning labs. Tools/Workflow: course slides, lab assignments with sample‑size calculations and replication critiques. Dependencies: curriculum time, access to anonymized summaries or synthetic data.
- Policy (journals, conferences): Reporting standards for online experiments — Mandate inclusion of power analysis/MDE, unit of randomization, SRM checks, and explicit “non‑randomized platform study” labels where applicable. Tools/Workflow: author checklists; reviewer rubrics. Dependencies: editorial board adoption, template updates.
- Policy (industry governance): Registered trials or pre‑analysis plans for impactful claims — Encourage AEA‑style registry entries (even with redactions) for high‑stakes commercial A/B tests. Tools/Workflow: lightweight registration forms capturing hypotheses, primary metrics, duration, and MDE. Dependencies: legal review, NDA‑compatible summaries.
- Industry/Academia (privacy & legal): Safe replication via summary statistics — Share vetted summaries (counts, conversion rates, CIs, p‑values) rather than raw logs to enable third‑party checks without privacy exposure. Tools/Workflow: standard summary-stat schemas, legal boilerplate NDAs, replication MoUs. Dependencies: counsel approval, platform ability to export vetted summaries.
- Industry (platform selection): Vendor due diligence — Prefer experimentation platforms that support always‑valid p‑values or fixed‑horizon modes with clear documentation, SRM alerts, and user‑level analysis options (plus Delta method support when analyzing visits). Tools/Workflow: RFP criteria and scoring sheets. Dependencies: vendor features, internal statistical governance.
- Daily life (critical consumption of research claims): Skepticism toward large effects from small studies — Treat “extraordinary” UI/behavioral claims (e.g., 50%+ lifts) as provisional unless supported by large, well‑powered replications. Tools/Workflow: quick checklist (sample size, randomization, power, independent replication). Dependencies: access to basic study details.
Long-Term Applications
These opportunities will benefit from further development, scaling, or research before broad deployment.
- Industry/Platforms (software): AI copilot for experiment design — A planner that ingests site traffic, historical variance, and priors (GoodUI, Evidoo, internal data) to propose MDEs, durations, and sequential vs fixed‑horizon designs; warns of underpower and winner’s‑curse risk. Tools/Products: experiment‑planning assistants integrated into A/B platforms. Dependencies: high‑quality priors, platform APIs, change‑management adoption.
- Research/Privacy tech (cross‑industry): Privacy‑preserving effect size repositories — Differential privacy or synthetic‑data pipelines that publish aggregated lift distributions by pattern (e.g., CTA visibility, payment options), enabling better MDE selection without leaking sensitive signals. Tools/Products: DP aggregation services; pattern taxonomies. Dependencies: DP expertise, company participation, standard schemas.
- Policy/Standards (industry and academic): CONSORT‑like reporting for digital experiments — A standard for A/B test reporting (power/MDE, assignment unit, SRM checks, data exclusions, analysis plan, sequential vs fixed). Tools/Products: open standard, validator tools, journal and vendor alignment. Dependencies: multi‑stakeholder consensus, piloting at major venues.
- Methods (statistics): Winner’s‑curse adjustments in routine reporting — Shrinkage/empirical Bayes estimates alongside raw lifts; “small telescope” diagnostics auto‑computed and displayed in experiment summaries. Tools/Products: shrinkage modules, telescope calculators, default in vendor dashboards. Dependencies: education of users, agreement on priors.
- Platforms/Ad tech: True randomization APIs within ad platforms — Native controls to allocate impressions randomly (pre‑delivery), plus transparent logs for SRM auditing, enabling causal inference in ads. Tools/Products: randomized delivery endpoints; audit APIs. Dependencies: platform incentives, advertiser demand, privacy regulation.
- Education (business, design, data): Reforms embedding power and reproducibility — Standard curricula in MBA, HCI/UX, and data programs that teach MDE setting, power, sequential testing, and replication economics. Tools/Products: modular courseware, certification tracks. Dependencies: accreditation bodies, faculty training.
- Industry consortia: Cross‑company registered replications — Scale the Trustworthy A/B Patterns model to run coordinated, registered trials on common patterns (e.g., button copy, contrast, layout), sharing summary results. Tools/Products: replication marketplaces, neutral coordinators. Dependencies: NDAs, governance, funding.
- Analytics (operations research): Decision frameworks that price in tiny effects vs implementation cost — Formal models weighing effect size distributions, detection power, engineering/design cost, and opportunity cost to optimize experimentation portfolios. Tools/Products: portfolio optimization systems for product experimentation. Dependencies: reliable cost data, leadership buy‑in.
- Vendor transparency: Robustness and seasonality explanations — Standardized, user‑visible explanations for p‑value dynamics (e.g., “stat reset,” time bucketing), with simulation aids for interpreting always‑valid vs fixed‑horizon results. Tools/Products: explainability modules in Stats Engines. Dependencies: vendor investment, UX clarity.
- Measurement science: Broader adoption of unit‑consistent analysis — Automated detection and correction (Delta method) when analysis is at visits but randomization is at users; guard against dependence violations. Tools/Products: auto‑unit alignment, visit‑to‑user adjustment libraries. Dependencies: correct instrumentation, platform analytics hooks.
Notes on feasibility across items
- Assumptions that matter: sufficient traffic to power tests at the chosen MDE; trustworthy randomization and logging; organizational willingness to de‑prioritize low‑ROI bets; access to prior effect distributions; legal frameworks for sharing summaries.
- Key dependencies: experimentation platform capabilities (SRM detection, unit handling, sequential testing), privacy‑preserving publication methods, and multi‑stakeholder adoption of reporting standards.
Glossary
- A/B test: A randomized online controlled experiment comparing two variants to measure causal effects on a metric. "in an A/B test simply rounding the corners of square buttons increased the online click-through rate by 55%"
- Always-valid-p-value: A p-value methodology that remains valid under continuous monitoring and peeking during an experiment. "Optimizely analyzes the data using an always-valid-p-value approach"
- Alpha: The significance threshold used to decide statistical significance for a test’s p-value. "the industry default of 0.05 for alpha (the p-value threshold for statistical significance)"
- Bayes Rule: The principle for updating probabilities with evidence; used to compute the chance a significant result is false. "Bayes Rule can then be used to compute the False Positive Risk"
- Call-to-action (CTA) buttons: Interface elements prompting users to take an action (e.g., “Buy,” “Add to cart”) to drive conversions. "call-to-action (CTA) buttons"
- Cohen's d: A standardized effect size for differences in means, commonly used in psychology. "In psychology, Cohen's d of 0.2 is considered small (Brysbaert 2019)"
- Cohen's h: An effect size measure for differences between proportions. "and Cohen's h for that are about 0.01-tiny."
- Confidence interval: A range of plausible values for a parameter that reflects uncertainty in estimation. "with 95% confidence intervals that include zero"
- CTR: Click-through rate; the fraction of users who click a given element or link. "CTR on the \"Contact Us\" button"
- Delta method: A statistical approximation technique to estimate the variance of functions of random variables. "requiring an analysis using the Delta method (Deng, Knoblich and Lu 2018)."
- Effect size: The magnitude of a treatment’s impact on an outcome metric, independent of sample size. "effect size estimates that were approximately two orders of magnitude smaller than initially reported"
- False Positive Risk: The probability that a statistically significant result is actually a false positive. "False Positive Risk, the probability that the statistically significant result is a false positive"
- Frequentist algorithm: An analysis approach based on frequentist statistics (not Bayesian), often used in A/B platforms. "Kameleoon's frequentist algorithm matches the lift and p-values computed in Excel"
- Guardrail: An automated experiment check that flags issues (e.g., SRM) which invalidate results. "a common guardrail in experimentation platforms (Kohavi, Tang and Xu 2020)."
- Hierarchy of evidence: A ranking of study designs by their reliability in establishing causal claims. "lower in the hierarchy of evidence than A/B tests"
- Internal validity: The extent to which a study reliably identifies causal effects without confounding. "Lacking internal validity, online platform studies should not be considered as randomized experiments."
- Lift: Relative improvement in a metric caused by a treatment compared to control. "We have a good sense of the lifts generated, typically under one percent, rarely over two or three percent."
- Meta-analysis: Statistical synthesis combining results across multiple studies to estimate an overall effect. "The meta-analysis combining all three experiments results in a (weighted-average) lift of 0.21% with p-value 0.08"
- Minimum Detectable Effect (MDE): The smallest effect size an experiment is designed to detect with specified power and alpha. "We also publicly called out the MDE (Minimum Detectable Effect) as relative 2% lift"
- NDA (Non-Disclosure Agreement): A legal contract restricting the sharing of confidential information. "operated under NDA (Non-Disclosure Agreement)"
- NHST (Null Hypothesis Significance Testing): The conventional framework for hypothesis testing using p-values. "Null Hypothesis Significance Testing (NHST) should require power calculations."
- OSF (Open Science Framework): A platform for pre-registering studies and sharing research materials. "pre-register the experiment publicly (e.g., on OSF or AsPredicted.org)"
- Post-randomization targeting algorithms: Platform algorithms that modify delivery after assignment, breaking clean randomization. "employ post-randomization targeting algorithms that prevent clean random assignment of participants ..."
- Power analysis: Planning analysis to determine sample sizes needed to detect a specified effect with desired probability. "findings underscore the critical importance of power analysis and experimental design"
- Pre-registration: Publicly documenting hypotheses and analysis plans before data collection to reduce researcher degrees of freedom. "We did not pre-register the experiment publicly (e.g., on OSF or AsPredicted.org)"
- Prior distribution: A probabilistic model representing beliefs about a parameter (e.g., effect size) before seeing data. "One has to establish some prior distribution on the expected effect size"
- Randomized controlled experiments: Studies with random assignment to treatment/control considered the most reliable for causal inference. "Randomized controlled experiments are considered the gold standard in establishing causality"
- Replication crisis: A widespread challenge in reproducing significant scientific findings across replications. "replication crisis"
- Reproducibility crisis: Systemic problems leading to difficulties in replicating or reproducing research results. "reproducibility crisis"
- Right-tail: The upper tail of a statistical distribution used in one-sided (right-tailed) significance tests. "The power shown is for detecting a positive treatment effect (right-tail; 2.5% is the type-I error rate)"
- Sample Ratio Mismatch (SRM): A significant deviation from the planned allocation ratio between variants indicating an invalid experiment. "suffers from a large SRM, or Sample Ratio Mismatch (SRM; Kohavi and Thomke 2017, Fabijan et al. 2019)"
- Seasonality and drift: Temporal fluctuations and trends in data that can affect experiment metrics. "seasonality and drift in the conversion rates."
- Small telescopes: A framework for judging whether the original study had sufficient power to be informative. "With the small telescope approach, we ask not whether the results of the replication are consistent with the results of the original study, but whether they are consistent with its design"
- Standard deviation: A measure of variability within a group’s observations. "where o is the standard deviation of the measurement within each group"
- Theory-test replications: Replications that apply similar interventions and methods in different contexts to test a theory. "these are theory-test replications"
- Time bucketing algorithm: A method that groups data into time intervals for analysis, affecting statistical updates. "due to the time bucketing algorithm and \"stat reset,\""
- Type-I error: The probability of incorrectly rejecting the true null hypothesis (false positive). "right-tail; 2.5% is the type-I error rate"
- Unique Selling Proposition (USP): A marketing message highlighting a product’s distinctive benefit near key actions. "Unique Selling Proposition (USP)"
- Winner's curse: The tendency of significant results from low-powered studies to overestimate the true effect. "Underpowered studies (below 50%) suffer from the winner's curse"
- Yates continuity correction: An adjustment to chi-square tests for small counts to improve approximation accuracy. "appropriate to apply Yates continuity correction"
- Z-score: A standardized value indicating how many standard deviations an observation is from the mean. "a z-score of 46"
