Check feasibility before testing

Video transcript

1. 0:00–0:06

   Choose a useful effect • A more frequent metric can change the question. • Accepted enquiries / eligible users • Clicks answer a different question • Hypothetical: 5% to 6% • 1 percentage point · 20% relative change

   The primary outcome is accepted enquiries divided by eligible users, with later qualification as a possible guardrail rather than silently replacing the outcome. A separate click metric answers a different question. The article hypothetical baseline of five percent and alternative of six percent means one percentage point absolute or twenty percent relative change. This is an example for planning, not a promise of gain or a recommended minimum effect. Choose an effect worth acting on, a metric and a suitable method before selecting duration.

2. 0:06–0:12

   Count eligible units • Repeated visits are not automatically independent users. • U17 • U18 • Variant A · stable assignment • Variant B · stable assignment • Invented visit tokens · random groups need not be exactly equal

   Three fictional visit tokens U17, U17 and U18 move into two distinct eligible-user positions. The repeated U17 merges and disappears, leaving one unit U17 and a separate U18. They illustrate stable assignment to A and B; return visits should not automatically become independent observations or receive a different variant. Actual randomization groups need not have exactly equal realized sizes. This two-user sketch is not a sample-size recommendation or proof of valid randomization. Verify exposure, eligibility, one binary outcome per independent user where assumed, data quality and usable observations.

3. 0:12–0:18

   Translate sample into time • Recruitment and outcome follow-up are separate. • 16,400 users • 400/day: 41 days • 80/day: 205 days • Then follow-up • Hypothetical fixed horizon · 16,400 total • Steady eligible recruitment · follow-up and minimum duration are additional

   The reviewed calendar figure shows 16,400 users, alternative rates 400/day for 41 recruitment days and 80/day for 205, followed by a separate follow-up bracket. These are the source article hypothetical fixed-horizon inputs: binary independent observations, five percent baseline, six percent alternative, equal allocation, two-sided significance level five percent, power eighty percent. An arcsine-square-root standardized effect with a normal two-proportion power approximation gives approximately 8,143 per version, rounded conservatively to 8,200, total 16,400. Recruitment is assumed steady with no usable-observation loss. Outcome follow-up, data processing, minimum duration and other constraints add time. This is not a prescription for a 205-day test or a universal method. Eighty percent power is not an eighty percent chance this page wins; five percent significance does not assign ninety-five percent truth probability to a result.

4. 0:18–0:24

   Follow the analysis plan • Operational stops are not a confirmed win. • Serious fault or harm • Operational stop • No confirmed winner • Planned decision rule · interim decisions need the matching method

   A serious fault or harm connects to an operational stop, while a separate boundary keeps the result at no confirmed winner. Monitoring broken behaviour and data integrity is necessary independently of statistical inference. Follow the prespecified decision rule. Repeatedly stopping at ordinary fixed-horizon significance checks raises false-positive risk. A suitable sequential method uses specific adjustments for interim decisions; it does not manufacture observations or guarantee feasibility. Verify the actual method and configuration rather than inferring it from a platform name.

5. 0:24–0:30

   Choose suitable evidence • Fix reproducible faults or investigate the task when needed. • Effect estimate • Uncertainty • Useful change? • Next evidence • Reproduce fault → correct → retest • Observe task → investigate • Inconclusive ≠ equivalent · no causal uplift claimed

   The reviewed uncertainty diagram contains an effect estimate, uncertainty, an open useful-change question and next evidence. Alongside it are two alternative investigation paths: reproduce a fault, correct it and retest; or observe the customer task and investigate friction. These may be appropriate when the experiment cannot support the decision in a practical period. They do not provide the same population causal effect as a valid randomized experiment. Inconclusive evidence is not proof of equivalence and may remain compatible with useful improvement, harm or little difference. Report uncertainty and test integrity; do not cherry-pick a favorable subgroup or change eligibility after seeing results. Preserve existing commercial scope and document what evidence would change the next decision.