Blog · Gate ritual
When sample size says wait: shipping with incomplete analytics
Product managers hear “we don’t have enough data” as “QA is stalling.” Analytics QA for App Releases needs a third outcome besides ship and hold-for-bug: hold-for-sample. It is a decision with a date, a number, and an owner — not a mood.
Incomplete is not the same as broken
Broken means the contract failed: events missing, types wrong, double-fire, identify race. Incomplete means the instrumentation looks right on the candidate and the observed n is too small to prefer 4.18 over 4.17. Those are different memos. Mixing them teaches engineering to ignore both.
Write the threshold before you look
In week six of Release Signal Lab we ask for the minimum n per OS × locale slice before anyone opens the dashboard. A Bangkok wallet might say: 800 identified checkouts on Android, 400 on iOS, at least 120 on th-TH. If the staged rollout cannot reach that in five days, the gate is hold-for-sample with a calendar invite, not a vague “let’s watch it.”
Thresholds that appear after seeing a scary dip are not thresholds. They are arguments.
Staged rollout is still a release
A 5% staged ship is not a moral victory if you cannot query that 5% by build number. Hold-for-sample plus a stage is legitimate. Stage without version filters is how blended charts return through the back door.
Tone on the memo
Example we like: “Instrumentation QA passed on 4.18.2. th-TH usable-session n=96 after 72h (threshold 120). Hold-for-sample until Friday 17:00 ICT. Owner: Ploy. If n unmet, extend stage; do not full-roll.” The PM may still dislike the delay. They cannot claim the studio hid behind uncertainty.
We used this exact shape on a travel bilingual onboarding rewrite — see the second case on reviews. The lab outline is on Release Signal Lab.