Comments on: Statistical significance & other A/B test pitfalls

By: Dossier metrics part4, pour aller plus loin: sources et ouvrages | Clement vouillon

Mon, 31 Oct 2011 13:58:08 +0000

[...] Statistical significance & other A/B test pitfalls [...]

By: Tutorial Pengujian A/B untuk Meningkatkan Nilai Konversi Website | Tutorial-Website.Com

Thu, 13 Oct 2011 02:13:38 +0000

[...] Statistical Significance and Other A/B Test Pitfalls [...]

By: A/B测试终极指南[转载] « DavidW

A/B测试终极指南[转载] « DavidW — Tue, 20 Sep 2011 06:50:13 +0000

[...] Statistical Significance and Other A/B Test Pitfalls [...]

By: A/B测试终极指南[转载] « DavidW.me

A/B测试终极指南[转载] « DavidW.me — Tue, 30 Aug 2011 04:17:49 +0000

[...] Statistical Significance and Other A/B Test Pitfalls [...]

By: Splitter: A split-testing plugin for Wordpress

Splitter: A split-testing plugin for Wordpress — Wed, 27 Jul 2011 17:17:32 +0000

[...] (I don’t know where the 65th one went.) This leads me into the biggest A/B testing trap, statistical significance. Crazy things can happen by chance, so it’s important to be rigorous about making sure your [...]

By: Facebook Ads and A/B Testing ‹ BMG – A Los Angeles Based Digital Agency

Wed, 20 Jul 2011 18:11:14 +0000

[...] The best way to produce significant results is by using a large sample size. I recommend reading Cennydd Bowles article on AB testing for more [...]

By: 网页A/B测试终极教程（三）-ab测试工具 « 排头兵 @ Talk

网页A/B测试终极教程（三）-ab测试工具 « 排头兵 @ Talk — Sun, 26 Jun 2011 13:31:18 +0000

[...] Statistical Significance and Other A/B Test Pitfalls [...]

By: Jonathan

Jonathan — Sun, 22 May 2011 20:57:58 +0000

Those having a tough time with A/B tests might also like to have a look at the phenomenon of “repeated significance testing errors” – something that can be even more problematic than the issue of statistical significance outlined in Mr Bowles’s post:

http://www.evanmiller.org/how-not-to-run-an-ab-test.html

“If instead of deciding ahead of time, ‘this experiment will collect exactly 1,000 observations,’ you say, ‘we’ll run it until we see a significant difference,’ all the reported significance levels become meaningless.”

With millions of dollars now pouring in to multivariate testing (eg Sitepect et. al.), repeated significance testing errors raise the stakes for test-driven design rather high.

By: The Ultimate Guide To A/B Testing — Mafuge

The Ultimate Guide To A/B Testing — Mafuge — Mon, 16 May 2011 15:05:46 +0000

[...] Statistical Significance and Other A/B Test Pitfalls [...]

By: 3 problemen van AB testen - Webanalisten.nl

3 problemen van AB testen - Webanalisten.nl — Fri, 13 May 2011 14:02:48 +0000

[...] aannames worden gedaan die mogelijk onjuist zijn. Ik wil het niet eens hebben over significantie (zoals vaak besproken door anderen) want ik ga er vanuit dat iedereen hier daar wel raad mee weet: Je moet wel genoeg observaties [...]