A/B Testing : a lesson in jumping to conculsions
January 20th, 2010 | Published in SEO
Last year I set up an A/B test on our ebook subscription page on Piehole.ie. We are a pretty low traffic site but I didn’t let that stop me drawing conclusions early on. My thought was that hard as nails sales copy was outperforming a very much more understated (and a little scrappy) subscription page.
It looks like I jumped to conclusions though. Checking back again this morning after some time I noticed that actually, the simple copy out-performs the snazzy buy now copy. This goes against conventional wisdom.
Of course google did tell me that the early results had a weak level of confidence but I was too impatient to wait for confirmation. It is just by luck that I forgot to stop the experiment early on.