Posted by on Mar 26, 2008 in SEO

I’ve found myself using this sample size calculator quite a bit recently. Thanks to Hawaii SEO for first telling me about this tool through Twitter. The calculator helps you determine the sample size required to reach a certain degree of confidence about the results of an experiment, or the confidence interval for a given sample size. This is extremely useful for internet marketers. It can be used to determine how certain you should be of the results of various campaigns and experiments you may conduct as an internet marketer, such as the click through rates in Adwords campaigns and ads on a website, conversion rates for various traffic sources and landing page versions, etc. The calculator does require a little knowledge of statistics to use effectively, but there are explanations on the page for each of the terms used to help you understand it all.

A common mistake I see among internet marketers is that they think that since they got two different results for two different campaigns, then in the long run those result will hold true, and they assume that the “winner” at that moment will be the true “winner” in the long run. I don’t blame marketers who make that mistake though, I have found that in general most people are clueless when it comes to Statistics. I am amazed at how often I see statistical fallacies used in the media from what are supposedly reputable sources. For an interesting read on this topic and a good introduction to statistics I recommend the book How to Lie with Statistics.

2 Comments

  1. 3-28-2008

    Thanks Chris – In the past I’ve often just run the a couple variants side by side for a week or so then hope I’ve got enough data to make an informed decision. Still, I would take that and re-run the tests every few months.

    Google’s Site Optimizer has built in statistical tools as well, so it will tell you within a certain range how accurate the result is. It’s fun to watch the accuracy band get tighter and tighter each day.

  2. 3-28-2008

    Yeah, I love how website optimizer give you a nice graphical display of confidence intervals for your landing page variations. It is both useful and just fun to see. It would nice to see the same thing built into Adwords and Google analytics.