I have two points for which I occasionally try to be persecuted:
- If you don’t know which tool to choose, then choose any tool. And this is most likely Google Optimize.
- You don’t need to know the under-the-hood math of A/B testing. Look at the tool’s “Win/Lose” messages.
So why do I think that?
- If you don’t know which tool to choose, chances are you have no or very little experience in A/B testing. To get that same experience, it’s important to start as soon as possible.
- If you don’t know which tool to choose, it means you have no requirements for the tool. If there were real requirements, a simple comparison of functionality would quickly solve the problem. And if there are no requirements, then anything will do.
- Knowing the under-the-hood math will help if you write your tool from scratch. In other cases, it’s practically useless: you can’t change the Google Optimize algorithm(s) and/or other tools.
It seems like picking the best tool and figuring out the under-the-hood math is very, very right. The devil lies in the definition of “right.” I insist that speed of motion (start early) is more important in most cases. In that time, you will have time to do more and get better results than that gain from a better tool and knowledge of math.
Run experiments whose wins will be reported to you by accounting, not by the analytics system.