A/B testing
A/B testing or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates. A classic direct mail tactic, this method has been recently adopted within the interactive space to test tactics such as banner ads, emails and landing pages.
Significant improvements can be seen through testing elements like copy text, layouts, images and colors. However, not all elements produce the same improvements, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest improvements.
Employers of this A/B testing method will distribute multiple samples of a test, including the control, to see which single variable is most effective in increasing a response rate or other desired outcome. The test, in order to be effective, must reach an audience of a sufficient size that there is a reasonable chance of detecting a meaningful difference between the control and other tactics: see Statistical power.
This method is different to multivariate testing which applies statistical modeling which allows a tester to try multiple variables within the samples distributed.
Contents |
[edit] Companies well-known for using A/B testing
Many companies use the designed experiment approach to making marketing decisions. It is getting more of a common practice as the tools and expertise grows in this area.
While it is widely used behind the scenes to maximize profits, the practice occasionally makes it into the spotlight.
- Amazon.com pioneered its use within the web ecommerce space. Also stirred controversy by testing into optimal price-points. [1]
- BBC.[2]
- Google. One of their top designers, Douglas Bowman, left and spoke out against excessive use of the practice.[3]
- Microsoft [4]
[edit] Other Terms Used
- A/B/N Testing: A/B testing with more than two alternatives ("N" cells)
- A/B/..Z Testing: Same as above
- A/B/A: Only two alternatives, but one is repeated. This enables a quick visual of when the test reaches significance.
- Multivariate Testing: A designed experiment where effects from two or more potential causal factors can be isolated from one another.
- Multivariant Testing: same as above.
