Multivariate testing

From Wikipedia, the free encyclopedia

In statistics, multivariate testing or multi-variable testing is a technique for testing hypotheses on complex multi-variable systems, especially used in testing market perceptions.^[1]

Contents 1 In internet marketing 2 Designs of experiment 3 See also 4 References 5 External links

[edit] In internet marketing

This section does not cite any references or sources. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (January 2008)

In internet marketing, multivariate testing is a process by which more than one component of a website may be tested in a live environment. It can be thought of in simple terms as numerous split tests or A/B tests performed on one page at the same time. Split tests and A/B tests are usually performed to determine the better of two content variations, multivariate testing can theoretically test the effectiveness of limitless combinations. The only limits on the number of combinations and the number of variables in a multivariate test are the amount of time it will take to get a statistically valid sample of visitors and computational power.

Multivariate testing is usually employed in order to ascertain which content or creative variation produces the best improvement in the defined goals of a website, whether that be user registrations or successful completion of a checkout process (that is, conversion rate). Dramatic increases can be seen through testing different copy text, form layouts and even landing page images and background colours. However, not all elements produce the same increase in conversions, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest increase in conversions.^[2]

Testing can be carried out on a dynamically generated website by setting up the server to display the different variations of content in equal proportions to incoming visitors. Statistics on how each visitor went on to behave after seeing the content under test must then be gathered and presented. Outsourced services can also be used to provide multivariate testing on websites with minor changes to page coding. These services insert their content to predefined areas of a site and monitor user behavior.

In a nutshell, multivariate testing can be seen as allowing website visitors to vote with their clicks for which content they prefer and will stand the most chance of them proceeding to a defined goal. The testing is transparent to the visitor with all commercial solutions capable of ensuring that each visitor is shown the same content on every visit.

Some websites benefit from constant 24/7 continuous optimization as visitor response to creatives and layouts differ by time of day/week or even season.

Multivariate testing is currently an area of high growth in internet marketing as it helps website owners to ensure that they are getting the most from the visitors arriving at their site. Areas such as search engine optimization and pay per click advertising bring visitors to a site and have been extensively used by many organisations but multivariate testing allows internet marketeers to ensure that visitors are being effectively exploited once they arrive at the website.

One form of multivariate testing is done through Page Tag (metadata); a process where the website creator inserts Javascript into the site to learn about visitor behavior. Page tagging typically tracks what a visitor viewed on the website and for how long that visitor remained on the site. A drawback to multivariate testing with page tagging is that it is time consuming and requires the assistance of an IT specialist or IT department and typically cannot be accomplished by a web marketer.^[3] Companies known to employ this method of multivariate testing are: Conversion Works, Omniture, Accenture Digital Optimization, Business Intelligence Group GmbH (B.I.G.), Amadesa, Maxymiser and Interwoven

Another form of multivariate testing is done with or without the need for page tagging. This approach uses a solution that is hosted externally as software as a service (SaaS) or run internally within an enterprise’s own datacenter. This could include testing variations of content such as headlines, images, copy, placement, for example, as well as targeting specific content to specific groups of users (also known as site-wide behavioral targeting) based on such criteria as recency and frequency of visits, referring URL, unique customer ID etc. This approach is superior to JavaScript because the logic sits server side rather than in the users browser. Therefore all other customer experience solutions can continue to work exactly as before. SiteSpect, Inc. is one company known to employ this method of multivariate testing.

Another area in multivariate testing comes from user adoption of the mobile web. Today, mobile phones and personal digital assistants (PDA) are increasingly powerful and are able to bring both regular and optimized websites into the hands of mobile visitors. Multivariate testing on mobile platforms can test multiple variations of every static and dynamic element of a company’s mobile web on the third screen, including image size, choice, specific words or phrases, placement, design, graphical elements, headlines, colors, and variations in functionality. Unfortunately, many mobile phones do not support Javascript and therefore preclude Javascript-based testing approaches.

[edit] Designs of experiment

This article may contain inappropriate or misinterpreted citations that do not verify the text. Please help improve this article by checking for inaccuracies. (help, talk, get involved!) (January 2008)

Statistical testing relies on design of experiments. Several methods in use for multivariate testing include:

1. Discrete choice modeling is the technique that won Daniel McFadden the Nobel Prize in Economics in year 2000. Choice modeling models how people make tradeoffs in the context of a purchase decision. By systematically varying the attributes or content elements, one can quantify their impact on outcome, such as a purchase decision. What is most important are the interaction effects uncovered, which neither the Taguchi methods nor Optimal design solve for^[4].
2. Optimal design involves iterations and waves of testings. Optimal design allows marketers the ability not only to test the maximum number of creative permutations in the shortest period of time but also to take into account relationships, interactions, and constraints across content elements on a website.^{[citation needed]} This allows one to find the optimal solution unencumbered by limitations.
3. Taguchi methods: with multiple variations of content in multiple locations on a website, a large number of combinations need to be statistically tested and medium/low traffic websites can take some time to get a large enough sample of visitors to decide which content gives the best performance. For example, if 3 different images are to be tested in 3 locations, there are 27 combinations to test. Taguchi methods (namely Taguchi orthogonal arrays) can be used in the design of experiments in order to reduce the variations but still give statistically valid results on individual content elements^[5]. Taguchi uses fractional factorial designs.

[edit] See also

• Web usability
• Multivariate statistics
• A/B testing
• Product optimization

[edit] References

[edit] External links

• Marketing Experiments Journal