Should I be doing A/B and multivariate testing?

The latest big buzz doing the rounds is all about A/B Testing and in particular, Multivariate Analysis. Applied to Content and Marketing Optimisation, it promises big returns. But just what is it and should one be doing it?

A few years ago I was invited to attend and speak at a Marketing Analytics conference. Being involved in a web analytics business, and having worked in digital marketing for ten years, it seemed on the surface a good match.

That day, I sat through several seminars where various learned fellows discussed the latest algorithm they had developed and how through an intense, twelve month process, they had managed to combine seventeen separate variables into a regressional data model of such deep thought and complexity, that it was capable... on a bad day... of designing rockets and the occasional lottery win. Through this, the various boffins had managed to extrapolate a 0.38 % improvement in conversions on their website, as well as settling a number of legal cases currently troubling the High Court, although no-one was really sure of all this because further study was required.

It was following this day,  that I vowed to focus on the marketing, and less on the analytics.

I came out of that conference understanding why so many marketers prefer to focus on their experience and gut instinct when making decisions, often at the exclusion of available data. These marketers speak of “Analysis Paralysis” but really, are saying, “Analytics is not for me”.

We’ve seen so many success stories from applying web data to site and marketing optimisation that we know this shouldn't be ignored , but, it is certainly the case that one can get lost in the data and/or the process.

So it is with considerable surprise that the latest big buzz doing the rounds is all about A/B Testing and in particular, Multivariate Analysis. Applied to Content and Marketing Optimisation, it promises big returns. But just what is it, and should one be doing it? With the former question in mind, here’s Wikipedia’s description of Multivariate Analysis:

Multivariate statistics is a form of statistics encompassing the simultaneous observation and analysis of more than one statistical variable. The application of multivariate statistics is multivariate analysis.

Well, I know various white coated men who think that sounds like fun, but it seems a long way away from web marketing! So what is the draw?

“	Have we finally met a situation where marketers and site designers are accepting that a new approach is needed if performance is to be improved?	”
	Mark Wilding Head of Product Development Cognesia Ltd

Firstly, it is necessary to put the analytical element of this to one side, and instead, focus on what it achieves creatively. It does seem that we have finally met a situation where marketers and site designers are accepting that a new approach is needed if performance is to be improved.

We’ve all had our say, we’ve all tried various things, but now, we need a more systematic approach. It’s also really accepting that the web has reached maturity, and to move forward, websites need to be more intelligent, and less manual. Also, it moves businesses away from an opinion based approach to design and development, and into a more scientific model of automated evolution.

Putting one page up against another is no big challenge. Trying one advert against another is logical and quite simple. Comparing different content and its performance is quite achievable. Using web analytics tools, it’s perfectly possible to work out which options work best and make decisions.