I get a weekly update feed from Quora on subjects that may pique my interest. One of them this week was a question posed about “What are the most unexpected things people have learned from A/B Tests”. That may sound as interesting as watching wet paint dry, but it was answered by Dan Siroker, an ex Google employee, who related his experience of raising money for Barack Obama during the run-up to his first Presidency term. It’s an hour long talk he gave at Stanford, albeit he finishes the formal part of the presentation (and opens the floor for questions) at around 35 mins in. The main “wow” surprise for me is about half way in.

If you have a spare hour, I commend a full viewing of this 60 minute video.

If you don’t, he gives some text book examples of trying different web page backgrounds, and different wording on web page buttons, to pull the most people to give their email addresses, and to donate to the Obama campaign. Combinations they tested ended up being 40% more effective than their initial assumptions of what they thought would have worked best.

They also found that they needed to post different wordings, based on where viewers were in their engagement process with the campaign. New visitors to the site, repeat visitors who’d previously left their email address but not donated, and repeat visitors who’d already donated, responded differently to different approaches to get them to donate (or donate again). And conscious that the could insult or be too pushy for asking previous donators for a fixed sum of money, they simply suggested the very same sum that the person had donated the first time around, something that maximised contributions.

Having built up an email database – which they reckoned their approaches had given them an extra 4 million names by doing efficient testing – a golden opportunity came up. Sarah Palin, the then Republican Vice Presidential candidate, gave a speech at the Republican Convention (which millions watched live) that tried to trivialise Obama as an inexperienced small-town Mayor who had yet to get to grips with what “responsibility” was. The Obama campaign management peeled off an email which they sent to their database, drawing attention to the negative stance and to request people show their distaste of “Out of Touch Politicians” by donating to the Obama campaign. In one day, an extra $10 million of contributions rolled in.

The video of Palin and the text on the letter are around 30 minutes into that video.

I think there is one message there for people who run negative campaigns, be it in politics or in business; not a good idea. And particularly so if your intended target have the resources to magnify the stupidity of you engaging in such practices. Better to sell your own positives, without being seen to try chopping the legs off folks you’re competing with.

There is another more subtle message to many folks (with the word “Marketing” in their job title) i’ve met down the years who think it’s a good idea to throw marketing campaigns into the ether without integrating any test to measure success, nor to learn something new every time. There is a reason why seminal works like Kotler “Principles of Marketing” or Drayton Birds “Commonsense Direct and Digital Marketing” each have several chapters on Testing. To leave that out, and deny yourself the ability to learn and improve, is the sign of an amateur.

Footnote: The presenter Dan Siroker can be found at email: [email protected], Twitter: @dsiroker. Dan is the Co-founder of company Optimizely that does the heavy lifting on learning from A/B tests on web sites.