Creative & Tech

Lessons Learned on Optimizing for Growth with A/B Testing [SXSW]

March 20, 2015

Among the copious panels, sessions and talks at SXSW this year focused on wearables, data and privacy, and other hot trends, there emerged subtler but just as important topics, including lessons from A/B testing. Todd Yellin from Netflix and Evany Thomas from Pinterest both touched on using A/B testing to improve the user experience of digital products and drive business results. Through the cases they shared highlighting their respective companies, there is much that other brands, digital marketers and agencies can learn from their experiences.

Both presenters started where all campaign planning and testing should – with identifying KPIs. Yellin stressed the importance of selecting as few metrics as possible, focusing on what’s most important to move the business, which for Netflix, KPIs were retention and the secondary streaming hours. In Thomas’ work, the main goal for Pinterest was to drive sign ups.

Both companies used incremental testing, making changes one at a time to key product features, and testing them against each other or against control groups to see how they affected the defined KPIs. And through that process, they learned and shared the following tips:

  • Focus on Behavior Data – It’s tempting to rely on demographic data like gender and age, or even on what your users are telling you and asking for. But what really matters if you want to drive the best results among all users is what they do en masse – i.e., their behavior.
  • Trust Your Experts – You can’t test every tiny detail, or you will never make a big enough impact. Instead, rely on the expertise of your team members to design changes grounded in research and data.
  • Keep The Faith – Certain ideas and changes won’t work at first, but don’t be discouraged! Keep iterating and testing if you believe in your idea and have data to back it up.
  • Be Bold— Testing can encourage boldness because it allows you to try something new. Don’t be afraid to mix it up and keep evolving to stay relevant. Remember though that sometimes you’ll have to kill your ideas depending on the results.

So go on and test! With these tips in mind, A/B testing can help you try new things and drive business results on everything from content, to media, to UX design. However, regardless of what you find through testing, don’t get carried away. Yellin ended his talk with a reminder to the audience – A/B testing is not always the answer. Brands must stay true to their values even if it doesn’t move their metrics as desired. After all, making it easier to cancel your subscription doesn’t exactly help with retention, but it’s important to Netflix to offer consumers that option without jumping through hoops.

Cover photo via SparkyHub