The goal of this interview series is to inspire and help people to transition their career into a new or next experimentation related role. In this edition Ronny Kohavi shares his journey. You can follow Ronny on LinkedIn, Twitter and his website.
Best way to learn from Ronny is by attending his class: Accelerating Innovation with AB Testing. Ronny provided a promo code for the first 10 people to use it: EXPMINDBENDER
When results look too good to be true, hold the celebratory dinner and double-check things. Learn about Twyman’s law.
Ronny Kohavi
I love helping companies become more data-driven by using the scientific method, and I have done that successfully for over 20 years at Microsoft (where I headed ExP, the Experimentation Platform), at Amazon, and in consulting engagements (but failed to do that at Airbnb, where Brian officially disclosed recently that he will not do A/B tests in most scenarios, but instead ship on “pride.”)
I co-authored the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing with Diane Tang and Ya Xu. The book sold over 20,000 copies in English, and was translated to Chinese, Japanese, Korean, and Russian.
I received the Individual Lifetime Achievement Award for Experimentation Culture in 2000. My papers have over 55,000 citations according to Google Scholar, and several of my papers are in the most cited A/B testing papers.
What is your current experimentation role and what do you do?
I consult in A/B testing and teach a 10-hour interactive Zoom class with Maven: Accelerating Innovation with AB Testing. If you’re interested in my class, here is a promo code: EXPMINDBENDER for the first 10 people to use it.
How did you enter the experimentation space? What was your first experimentation related role?
When I was at Amazon, back in 2003, we ran controlled experiments with our platform called Weblab, which was one of the teams I led.
I remember looking at the statistics of our success/failure rate and realizing that more than 50% of our ideas failed to move the metrics they were designed to improve. That was a big epiphany for me; very humbling.
How did you start to learn experimentation?
I learned some from statisticians who worked with me, and from reading multiple statistical books. I then co-authored one of the early papers on experimentation in 2007 (http://bit.ly/ExPHiPPO), which was one of the earlier mentions of HiPPO, the Highest Paid Person’s Opinion (http://bitly.com/HIPPOExplained).
How do you apply experimentation in your personal life?
Controlled experiments are hard to do without larger counts. I point to the hierarchy of evidence when we discuss interesting findings in the news (most are unreliable).
What are you currently doing to keep up with the ever-changing industry?
I read books, articles, and posts.
Several books I recommended for the data science skeptics who believes in Twyman’s Law: https://bit.ly/bestBooksRonnyk
I created a spreadsheet with the most cited papers in A/B testing, and update it every few months, a forcing function to look at relatively recent papers: https://bit.ly/mostCitedSourcesinABTesting
I was recently involved in writing a survey article on statistical challenges in online controlled experiments: https://bit.ly/ABTestingStatChallenges. There are a lot of good references to new resources about recent advances.
I read several newsletters, including Experimental Mind, I have some search alerts for keywords, and I attend or read publications from some conferences (e.g., KDD, CODE, Conversion Hotel). In the next few months I will speak at ConvEx and Conversion Jam.
What recommendations would you give to someone who is looking to join the experimentation industry and get their first full-time position?
Make sure to internalize the key statistical definitions and their meaning. Brad Efron at Stanford used to say: Those who ignore Statistics are condemned to reinvent it. See http://bit.ly/ABTestingIntuitionBusters.
Think about the OEC, the Overall Evaluation Criterion. Don’t just optimize conversion.
How do you think experimentation will develop (in the next 10 years)
The treatment effect is important, but it’s one number. Identifying interesting segments, or as the statisticians call them: heterogeneous treatment effects, calls for using machine learning, and has provided amazingly interesting insights at Microsoft, where we implemented algorithms to identify these. Very few platforms out there provide such capabilities.
With generative AI, we may have automated algorithms that might suggest treatments, especially if they have access to patterns, such as GoodUI. The first attempt I’m aware of is CRO Professor, which ingested our book and other resources and can answer questions.
Do you want to share anything else?
When results look too good to be true, hold the celebratory dinner and double-check things. Learn about Twyman’s law.
Which other experimenters would you love to read an interview by?
Guido Imbens
Thank you Ronny for sharing your journey with the community.
