The goal of this interview series is to inspire and help people to transition their career into a new or next experimentation related role. In this edition Lukas Vermeer shares his journey.
Lukas has a background in machine learning and computational science. Over the course of his career he has been fascinated by experimentation, how it affects organizations and how it can be used to build products for customers. Lukas has been writing about related topics so that others can learn from it.
I think sharing what you have learned is an important element of the experimentation mindset.
Lukas Vermeer
What is your current experimentation role and what do you do?
I’m currently senior director of experimentation at Vista. I joined three years ago to help accelerate their experimentation practice. They were already running some experiments in silos but thought they could do more. I built up a team that is doing most of the experimentation platform, education, and support work independently now. I’m spending more of my own time in the Product organization more broadly on topics that enable experimentation, such as building up a strong practice of empowered product teams.
How did you enter the experimentation space? What was your first experimentation related role?
I started in business intelligence at Oracle working on a next best action product they had acquired that used reinforcement learning and had a built-in control group to measure impact. I didn’t really understand the power of that approach at the time. When I joined Booking.com as a data scientist, I quickly realized from their extensive A/B testing that most things I was building had no impact on customer behavior. That confrontation with the power of a control group fascinated me. When the product manager for the experimentation platform Jonas Alves left the company, I took over his role on the experimentation platform team. We built out the product to support more types of experiments on more parts of the product, as well as made improvements to the infrastructure and user interface. Together with Roderik Koenders I also facilitated the experimentation training program at Booking for years. This helped me internalize the concepts better, and allowed me to practice explaining them to a broad audience.
How did you start to learn experimentation?
I first encountered experimentation concepts during university, but my real journey began at Booking.com. As a data scientist there, I quickly realized the importance of controlled experiments in understanding what actually works. I became fascinated by experimentation, spending time analyzing other teams’ experiments and helping improve them. My learning came from reading books and papers, but teaching others through regular training sessions as well as providing day-to-day support to experimenters really solidified my understanding. It was a gradual process of hands-on experience, self-study, and teaching that built my expertise over time. I didn’t set out to become an experimentation expert – it grew out of genuine fascination and seizing opportunities to learn and apply it in my work.
How do you apply experimentation in your personal life? (what are you tinkering with or always optimizing?)
I don’t deliberately run rigorous experiments, but I do tinker with things out of curiosity to learn as much new stuff as I can. For example I got into home automation, hydroponics, and making clothes. I have short bursts of fascination with a topic until I hit a performance plateau where significantly more effort would be needed to improve. Then I tend to move on to something else new and intriguing. The goal for me is not to become highly skilled, just to explore and learn.
What are you currently doing to keep up with the ever-changing industry?
I question whether “keeping up with the industry” is a useful framing. There are new things I don’t know, but also old, time-tested knowledge I have yet to learn. I try to prioritize learning based on what I think will be most useful, whether old or new. I follow some blogs, newsletters and Twitter, but I prefer reading older books over the latest hype.
What recommendations would you give to someone who is looking to join the experimentation industry and get their first full-time position?
I’m jealous of people who know exactly what they want. Seeking one narrow field reduces your options. Question why you so desperately want to join this one community. That said, experimentation tends to be an open, transparent community willing to share knowledge. By putting in the time you likely can become part of it. But that alone may not get you a job.
Which developments in experimentation excite you? How do you see the field changing in the next 5 to 10 years?
I’m excited to see more focus on making experimentation an integral part of product development rather than just a marketing activity on the side. Things that give teams the full context to experiment themselves rather than just tools to run an experiment. This means I’m spending less time directly on experimentation, though it’s still part of a larger whole for me.
Is there anything people reading this can help you with? Or any parting words?
I have more to learn on topics adjacent to experimentation like product development, user research and bridging quantitative and qualitative disciplines. Anyone skilled in those areas I would love to chat with, especially people spanning multiple domains.
Which other experimenters would you love to read an interview by?
Matt Gershoff, Sebastian Honores Espejo, Aleksander Fabijan, Zoé van Havre, Mats Stafseng Einarsen, Jonas Alves, Roderik Koenders, Roger Longbotham, Diane Tang, Alex Deng, Andrea Burbank, Hilary Mason, Annemarie Klaassen, Lizzie Eardley, Emilie Robinson, Emily Sommer, Eytan Bakshy, Colin McFarland, Dean Eckles, Susan Athy and Guido Imbens.
Thank you Lukas for sharing your journey and insights.
