The goal of this interview series is to inspire and help people to transition their career into a new or next experimentation related role. In this edition Erin Weigel shares her journey. You can follow Erin on LinkedIn, Twitter and erindoesthings.com.
Making things for people is a privilege and responsibility. We need to handle it with care.
Erin Weigel
Hi there 👋 My name is Erin, and I do things. A thing I do that you’re probably most interested in is experimentation. (Though I also do design systems, accessibility, and am fixing up an abandoned villa in France. I also like to garden.)
I’ve been running experiments and learning from them for more than 12 years. I learned most of what I know about data-informed design from my time working at Booking.com. I started as a UX designer, then grew into a Principal designer over my nine years there. I also worked as a data science product manager and eventually in the leadership role responsible for design systems and accessibility.
During my time at Booking I ran more than 1,400 experiments and helped grow the design team from about 30 people to over 350. I learned not only a TON about experimentation, but also about building and growing an experimentation culture. In my “free time” I’m writing a book about experimentation and the conversion design process called, “Design for Impact.” It’s published by Rosenfeld Media and will be available sometime in 2024!
What is your current experimentation role and what do you do?
I’m currently a senior design manager at Deliveroo. It’s a UK-based food delivery company that has a strong presence in Asia.
I’m responsible for the company’s design system. I set the strategy and priorities as we work on optimizing the design foundations (color, type, spacing, UI) at scale.
I’m also responsible for the Rider Kit. It’s what delivery people wear while they work. My teammate, Sam, does the physical product designs. Then, we collaborate with Rider Operations and Data Science to run experiments in the real world. This helps us understand if the products we provide have the impact we aim for. I love the challenge of experimenting in the physical world.
How did you enter the experimentation space? What was your first experimentation related role?
My first experimentation role was at a small weight-loss company in the USA. But, I’ve never had the word “experimentation” in my title. I’ve always been a designer. Experimentation is just a core part of how I make good design decisions. Without it, there’s too much subjective opinion. And more often than not—opinion leads people astray.
My first experiments were run on email marketing campaigns. I noticed they used very small font sizes and had many images with text without alternative text. So, I redesigned their templates to use a minimum 16px font-size and used plain text as much as possible.
When the marketing team reviewed the results they saw a 120% improvement on sales.
The director asked, “What changed?” The analyst replied, “Erin made some design changes.” Then, I explained what I did that had the impact.
It was that moment when I realized that design implementation can be the difference between failure or success. I got hooked on learning if the changes I made had the impact I aimed for. As luck would have it, my next job was at Booking.com where I had the freedom and ability to learn as much as I wanted about user-centered design through experimentation.
How did you start to learn experimentation?
I didn’t realize this was a unique way to work. It seemed obvious to me, but I had no idea how things worked “under the hood.” I relied a lot on data scientists to help me analyze the results. The main person I learned from was Lukas Vermeer. We started at Booking on the same day and were in the same onboarding group. I learned how to read the data myself by walking up to Lukas’s desk and saying, “LUKAS WHAT IN THE WORLD IS GOING ON HERE?” He helped me understand what all the visualizations meant and when results were and were not reliable. Eventually, I’d walk up to him and tell him what I thought I saw in the results. When he confirmed my analyses more than he said, “uhhh no” I felt confident enough to not bug him so much about data analysis. Instead, we talked about other things like Scurvy, Ebola, cannibalism, and other weird stuff (we like to read weird books).
What are you currently doing to keep up with the ever-changing industry?
I’m lucky cause I have a lot of super smart friends.
Lukas Vermeer isn’t my only Lucas data scientist friend. I also hang out with my friend, Lucas Bernardi, who is a Principal Data Scientist who used to work at Booking, too. A fun fact is that not only are they both data scientists named Luk/cas who used to work at Booking, but they also both have twins.
Anyway, Lucas Bernardi and I work on my upcoming book together. He often sends me interesting research reports and talks about the latest developments in the field of mathematics and machine learning.
What recommendations would you give to someone who is looking to join the experimentation industry and get their first full-time position?
Be curious and learn as much as you can. Ultimately, experiments is about learning from high-quality evidence. If you don’t have access to testing tools, then read research. Find other ways to move your access to information further up the hierarchy of evidence. I’d also practice using a design process that’s rooted in clear, structured thinking that measures impact in whatever way you can. Getting fluent in the language of data and research-informed design are core skills of the craft.
How will AI change how experimenters work?
I think it could be useful for meta analysis. If you upload the results of all your experiments, it could be useful to extract success patterns or experimentation themes. It can also help people write the code needed to run experiments. It could also be used to summarize and translate academic papers into something more accessible to the average person. Gatekeeping is a side effect of the precise language used in science. Big words combined with long, complex sentence structure about complicated topics scare a lot of people away. Hopefully AI can open the learning door to more people by using more plain language with simpler explanations and examples than we currently have.
Do you want to share anything else?
I think experimenters are weird and wonderful. We tend to have broad interests, which allow us to combine our areas of expertise with different ideas to create something new. Innovation happens on the edges of disciplines. So, the more we can lean on and learn from one another, the more we can move the quality of the internet forward. With that we have a responsibility to learn about ethics and social responsibility. At the end of the day our purpose is to make things better—not just different (and hopefully not worse!).
Don’t let the data do the thinking for you. Your role is to use all the data you gather to do a thoughtful and human analysis. Your goal is to make things better for people. Hold yourself to a high standard and try not to let your cognitive biases lead you astray.
Surround yourself with intellectually honest people. Find people who aren’t afraid to question and call out things they think aren’t right. Take the time to learn about human psychology and reflect often on your own motivations and behaviors and how they impact the decisions you make. Making things for people is a privilege and responsibility. We need to handle it with care.
Thank you Erin for sharing your journey with the community.
