The goal of this interview series is to inspire and help people to transition their career into a new or next experimentation related role. In this edition Sven Schmit shares his journey. You can follow Sven on LinkedIn, Twitter (now X) or read more on his website.
Working in experimentation? Why not participate in the State of Experimentation survey which Sven setup with other experts from the industry.
The real power of experimentation does not come from a p-value, but from the culture it builds.
Sven Schmit
Hello there, I’m Sven Schmit. For as long as I can remember, I have been interested in solving data problems, whether that’s in statistics, machine learning, optimization, or related fields. In the ideal scenario, it’s about using tools from mathematics first to gain a new or better understanding of a particular problem, and then leveraging that insight to act differently and improve outcomes. Of course, that is quite broad so I consider myself a bit of a jack of all trades and very much a master of none.
Outside of work I enjoy running and cycling, trying to keep up with the much more talented athletes at Eppo on Strava.
What is your current experimentation role and what do you do?
Currently, I’m Head of Statistics Engineering at Eppo. To provide some context, Eppo is an experimentation platform made for the modern data stack. Many major tech companies, such as Microsoft, Meta, Airbnb, Booking, and Netflix, have invested significantly in creating their in-house experimentation platforms. However, as data infrastructure has moved from bespoke systems to a more unified architecture on top of Snowflake, BigQuery, Databricks or Redshift, it makes little sense for everyone to build their own in-house experimentation platform. We aim to make the benefits of state-of-the-art experimentation platforms available to everyone.
As far as I am aware, the statistics engineer role at Eppo is quite unique. As the name suggests, it encompasses a blend of statistics and software engineering. Obviously there are plenty of statistical problems to solve when building an experimentation platform, and those are clearly in our purview. But we also put on our software engineering hats, and implement our solutions into production. Finally, there’s an aspect of product-oriented thinking, where we leverage our domain expertise and collaborate with the product and design teams to build a platform that is accessible and meaningful to both individuals with a PhD in Statistics and those who have less fond memories of their statistics 101 class in college.
How did you enter the experimentation space? What was your first experimentation related role?
I definitely come to the experimentation space from the theoretical side. As part of my first year of undergrad in Econometrics I learned about hypothesis testing and for the next decade I piled more maths on top of that.
In graduate school I stumbled upon Stitch Fix, a ecommerce company that combines human curation with machine learning and data science. Initially joining them as an intern, I focused on improving recommendation systems. This leads naturally to experimentation, as it is the only trustworthy way to quantify improvements, and nothing made it to production unless it proved its value in an experiment. After finishing graduate school, I joined the team full-time and spent several more years working on a variety of problems within this domain. One of the most fun projects to work on was the virtual warehouse, where we needed to account for spillover effects among inventory items, but it also paved the way to experiment with inventory strategies themselves.
But more than that, leaders of the algorithms team such as Eric Colson and Brad Klingenberg really fostered a bottoms-up culture of innovation and creativity (or “tinkering”) over efficiency; the embodiment of an experimentation culture in the broadest sense. It’s a way of working that really resonates with me. The real power of experimentation does not come from a p-value, but from the culture it builds.
How did you start to learn experimentation?
Let’s take a step back and examine why we engage in experimentation in the first place. Experimentation is a means to an end. We are really interested in making good decisions. To do so, we need to understand the causal impact of actions. This is generally hard because we cannot observe the counterfactual (what would have happened if we acted differently). We can make progress with experimentation if we have many units to randomize and compare. Unfortunately, that often does not translate well to personal life. That’s a long way of saying I do not experiment much in my personal life, and I think that is okay; it is just not the right tool for the job.
That said, I do think there’s a beautiful principle that comes from multi-armed bandits that applies equally well to personal life: optimism in the face of uncertainty. Not many would call me an adventurous person by nature, but this principle nudges me in the right direction. There exists a robust mathematical foundation supporting the idea that exploring the unknown is essential for making optimal decisions.
What are you currently doing to keep up with the ever-changing industry?
On one hand, there’s been a lot of interest in experimentation and causal inference more generally recently, with many advances and surely many more to come. On the other hand, a lot of the statistics we use most often are at least half a century old by now; it’s an interesting dichotomy.
Of course, we all like to read books, paper, and articles, but it’s a challenge to make sure these don’t linger at the bottom of an ever evolving priority list. One habit that we used both at Stitch Fix as well as Eppo is to set aside one hour a week to host a “reading group” – an informal, small group that gets together and discusses some work (usually a paper, but could also be a book chapter or a presentation) led by one of us. It’s a great forcing function to make sure we expand our horizon beyond the day-to-day concerns, and sometimes that means getting back to the fundamentals and other times we explore recent advances.
I am also a big fan of the CODE conference that is organized by MIT every year. It’s a conference where academics and industry experts come together to share their latest ideas. Usually I come away inspired with new ideas to work on.
What recommendations would you give to someone who is looking to join the experimentation industry and get their first full-time position? Especially for someone with a background in academia.
Frankly speaking, I think it has gotten more difficult and competitive over the last 5 years. In my experience, many companies prioritize prior experience which obviously is difficult when you are switching from academia. Part of the problem is that while there is always a need to hire great people, it is difficult to signal that based on only academic experience.
A primary concern that often comes up for candidates with an academic background is their proficiency to deliver production quality code in an industry environment. It’s not something academia trains you for (and I think rightly so). Luckily, we nowadays have access to a great set of resources. Of course you can spend time on websites such as leetcode to focus on technical interviews, but I think you should also consider doing some side project that interests you and then writing about it.
Finally, remember that you are looking for a single job, so you don’t necessarily care about a high conversion rate. Also something that’s much more clear to me now then when I was looking for my first job is that a rejection is rarely a judgment on whether you as a person are “good” or “bad”, but rather the company is looking to find someone to complement the team and fill a weakness. So do not take it personally and just work on increasing N.
Which developments in experimentation excite you? How do you see the field changing in the next 5 to 10 years?
Based on talk with customers, it is pretty clear that the experimentation field as a whole is moving towards adoption of more sophisticated experimentation technologies. Speeding up experiments using CUPED is a good example of a capability once reserved to the happy few that has become table stakes for many in the last year or two. I imagine we will see the same trend in the next 5 to 10 years for things like estimating long term effects, heterogeneous treatment effect estimation, more general causal inference beyond A/B tests, etc.
It’s not quite moving at the same pace, but I also see more companies generally embrace a culture of experimentation and increased buy-in from leadership to make data-driven decisions on sound causal analysis.
Finally, it is a bit of an open door but clearly the AI space is advancing rapidly and it will likely have an impact on the experimentation practice. However I think it is still too early to tell exactly how this will shake out.
You are Dutch. How is it like to live in the Bay Area? What are three things you miss from The Netherlands?
That’s right, I grew up in the Netherlands, and moved to the Bay Area for graduate school a little over 10 years ago. Initially, it felt like an amazing opportunity to experience living in the US for a couple of years. But then “data scientist” became the “sexiest job of the 21st century” and it turns out there are all these amazing and innovative job opportunities out there (who would have thought? Hence, I am still here and having a good time.
Comparisons are difficult as I have not worked in the Netherlands, and giving a nuanced view is beyond the scope of this interview, so let me just highlight one thing I really appreciate about the Bay Area: there is a constant influx of ambitious people from all over and ambition in general is encouraged and amplified. I imagine this sounds at least grandiose and potentially delusional to any Dutch reader, but I really appreciate the inspiring environment it creates. Even though most efforts do not make it “to the moon”, you only live once so you might as well give it a shot. And even if things do not work out, that is not considered failure, but rather a valuable lesson learned. That said, every advantage has its disadvantages that perhaps we can discuss another time (maybe with a drink).
Of course, there are certainly things I miss from the Netherlands are:
- Seasons: California is known for its fantastic weather, and rightly so. However, there is very little difference between the seasons, removing the obvious progression of time. For example, currently it is almost October, but it still feels like Spring to me.
- As a cyclist, I miss the bike infrastructure of the Netherlands. There is some fantastic scenery around here but it is much less (safely) accessible. And while it does not affect me personally, the car-centric nature also really limits the freedom of children who need to be driven around most places. It now makes me appreciate the freedom such infrastructure brings.
- Finally, from the Netherlands it is easy to travel to different European cities, each with its own unique culture and history. It’s something that is easy to take for granted but something I now wish would be a lot easier to do.
And as a bonus, I can’t wait to have a broodje kroket and a patatje oorlog next time I’m back.
Is there anything people reading this can help you with? Or any parting words?
Thanks Kevin for putting these interviews together, I have really enjoyed reading about the experiences of others. Also whether or not anything I said has resonated, please do not hesitate to reach out; I am always happy to chat about experimentation.
Which other experimenters would you love to read an interview by?
There are a couple of experimentation experts that come to mind, and I would enjoy reading their stories:
- Bradley Fay (Draftkings)
- Michael Lindon and/or Martin Tingley (Netflix)
- John Meakin (Meta)
- Demetri Pananos (Zapier)
- Kelly Pisane (Booking)
- Gosia Poplawska (Miro)
Thank you Sven for sharing your journey with the community.
