The goal of this interview series is to inspire and help people to transition their career into a new or next experimentation related role. In this edition Matteo Courthoud shares his journey. You can contact Matteo via LinkedIn, Twitter/X and GitHub.
AI will definitely impact experimentation, but, if anything, it will only increase the need for people who can study causal questions.
Matteo Courthoud
Hi everyone! I am Matteo and I work, do research, and write about causal inference and experimentation. I recently defended my Ph.D. in Economics in Zürich and started working for Zalando in Berlin. Outside of work, I am a big fan of mountains, basketball (watched), and historical novels.
What is your current experimentation role and what do you do?
I am currently an Applied Scientist at Zalando and I would describe my role as a swiss army knife in causal inference. I don’t have a fixed task, but rather I help wherever I am needed. In general, my duty is to come up with solutions for settings that go beyond “standard” experimentation. Sometimes experimentation is possible, but a difference-in-means estimator is not powerful enough. Sometimes experimentation is not possible, too expensive, or it does not identify an object of interest. In those cases, one has to resort to quasi-experimental methods which ultimately means making stronger assumptions. Examples of quasi-experimental methods include difference-in-differences, synthetic control, regression discontinuity design, and methods that rely on “selection on observables” (matching, propensity weighting, meta-learners, …).
How did you enter the experimentation space? What was your first experimentation-related role?
My first experimentation-related role was a research internship at Google during my PhD, where I worked on geographical experiments. As in my current role, my task was to go beyond standard experimentation and increase the power of existing AB tests. In fact, while geographical experiments have great properties such as tracking offline metrics and preserving privacy, they are severely underpowered. Methods like CUPED can help, but they are just a foot in the door of improving estimators’ efficiency.
The internship was a great experience and one of the main drivers behind my decision not to stay in academia. I realized that the industry offers plenty of interesting challenges and a more balanced lifestyle. There is often this misconception in academia (especially in Europe), that research internships involve warming coffee or doing photocopies. Instead, as a colleague once said, research internships are often the most enjoyable roles in the industry since you get all the fun, open-ended research, and none of the hurdles (stakeholder management and bureaucracy). In general, I think research internships are a great opportunity for Ph.D. students to make a more informed decision between industry and academia.
How did you start to learn experimentation?
I started learning about experimentation and causal inference during my master’s in economics. The economics curriculum contains more statistics than most people think, with a strong focus on causality. However, it was only during the Ph.D. that I became seriously interested in causal inference and econometrics. While it was not my main research topic, I enjoyed working with data and learning about causality and the PhD is a great opportunity for a curious person to learn and explore new ideas.
How do you apply experimentation in your personal life?
I don’t know about experimentation, but I deal with data a lot in my personal life. I have a terrible memory, so I need to log anything I want to keep track of. I am also a great basketball fan, and basketball is a golden mine for people who like tinkering with numbers.
What are you currently doing to keep up with the ever-changing industry?
This is a very timely question! During the PhD, keeping up with research was my job and I could dedicate several hours (or days) a week to reading and testing new methods. However, with a full-time job, it is not trivial to fit learning into the daily routine, even in an applied research role.
I am still experimenting with different options. At a certain point, I started a newsletter, as a commitment device to keep myself informed. However, I realized the pace at which new causal inference knowledge is generated is too stochastic for a regular newsletter. However, I am sometimes active on my blog on Medium, where I post deep dives on causal inference topics. It started when preparing for interviews for the internship, as I was making up industry use cases for theoretical methods. However, I soon realized there was a wide audience interested in hearing more and it was also a great stimulus to dig deeper into certain topics. My writing has been less regular now that I have a full-time job, but I am looking forward to finding some balance.
What recommendations would you give to someone who is looking to join the experimentation industry and get their first full-time position?
First of all, I don’t want to give the impression that you need a PhD to join the experimentation industry. A solid undergraduate background in statistics is enough, but I would recommend investing in causal inference. There are plenty of resources out there, in any form, at any level, and of very high quality. If you don’t know where to look, I have collected causal inference resources in a GitHub repository.
If you are a curious person, a PhD can be both fun and a great career boost. I feel there is more and more demand for data scientists who are not only familiar with experimentation but can also deal with settings where experimentation is not sufficient or possible. Moreover, these positions usually involve a larger degree of intellectual stimulation and freedom.
You recently changed roles (or are in the midst of changing). What made you look for something else? How did you approach your job hunt?
I recently completed my Ph.D. and decided not to stay in academia, but move to the industry instead. It was not an easy choice since there is a lot of inertia in academia and a lack of information on alternative paths. What helped me was doing a visiting semester in the U.S. where the connection between academia and industry is much tighter and I saw many companies advertising research internships and full positions to PhD students.
I was curious and, once I came back to Europe, I started looking around. Unfortunately, while research internships are getting more and more popular in the U.S., Europe is lagging behind. Part of the problem is supply-driven – most research positions are located in companies’ headquarters in the U.S. -, but part is demand-driven as they are perceived as a waste of time in academia. As far as I know, the companies that offer research internships in experimentation and causal inference in Europe are Google, Zalando, Spotify, Booking, Amazon, and QuantCo.
The main keywords that I was using to filter job offers were “causal inference” and “AB testing”. The problem is that most experimentation roles are hidden behind a Data Scientist title that can encompass very different duties, from cleaning data to designing estimators. Experimentation Jobs is a very timely resource!
How do you think experimentation will develop (in the next 10 years)? How will AI change how experimenters work?
I think in the future we will see a consolidation of experimentation libraries and platforms. So far, experimentation has been a prerogative of big tech companies, which have developed their own in-house tools. However, as experimentation spreads, the demand for software from mid-size companies will increase. Indeed we have just seen Spotify announcing the release of their own experimentation library. I am sure other big tech companies will follow.
For what concerns AI, it will definitely impact experimentation, but, if anything, it will only increase the need for people who can study causal questions. In the long term, when AI will be able to really replace humans in causal reasoning, I think we will have bigger concerns than how experimenters work.
Is there anything people reading this can help you with? Or any parting words?
First of all, thanks Kevin for the great initiative! It’s a terrific series of interviews. For everyone else, please don’t hesitate to reach out if I can be of any help, especially if you are a Ph.D. student. I was one of you until yesterday and I know the huge information gap and uncertainty.
Which other experimenters would you love to read an interview by?
I have to say you already have an impressive line-up of interviews and I feel humbled to appear among them. The first names that come to mind are Matheus Facure (Nubank), Pedro Sant’Anna (ex-Microsoft), Alex Deng (Airbnb), Apoorva Lal (Netflix), Rose Tan (Linkedin), Jean Pouget-Abadie (Google).
Thank you Matteo for sharing your journey and insights with the community.
