The goal of this interview series is to inspire and help people to transition their career into a new or next experimentation related role. In this edition Leihua Ye shares his journey. You can follow Leihua on LinkedIn, his blog and YouTube channel.
Learning takes time, and it is not a sprint. If you want to step into the field, start learning and reading today.
Leihua Ye
My name is Leihua Ye, and I enjoy nature, outdoor hiking, traveling, and reading good books. Besides, I hold a PhD degree from the University of California, Santa Barbara.
A fun fact about my educational background, I have degrees in three disciplines, including Humanity, Social Science, and STEM. Also, I obtained these degrees in three countries: China, the UK, and the US.
What is your current experimentation role and what do you do?
Currently, I’m a Senior Data Scientist at the Experimentation Team (called Expo) for Walmart. As the Fortune 1 company, Walmart has a huge need for trustworthy test results. Expo administers and delivers high-quality A/B testing data points for decision-making.
I wear two hats:
- On the technical side, I provide technical support and integrate best practices in various fields – Big Data, Data Science, Causal Inference, and Reinforcement Learning – into our platform.
- On the culture side, I’m an experimentation evangelist who preaches the value of experimentation.
Running tests is easy; running trustworthy tests is hard!
Experimentation is such an interdisciplinary field that requires domain knowledge from various subjects. For example, Online Experiments typically have millions of users, a typical big data setup. Doing any type of big data calculation and obtaining the results timely present a unique challenge to both Data Scientists and Data Engineers. As a Senior Data Scientist, I need to validate the statistical rigor and provide alternatives if the method does not work. This is just one simple example of the interdisciplinary nature of A/B testing. There are so many more use cases that need unique solutions.
As a Senior DS, I need to ensure the methodology is the “right” choice for the problem statement. The reason for the quotation marks is that being right is not equal to being fancy or perfect. There are so many new methodologies coming out every year, but only very few of them provide practical value. Putting on the DS hat, my thought process starts with the business problem and then moves back to the technical arsenal, trying to identify a good enough solution. It’s a fun process but not easy.
Referencing Microsoft’s Flywheel Theory (Fabijan et al., 2021), I see experimentation as a dynamic state: in order to make the flywheel spin, we need a strong culture of experimentation to facilitate collaboration, change how key decisions are made, and address concerns. Technical support alone is insufficient, and we need someone who understands customer pain points and helps them understand the value of experimentation.
How did you enter the experimentation space? What was your first experimentation related role?
I received a ton of training in quantitative methodology from my PhD program. While trying to locate an industry job after graduation, I applied for different quantitative positions, including UX researcher, Quantitative Financial Analyst, Machine Learning Researcher, and Data Science. Unfortunately, this strategy was too generic and broad and did not work out well. So, I re-examined my strongest strengths and identified the experimentation space as the best fit, considering my background in Experimentation and Causal Inference. After deciding the direction, I spent thousands of hours reading and researching how different companies build up their experimentation platforms. Up until now, reading has become my daily routine.
This is my first experimentation related job in the industry, but I used a lot of statistical and causal inference methods in my PhD years. Landing the first industry job is particularly challenging for PhD candidates. The available number of positions is limited, but the candidate pool is large. My initial strategy was not effective and struggled for a while. Later on, I found a few strategies that helped me land the first job:
- Create content online and build up your network. I spent hundreds of hours reading/researching technical articles/blog posts and talking to other DS in the field.
- Practice your interview skills. Like any other skills, interview skills are obtainable after deliberate practice. Technical interviews have several components, and practice each component extensively.
- Interviewing is a game of numbers: success rate = 1 offer / the total number of interviews. You only need 1 offer but need dozens of interview opportunities to succeed.
How did you start to learn experimentation?
My PhD program is a great start. By its completion, I have been using the same statistical tools for 5+ years. Also, reading blog posts/research papers/talking to other experimenters help me build a very solid foundation in the field. It’s like Learning by Reading.
After joining Expo, I got the opportunity to be more hands-on and implement a lot of cool methods. It’s kind of like Learning by Doing.
What are you currently doing to keep up with the ever-changing industry?
- Extensive reading across multiple disciplines;
- Talking to others and seeking for suggested readings;
- Learning to be an expert in an adjacent field. As a DS, I’m familiar with other job functions and choose Data Engineering as the adjacent field.
What recommendations would you give to someone who is looking to join the experimentation industry and get their first full-time position?
First, you can actually learn almost everything about experimentation from reading (e.g., published blog posts, research papers). The main reason I got the job was because I was super familiar with the field and its methodology. The familiarity comes from extensive reading and researching.
Second, how’s your depth of network? How many insiders can vouch for your credibility? Job searching is about networking, and you can land a job easier with insider endorsements.
Last, sharpen your interview skills. Practice coding in Python and SQL. Explain your thought process and answer follow-up questions.
By the end of the day, job searching is a numbers game: there will be a lot of failures before landing an offer.
How will AI change how experimenters work?
Generative AI (GenAI) will open up new avenues of testing. AI can generate content at marginal costs, and A/B testing can tell which type of content is better. I’m excited to see how GenAI and A/B testing can work together.
Do you want to share anything else?
Learning takes time, and it is not a sprint. If you want to step into the field, start learning and reading today. Take note of everything you have learned today, and review the learnings after 6, 12 months. You will be surprised by how much progress you have made.
Thank you Leihua for sharing your journey with the community.
