The goal of this interview series is to inspire and help people to transition their career into a new or next experimentation related role. In this edition Vineeth Madhusudanan shares his journey. He is Product Manager at Statsig.
Talking to people! I get to talk to dozens of experimenters each month, working in very different problem spaces, and I learn more out of this than all the newsletters and talks in the same timeframe.
Vineeth Madhusudanan
Please introduce yourself to our readers.
I’m a Product Manager at Statsig, a modern product development platform. Most of this time, I’ve worked on our experimentation tools. I previously worked at Facebook on video infrastructure, helping to figure out how we evolve the internet towards video (it was originally built for text and images). Before that, I worked at Microsoft as an early PM on Office 365.
What is your current experimentation role and what do you do?
I was Statsig’s first PM. Statsig was started by a group of ex-Facebook employees. Facebook’s world-class experimentation tooling shaped the product-building culture and inspired similar tools at companies like Airbnb and Uber. At Statsig, we learnt how to scale this to thousands of companies – big and small. Some of this learning was scaling past B2C into B2B and learning about problems unique to verticals like healthcare, eCommerce and banking.
While I get to work with some very sophisticated experimenters, a particular joy is to see teams that built experimentation culture from scratch, and see them succeed and scale over the last 3 years.
How did you enter the experimentation space? What was your first experimentation related role? Share your origin story here.
When I worked on Office 365 (at Microsoft), experimentation was still infrequent. An experiment was expensive to start, and we were thoughtful about when we used it. After joining Facebook, I felt like I had a new way to build software. Even a bug fix was rolled out gradually, and the tools automatically treated it as an A/B test and measured impact on the metric.
Seeing the impact measured on “great ideas” you had was humbling. But I can’t imagine working on a product that’s used at scale without having tools that enable this culture of measurement.
How did you start to learn experimentation?
I learn by doing. I primarily learnt about experimenting through experiment reviews. There’s nothing more useful (and humbling) than running experiments, documenting your observations and conclusions, and having a set of peers critique and improve this. I’ve since written extensively about the value of experiment reviews – link. More than any training class you send people to, this is how people will learn.
How do you apply experimentation in your personal life? (what are you tinkering with or always optimizing?)
I love doing things outdoors – whether climbing, hiking, or backpacking. When I got into going ultralight, one of my favorite things to do was to try new gear setups and tradeoffs in the real world. I’d do “shakeout” trips in controlled environments just to try out new gear configurations. Yes, you can sleep in 20°F weather without a sleeping bag if you’re willing to pair an ultralight down quilt with your warmest jacket when you sleep – and save a pound of gear! My first time climbing Mt. Rainier (Seattle’s backyard mountain) – I had a 45lb overnight pack. My lightest pack doing that – with a similar margin of safety – was less than half that!
What are you currently doing to keep up with the ever-changing industry?
Talking to people! I get to talk to dozens of experimenters each month, working in very different problem spaces, and I learn more out of this than all the newsletters and talks in the same timeframe.
What recommendations would you give to someone who is looking to join the experimentation industry and get their first full-time position?
The best products are built by people with a very strong mental model about their users and the problems to solve for them. Experimentation doesn’t change that.
If your approach is experiment -> decision, growth will peter out rapidly. The right framework is to experiment -> update your mental model -> use this to make decisions. Being able to understand and explain experiment results is a key skill to build. Most A/B testing nay-sayers are actually critiquing the approach of throwing spaghetti at the wall and shipping what sticks.
Which developments in experimentation excite you? How do you see the field changing in the next 5 to 10 years? What will stay the same? What’s not going to change in the next ten years?
What’s not going to change: “Good” experimentation practices matter much more than fancy math. People love to work on “fancy math” because it’s intellectually interesting and also very clean and precise to work with. Fancy math is helpful, but good decision-making trumps any impact from fancy math. Unfortunately, good decision making is a messy people problem, and is much harder to work through. It’s way easier to roll out sequential testing in a tool than build a culture of experiment reviews. The latter matters much more. Fun fact – some big-tech companies don’t bother with tools like sequential testing (since it causes power loss), and instead focus on good decision-making practices.
My crystal ball is very cloudy 5-10 years out, but I’m excited about how AI will help better decision-making over the next year! Good decision making comes from having context (what have we tried previously, was this metric easy/hard to move) and from mechanical tasks (is the hypothesis well formulated, have we looked at key slices of data). AI has become good enough to be an excellent co-pilot on much of this. It won’t completely replace experiment reviews (just yet), but the ability to get 80% of the feedback instantly and only wait on the humans for the last 20% will be a game changer for scaling experimentation culture.
Is there anything people reading this can help you with? Or any parting words?
Re-read the blurb above about the importance of building strong mental models about your users and their problems 🙂
Which other experimenters would you love to read an interview by?
Aleksander Fabijan (Microsoft Experimentation Leader), Dylan Lewis (Atlassian Experimentation Leader).
Thank you Vineeth Madhusudanan for sharing your journey and insights.
