Takeaways from 31 Experimentation Experts in 2024
I’m taking on a bunch of new experimentation work for a Fortune 500 company—new teams, new workflows, and lots to learn about the business! As a result, the newsletter will move to once a month instead of twice (though sometimes twice). Wish me luck!
In 2024, experimentation teams have been tackling big challenges—refining their processes, embracing AI, and building cultures that drive innovation. I interviewed 31 leaders across industries to uncover how they’re navigating these shifts, from securing leadership buy-in at Nike to tackling transparency at Refurbed. Microsoft is exploring AI-driven workflows, while Verizon pushes to become the “best AI company.” These bold ideas and strategies, from prioritizing the right tests to balancing metrics that foster growth and stability, are shaping the future of experimentation. Here are a few of my favorite takeaways and moments from last year’s interviews (with links to the full discussions) that will, hopefully, inspire you to elevate your programs in 2025.
Building a Culture That Makes Experimentation Thrive
Experimentation success depends not only on how teams are structured but also on how they go about creating a culture of experimentation and inclusivity. At Nike in EMEA, Nic English explains how leadership buy-in has been a recent focal point for program adoption, creating an environment where teams feel supported—even when experiments may fail. Mike De Boer from Comcast and Manjot Jaswal at RS Group both discuss the power of collaborative sessions in our roundtable event—whether through open ideation or structured workshops—to spark creativity and bring in ideas from all levels. At eBay, Ben Skrainka described their maturity model, which integrates culture, infrastructure, and methodology to democratize experimentation which drives cohesive decision-making across the organization.
What I’ve found is that these varied approaches underscore that success in experimentation is not about following a formula or best practice, but about evolving strategies that align with your organization's unique culture, challenges, and aspirations. Each of these interviews offers unique takeaways.
AI and Automation Are Set to Revolutionize Experimentation Teams
Generative AI will eventually permeate every experimentation team, regardless of size. While many of us are aware—at least from an outsider’s perspective—of how advanced FAANG companies are, there are still teams within those organizations working to figure things out. Prerit Saxena from the Microsoft Edge team, for example, envisions a future where AI is used to generate test ideas and streamline reporting, freeing teams, including data scientists, to focus on more strategic and complex questions. However, when I spoke with the Meta experimentation team, this might not have been the initial use case, as data hygiene and integrity remain ongoing challenges. More details will follow as we work through HR approvals for an upcoming exclusive interview.
Meanwhile, eBay has already leveraged AI to make testing accessible to non-technical business users, while Javier García Latorre from Delivery Hero discusses using automation (not necessarily AI) to speed up testing workflows and processes without compromising quality. Additionally, Burak highlighted Verizon’s ambitious goal to become the “best AI company,” underscoring how much the organization values experimentation as a driver of business growth.
YinYin Yu at LinkedIn envisions AI enhancing systemic vetting processes which I found interesting, including testing algorithms to ensure recommendation systems are free from bias. While this remains an aspirational use of AI (and a fun part of our conversation), it highlights the potential for aligning experimentation with broader organizational goals to ensure fairness and accountability.
Across all interviews, one thing is clear: workflow automation is a top priority for AI use cases. Companies are using—or aspire to use—AI for automation to refine testing processes and save time and resources. In the not-so-distant future, this will reshape the roles of CROs, experimenters, and program managers.
The Best Prioritization Strategy Is the One You Use
With limited resources and competing goals, test prioritization is critical but all teams seem to have their own frameworks and methods for prioritization. My conversation with Andrey Prokhorov at ASICS in EMEA focused on their six-point framework for evaluating experiments, which balances effort, reach, and impact. His thoughtful approach ensures teams concentrate on experiments with the highest potential value not the most popular or political which I appreciate. While I didn’t discuss similar frameworks with other companies, ASICS’ strategy stands out for maintaining momentum and keeping the focus on program outcomes.
Andrey has put significant thought into his prioritization framework, but the key takeaway is this: the best framework is the one you actually use. I hope this conversation inspires you to adapt these insights to create or refine your own approach.
Tackling Bugs and Buyer Needs Through Smarter Experimentation
Experimentation isn’t just about optimizing performance; it’s also about understanding the “why” behind user behavior and solving complex challenges. Florent Buisson, formerly at Allstate, discussed his causal behavior framework, which maps the drivers of user actions to design more meaningful experiments. For example, while Net Promoter Score (NPS) can reveal broad trends, he has found that it often lacks the granularity needed to drive results—an interesting insight from our conversation. The key takeaway from our conversation is to develop a framework that prioritizes metrics directly influencing behavior. Have a listen if you’d like to hear Florent’s approach.
In a roundtable event with Target, Drizly, and SEPHORA earlier this year, Divya Al shared valuable but often overlooked advice: leverage experimentation and A/B testing to catch bugs early when rolling out new features and experiences. Testing can be a great risk mitigator, saving time and resources by preventing issues from escalating.
Refurbed’s Daniel Slowacek explained how his UX team redesigns product pages to address buyer concerns around quality and transparency. Their challenge was designing for multiple refurbished product categories, each catering to very different buyer needs—an interesting and unique problem to solve.
Conclusion
As we move into 2025, these interviews show one clear thing: experimentation is about more than just tools—it’s about teamwork and new ideas. From using AI to speed up workflows to finding better ways to test ideas, success comes from trying what works best for your team.
The leaders I spoke with showed that there’s no single way to succeed. It’s about growing and improving in ways that fit your goals. I hope these lessons inspire you to try new things, stay curious, and make 2025 a great year for learning and growing through experimentation!
📚 Worth Exploring
Want Your Company to Get Better at Experimentation?
The article explores how companies like Netflix and Airbnb scale innovation through democratized experimentation platforms, minimal data scientist involvement, and a shift from individual tests to program-wide analysis for greater impact. Link to article
Deep vs. Broad Context Traps
This article explains that knowing a lot about one thing can make you miss other ideas, and knowing a little about many things can make solving problems hard. It’s a good reminder for all of us, especially those in CRO, who often know a little about many things, to find better balance for doing their best work. Link to article
Top 1% E-Commerce UX awards by Baymard Institute
The article highlights the top-performing e-commerce websites for 2024, showcasing best-in-class user experience practices based on rigorous UX evaluations. Link to article
How Meta scaled regression adjustment to improve power across hundreds of thousands of experiments on our AB testing platform
Meta scaled the use of CUPED (Controlled and Pre-Post Experiment Design) regression adjustment, reducing confidence interval widths by over 30%, enabling more precise and efficient experimentation. This approach allows for faster product iterations by detecting smaller effects and testing on smaller user cohorts. Link to article
ChatGPT for Data Analysis: A Beginner’s Guide
The article explains how to use ChatGPT's Advanced Data Analysis (ADA) tool for tasks like processing data files, creating visualizations, and conducting advanced analyses, all without coding. Link to article
Should you do A/A, A/A/B or A/A/B/B tests in CRO?
The article critiques these tests in CRO, highlighting their inefficiency and recommending a focus on well-designed A/B tests with proper statistical rigor. Link to article
2025 will be the year of the CX Leader
2025 could indeed become the year of the CX Leader, as businesses increasingly recognize the need for customer-centric cultures that permeate every layer of their organization, especially as AI continues to gain adoption. True CX leadership requires embedding customer obsession into every decision, empowering employees, and prioritizing long-term relationships over short-term gains. Link to post
Behind the product: Duolingo streaks
Lenny’s post explains how Duolingo’s streak feature became a powerful tool to help the company grow and shows how trying out new ideas over and over helped them succeed. It shares interesting lessons about keeping users interested and making products better through testing. Link to post
👋 Hi, I’m Brian Poe, Founder and Chief Optimization Officer of Mammoth Insights, a boutique agency with 15+ years of experience helping mid-sized to enterprise companies improve digital strategies through affordable, high-quality Conversion Rate Optimization (CRO) research and experimentation. I’ve had the pleasure of working with leading global brands such as Best Buy, Verizon, Target, Ulta Beauty, Kia Motors of America and more.
Whenever you're ready, there are a few ways I can help you:
1. EXO Field Notes Participation: Want to elevate your personal brand on EXO Field Notes? Here’s how: 1:1 Interviews spotlight top experimentation and product leaders (see Nike example), Micro-Interviews are perfect for newcomers (view archive here), and Roundtable Discussions offer peer-to-peer discussions for 2-3 in-house brands (see roundtable example). Click here to message me on LinkedIn to inquire.
2. Free 90-Minute CRO Workshop: Looking to kickstart your CRO program? I’m offering a free 90-minute workshop for in-house teams new to CRO and A/B testing. It’s not about tool training—it’s about building a strong foundation with topics like CRO fundamentals, prioritization frameworks, and behavioral design principles. Designed for teams using or exploring tools like Optimizely, VWO, or AB Tasty, this workshop also includes exclusive vendor discount codes (no sales pitches, just resources). Space is limited through February, so reach out to grab a spot!
3. Full-Service Experimentation: Looking for an all-in-one solution? Our experimentation service at Mammoth Insights covers everything from crafting the strategy to executing the tests. We’re tool-agnostic, so there’s no need to change your setup. With all team members with 10+ years of experience working with companies like Best Buy and Verizon, we guarantee fast and effective results. Click here to learn more.
4. Conversion Research: Stop guessing and start converting. Our CRO service uses a mix of quantitative and qualitative UX research, including digital analytics and usability tests, to find what works best for your audience. I’ve helped brands like Capella University and Verizon Wireless achieve more with their existing traffic. Click here to learn more.
5. Brand Booster Roadmap: Future proof your brand experience. Work directly with Brian, Chief Optimization Officer, and Jennie, Head of Brand and former Head of Brand Strategy & Management at Wayfair. You’ll receive a comprehensive, tailored audit and brand strategy guide with data-driven recommendations. This includes brand positioning, online experience analysis, omni-channel content analysis, and a detailed brand experience roadmap. Contact us for a free consultation and to learn more.