Navigating a Reorg as an Experimentation Team
We’re almost through the first quarter of the year (which is crazy to think about), and for many, that means organizational shifts and reorganizations. If you’ve been in the experimentation space for a while, you know how these shake-ups can affect testing programs—sometimes for the better, sometimes… not so much.
Right now, I’m helping a Fortune 500 company and their experimentation team (my largest client) navigate a complex reorg. It’s highly matrixed, but we have leadership support to push through necessary changes. That makes a big difference. However, not everyone is in that position, and I know that for many, reorgs can feel like being caught in a riptide—you’re not sure whether to swim with it, against it, or just try to stay afloat.
So, if your experimentation program is feeling the impact of a reorg, here are five key considerations to help you navigate:
1. Stakeholder Alignment & Advocacy
New leadership? Shifting priorities? Budget shake-ups? All of the above? Welcome to reorg season. One of the first things you should do is identify key stakeholders and ensure they understand how experimentation aligns with business goals. A well-positioned program doesn’t just survive a reorg—it proves its value as a critical driver of data-backed decision-making. Be your program’s best advocate.
2. Knowledge Transfer & Documentation
When teams shift, valuable insights can disappear overnight. Keep a well-documented experimentation roadmap, results, and key learnings. Think of it like a survival guide—if leadership changes or team members move on, you don’t want to lose the context of past experiments or risk running the same tests again. A strong documentation process makes onboarding new team members easier and ensures continuity.
3. Cross-Functional Collaboration Shifts
Experimentation is a team sport. It thrives on collaboration between product, marketing, UX, and data teams. A reorg might change reporting lines, create new dependencies, or introduce silos. Get ahead of these shifts by mapping out how the new org structure impacts workflows. Proactively rebuild bridges and establish clear processes to keep experimentation running smoothly.
4. Cultural Buy-In & Experimentation Maturity
Not all leadership changes come with the same enthusiasm for experimentation. Some see it as a business accelerator, while others view it as “nice to have.” If your new leadership isn’t familiar with the power of testing, you may need to reframe the narrative. Position experimentation as a way to de-risk decisions and optimize performance, rather than as a bottleneck or an extra step in the process.
5. Resource & Tooling Stability
Budget reallocation, tool consolidation, or shifting priorities can impact the resources available for experimentation. If your team relies on a specific testing platform or analytics tool, confirm what will remain in place. If cuts are coming, look for alternative solutions and make a case for why maintaining critical tools is necessary for driving business impact.
Reorgs can be messy, unpredictable, and, at times, frustrating—but they also create opportunities to reset, realign, and reinforce the value of experimentation. As the year picks up speed, our job isn’t just to keep moving—we need to move with intention, ensuring our teams stay aligned, our work remains impactful, and we continue to advocate for data-informed decisions.
Which of these considerations feels most relevant to you right now? What’s been the biggest challenge in your organization? Let me know by leaving a comment—I’ll share my perspective, too!
Know When to Personalize and When to Pass
Josh is an experienced professional with a robust foundation in data science, modeling and personalization. We had a really fun chat about personalization and in this clip, he shares some great tips on when to use it and when to avoid it.
Josh and I go on to discuss personalization considerations, the importance of Incrementality, and frameworks for assessing your personalization readiness.
Watch the full interview with Josh (35 minutes)
📚 Worth Exploring
The Duolingo Handbook
Duolingo, a self-proclaimed quirky business, writes about their love for experimentation, highlighting how they’re all about trying crazy stuff, learning on the fly, and giving folks room to test new and inventive ideas. Interesting read. Link to PDF
Experimentation at Nike (43 minutes)
In this article, Nic English shares his experience growing Nike’s experimentation program in EMEA, highlighting the importance of starting with simple tests, aligning metrics with business goals, and embracing bold risks. Link to interview
Extensible Experimentation Platform: Effective A/B Test Analysis at Scale
Microsoft shared a white paper on how they run ~100k A/B tests annually using their internal Experimentation Platform (ExP). Link to paper
Interpreting A/B test results: false positives and statistical significance
Netflix’s team digs into the mess of false positives in A/B testing, showing how nailing down statistical significance and setting firm experiment rules upfront can sharpen the accuracy of results. Link to article
Bayesian vs. frequentist statistics: Not a big deal?
In this article, the Statsig team breaks down the clash between Bayesian and Frequentist statistics, highlighting how Bayesian methods can mix past knowledge with new data to give sharper answers in A/B testing, while Frequentist approaches stick to strict rules based only on what’s collected. Link to article
The CRO Shift Is Here: Future-Proof Your Career at Conversion Live (Sponsor)
AI is transforming marketing faster than ever, and 2025 is the year we either use it or lose it. At Conversion Live 2025, we’re bringing together the brightest minds in CRO, AI, and digital strategy to unpack the tools, tactics, and trends that will define the future. How will you optimize in a world where search traffic is shrinking, ad costs are rising, and AI is rewriting the rules? Find out at this one-of-a-kind virtual conference.
3 days of expert-led sessions
Live networking in our virtual lounge
Access to ConversionMind, an AI trained on the event’s content
Seats are limited so Save your spot today.
A Micro-Interview with Rosario Toscano
This week I’m joined by Rosario Toscano, a CRO Consultant from Treviso, Italy.
Beyond the great opportunities offered by artificial intelligence, take a few minutes each day to reflect on your day: what did you do well, and what can you improve? Be honest with yourself, because there is always room for growth.
Read the full interview with Rosario
Interested in sponsoring the newsletter?
Interested in sponsoring or advertising on this newsletter? I’m always looking to partner with brands and businesses that align with my audience. Reach out to me at brian@mammothinsights.io to inquire about sponsorship opportunities!
About Me & Mammoth Insights
I’m Brian Poe, Chief Optimization Officer at Mammoth Insights, where we help mid-sized to enterprise companies improve digital strategies through data-driven Conversion Rate Optimization (CRO), experimentation, and conversion research. With 15+ years of experience working with brands like Best Buy, Verizon, and Target, I specialize in turning insights into revenue and growth.
Want to level up your CRO program? Join my 90-minute CRO Kickstarter Workshop—a hands-on session covering CRO fundamentals, prioritization frameworks, behavioral design, and more. I’ll show you everything I’ve learned over 15+ years in CRO and how to maximize your program. Only $499 USD. Let’s get started—reach out today!