Why experimenting well matters (and how to do it better)

16/4/2025 ☼ experimentation ☼ strategy ☼ uncertainty ☼ methodology ☼ research

tl;dr: Today’s business environment is changing ever more quickly. It’s becoming too unpredictable for conventional planning. How can organisations adapt and innovate when traditional long-cycle planning isn’t good enough anymore? The answer — across governments, established businesses, startups, and NGOs — is to experiment more and do it better.

Experimentation is closely associated with product teams and product development — practices like A/B testing, or beta launches, or conversion funnel testing. This narrow view misses the much bigger picture: Experimentation can be a powerful strategic capability across the whole organisation.

I recently worked with a client whose sales team struggled to identify good leads, qualify them properly, and convert them into customers. Rather than completely overhauling their sales processes based on theoretical “best practices” developed in other quite different industries — a big investment of time and effort with unclear returns — we identified ways to run quick, cheap, and informative experiments. These took the form of small changes to specific parts of their existing sales process. Within two months, they had found a way to increase qualified leads by more than a third, and shortened the average sales cycle by nearly three weeks.

Here are two other examples:

Experimenting with hiring process: An AI startup working in a novel application space was having trouble hiring staff to join its R&D team because the application space was poorly understood. They designed a way for potential hires to experiment with being at the startup without signing a full-time employment contract (by using a fast-to-execute standard consultancy agreement), and a series of experiments to test different audiences in which to advertise their job availabilities. Result: Applicant pool size increased by ~200% within 2 months of starting these experiments; 18 months after the experiments, 40% of the full-time R&D staff converted from the experiment.
Experimenting with new product development: A packaged ready-to-eat food subscription company used a low-volume fast-turnaround test kitchen to rapidly test different recipe formulations and price points with a large number of clients in the target customer demographic. Result: Reformulated recipes to increase customer satisfaction and willingness-to-pay by ~15%, while reducing unit production cost by ~30% at scale.

The insight here is: Experimentation isn’t just for creating product prototypes — it’s a general principle for learning and adapting when you don’t immediately know the best path forward.

Whether you run an established company looking for operational improvements, a startup refining your market fit, or a government agency improving service delivery, well-structured experimentation is an action framework that helps your team navigate uncertainty and find solutions that work.

Experimentation’s untapped potential

Organisations that develop the capacity to experiment well gain many competitive advantages:

Strategic adaptation capability: When regulations change unpredictably, consumer preferences shift rapidly, and supply chains face disruption, experimentation provides the only reliable way to navigate uncertainty. Companies skilled in systematic experimentation build resilience to change that competitors cannot match.
Cross-functional applicability: Experimentation’s greatest untapped potential extends beyond product development — into sales strategies, hiring processes, investor pitches, business models, and daily operations. These benefits compound when experimentation skills spread throughout your organisation.
Less unnecessary downside: Well-designed experiments transform big, monolithic high-stakes decisions into structured explorations where each small, low-downside stage reduces the downside exposure from subsequent stages. Instead of betting big on untested assumptions, experimental approaches allow you to explore widely, validate ideas incrementally, and control downside.
Faster learning: Organisations that experiment effectively learn faster than competitors. This creates market advantages. They spot promising opportunities sooner, abandon unproductive paths earlier, and adapt to changing conditions more quickly.

Unfortunately, most organisations don’t enjoy these advantages because they don’t know how to experiment well.

Some signs of this include teams that insist on “finishing” or “polishing” work product for months (whether it is a piece of software or an analytic memo) before showing them to other parts of the organisation, and employees that focus on a small number of big innovation projects that are “pretty likely to work” instead of trying to build a large number of inexpensive small bets to explore the terrain of possibility.

Despite experimentation’s clear advantages, most organisations don’t do enough of it, and almost none experiment in a strategically effective, learning-optimising way.

Why organisations struggle with experimenting well

Over the last decade, I’ve worked with government agencies, established companies, nonprofits, and startups. I’ve repeatedly seen the same set of barriers to good experimentation:

Inadequate training: Most organisations don’t have teams trained to think rigorously about research — one of the consequences is that struggle with experiment basics. Their research questions are too broad, the problems they want to investigate are poorly structured, their experiment designs don’t address the problems directly or produce hard-to-interpret data; the list goes on and on.
The allure of the big bet: There’s often a strong instinct to focus efforts on refining an experiment on which most hopes can be pinned. Teams aren’t trained and incentivised to continually redesign experiments to be as small, cheap, and fast to run as possible, while preserving their learning value.
Misaligned incentives: Traditional organisational metrics and reward systems discourage experimentation by visibly rewarding those who meet legible KPIs and punishing “failures,” rather than incentivising work that is designed to promote learning regardless of outcome.
Implementation challenges: Experiments that are poorly thought-out and designed produce insights that may be valid but are hard to translate into operational changes, missing the full value of their learning.
Cultural resistance: There is a mistaken assumption that experiments always create major downside exposure or require big investments without guaranteed returns. Teams caught up in day-to-day firefighting incorrectly believe that they don’t have time to experiment.

Being strategic about effective experimentation

Getting over these challenges requires a systematic approach to developing experimentation as an organisational capability. Here are some key aspects of this capability for experimentation excellence, based on my work with organisations ranging from international NGOs with offices around the world, to companies with operations on multiple continents, to governments, to seed-funded startups:

Ask good questions: Begin with questions that clearly identify strategic uncertainties.
Frame problems well: Build on good questions to develop well-structured problem statements that enable targeted experimentation.
Design valuable experiments: Design experiments to directly address well-framed problems and provide meaningful insight regardless of experimental outcome.
Refactor experiments for efficacy: Systematically redesign experiments to increase their speed, and reduce their cost and downside potential, while preserving their learning value.
Build learning systems: Establish processes to capture, synthesise, and share experimental capability across your organisation.

Developing this capability requires systematic and focused investment

A path forward

Adapting quickly to changing conditions is rapidly becoming essential for survival and success. Developing strong experimentation capabilities has become a strategic and competitive imperative — whether you’re looking to design a new product, improve your sales funnel, fix a broken hiring process, do better on investor pitches, find a new business model, or do some other important thing that your organisation doesn’t know how to do yet.

The organisations that will thrive in the coming years will be those that build their internal capacity to learn quickly and effectively through well-designed experimentation.

Get in touch if you’d like to talk about building experimentation capacity in your organisation.

I’ve been working on turning discomfort into something productive. idk is the first of these tools for productive discomfort.

And I’ve spent the last 15 years investigating how organisations can succeed in uncertain times. The Uncertainty Mindset is my book about how to design organisations that thrive in uncertainty and can clearly distinguish it from risk.