What it means
The distinction people miss is between data-informed and data-directed. Data-informed means the numbers are one input alongside judgement and strategy, whereas data-directed means the organisation commits in advance to following what the measurement says.
The second is more disciplined but only works where the question can be measured cleanly and cheaply. That makes it best suited to frequent, reversible decisions: pricing pages, email subject lines, checkout flows, stock levels, credit limits.
Rare, strategic and irreversible choices such as entering a new country or buying a competitor cannot be tested this way and have to remain judgement calls supported by analysis. Applying a directed approach to the wrong class of decision produces false confidence.
The mechanics are straightforward. Write down the decision and the criterion before looking at the data, choose a metric that reflects the business outcome rather than a convenient proxy, run the test long enough to reach a reliable result, then act on it whatever the answer turns out to be.
Writing the criterion first is what stops people reinterpreting the numbers once they can see them. The most common failure is measuring the wrong thing.
A change that lifts click-through but lowers order value looks like a win on the dashboard and a loss in the accounts, so metrics should be tied to contribution or cash wherever possible. The second failure is stopping a test the moment it looks positive, which reliably manufactures results that do not repeat.
Culturally the hard part is agreeing that the evidence outranks the most senior opinion in the room. Organisations that manage it usually do so by making the criterion public before the test and by treating a disproved favourite idea as a good outcome.
The payoff compounds, because many small verified improvements beat a few large unverified ones.
In practice
Real-world examples.
Example
A software company is split on whether to offer a free trial or a demo-led sales process. It runs both for a quarter across matched segments and measures 90-day paid conversion, agreeing in advance to adopt whichever wins. The free trial converts at 7.4% against 5.1%, so the sales team is restructured around it even though the head of sales had argued the other way.
Example
A restaurant group tests removing eight low-selling dishes from the menu in six of its 40 sites. Average spend per head rises by $1.90 and kitchen waste falls by 11% in the test sites, with no drop in covers. The change is rolled out across the whole estate the following quarter.
Example
A haulage business must choose between two depot locations. Because the decision is once in a decade and cannot be tested, the team runs route modelling on two years of actual delivery data and treats the output as strong evidence rather than a verdict. The site chosen is second best on modelled cost but far better on driver retention, which shows where a directed approach has to give way to judgement.
Formula
Calculation
Value of a tested change = additional conversions x contribution a conversion, scaled to full traffic, less the cost of building it
An online retailer tests a redesigned checkout page, splitting traffic evenly with 20,000 visitors seeing each version.
Control: 20,000 x 3.2% = 640 orders
Variant: 20,000 x 3.8% = 760 orders
Difference = 760 - 640 = 120 extra orders from 20,000 visitors, a lift of 0.6 percentage points
Average order value is $60 and gross margin is 40%, so contribution an order is $60 x 0.40 = $24.
Annual traffic to the checkout page is 1,200,000 visitors.
Extra orders at full scale = 1,200,000 x 0.006 = 7,200
Extra contribution = 7,200 x $24 = $172,800
Build and testing cost = $40,000
Net annual benefit = $172,800 - $40,000 = $132,800
The decision rule was written before the test: adopt the variant if it delivers at least a 0.3 percentage point lift. It delivered 0.6, so the variant ships.Case study
Seen in the real world.
Marlowe and Vane is a fictional online furniture retailer, used here as an illustrative example of the approach and its limits. Its executive team had spent two years arguing about whether free delivery would grow the business or destroy the margin.
Instead of settling it in a meeting, they agreed a criterion in writing: free delivery on orders over $400 would be adopted permanently if, over eight weeks, it raised contribution per visitor by at least 5%. The test ran across half of all traffic, 320,000 visitors, and produced a 9% increase in contribution per visitor, from $2.10 to $2.29.
Applied to annual traffic of 4,000,000 visitors, the extra $0.19 a visitor was worth 4,000,000 x $0.19 = $760,000 a year, and the policy was adopted the following month. The illustrative lesson arrived a year later, when the same team tried to settle a brand repositioning the same way and found no test that could answer it, which is precisely where data-directed decision making stops and judgement starts.
Watch out
Common mistakes.
- Choosing a metric that is easy to measure rather than one that reflects profit, so the winning variant improves clicks while reducing contribution.
- Peeking at results and stopping the test as soon as they look favourable, which inflates apparent wins that do not survive a full rollout.
- Applying the approach to rare, irreversible strategic decisions that cannot be tested, and then calling the resulting analysis proof.
Questions
People also ask.
How is this different from being data-informed?
Data-informed treats evidence as one input among several, while data-directed commits in advance to acting on the measured result.
What if the test result is inconclusive?
Treat that as an answer, because it usually means the change is too small to matter, so keep the simpler option and spend the effort elsewhere.
Do we need a large team to do this?
No. The essentials are a written decision rule, one honest metric and the discipline to run the test to completion, none of which require specialist staff.
From the founder's library
Take it further with the book.
Build your financial confidence beyond this definition. Shihan's full-length guide, Accounting Fundamentals, takes the same plain-English approach and turns it into a complete, practical playbook for non-finance managers, business owners and students - with chapter-end quiz answers and presentation slides included.
25% off with code MMHQ25, applied at checkout. Priced in USD - checkout may show the equivalent in your local currency.
View the book and save 25%