We picked a headline metric, put it on a dashboard, and told everybody it was the thing that mattered. It improved every month for eight months. Revenue did not move.
What happened is a fairly ordinary failure and it is worth describing because the mechanism is general.
The metric we chose
Cost per acquisition, defined as media spend divided by recorded conversions.
Reasonable on its face. It is comparable across channels, it is easy to calculate, and it points in an obviously desirable direction.
The problem is that it is a ratio, and a ratio can be improved by making the numerator smaller or the denominator larger, and only some of the ways of doing that are good for the business.
How it got gamed without anybody trying
Nobody set out to manipulate anything. The optimisation happened because the systems did what they were told.
Budget shifted toward audiences that converted cheaply. Those turned out to be people already close to purchasing — existing customers, people who had already visited, branded search.
Which meant we were increasingly paying to reach people who were going to convert anyway. The recorded cost per acquisition fell beautifully. The number of conversions that would not have happened without us fell too, and that number was not on the dashboard.
Simultaneously, we cut spend on the audiences that converted expensively, which were people unfamiliar with us. Those are the people who become next year's cheap conversions, and we stopped acquiring them.
The metric improved for eight months and the pipeline of future customers was quietly drained.
The attribution problem underneath
The deeper issue and the one that makes this hard to avoid.
Attribution assigns credit for a conversion to touchpoints. Whichever model you use, it distributes credit among things that were observed.
What it cannot do is tell you what would have happened without them. A conversion attributed to a paid search click may have been going to happen regardless, and attribution has no way to represent that.
Which systematically favours channels that appear late in a journey, close to the moment of decision, and systematically undervalues anything that creates awareness earlier.
The consequence, across the industry, is a persistent drift of budget toward the bottom of the funnel, which looks efficient and hollows out the top.
What we did about it
Three things, of which one worked well, one worked partially, and one was too expensive for us.
Incrementality testing. Holding out a portion of the audience and measuring the difference in outcome. This is the only method that answers the question directly, and it works. The cost is that you have to deliberately not advertise to some people, which is politically difficult and which finance departments dislike.
Adding counter-metrics to the dashboard. Alongside cost per acquisition, we tracked total new customers, share of conversions from audiences new to us, and total volume. A ratio next to its components is much harder to game accidentally.
And marketing mix modelling, which is the statistical approach to estimating channel contribution without relying on individual tracking. It works and it requires more data history and more analytical capacity than we had at the time.
The general lesson
Any single metric that a team is judged on will be optimised, including in ways nobody intended.
This is not a new observation and it keeps happening because a single number is genuinely useful for coordination and a dashboard with fifteen numbers does not coordinate anything.
What has worked for us since is a small number of metrics that pull against each other. Efficiency next to volume. Cost next to total new customers. Short-term response next to a longer-term brand measure.
Any one of those alone can be improved in damaging ways. Improving all of them simultaneously is difficult to do accidentally.
The thing I would tell somebody
Before adopting a metric, spend twenty minutes asking how you would improve it if you were being dishonest.
Then assume that a system optimising against it will find those routes without any dishonesty involved, because that is what optimisation does.
If the dishonest routes are easy and the honest ones are hard, the metric will get worse for the business as it gets better on the dashboard, which is exactly what happened to us for eight months while everybody was pleased.
The organisational half of the problem
Worth naming because the analytical fix is not sufficient on its own.
The reason we optimised toward a damaging metric for eight months was that people were reviewed against it. Nobody was going to raise a concern about a number that was making them look competent.
Changing the metric without changing what people are assessed on produces a dashboard nobody acts on.
What worked was moving the team's objectives to the counter-metrics — total new customers, incremental revenue — and treating efficiency as a constraint rather than a goal.
That is a harder conversation than adding a chart, and it is the one that actually changes behaviour.