How to build a paid media agency scorecard (with a weighted template)

Most brands do review their agencies. As far back as 2009, an ANA survey found that 82% of marketers ran formal agency performance evaluations. The difficulty lies in what those reviews actually involve: in the same survey, around two-thirds of firms evaluated their agencies only once a year.

A year is a long time in paid media. By the time the review comes round, the strong first quarter has been largely forgotten, while the tracking outage in November is remembered by everyone.

There is a second sign that evaluation is not working as it should. Performance-related pay for agencies has been in decline for a decade. The ANA's Trends in Agency Compensation study found that 41% of marketers used performance incentives in 2022, down from 48% in 2016 and 61% in 2013. There are several reasons for this, but in our experience one of them is fairly straightforward: neither side trusts the scorecard enough to attach money to it.

A scorecard will not rescue a poor relationship on its own (we have written about the wider causes in Why most brand-agency relationships underperform). What it does provide is a shared, written view of what good looks like, which is where most improvements begin.

What a scorecard is for

It helps to be clear about the purpose before building anything. A useful paid media scorecard should:

• Separate what the agency controls from what it does not. Auction prices, stock levels, seasonality and the conversion rate of your own website all affect results.
• Be agreed at the start of the period. One written at the end becomes a verdict on the past, and the agency will, understandably, treat it as such.
• Work in both directions. The agency should have the opportunity to score you as well.
• End in an action list. The score itself matters rather less than the conversation it prompts.

It is not a substitute for day-to-day account management, nor should it be a pitch in disguise. If the decision to move the business has already been made, a proper review is the more honest route.

Four pillars, and how to weight them

We build paid media scorecards around four pillars. The weights below are our starting point, and most clients end up adjusting one or two of them.

• Business outcomes, 40%. Did the activity move the numbers that were agreed? This is the heaviest pillar, but deliberately below half, because results depend partly on factors outside the agency's control.
• Operational excellence, 25%. Account hygiene, accurate and timely reporting, pacing against budget, speed of response, and how well things are handled when something goes wrong.
• Strategic thinking, 20%. Proactive ideas, the quality of testing, sensible use of new platform features, and planning that reflects your commercial position.
• Partnership, 15%. Transparency on fees and costs, the stability of the team, communication generally, and how the agency handles bad news.

The ANA data offers a useful check on what marketers already value. Its most commonly used qualitative criteria were innovation (85%), strategy (82%), implementation and follow-through (82%) and fiscal stewardship (81%), all of which sit comfortably within those four pillars.

As for adjusting the weights: if the agency mainly executes and planning happens in-house, it makes sense to move five or ten points from strategic thinking to operational excellence. If you appointed them chiefly for their ideas, the reverse applies. The one thing we would avoid is changing the weights partway through the year.

Measuring outcomes without fooling yourself

The outcomes pillar is where most scorecards go astray, largely because it is so tempting to use whatever the ad platform reports. Platform ROAS is perfectly useful for optimisation. As evidence of agency performance, however, it amounts to the agency marking its own homework, using a model built by the company selling the media.

We use a simple measurement ladder: the higher the rung on which you can assess the agency, the more the scorecard is worth.

The higher rungs have become considerably more accessible. In September 2026 Google made Meridian GeoX, the geo-experiment tool within its open-source marketing mix modelling package, generally available worldwide. It compares test and control regions without relying on user-level tracking, and works across platforms rather than only Google's own. If your agency has never proposed an incrementality test, that is telling in itself.

There are two things worth bearing in mind when setting targets.

The first is Goodhart's law, usually paraphrased as "when a measure becomes a target, it ceases to be a good measure". Reward CPA alone and you will probably see brand search creep up as a share of spend, since brand terms are cheap and convert well whether or not the advertising caused the sale. Any efficiency metric is best balanced with a volume metric and, where possible, an incremental one.

The second is time horizon. Les Binet and Peter Field's analysis of the IPA Databank found that sales activation produces short-term peaks, while the effects of brand building accumulate over longer periods, and that the strongest results came from a balance of the two. A scorecard built entirely on monthly CPA will nudge the agency towards activation every time. If they run video or upper-funnel programmatic on your behalf, reach and brand measures belong in the outcomes pillar too.

The template

The template below is the one we start from. Each criterion is scored from 1 to 5. We prefer a five-point scale to a ten-point one, partly because people find it difficult to distinguish a 6 from a 7, and partly because five points only require you to describe what a 1, a 3 and a 5 look like.

The overall score is the sum of each criterion's score multiplied by its weight, with a maximum of 5.00. We read the result in four bands:

A worked example

The table below shows a hypothetical quarter for an agency handling paid search and social.

A total of 3.35 places this agency in "needs work", if only just. The total, though, is not the most interesting part; the pillars are. Partnership averages 4.00 out of 5 and the operational basics are in reasonable shape. The weakness lies in strategic thinking, at 2.90, and within outcomes the incrementality score is a 2. Put simply, this is a pleasant and reliable agency that hits its CPA but rarely pushes. That is a far more useful finding than "3.35", and it gives a clear brief for the following quarter: a testing roadmap and at least one incrementality study.

The other half: the agency scores you

In the same ANA survey, 59% of marketers with formal programmes used two-way evaluations, in which the agency also rates the client. In paid media this matters more than is often acknowledged. A slow approvals process or a missing offline conversion feed can easily cost more than a mediocre account manager.

The same 1 to 5 scale works well here, and this side can be left unweighted. In the UK, ISBA and Alchemists have published a best-practice guide to evaluating client-agency relationships, which ISBA members will find worth reading.

How often to score

• Monthly pulse. Half an hour on the operational pillar only, scored from logs and dashboards, with the aim of catching problems while they are still small.
• Quarterly business review. The full scorecard, in both directions, ending with an agreed action list. This is the review that carries most weight.
• Annual review. A broader 360 involving stakeholders beyond the day-to-day team, a look at the commercial terms, and targets reset for the year ahead.

Linking the scorecard to fees

We would only do this once the scorecard has run for at least two quarters and both sides trust the numbers. Tie money to a scorecard that nobody believes, and every review becomes an argument about the scores rather than a discussion of the work.

When the time comes, a few principles help:

• Keep the at-risk share modest. We would usually start at 10 to 15% of fees, with upside as well as downside.
• Pay on what the agency controls. That means the operational and strategic pillars, plus outcomes measured at the verified or incremental rung rather than platform ROAS.
• Keep the calculation mechanical. An incentive that has to be renegotiated every quarter will erode trust faster than it improves performance.

This is becoming more relevant over time. The same ANA study found that 53% of large advertisers now use fixed or output-based fees, up from 5% in 2016. When you pay for outputs, the scorecard is how you define what a good output looks like.

Mistakes we see often

• Too many criteria. Beyond fifteen or so, nobody reads it properly and every score drifts towards a 3.
• Scores without evidence. Each score should point to something that can be opened and checked: a log, a report or an audit.
• Changing the weights mid-year. It looks like moving the goalposts, largely because it is.
• Letting one poor month dominate. Score the quarter, not the last fortnight.
• Treating the number as the result. What you are really after is the two or three things that will be different next quarter.

Where to start

Take the template, trim it to the criteria that matter for your account, and agree the weights with your agency before the quarter begins. Then run a baseline quarter with no consequences attached. That first conversation will very likely tell you more about the relationship than a year's worth of monthly reports.