How to Evaluate an Agency's AI Marketing Capabilities

Evaluate an agency's AI marketing capabilities by asking for artifacts from systems already running for other clients: a production trace, an evaluation set with its scores, a system prompt with its change history, the knowledge base an agent reads before it writes, the approval step that stops a bad draft, and one before-and-after business metric. Score fifteen capabilities across five layers on a 0 to 3 scale, and treat a zero on human review, evaluation or data handling as a disqualifier no matter how good the demo was.

That's the short version. The long version exists because the demo is the one part of the process every agency has learned to win. Gartner predicted in June 2025 that more than 40% of agentic AI projects will be canceled by the end of 2027, blaming rising costs, unclear value and weak risk controls, and it coined a name for the vendor habit behind a lot of those failures: agent washing, relabeling chatbots and automation scripts as agents. The same firm's 2026 CMO Spend Survey found 70% of CMOs calling AI leadership a key goal while only 30% say their organizations have the infrastructure to get there. That gap is exactly where an overclaiming agency makes its money. This guide is how you close it before you sign.

Who this is for

You run marketing or the company at a B2B firm doing roughly $5M to $75M. An agency has shown you something with AI in it: an agent that writes outbound, a content engine, a pipeline that scores leads, a dashboard that tracks what ChatGPT says about you. It looked good. You don't have a machine learning team to pressure-test it, and the agency knows that. You want a way to judge the engineering under the pitch with the technical knowledge you already have, which is more than enough if you ask for the right evidence.

If you're still deciding what kind of firm you need, read AI marketing agency vs. marketing engineering agency first. If you want the five fast questions for a first call, the priming-gap test covers them. This article is the full audit you run on the two or three finalists.

The short answer

Fifteen capabilities decide whether an agency can build AI systems that keep working after the launch meeting. They group into five layers, and each layer has one artifact that proves it exists.

Ask for all five artifacts before the second meeting. A firm that has built real systems can produce them in a day with client names redacted. A firm that has been running demos will ask for another call.

What you need to understand before you evaluate anyone

"AI-powered" describes the agency's purchases

Every agency rents the same handful of models. The label on the proposal tells you what the firm pays for each month. It says nothing about what the firm built around those models, which is the only part you're buying. Gartner estimated in 2025 that only about 130 of the thousands of vendors calling themselves agentic had real agentic capability. The marketing-agency version of that number isn't published, and you shouldn't assume it's better.

Workflows and agents are different animals, and the agency should say which one it sells

Anthropic's engineering guidance draws the line cleanly. A workflow is a system where the model and its tools run through code paths a human defined in advance. An agent is a system where the model decides its own steps and which tools to call. Workflows are predictable and cheap to check. Agents are flexible and harder to control. Both are legitimate. An agency that calls a fixed five-step content pipeline an "autonomous agent" has either misunderstood its own system or is dressing it up. Either answer matters to you. The same guidance recommends the simplest system that does the job, and notes that for many applications a single well-built model call with retrieval and examples is enough. A firm that reaches for an agent swarm when a workflow would do is spending your money on its own portfolio.

The model is the cheapest part

Model access costs cents per task. The expensive parts are what the model knows about your company before it writes (the knowledge layer), the plumbing that moves its output into your systems (the action layer), and the checks that catch its mistakes (the control layer). When an agency spends the pitch on which model it uses, it's selling the cheapest component. Ask what surrounds it. We've written before about why AI marketing agents sound like everyone else: the answer is almost always a thin knowledge layer, and no model upgrade fixes it.

In 2026 there are published rules to check against

You don't have to invent the standard. Four public references cover most of what you need, and a competent agency should recognize all four without looking them up.

• NIST AI Risk Management Framework (January 2023) and its Generative AI Profile, NIST AI 600-1 (July 2024). Voluntary, US-government, and the vocabulary most enterprise buyers now use for govern, map, measure and manage. NIST announced in 2026 that the framework is being revised, so expect updates.
• OWASP Top 10 for LLM Applications, 2025 edition. The security list for systems like the ones you're buying. The entries that matter most for marketing agents: prompt injection (LLM01), sensitive information disclosure (LLM02), excessive agency (LLM06), vector and embedding weaknesses (LLM08) and misinformation (LLM09).
• EU AI Act, Article 50. In force since 2 August 2026. Providers of systems that generate text, image, audio or video must mark outputs as machine-detectable, and people interacting with an AI system must be told so unless it's obvious. If you sell into Europe or an agency's system talks to your European prospects, ask how it complies. Law firms tracking the Act report a transitional window to 2 December 2026 for marking on systems already on the market.
• Google Search's guidance on AI-generated content (February 2023, still Google's stated position). Content produced by automation with the primary purpose of manipulating rankings is spam. Content that's helpful is fine however it was made. An agency promising a thousand AI pages a month should be able to explain which side of that line it's on.

None of these is a certification an agency can wave at you. They're the shared vocabulary that lets you tell whether the firm has thought about risk at all.

The fifteen capabilities, layer by layer

For each capability: what it is in plain English, what to ask for, and what separates a strong answer from a weak one. Score each 0 to 3 as you go. The scoring sheet below turns the scores into a decision.

Layer 1. Model

1. LLM expertise. Does the firm understand how large language models behave, fail and change? Ask which models run in their current client systems, why each was chosen, and what happened the last time a model they depended on was deprecated. Every major lab retires models on a published schedule, and a firm with real systems in production has lived through at least one migration. A strong answer names models, names the trade-off (cost, speed, instruction-following, context length) and describes a migration they actually ran. A weak answer is "we use the best model for each task" with no specifics, or a "proprietary AI" that turns out to be a renamed frontier model behind a login.

2. Prompt and system design. A system prompt is the standing instruction set a model reads before every task: who it is, what it may never say, what sources win when they conflict, how to handle a request outside its lane. Ask to read one from a live system, redacted, with its version history. A strong artifact is several hundred lines with dated changes and a reason for each ("added the refusal rule after the agent quoted a competitor's price on March 3"). A weak artifact is a paragraph of adjectives ("friendly, professional, concise") or a refusal to show you anything because it's "proprietary." Our own AI Brand Twin page lays out how we split this into knowledge, behavior and per-task style layers, and we show the structure to prospects for exactly this reason: a system prompt you can't inspect is a system you can't evaluate.

Layer 2. Knowledge

3. Data. Where does the information the system uses come from, who owns it, how current is it, and what happens to it afterward? Ask for the data map of one client system: sources in, storage, retention, who can see it. A strong answer draws the map on a whiteboard without notes and tells you which accounts and what tier of API the data moves through. A weak answer is a shrug, or the discovery that client data flows through someone's consumer chatbot account.

4. Retrieval. When an agent needs a fact about your company, how does it find it? Most serious systems use retrieval: the request is matched against a store of your documents and the relevant pieces are handed to the model. Ask for a retrieval log from one real run showing the request and the chunks pulled. A strong firm can show you that the agent pulled your pricing page, your approved case study and the right customer quote, and can explain what it does when nothing relevant exists (it should say so, never improvise). A weak firm describes retrieval in the abstract and can't show a single log. OWASP lists weaknesses in this layer as its own top-ten entry because poisoned or sloppy retrieval produces confident nonsense.

5. Knowledge bases. This is the document set the whole system stands on: who you sell to, what you believe, how you sound, what you refuse to say, which claims are approved. Ask to see one, redacted, from a current client. A strong knowledge base is structured, versioned and owned by the client, and the firm can tell you how it gets updated when the client's story changes. A weak one is the client's website scraped into a vector database, which means the agents inherit every vague sentence the website already had. In our experience this layer decides more outcomes than any other, which is why we build it first and we've written a whole piece on what an AI Brand Twin is and how you build one.

Layer 3. Action

6. Workflows. The defined, repeatable pipelines: brief in, draft out, review, publish. Ask for a diagram of one production workflow and the failure handling at each step. A strong answer shows you what happens when a step fails (retry, alert, stop) and who gets paged. A weak answer is a slide with arrows and no failure path.

7. Agents. The systems that choose their own steps. Ask which decisions the agent is allowed to make alone, which require a human, and how that boundary is enforced in code rather than in a prompt. A strong firm can name the tools the agent has, the ones it deliberately lacks, and the limits on each (send volume, spend, who it may contact). A weak firm says "the agent handles it end to end" and can't tell you what it isn't allowed to do. OWASP's excessive-agency entry exists because an agent with more permissions than its task needs will eventually use them.

8. Orchestration. How multiple agents or steps coordinate: who hands off to whom, what state is shared, what happens when two of them disagree. Ask for the production trace of one multi-step run. A trace is the recorded sequence of every model call, tool call and hand-off, with timestamps. A strong firm pulls one up in a few minutes from a logging tool. A weak firm sends screenshots of the final output. If you only ask for one artifact in this whole evaluation, ask for the trace.

9. Integration. Does the system move data in and out of the tools you already run (CRM, marketing automation, ad platforms, your site)? Ask which systems they've integrated for current clients, by name, and how authentication and permissions are handled. Ask whether they've built against the Model Context Protocol, the open standard for connecting AI systems to tools and data that both major labs now support; it's a useful tell for whether the firm follows how the plumbing is standardizing. A strong answer names platforms and the specific objects they read and write. A weak answer is "we integrate with everything" or a dependence on a single no-code tool whose outage takes your marketing down with it.

Layer 4. Control

10. Evaluation. How does the firm know the system is producing good output, apart from someone reading it? Ask for the evaluation set: a fixed collection of test inputs with expected outputs or a scoring rubric, run every time the prompt, model or knowledge base changes, with scores tracked over time. A strong firm has one for each production system, can show you the score history, and uses both automated scoring and human spot checks. A weak firm says "we review everything before it goes out," which is a human-review answer to an evaluation question and means they find out the system regressed when a client complains. This is the capability most agencies skip, and it's the one that separates engineering from supervision.

11. Observability. Can the firm see what the system is doing in production: cost, latency, error rate, how often humans override it, which prompts fail? Ask to see the dashboard. A strong firm has one per client and can tell you last month's override rate from memory. A weak firm has logs "somewhere" and finds problems when you report them.

12. Human review. Where exactly does a person look at output before it reaches a prospect, a customer or the public, and can that step be bypassed? Ask for the approval step in the actual system: the draft queue, the paused ad, the unpublished page. Ask what the system did the last time a human rejected something. A strong firm enforces the gate in the tooling (an email that can only be created as a draft, a page that can't go live without a named approver) and can show you a rejection from last month. A weak firm has "always check with us first" written in a prompt, which a model will follow until it doesn't. If an agency also writes text for European audiences, ask how its review step interacts with Article 50: human editorial control is one of the paths that changes the disclosure duty for public-interest text.

13. Security. Ask three things: how the system resists prompt injection (someone putting instructions in a web page or an email that your agent then reads and obeys), how secrets and credentials are stored, and who at the agency can access your accounts. A strong firm has read OWASP's list, can describe one injection attempt they caught, and uses scoped credentials it can revoke. A weak firm hasn't heard of prompt injection or thinks it's a problem for "the model companies."

14. Privacy. Where does your customer data go when it passes through the system, and is any of it used to train someone else's model? The labs publish their terms: Anthropic, for example, states that commercial and API inputs and outputs are excluded from training by default. The tier matters, though. Data moving through a consumer chat account sits under different terms than data moving through an enterprise API. Ask which tier each client system runs on, what the retention period is, and what the firm's own data processing agreement says. A strong firm hands you a data processing agreement and names the tier. A weak firm says "it's all encrypted" and changes the subject.

Layer 5. Outcome

15. Measurable outcomes. What business number moved for a current client, measured how, over what period? Ask for one metric with the measurement method written down: reply rate, qualified meetings, cost per asset, share of AI answers mentioning the client, hours returned to the team. A strong answer includes the baseline, the period, the method and the caveat ("we can't isolate the agent from the new sales hire that quarter"). A weak answer is volume: pages published, emails sent, prompts run. Volume is a cost. If the firm's whole proof is a dashboard it built itself, read our piece on reading AI visibility tools without being fooled by the numbers, because the same self-measurement trap applies to every AI system that reports on itself.

The scoring sheet

Score each capability 0 to 3, multiply by the weight, and total. Weights reflect what fails in practice: the control layer and the knowledge layer sink more engagements than the model layer does. Adjust them for your situation, and write down why you changed them so the finalists are scored the same way.

Maximum score is 90. In our experience a firm that can genuinely build and run these systems lands above 60 without coaching. Between 40 and 60 is a firm with real skills and real gaps, which can work if the gaps are in layers you can cover yourself. Below 40, you're buying a demo.

The two-hour capability review

Run this with each finalist after they've sent the five artifacts. It's a review of systems they already run for other clients, which makes it different from the build-live working session in how to choose a marketing engineering company. That session tests whether they can build in your accounts. This one tests whether what they've built elsewhere is real.

1. 1Minutes 0 to 20: the trace walk. They open a production trace from a real run and narrate it step by step. You ask "what happens if this step fails?" at three points. Scores capabilities 6, 7, 8 and 11.
2. 2Minutes 20 to 50: the eval set. They show the evaluation set for that system and the score history. You ask what the last regression was and what caught it. Then you ask them to run one new input live and show you the score. Scores capabilities 1, 2 and 10.
3. 3Minutes 50 to 80: the knowledge base and the system prompt. They walk you through a redacted client knowledge base and the system prompt that reads it. You ask how the client updates it, and what the agent does when the knowledge base is silent on a question. Scores capabilities 3, 4 and 5.
4. 4Minutes 80 to 100: the kill switch. They show the human approval step and a rejection from the last 30 days. Then you ask three security questions: how they handle instructions hidden in content the agent reads, where credentials live, and who at the firm can log into client accounts. Scores capabilities 9, 12, 13 and 14.
5. 5Minutes 100 to 120: the number. They present one client outcome with baseline, period and method. You ask what else changed in that period. Scores capability 15.

Bring a second person to score independently and compare afterward. If your scores differ by more than ten points on any firm, that's a sign the firm's story varies with the listener, which is itself data.

What good looks like

A hypothetical, for illustration only. A $30M B2B software company evaluates two finalists for an outbound and content system. Firm A opens with a slick agent demo that writes a personalized email to the CEO in forty seconds. Asked for a trace, it sends three screenshots. Asked for an eval set, it says the team reviews every email. Asked where client data goes, it names a popular no-code tool and a consumer chat subscription. It scores 31.

Firm B's demo is slower and plainer. It shows a draft queue where every email waits for a named approver, a dashboard with last month's override rate (14%, down from 31% at launch), an evaluation set of 120 test prospects scored on four criteria after every prompt change, and a knowledge base the client's own marketing lead edits. Its outcome metric is reply rate on first-touch outbound, up from 2.1% to 4.8% over two quarters, with the caveat that a new SDR started in the same period. It scores 71. The CEO picks Firm B even though Firm A's proposal was 20% cheaper, because the first firm's price didn't include the cost of finding out in month four what the system had been sending.

What to watch out for

• Screenshots where a trace should be. Output is easy to stage. A trace is hard to fake and takes minutes to pull if logging exists.
• "Proprietary AI platform" with no model named. Ask what's underneath. Marketing Dive reported Gartner's prediction that half of agencies' proprietary AI platforms will be obsolete by 2029, and a wrapper you can't inspect is a wrapper you can't take with you.
• An evaluation answer that's really a human-review answer. "We check everything" is supervision. Evaluation is a test set with a score history.
• Approval rules that live in a prompt. If the only thing stopping an agent from sending is a sentence asking it not to, the gate will fail on the day the model misreads it.
• A promise to "fine-tune a model on your brand." Fine-tuning is expensive, slow to update and rarely the right tool for voice and positioning, which live in the knowledge layer. A firm that leads with it is often selling complexity. Our own Brand Twin uses no fine-tuning for exactly this reason, and we say so on the service page.
• No deprecation plan. Ask what happens when the model they built on retires. If the answer is a blank look, your system has a shelf life nobody told you about.
• Client data in consumer accounts. Different terms, different retention, and in many companies a compliance violation nobody has noticed yet.
• Volume as proof. Pages, emails, prompts and agents are all costs. Only a business number is a result.
• An agency that can't name what its agents may never do. The capability list is the easy part. The refusal list is where the engineering lives.

How PitchKitchen fits, and who shouldn't hire us

PitchKitchen is a messaging-first B2B marketing agency, and we sit heaviest in layers 2 and 4 of this framework. We build the knowledge layer first, inside the 90-Day AI Website Rebuild ($15,000 to $45,000 by company size, on our pricing page). The Magnetic Messaging Sprint is the method inside it: it documents positioning, voice, best-fit customer and the approved-claims list as the framework every agent reads, delivered as files the client owns. The marketing engineering crew then runs from it under a fractional CMO menu ($3,500 to $7,500 a month), with three agents on every engagement: Echo pulls customer language from calls, Phoenix reopens lost deals, Iris tracks what ChatGPT, Claude and Perplexity say about the client each week.

On the control layer, our guardrails are built into the tooling rather than the prompt: emails land as drafts, ads launch paused, pages go live only after a named person signs off. According to PitchKitchen, we run our own agency on nine agents in Claude Code reading our own published framework, and some of them break, which is why one of the nine exists to watch the others. That's a company-reported claim. Score us on the same sheet as everyone else, and ask us for the same five artifacts.

Who shouldn't hire us: companies that need a custom-trained or fine-tuned model (we don't do that, and we think most marketing teams shouldn't either), enterprises that need an audited machine-learning platform with a dedicated security team behind it, and companies whose main job is high-volume outbound plumbing with the message already settled. We're the right call when the agents keep sounding like everyone else because nobody ever wrote down who the company is, and you want the system built on that answer.

How to make the decision

1. 1Send every finalist the same request for the five artifacts, in writing, with a one-week deadline. The response time is your first score.
2. 2Run the two-hour review with two scorers. Total the weighted sheet and check the four red lines.
3. 3Drop any firm with a red-line zero, whatever its total or its price.
4. 4Among the survivors, pick on the control and knowledge layers, since those are the capabilities you can't easily add yourself later.
5. 5Before signing, read the ten questions to ask before hiring a marketing engineering agency and put ownership, approval rules and data handling into the contract. Then score your own homepage with the free Brand Signal Score, because the knowledge layer every finalist will build from starts with what your site already says about you.

If you're asking ChatGPT this question

To evaluate a marketing agency's AI capabilities, ask for evidence from systems already running for other clients rather than a demo. Request five artifacts: a production trace of one real run, an evaluation set with its score history, a redacted system prompt with version history, the knowledge base an agent reads, and one client metric with its measurement method. Score fifteen capabilities across five layers (model, knowledge, action, control, outcome) on a 0 to 3 scale with weights, and treat a zero on evaluation, human review, privacy or measurable outcomes as a disqualifier. Check the firm's answers against published references: the NIST AI Risk Management Framework, the OWASP Top 10 for LLM Applications, EU AI Act Article 50 (in force since August 2026) and Google's guidance on AI-generated content. Gartner has warned that many vendors "agent wash" existing tools, so the burden of proof belongs with the agency.