How to choose an AI agency

An AI agency designs, builds, and maintains AI applications for other companies. That is not the same as an agency that buys existing tools and configures them. At an AI development agency, software gets written — shaped around your data and your systems.

The distinction sounds semantic. It is not. Configuring a tool takes days; building an application that reads your contracts, hangs off your CRM, and can demonstrate its answers are right takes months and costs accordingly.

So ask early which of the two you are sitting across from. Both are legitimate. The price, the timeline, and what you are left holding are completely different.

This question separates agencies faster than anything else. Not what has been built — what is running now, who uses it, and how long it has survived.

Building a demo is an afternoon's work. The difference lives in the months after: permissions and access, error handling, cost per request, and being able to show the answer means something. Agencies that have never been through that phase underestimate it consistently, and you pay the gap.

Listen for what is not said, too. If the conversation stays on pilots and proofs of concept, you already have your answer.

Ask it literally. The answer tells you more than a portfolio does.

A workable rule: AI is promising when the work is language or documents, when there is real repetition in it, and when no rule can be written that solves it. If your process fits into a handful of if-then rules, ordinary software is cheaper, faster, more predictable — and you can explain afterwards why something happened.

An agency that answers every question with AI is selling its inventory. The ones willing to turn work down usually have enough work.

This is where agencies without production experience run out of road.

What you want to hear has three parts. A test set built from your own edge cases, so every change is measurably better or worse. Answers that cite the source they came from. And an explicit floor: too little grounding means the system says it does not know, rather than producing something plausible.

Miss those three and you do not have an application, you have a well-dressed guess. On Wally, the assistant we built for accounting firms, this was the heaviest part of the work — a tax answer is worthless without the official source underneath it. A confidently wrong answer about VAT does more damage than no answer.

With AI the build is not the only invoice. Every request costs money in model usage, and that figure grows with your usage instead of staying flat.

An agency that has done this before works it out at realistic volumes up front: what does this cost per month at a hundred users, and at ten thousand? An application that works technically but loses money per user is still a problem. You just discover it at the moment things start going well.

Ask who can move that number, too. Caching, a smaller model where one will do, how much context you send per request — those are architecture decisions. Retrofitting them is expensive.

Not for the code. For the project, yes — though not for the reason people usually give.

AI projects live or die on access to the people who currently do the work. The questions that decide the outcome are about exceptions, habits, and unwritten rules. Those surface when you sit next to someone for a morning. Not in an hour-long video call with eight attendees.

If your agency is far away, ask how they arrange that access instead. “We work remotely” only counts when there is a plan behind it.

These fit into a single half-hour meeting. Together they give a surprisingly complete picture.

• What is in production today, and for how long?Demos say nothing about the phase where the money goes.
• When would you advise against AI?No answer means you are talking to a salesperson.
• How do you prove the answer is right?A test set from our edge cases, source citation, and an explicit floor. All three, not one of three.
• What does this cost per month at ten thousand users?Model usage recurs and grows. If they have not worked it out, they have not shipped it.
• How will you reach our people?The exceptions that make or break the project are documented nowhere.
• Who owns the code afterwards?And where does it run. Ask now, not at handover.