Questions to ask an AI agency before you hire one
Every second consultancy now has “AI” somewhere on its homepage. There is no licence or accreditation for calling yourself an AI agency in Australia — anyone can put it on a business card, and plenty have. That would matter less if the work reliably paid off. It doesn’t: RAND’s late-2025 analysis of enterprise AI projects found 80% fail to deliver the business value promised, and MIT’s Project NANDA put the figure for generative AI pilots even higher — about 95% show no measurable return on the P&L. So before you sign anything, these are the questions to ask an AI agency before you hire one, and the answers that should make you walk away.
Questions to ask an AI agency before you hire one
“Show me a working system, not a case study.” A slide with a logo and a percentage proves nothing. Ask to see something they built running, even a screen-share of a similar client’s setup with names removed. If everything they’ve done is “confidential” and they can’t even describe the category of result, that’s a sales pitch, not a track record.
“What exactly will you connect to, and who owns the output afterwards?” Get specific about which of your systems the work touches — your CRM, your quoting tool, your accounting package — and ask plainly: if we stop paying you, do we keep what was built, or does it stop working? Some agencies build on top of tools you can walk away with. Others build something that only runs through them.
“What data of ours will this see, and where does it go?” This is the one question almost nobody asks and it’s the one that matters most. Under the Privacy Act 1988, you’re responsible for what happens to your customers’ personal information even after you’ve handed it to a vendor. Ask which AI model or provider processes your data, whether it’s used to train anything beyond your own results, and whether it’s stored or processed overseas. A competent agency answers this in one sentence. A vague one starts talking about “enterprise-grade security” instead.
“What’s the one thing that would tell us this worked?” If the first meeting doesn’t include a question about what a good outcome looks like for you — a number, a time saved, an error rate — you’re being sold a tool, not a result. Agencies worth hiring ask about your business before they pitch their capability.
“What happens if it underdelivers?” Ask for an example of a project that didn’t go to plan and what they changed. Nobody bats a hundred. An agency that can’t name one either hasn’t done enough real work yet, or isn’t being straight with you.
“Can we start small, and what does the exit look like?” Watch for long lock-in contracts, no trial or pilot phase, and no clear way to get your data back out in a usable format. A short scoped first stage — priced separately from any bigger build — is a normal, healthy way to start.
The pattern is the same for every question on that list — it’s not the technology that disqualifies an agency, it’s how they answer a direct question about your business:
Red flags worth ending the meeting over
- The pitch leads with a deck about them, not questions about you.
- “Proprietary AI” they can’t explain in plain terms — often a wrapper around an off-the-shelf model with a markup attached.
- No answer on where your data goes or who else’s systems it touches.
- Every result is unquantified (“clients love it”) rather than measured.
- Twelve-month-plus contracts before you’ve seen a single working outcome.
None of this means AI agencies are a bad idea — most SMEs genuinely need outside help to move past the demo stage. It means the burden of proof sits with them, not with your gut feeling in the meeting.
Ask the questions of yourself first
The strongest position to walk into that meeting from is already knowing your own foundations — what your data actually looks like, where it lives, and what “good” would mean for your business — so an agency’s answers have something real to be measured against. That’s exactly what our complimentary AI-Readiness Data Check is for: twelve questions, about ten minutes, no sales call required to get the result. Most people who run it walk into their next agency conversation asking sharper questions than the agency was expecting.