10 Questions to Ask Any Software Vendor Who Says “We Use AI”

The company that said its AI was building your app

For several years, Builder.ai told businesses something they badly wanted to hear. You didn't need a development team. You described your app to an AI assistant called Natasha, and the platform assembled it for you — faster and cheaper than any agency could.

The customers were ordinary, sensible organisations: founders with a product idea and no engineers, established firms with an app they had been postponing for years. Their external problem was simple. They needed software and couldn't build it themselves. Their internal problem was harder. They weren't technical enough to judge the claims, and they knew it. And underneath sat a reasonable belief: buying software honestly shouldn't require a computer science degree.

The guide they chose looked impeccable. Builder.ai raised hundreds of millions of dollars from investors including Microsoft and the Qatar Investment Authority, and was valued at around $1.5 billion in 2023.

But the doubts had started early. As far back as 2019, The Wall Street Journal questioned how much of the work was really done by AI, reporting that the company relied heavily on human developers. In 2025 the rest unravelled. Bloomberg reported that the company had overstated its sales to creditors by around 300%, a lender seized its cash, and in May it entered insolvency proceedings. Customers were left trying to recover their code and their money from a company that no longer answered.

Here is the uncomfortable lesson. Nothing that went wrong required deep technical expertise to spot. It required a handful of plain questions, asked early and insisted upon. Who actually does the work? Can I see my code today? How do you measure what you claim? What happens to my project if you disappear?

Builder.ai is an extreme case. Most vendors who say they use AI are telling the truth. The real problem is subtler: “we use AI” now appears in almost every proposal, and it can mean anything from a transformed delivery method to a few developers with a coding assistant. The ten questions below separate the two, and they work whether or not you are technical.

Why this question is harder than it looks

Most guides on this topic are written for a different situation: hiring a company to build an AI product, such as a chatbot or a prediction model. This one is about something more common. You are commissioning ordinary business software — a portal, a workflow system, an integration — and the vendor says AI makes them faster, cheaper or better at building it.

That claim deserves scrutiny because the evidence is genuinely mixed. Controlled studies have found AI coding tools make developers anywhere from about 20% to 55% faster on individual tasks, while one rigorous trial found experienced developers were slower on their own codebases, despite believing they were faster. Telemetry from 22,000 developers, published by Faros AI in 2026, showed task throughput rising while bugs, incidents and review times rose faster. A vendor can be sincere and still be wrong about their own productivity.

So the aim is not to catch anyone out. It is to make the real shape of their delivery visible before you sign.

The ten questions

1. Where exactly in the project do you use AI?

A good answer is specific and covers the whole lifecycle: drafting requirements from your existing documents, generating tests, producing documentation, writing routine code under review. A weak answer is “everywhere” or a list of tool brand names. If AI only touches the coding, expect modest gains, because coding is usually less than half of a project's effort.

2. Who reviews what the AI produces, and how do you stop review becoming the bottleneck?

Every AI output needs a responsible human. Ask who that is, at what stage, and how they cope with volume. Research shows review is where AI-accelerated teams jam: median review times have grown several-fold in heavily AI-assisted teams, and more code is merging without any review at all. A vendor with a real answer will talk about small changes, automated checks, and senior engineers reserved for architecture and security.

3. Can you show estimated against actual hours on a real project?

This is the most revealing question on the list. Self-reported productivity is unreliable; hours estimated at scoping against hours actually logged is not. Ask for a named project, or an anonymised one with the method explained. Ask who produced the original estimate. A vendor who has never measured their own AI gains is selling a feeling.

4. Which AI models do you use, and why those?

There is no single right answer, and you should be wary of anyone who claims there is. The market is global and moves monthly: closed models from OpenAI, Anthropic, Google and xAI; open-weight families such as Meta's Llama, Mistral, DeepSeek and Qwen; regional models such as the UAE's Falcon and Jais for Arabic. A strong vendor chooses by task, cost and data rules, and can switch when something better or cheaper appears. Red flags are total dependence on one provider with no fallback, or refusal to say.

5. Where does our data go when AI touches it?

Ask whether your documents, code or customer data are sent to external AI services, in which region they are processed, whether they are retained, and whether they can be used to train anyone's model. For organisations bound by GDPR in Europe, or by data protection laws in the UAE and Saudi Arabia, the answer may decide the vendor on its own. Good vendors can offer options, including open-weight models hosted in your own environment when data cannot leave it.

How AxON approaches this

These questions shaped how we built AxON, our AI delivery platform. AI works across scoping, design, build, testing and documentation, with engineers accountable for every output. It is model-agnostic: tasks are routed to whichever model fits the work, the cost and the client's data rules, rather than to a single provider. We measure every project as estimated against actual hours, which is how we can say our lab operations build came in at 5,300 hours against 10,000. And AxON is how we build, not something clients license: they own standard code and full documentation from day one.

6. Will we have access to our code from day one?

This is the Builder.ai question. You should have access to the source code repository throughout the project, not just at handover, and the contract should state that you own the code, the documentation and any project-specific AI configuration. If the vendor's platform is required to run or change what they build, you are not buying software. You are renting dependence.

7. What happens to test coverage and documentation on AI-assisted projects?

Speed that comes from skipping tests is not speed. Veracode's 2025 analysis of more than 100 models found AI-generated code introduced well-known security vulnerabilities in 45% of test cases. Ask for the vendor's typical test coverage, how tests relate to requirements, and what documentation you will receive. Good AI-assisted delivery should increase both, because generating tests and documentation is exactly the repetitive work AI handles well.

8. How does the AI saving reach us?

If AI genuinely makes a vendor faster, the benefit should appear somewhere you can see it: a lower fixed price, more scope for the same budget, or an earlier delivery date. Be alert to an awkward incentive. A vendor billing by the hour who becomes twice as fast earns half as much for the same work. Ask the question directly and expect a direct answer.

9. If you build AI features for us, what will they cost to run?

Many projects now include AI inside the software itself — search over documents, assistants, automated reports. Those features carry a monthly bill that grows with usage. Surveys in 2026 found most enterprises had overrun their AI budgets, largely because usage grew faster than token prices fell. Ask for a written monthly estimate at your expected user numbers, and ask what design choices keep it down: routing simple tasks to smaller models, caching repeated answers, limiting how much text is sent per request.

10. What do you not use AI for?

This question is the best single test of judgement. A credible vendor will name things: final security decisions, architecture, anything a regulator treats as critical, and any place where a wrong answer is costly and hard to detect. In pharmaceutical manufacturing, for example, Europe's draft GMP Annex 22 keeps generative AI out of critical applications while allowing it in supporting work under human oversight. A vendor who says they use AI for everything has not thought hard about where it fails.

How to score the answers

You do not need to understand the technology to assess the replies. Listen for three things.

1. Specificity. Named stages, named reviewers, named projects and numbers. Vague confidence is the loudest warning sign.
2. Evidence. Anything they can show rather than tell: repository access, test reports, a real estimate-versus-actual comparison, a sample running-cost estimate.
3. Limits. Honest vendors volunteer what AI doesn't do and what it doesn't speed up, such as approvals, third-party integrations and regulatory review.

A vendor who scores well on all three is worth serious consideration. A vendor who scores well only on enthusiasm is worth a second opinion.

Questions that matter less than you'd think

Some questions come up often and reveal very little. “Do you use the latest model?” says nothing about delivery, since the latest model changes monthly. “How many AI engineers do you have?” rewards job titles, not results. “Can you show a demo?” proves a prototype exists, not that a production system will be delivered, maintained and secured. Spend your limited meeting time on the ten above.

When to ask them

Timing matters as much as wording. Put questions 1, 4, 5 and 10 in your request for proposal, so every vendor answers them in writing and you can compare like with like. Save questions 2, 3 and 7 for the first technical meeting, where you can ask for evidence on screen rather than on paper. Raise questions 6, 8 and 9 before the commercial negotiation, because their answers belong in the contract. And ask one of the vendor's existing clients the simplest version of all: did the project arrive when they said it would?

The bottom line

“We use AI” is no longer a differentiator. It is a claim, and like any claim it should be tested. The Builder.ai story is extreme, but its lesson is ordinary: the questions that protect buyers are simple, and the time to ask them is before signing. Ask where AI is used, who is accountable, how gains are measured, where data goes, who owns the code, and how any saving reaches you.

CTA: If you'd like to put these questions to us, we're happy to answer every one in writing — and to show the numbers behind the answers. A two-week Fit Check on one of your processes is the simplest way to see them in practice.

Frequently asked questions

How do I evaluate a software vendor that uses AI?

Ask where AI is used across the project, who reviews its output, how gains are measured, which models are used and where data goes, who owns the code, and how the saving is passed on. Look for specific answers backed by evidence.

Should AI make software development cheaper?

It can, mainly by reducing hours in requirements, testing and documentation. Ask the vendor to show how the saving appears in price, scope or delivery date.

Is it risky to use a vendor that relies on one AI model?

It can be. Dependence on a single provider creates cost, availability and data-residency risk. Vendors who can route work across several models are more resilient.