The Questions to Ask an AI Vendor Before You Sign

Every software vendor you deal with has added AI to their pitch in the last eighteen months. Some of them built something. Some of them added a chatbot to a settings page. From the outside, the demos look identical.

The difference shows up eleven months later, in a renewal you cannot get out of.

Six questions that do the work

1. What model is underneath, and who runs it? A straight answer sounds like a name and a hosting arrangement. A vague answer — "proprietary technology," "our own algorithms" — usually means they are reselling someone else's model with a wrapper. That is not disqualifying, but it changes what you are paying for and who actually controls your data.

2. Does our data train your model? Ask it exactly that way, and get the answer in the contract rather than in an email. "We take privacy seriously" is not an answer. The answer is yes or no.

3. What is the accuracy rate, measured how, on what data? Vendors quote accuracy from a benchmark that resembles nothing in your business. Ask what it does on inputs like yours. If they have never tested that, you now know something important.

4. What happens when it is wrong? Not whether — when. Is there a confidence score? Does it flag uncertainty, or state a wrong answer with the same tone as a right one? Systems that fail loudly are far safer than systems that fail smoothly.

5. Can we export everything and leave? In what format, over what period, at what cost. Ask before signing, because the answer after signing is whatever they decide it is.

6. Who is accountable for output that harms a customer? Watch the room. This is the question that ends the demo energy, and the reaction tells you more than the answer.

The contract terms that matter

• Data ownership and reuse — yours stays yours, stated explicitly
• Training rights — whether your data may be used to improve their product, for anyone else's benefit
• Performance commitments — with a remedy attached, not just an aspiration
• Human review — where it is required, and who is responsible
• Public records handling — critical for any public entity, routinely missing
• Security and breach notification — timelines in hours, not "promptly"
• Exit and data return — format, window, and cost, in writing

The trap in renewals

The most common way organisations acquire an AI tool is not by buying one. It is by renewing a contract for something they already use, where AI features were added quietly and the terms changed with them. Nobody evaluates it, because nobody experiences it as a purchase.

Read what changed. That is where the training-data clause usually appears.

Write the decision down

Whatever you decide, record why: what you asked, what they answered, what you concluded. If anyone questions the purchase later — an auditor, a board, a council, a client — a one-page record turns an interrogation into a conversation.

If you want a scorecard and a full contract-provision checklist you can hand to a colleague, The AI Vendor Evaluation Field Guide is built for exactly that, and AI Policy Templates covers the internal policy the purchase has to fit inside.