Decide the problem before you meet the vendor. A services business had three quotes for “an AI system” and no way to compare them; two weeks of looking at how the business actually ran showed the biggest win was automating quote follow-ups, which none of the three had proposed and which cost a fraction of the cheapest quote.
Here are the questions for when you do meet them.
1. Which of our workflows does this change, and by how much? A good answer names a specific hand-off in your business and an honest estimate of time or money. A weak answer is a capability list. If the vendor cannot say what changes for the person doing the work, nobody has looked.
2. What data does it need, where does it go, and under what agreement? You are listening for named systems and named models, whether your data is used to train anything, where it is stored, and how long it is kept. “It is all secure” is not an answer. A written data-processing agreement is.
3. What can it see, and who decided? If the feature touches records belonging to different customers, staff or departments, ask where the rule about who can see what is enforced. The good answer is “as close to the data as possible, so every screen inherits it”. The bad answer is “each screen checks”.
4. How will we know it is working? Not “you will have a dashboard”. You want to hear about checks that run on real data and state things that must always be true, and about being told the same day when one fails. You also want to hear that every error carries the version that produced it, so problems can be traced to a change.
5. What happens when it is wrong? AI features are wrong sometimes. Ask where a person is in the loop, whether a wrong answer can be caught before it reaches a customer, and whether the vendor keeps humans on anything that changes data or money. A feature with no human step on a money-moving action is a feature you will read about later.
6. When is it reviewed for security, and by whom? The answer should include a scheduled, automated pass that runs on every change, and independent human testing at least annually for anything handling money or personal data. “We take security seriously” is a slogan. A cadence is an answer.
7. What do we own when this ends? The data, in a usable export. The configuration. Whatever was built for you specifically. And a clear statement of what you cannot take with you. Ask before signing, because the answer changes the price of leaving.
If they can answer all seven
You are probably talking to a good vendor, and you are certainly a better-prepared buyer than most. Write the answers into the AI plan, under the feature they concern, so that the next person to ask does not have to start from the demo.