The three parts that decide whether an AI feature is trusted are invisible, and they have to be planned before the feature ships: knowing it is working, deciding what it can reach, and having planned its security.
The visible part is what gets built. An AI feature goes live, answers customer questions or drafts the tickets or fills in the invoices, and for a few weeks it is the most talked-about thing in the company. Then a customer mentions, in passing, that a total was wrong last month. Nobody can say when it started, who was affected, or whether it is still happening.
You need to know it is working
Software built with AI tools tends to be right on the cases somebody thought of and quietly wrong on the ones nobody did. Nothing crashes. The number is just off. Testing before launch catches the first kind of mistake; only monitoring catches the second.
Monitoring here does not mean a dashboard of green lights. It means checks that run against real data all the time, stating things that must always be true:
- an invoice total equals the sum of its lines
- a draft cannot become paid without being approved
- a record never points at another customer’s data
- the count in our system matches the count in the accounting system
When one of those fails, someone is told, the same day. And every error the system produces carries the version of the software that produced it, so “when did this start” has an answer that takes a minute, not an afternoon.
Without this, the company does not know whether its AI feature is working. It knows whether anyone has complained.
You need to decide what it can reach
The second invisible part is guardrails around data. These are decisions, not technology, and if the company has not made them its vendors and staff have made them by default.
Which data may go to which AI model, and under what agreement. What must never leave the building. Which records a given screen, user or automation is allowed to see, and where that rule is enforced so that the next feature cannot forget it. What gets logged, and for how long.
The best place for the “who can see what” rule is as close to the data as possible, so every screen, report and automation inherits it rather than re-implementing it. A company does not need to know how that is done. It needs to ask whether it was, and to hear a clear answer.
You need security to have been planned
Software built with AI is built fast, which is the point, and reviewed rarely, which is the problem. The things that go wrong are structural: who can see what, what the system will fetch on a user’s behalf, what happens when a setting is missing. They are cheap to design in and expensive to bolt on.
So security has to be two things. A design conversation before the feature is built, covering those structural questions. And a scheduled pass afterwards that runs whether or not anyone remembers: scans on every change, a regular check of the database’s own configuration, dependencies matched against known vulnerabilities each night, findings turned into fixes that a person reviews. Annual human testing for anything that handles money or personal data.
The question to ask
Whether you are building or buying, ask one question of whoever is delivering the feature: how will we know it is working, what data can it reach, and when does it get reviewed. If the answer is a demo of the feature, you are being shown the visible part.