What to Ask an AI Vendor After the Demo

The demonstration ends with a convincing answer on screen. The buying decision begins with everything the demonstration did not show: where the input travelled, whose credentials were used, which model produced the answer, and what the customer could recover if the service changed tomorrow.

Where does our information go?

Ask the vendor to trace one representative transaction. Follow the prompt, retrieved records, response, diagnostic logs, backups, and support access. Identify the model providers and subprocessors along the route. Data residency, retention, and training use should be explicit for each relevant service and configuration.

A setting in an administration console establishes a configuration option. A contract establishes a commitment. Ask which protections are enabled by default, which require a paid tier, and which the supplier is prepared to warrant. Keep those answers attached to the version and service being purchased.

What can the product actually change?

For an agentic product, inspect a tool call. A friendly interface may sit over a service account with broad permissions. Ask how the product identifies the acting user, limits delegated authority, handles denied actions, and prevents a retry from widening the operation. Request an example log from the configuration you expect to deploy.

What will we receive when something goes wrong?

Agree on the records available during an investigation: timestamps, request identifiers, model and software versions, tool events, and relevant access history. Establish how long those records remain available and how the customer requests them. A general promise to cooperate can be difficult to use during a time-sensitive incident.

Who decides when the product changes?

Suppliers need to improve their services. Buyers need to know when a change affects the approved use. Discuss model substitutions, new subprocessors, changed data use, expanded tool permissions, and altered service locations. Decide which changes require notice, testing, or agreement before they reach the customer’s workflow.

NIST’s AI Risk Management Framework provides a broader basis for examining risk throughout the system lifecycle. The questions here are procurement recommendations informed by that approach; the exact contractual rights need to fit the transaction and the buyer’s obligations.

Can we test our own difficult cases?

Bring a small acceptance set that includes ambiguous requests, restricted information, unavailable dependencies, and actions the system should refuse. Write down the expected behaviour before running it. Evaluate the path the product takes as well as its final answer, and preserve unresolved failures in the purchase decision.

Can we leave with something usable?

Before signing, examine the export format, migration assistance, access to relevant logs, and deletion arrangements. Run a sample export if the product allows it. The exit terms are part of the architecture decision: they determine how much of the operating record, configuration, and business continuity remains under the buyer’s control.