Start by naming the problem you are buying for
The fastest way to waste a vendor evaluation is to compare tools that solve different problems. AI governance vendors split into three jobs: documenting AI systems, folding AI into enterprise risk, and enforcing policy on live requests. A registry and a runtime enforcement layer can both call themselves AI governance and barely overlap. Before you score anyone, write down whether your gap is organisational (no inventory, no owners, no paper trail) or operational (data leaving for external models, agents acting unsupervised). The answer tells you which category to evaluate and saves you from a tool that reports beautifully on a risk it cannot touch.
Test on live traffic, not slideware
Feature lists describe intent. Behaviour is what you are buying. Insist on a trial against real or realistic traffic and run three checks. Send a request that policy says to block, and confirm the vendor stops it before the model runs rather than logging it afterward. Paste data that must be redacted, and verify it never reaches the provider. Have an agent attempt an out-of-scope action, and watch whether it is prevented or merely recorded. A vendor comfortable with this test is selling enforcement; one that steers you back to dashboards is selling documentation.
Demand evidence mapped to obligations
Compliance value comes from evidence an auditor accepts. Ask the vendor to produce the audit record for one specific request and show how it maps to a named obligation under the EU AI Act or ISO 42001. Check whether that record is generated automatically as the request runs or assembled later from logs, because reconstructed evidence has gaps exactly where enforcement was absent. The standard to hold is simple: can you trace what the system did, which policy applied, and what was redacted, for any request, without a manual project.
Score the total cost, including your own labour
Licence price is the visible cost. The hidden cost is the work the tool leaves to you. A documentation-led vendor still expects your engineers to enforce policy and assemble evidence, and that recurring labour belongs in the comparison. A runtime layer folds enforcement and evidence into the product. When you build the scorecard, weight ongoing operational burden alongside the annual fee, and normalise on what each vendor actually does per dollar rather than on the headline number.
Check fit for agents and multiple teams
Two questions separate durable choices from ones you outgrow. First, can the platform govern agentic AI, where actions happen at runtime and identity must be scoped per agent and verified per call. Second, can one layer hold different policies for different departments without a separate deployment per team. Vendors built only for static models and single-team use will strain as your AI footprint grows. Evaluate against where your usage is heading, not only where it sits today.
Frequently asked questions
What is the single most useful test of an AI governance vendor?
Send a request that should be blocked on live traffic and see whether the vendor stops it before the model runs or only records that it happened. That divides enforcement from documentation.
How do I compare governance vendor costs fairly?
Include the labour each approach leaves to your team after go-live, not just the licence fee, and normalise on what each vendor actually enforces and evidences per dollar.