The category is being defined by people who already owned a proxy
Look at who ranks for "ai gateway" and the shape of the AI gateway market becomes obvious. The first page is Cloudflare, Kong, Vercel, Microsoft's API Management documentation, the Envoy AI Gateway project, MLflow, and IBM's explainer. Search demand sits around 1,900 queries a month for the term and its plural, and almost none of that page is occupied by anyone whose starting point was governance. That isn't a conspiracy, it's gravity. The vendors who already ran your traffic added an AI mode and named it after the thing they already sold you. It does mean the definition of the category you're buying into was written by infrastructure companies, and infrastructure companies answer routing questions. The question a CIO is actually being asked in the audit is a different one.
Camp one: the proxy that grew an AI mode
Cloudflare's AI Gateway, Kong's AI Gateway, and Microsoft's GenAI gateway capabilities in Azure API Management are all the same move: an existing edge or API layer extended to sit in front of model providers. What you get is real and worth having. Caching, rate limiting, retries, failover between providers, and one endpoint instead of six SDKs. If your problem is that four teams each hold a different OpenAI key and your bill arrives as one number, this camp fixes that in a fortnight. The blind spot is what these products optimise for. They answer "how do I route model calls reliably". They do not, on their own, answer "can I show what the AI did, for which use case, on whose behalf, in March". Traffic control and evidence are not the same product, and the pitch decks blur them.
Camp two: the developer routing layer
Vercel's AI Gateway, Portkey's open-source gateway, and MLflow's AI Gateway serve the person building the application. Model switching, per-key spend, fast local adoption, no procurement cycle. A team can be through the door in an afternoon, which is exactly why they are. That speed is the trap. These get adopted per team, so eighteen months in you own four gateways, three of them on somebody's personal account, and you're back where you started: no single record, no single policy, and now four vendors. The tool did its job. It just wasn't ever the enterprise's tool.
Camp three: the control plane nobody markets well
The third camp is defined by what the other two don't do: attribute cost to a use case instead of to an API key, retain every prompt and response as evidence, apply policy the same way regardless of which team wrote the app, and fail closed when something falls outside scope. This camp has fewer logos, harder demos, and a much worse thirty second pitch, because "you can prove what happened" doesn't screenshot well. It also happens to be the thing the person signing the cheque needs, which is why the market keeps mispricing it.
The buying mistake
Here's the pattern, and it repeats: an engineering team picks a routing layer to solve what is actually a governance problem. Six months later finance asks which use case burned $40,000 last quarter and the honest answer is "key number three did". Risk asks for the prompts behind a decision a customer is complaining about, and the logs have a 30 day retention window that expired. The gateway was never the problem. The decision about what the gateway was for got made by whoever was in the room, and the person who needed evidence wasn't in it. Pick the control point before you pick the vendor. A gateway is a delivery and accountability decision that happens to be implemented in networking, and treating it as a networking purchase is how it ends up owned by the team least able to answer for it.
The questions that actually separate these products
Run these at the demo and watch which camp the vendor is in. Can you attribute spend to a use case rather than to an API key or a team? If the answer needs a spreadsheet, that's a no. Can you produce every prompt and response for a named user for a named week, twelve months from now? Ask what the retention default is, then ask what it costs to extend it. What happens when the provider silently ships a new model version? Does anything in the stack notice and tell you, or does your eval score drop and nobody finds out until a customer does? When a call breaks policy, does it fail closed or log and continue? Most default to log and continue, which is a decision your risk committee should be making, not your vendor. Who owns it: the platform team, or the app team? A gateway owned by the app team is a routing library with ambitions.
Where the regulation lands on this
Record keeping is the part of the EU AI Act that touches this purchase directly: systems that inform consequential decisions are expected to keep logs that make what happened reconstructable, and the penalty ceiling for the serious breaches runs to EUR 35 million or 7% of global turnover. The Commission's Digital Omnibus proposal would defer the Annex III high-risk obligations to December 2027, so the calendar has more room in it than most 2025 slide decks assumed. That extra time changes nothing about the design. The evidence you need is the same evidence a CFO wants when she asks what the AI spend returned, and the same evidence an engineer wants at 2am when an agent did something surprising. Regulation is not the reason to want a single intercept point for every model call. It's just the deadline that eventually makes you build one.
What we would do
Decide what you need to prove before you decide what you need to route. Write the four questions above on one page, take them to the shortlist, and let the answers sort the camps for you. One intercept point for every model call is the cheapest control you'll ever build, and it gets more expensive every month you run without it, because a control layer bolted on after the pilot isn't a control layer. It's a rewrite.
Sources
Cloudflare AI Gateway product documentation (cloudflare.com/products/ai-gateway). Kong, "What is an AI Gateway" (konghq.com/blog/enterprise/what-is-an-ai-gateway). Vercel AI Gateway (vercel.com/ai-gateway). Portkey AI Gateway, open source (github.com/portkey-ai/gateway). Envoy AI Gateway (aigateway.envoyproxy.io). Microsoft Learn, GenAI gateway capabilities in Azure API Management (learn.microsoft.com/azure/api-management/genai-gateway-capabilities). MLflow AI Gateway (mlflow.org/ai-gateway). IBM, "AI Gateway" (ibm.com/think/topics/ai-gateway). Regulation (EU) 2024/1689 (the EU AI Act), Articles 12 and 99, and the European Commission's Digital Omnibus proposal of November 2025. Search volume figures from DataForSEO, July 2026.