Many organizations are rushing to adopt Generative AI. New copilots, chat interfaces and LLM-powered applications are being launched across departments. But while most discussions focus on models, prompts and use cases, a different challenge often emerges behind the scenes: the infrastructure supporting these AI services was never designed for them. One global bank learned this the hard way.
What started as a successful LLM pilot quickly exposed limitations in its API management platform, forcing the team to rethink how AI traffic should be governed, secured and monitored.
Within the bank's API First department, the mission was clear: provide a standardized API platform that allows teams across the organization to build and consume services while adhering to strict governance standards.
For years, Apigee OPDK successfully fulfilled that role. Then the organization started rolling out Large Language Model services. Adoption accelerated rapidly, generating a surge in Server-Sent Events (SSE) traffic. Unlike traditional API calls, LLM interactions often involve long-lived streaming connections where responses are generated token by token in real time.
The existing platform was not built for that traffic pattern.
The bottleneck: traditional message processors attempting to buffer streaming LLM responses, causing performance degradation and connection failures.
The challenge was no longer limited to the GenAI service itself. The entire API ecosystem was at risk.
Fixing the streaming issue alone would not be enough. Internal teams wanted to keep all the governance capabilities they were already accustomed to.
In other words, they wanted the flexibility of modern AI services without sacrificing the governance standards of an enterprise API platform.
The team decided to introduce Envoy Proxy into the architecture. Unlike traditional API gateways, Envoy handles streaming traffic natively and does not rely on buffering long-running responses.
The impact was immediate. Performance stabilized and streaming reliability improved significantly.
But another challenge quickly appeared.
The governance gap: request-based controls remained intact, but token usage inside the streaming response was invisible.
The standard adapter focuses primarily on the request path. The information the bank actually needed to govern — token consumption — only became available inside the response stream.
That created a blind spot.
To bridge the gap, the team extended the Apigee Remote Service.
By leveraging Envoy's external processing capabilities, they created a mechanism that could inspect streaming responses, extract token usage information and synchronize that data with quota management policies.
The solution: using Envoy's external processing capabilities to intercept streaming responses and synchronize token usage with governance policies.
The result was a platform that could support modern AI workloads without compromising security, governance or operational stability.
This story highlights a challenge many organizations will face over the next few years. AI workloads behave differently from traditional applications. Streaming responses, token-based billing models and agentic interactions introduce new requirements that existing API architectures were never designed to handle.
The answers often determine whether an AI initiative can successfully move beyond the pilot phase.
Generative AI is forcing organizations to rethink API architecture. Traditional governance models remain important, but they need to evolve alongside new traffic patterns, new consumption models and new operational requirements.
The organizations that succeed will be the ones that combine innovation with control. Not by abandoning governance, but by adapting it to the realities of AI.
As GenAI adoption accelerates, many organizations discover that their existing API platforms were never designed for streaming workloads, token-based consumption models and AI-specific governance requirements.
AppyThings helps enterprises modernize their API architecture, evaluate AI readiness and build integration platforms capable of supporting the next generation of AI-powered services.
Want to understand whether your API platform is ready for GenAI? Talk to our experts.