Busk at 16:28, 17 August 2026

2026-08-17T16:28:36Z

Busk: Created page with "What Kong AI Gateway Does AutomaticallySemantic Caching: If a user asks the exact same question tomorrow, Kong serves the cached response instantly without incurring costs or..."

2026-08-17T16:25:01Z

Created page with "What Kong AI Gateway Does AutomaticallySemantic Caching: If a user asks the exact same question tomorrow, Kong serves the cached response instantly without incurring costs or..."

New page

What Kong AI Gateway Does AutomaticallySemantic Caching: If a user asks the exact same question tomorrow, Kong serves the cached response instantly without incurring costs or hitting the LLM again.Prompt Guardrails: Kong blocks malicious prompt injections or sensitive company data from leaking out to the LLM providers.Failover: If OpenAI drops offline mid-execution, Kong can automatically route GPT requests to Azure or Anthropic without breaking your Volcano code.Would you like to explore how to set up the semantic caching plugin inside Kong, or do you need help writing a custom MCP server for your specific database?

## Connecting Volcano to Kong AI Gateway
Volcano SDK is designed to work seamlessly with the Kong AI Gateway (via Kong Advanced Gateway Services). By routing your agent's LLM calls through Kong, you get instant access to enterprise-grade security, request auditing, rate-limiting, and semantic caching.
Here is a conceptual example of how to connect the two and build an MCP-native workflow.
------------------------------
## Step 1: Configure the Volcano Client
Instead of hitting the LLM provider (like OpenAI or Anthropic) directly, you point the Volcano client to your Kong AI Gateway endpoint. Kong will manage your API keys, load-balancing, and security controls behind the scenes.

import { Volcano, ModelProvider } from '@kong/volcano-sdk';
// Initialize Volcano to route through your Kong AI Gatewayconst volcano = new Volcano({
baseUrl: "https://your-kong-gateway-domain.com",
apiKey: process.env.KONG_AI_GATEWAY_TOKEN, // Your Kong access token
});

------------------------------
## Step 2: Define an MCP Tool
The Model Context Protocol (MCP) lets you expose external tools and data to your agent. In this example, we define an MCP tool that connects to a internal company database to fetch shipping updates.

import { McpServer } from '@modelcontextprotocol/sdk/server';
const mcpServer = new McpServer({
name: "shipping-tracker",
version: "1.0.0"
});
// Register a tool on the MCP server
mcpServer.tool("get_shipping_status", { orderId: z.string() }, async ({ orderId }) => {
// Logic to fetch real-world data
const status = await fetchInternalDb(orderId);
return { content: [{ type: "text", text: `Order status: ${status}` }] };
});

------------------------------
## Step 3: Run the Multi-Model Workflow
Volcano allows you to chain multiple models together. In this workflow, Claude 3.5 Sonnet acts as the high-reasoning "brain" to understand the customer's request and call the MCP tool. Then, GPT-4o mini handles the cheaper, faster task of drafting a polite email response.

async function handleCustomerInquiry(userPrompt: string) {

// 1. High-reasoning model analyzes prompt and triggers the MCP tool
const agentRunner = await volcano.agents.create({
model: "kong-managed-claude-sonnet", // Configured routing inside Kong
tools: [mcpServer.getTool("get_shipping_status")],
prompt: userPrompt
});

const toolOutput = await agentRunner.execute();

// 2. Chained workflow shifts to a faster, cheaper model for drafting
const finalResponse = await volcano.chat.completion({
model: "kong-managed-gpt4o-mini", // Routed and cached by Kong
messages: [
{ role: "system", content: "You are a polite customer support agent." },
{ role: "assistant", content: `Internal tool data: ${toolOutput}` },
{ role: "user", content: "Draft an email update to the customer based on this." }
]
});

console.log(finalResponse.text);
}
// Example usage
handleCustomerInquiry("Where is my package for order #98765?");

------------------------------
## What Kong AI Gateway Does Automatically

* Semantic Caching: If a user asks the exact same question tomorrow, Kong serves the cached response instantly without incurring costs or hitting the LLM again.
* Prompt Guardrails: Kong blocks malicious prompt injections or sensitive company data from leaking out to the LLM providers.
* Failover: If OpenAI drops offline mid-execution, Kong can automatically route GPT requests to Azure or Anthropic without breaking your Volcano code.

Would you like to explore how to set up the semantic caching plugin inside Kong, or do you need help writing a custom MCP server for your specific database?

@@ Line 1: / Line 1: @@
-What Kong AI Gateway Does AutomaticallySemantic Caching: If a user asks the exact same question tomorrow, Kong serves the cached response instantly without incurring costs or hitting the LLM again.Prompt Guardrails: Kong blocks malicious prompt injections or sensitive company data from leaking out to the LLM providers.Failover: If OpenAI drops offline mid-execution, Kong can automatically route GPT requests to Azure or Anthropic without breaking your Volcano code.Would you like to explore how to set up the semantic caching plugin inside Kong, or do you need help writing a custom MCP server for your specific database?
+## What Kong AI Gateway Does Automatically
 ## Connecting Volcano to Kong AI Gateway
@@ Line 67: / Line 70: @@
 * Prompt Guardrails: Kong blocks malicious prompt injections or sensitive company data from leaking out to the LLM providers.
 * Failover: If OpenAI drops offline mid-execution, Kong can automatically route GPT requests to Azure or Anthropic without breaking your Volcano code.

Kong volcano sdk - Revision history

Busk at 16:28, 17 August 2026

Busk: Created page with "What Kong AI Gateway Does AutomaticallySemantic Caching: If a user asks the exact same question tomorrow, Kong serves the cached response instantly without incurring costs or..."