Kong volcano sdk

What Kong AI Gateway Does Automatically

• Semantic Caching: If a user asks the exact same question tomorrow, Kong serves the cached response instantly without incurring costs or hitting the LLM again.
• Prompt Guardrails: Kong blocks malicious prompt injections or sensitive company data from leaking out to the LLM providers.
• Failover: If OpenAI drops offline mid-execution, Kong can automatically route GPT requests to Azure or Anthropic without breaking your Volcano code.

Connecting Volcano to Kong AI Gateway

Volcano SDK is designed to work seamlessly with the Kong AI Gateway (via Kong Advanced Gateway Services). By routing your agent's LLM calls through Kong, you get instant access to enterprise-grade security, request auditing, rate-limiting, and semantic caching.

Step 1: Configure the Volcano Client

Instead of hitting the LLM provider (like OpenAI or Anthropic) directly, you point the Volcano client to your Kong AI Gateway endpoint. Kong will manage your API keys, load-balancing, and security controls behind the scenes.

import { Volcano, ModelProvider } from '@kong/volcano-sdk'; // Initialize Volcano to route through your Kong AI Gatewayconst volcano = new Volcano({

 baseUrl: "https://your-kong-gateway-domain.com", 
 apiKey: process.env.KONG_AI_GATEWAY_TOKEN, // Your Kong access token

});

---

Step 2: Define an MCP Tool

The Model Context Protocol (MCP) lets you expose external tools and data to your agent. In this example, we define an MCP tool that connects to a internal company database to fetch shipping updates.

import { McpServer } from '@modelcontextprotocol/sdk/server'; const mcpServer = new McpServer({

 name: "shipping-tracker",
 version: "1.0.0"

}); // Register a tool on the MCP server mcpServer.tool("get_shipping_status", { orderId: z.string() }, async ({ orderId }) => {

 // Logic to fetch real-world data
 const status = await fetchInternalDb(orderId); 
 return { content: [{ type: "text", text: Order status: ${status} }] };

});

---

Step 3: Run the Multi-Model Workflow

Volcano allows you to chain multiple models together. In this workflow, Claude 3.5 Sonnet acts as the high-reasoning "brain" to understand the customer's request and call the MCP tool. Then, GPT-4o mini handles the cheaper, faster task of drafting a polite email response.

async function handleCustomerInquiry(userPrompt: string) {

// 1. High-reasoning model analyzes prompt and triggers the MCP tool

 const agentRunner = await volcano.agents.create({
   model: "kong-managed-claude-sonnet", // Configured routing inside Kong
   tools: [mcpServer.getTool("get_shipping_status")],
   prompt: userPrompt
 });

const toolOutput = await agentRunner.execute();

// 2. Chained workflow shifts to a faster, cheaper model for drafting

 const finalResponse = await volcano.chat.completion({
   model: "kong-managed-gpt4o-mini", // Routed and cached by Kong
   messages: [
     { role: "system", content: "You are a polite customer support agent." },
     { role: "assistant", content: Internal tool data: ${toolOutput} },
     { role: "user", content: "Draft an email update to the customer based on this." }
   ]
 });

console.log(finalResponse.text); } // Example usage handleCustomerInquiry("Where is my package for order #98765?");

---

What Kong AI Gateway Does Automatically

• Semantic Caching: If a user asks the exact same question tomorrow, Kong serves the cached response instantly without incurring costs or hitting the LLM again.
• Prompt Guardrails: Kong blocks malicious prompt injections or sensitive company data from leaking out to the LLM providers.
• Failover: If OpenAI drops offline mid-execution, Kong can automatically route GPT requests to Azure or Anthropic without breaking your Volcano code.