Building LLM Applications in Production: RAG Architecture, Vector Math & Latency Optimization

It was a Monday morning at 10:15 AM in an HSR Layout tech office. The air conditioning was humming quietly, but the engineering Slack channel was on fire. The company had launched its flagship "AI Copilot" feature over the weekend — a natural-language search bar meant to let enterprise customers query their internal PDF contracts and invoices.

The product manager walked up to my desk with a screen recording. "Renjith, look at this. The user types a simple query: 'What is our penalty clause for late delivery in the Flipkart contract?'. The spinner turns. And turns. And turns."

He paused the video at the final response card. Time to First Token (TTFT): 14.2 seconds. Total roundtrip: 22.8 seconds.

To make matters worse, the CFO had just posted a screenshot of the cloud spending dashboard. The OpenAI API bill for the 48-hour launch weekend was $4,280 (₹3.57 lakhs) — on fewer than 6,000 queries. The junior developer who built it had stuffed the entire 80-page PDF directly into the LLM system prompt on every single user request.

This is the dirty secret of building AI features in 2026: **calling an LLM API in a POST handler takes 10 minutes to code, but engineering a production AI system that responds in <800ms for ₹0.10 per query requires serious backend architecture.**

In this post, we're taking off the marketing goggles and building a real production-grade Retrieval-Augmented Generation (RAG) pipeline from first principles.

Why did that startup's feature cost $4,200 and take 14 seconds? Because it was a **Naive API Wrapper**. Here is the architectural difference between how beginner devs integrate AI vs how enterprise systems engineers build it:

🔧 Engineering Insight

LLM inference time scales with the number of generated output tokens, while input processing latency scales with prompt length. Dumping 100,000 tokens into prompt context forces the transformer model's Attention mechanism to calculate a matrix multiplication of size N² over millions of tokens. Filtering prompt input down to 2,000 highly relevant tokens drops prefill compute time instantly.

At the core of modern semantic retrieval is the **Vector Embedding** — converting text chunks into dense mathematical vectors in high-dimensional floating-point space (typically 768 to 1,536 dimensions).

When a user asks a question, your search engine calculates how close the query vector is to millions of chunk vectors stored in your database.

The Mathematical Distance Functions

There are three primary ways vector databases compute similarity:

Vector Database Indexing Trade-offs

Brute-force vector comparison is O(N × D) — checking a query vector against 1,000,000 vectors takes 450ms. That is too slow for production. We use **Approximate Nearest Neighbor (ANN)** indexing algorithms like HNSW (Hierarchical Navigable Small World):

Let's look at the math of **Prompt Caching** — one of the most powerful latency and cost levers introduced in modern LLM API infrastructure.

When an API receives a prompt, it must process the system instructions, tools schema, and retrieved context chunks before generating the very first word. If the prefix of your prompt (system prompt + static context) is identical across requests, the provider can cache the pre-computed Key-Value (KV) attention states in GPU memory.

The Economics of Prompt Caching

💰 Pro Tip for AI Engineers

Structure your prompts with **static content first** and **dynamic content last**. Put your system instructions, tool definitions, and long knowledge docs at the absolute top of the request payload. Put the user's latest message at the very bottom. If you flip this order, you bust the cache on every request!

Let's look at two real approaches to building a legal document search assistant serving 100,000 user queries per month.

Vikram: The Naive LLM Wrapper

• Sends 40,000 raw text tokens per prompt directly to GPT-4o on every user query
• No vector database; does string matching in Node.js memory
• No response streaming; waits for full JSON payload to return before rendering
• Monthly Cost: $4,500 | Average Latency: 12.4 seconds | User Feedback: "Slow & unusable"

Swati: The Production RAG Systems Engineer

• Indexes documents in pgvector using HNSW index with L2-normalized embeddings
• Uses Hybrid Search (BM25 + Dense Vectors) with Cohere Re-ranker to pick top 3 relevant chunks (1,500 tokens total context)
• Applies Semantic Caching in Redis — 38% of queries answered instantly from cache without calling LLM
• Streams responses via Server-Sent Events (SSE) over HTTP/2
• Monthly Cost: $320 | Average TTFT: 380 milliseconds | User Feedback: "Instant & accurate"