LLM Context Window Optimization: Cut Costs Without Sacrificing Quality

Why Context Windows Are Your Biggest Cost Lever

When engineers think about LLM cost optimization, they typically focus on model selection — switching from GPT-4o to GPT-4o Mini, or from Claude 3.5 Sonnet to Haiku. But our analysis at StackPulse shows that context window management typically delivers 40-70% cost reduction on inference spend, compared to 15-30% from model downgrades.

The math is straightforward: every token in a context window costs money. A 128K-token context window sounds generous until you are paying for 128K tokens on every single request. Many production LLM applications use fewer than 2,000 tokens of actual meaningful context — but pay for 128K because the tooling does not give you fine-grained control.

This guide covers the five layers of context window optimization: truncation strategies, semantic compression, dynamic context sizing, caching at the context level, and architectural patterns that reduce context requirements.

The Token Economics of Context Windows

Before diving into techniques, let us establish the cost baseline. Here is what different context window sizes actually cost on leading providers as of Q2 2026:

Provider	Model	1K input	128K context	Max context
OpenAI	GPT-4o	$2.50 per 1M input tokens	$3.20 for full 128K context	128K
OpenAI	GPT-4o Mini	$0.15	$0.19	128K
Anthropic	Claude 3.5 Sonnet	$3.00	$15.00	200K
Anthropic	Claude 3.5 Haiku	$0.80	$3.50	200K
Google	Gemini 1.5 Pro	$1.25	$1.25	1M

The per-token cost looks small, but at scale it compounds. A RAG pipeline processing 100,000 queries/day with 60,000 tokens average context pays:

• 100,000 × 60,000 × $0.000003 = $18,000/day in input tokens alone

Cut that context from 60K to 8K effective tokens (more on how below), and you pay:

• 100,000 × 8,000 × $0.000003 = $2,400/day

That is a $15,600/day savings — over $5.6M/year.

Layer 1: Intelligent Context Truncation

The bluntest tool is truncating context to a fixed maximum. Do not do naive truncation — cutting from the end — but do semantic truncation, keeping the beginning and end of documents where key information concentrates.

Naive vs. Semantic Truncation

Naive (bad): Cut from the middle of the document — the most semantically dense part gets destroyed.

Semantic (good): Score documents by relevance to the query, then pack the context window with the most relevant content, from the start of the document where it matters most.

Implement semantic truncation with a sliding window + relevance scoring approach:

import tiktoken
from openai import OpenAI

client = OpenAI()
MAX_TOKENS = 8000  # your context budget per request

def semantic_truncate(documents: list[dict], query: str) -> list[dict]:
    """
    Truncate documents to fit within token budget while preserving
    the most semantically relevant content for the given query.
    """
    enc = tiktoken.get_encoding("cl100k_base")  # GPT-4 tokenizer

    # Score each document by semantic relevance to the query
    scored = []
    for doc in documents:
        tokens = enc.encode(doc["content"])
        # Simple proxy: check keyword overlap (production would use embeddings)
        relevance = sum(1 for kw in query.lower().split()
                        if kw in doc["content"].lower())
        scored.append((relevance, len(tokens), doc))

    # Sort by relevance descending
    scored.sort(key=lambda x: x[0], reverse=True)

    # Greedily select documents until we hit the budget
    selected = []
    total_tokens = 0

    for relevance, doc_tokens, doc in scored:
        if total_tokens + doc_tokens <= MAX_TOKENS:
            selected.append(doc)
            total_tokens += doc_tokens
        elif not selected:
            # If nothing fits, take a slice of the most relevant doc
            new_doc = dict(doc)
            new_doc["content"] = enc.decode(enc.encode(doc["content"])[:MAX_TOKENS])
            selected.append(new_doc)
            break
        break

    return selected

Production-Grade Truncation with Embeddings

For production systems, use semantic similarity instead of keyword overlap:

from openai import OpenAI
import numpy as np

client = OpenAI()

def semantic_truncate_embeddings(
    documents: list[dict],
    query: str,
    max_tokens: int = 8000
) -> list[dict]:
    """
    Use embeddings to select the most relevant chunks for the query.
    This preserves semantic meaning rather than just keyword matching.
    """
    # Get embedding for the user query
    query_emb = client.embeddings.create(
        model="text-embedding-3-small",
        input=query
    ).data[0].embedding

    # Score each document by cosine similarity to query
    scored = []
    for doc in documents:
        doc_emb = doc.get("embedding")
        if not doc_emb:
            doc_emb = client.embeddings.create(
                model="text-embedding-3-small",
                input=doc["content"]
            ).data[0].embedding

        similarity = np.dot(query_emb, doc_emb)
        tokens = len(tiktoken.get_encoding("cl100k_base")
                      .encode(doc["content"]))
        scored.append((similarity, tokens, doc))

    # Sort by similarity and greedily pack the context window
    scored.sort(key=lambda x: x[0], reverse=True)

    selected = []
    total_tokens = 0

    for sim, doc_tokens, doc in scored:
        if total_tokens + doc_tokens <= max_tokens:
            selected.append(doc)
            total_tokens += doc_tokens

    return selected

Layer 2: Semantic Compression

Rather than truncating documents, compress them while preserving meaning. Two approaches work well:

A. LLM-Based Summarization

Summarize each retrieved document before adding it to context using a cheap model:

def compress_with_summary(documents: list[dict], query: str) -> str:
    """
    Use a cheap model to summarize each document
    in the context of the specific query.
    """
    summaries = []
    for doc in documents:
        prompt = f"""Given this user query: "{query}"

Summarize the following document in 3-5 sentences, preserving any facts,
statistics, or specific claims relevant to the query.

Document: {doc['content']}

Summary:"""
        response = client.chat.completions.create(
            model="gpt-4o-mini",  # cheap model for compression
            messages=[{"role": "user", "content": prompt}],
            max_tokens=200,
            temperature=0
        )
        summaries.append(f"[Source: {doc.get('source', 'unknown')}] " +
                         response.choices[0].message.content)

    return "\n\n".join(summaries)

Cost analysis: compressing 100 documents via gpt-4o-mini at ~$0.15/1M tokens adds ~$0.003 per query. On a 100K query/day system paying $18K/day in context costs, that is a 0.02% overhead for a 60% context reduction.

B. RAG-Specific Compression with Contextual Clustering

A more systematic approach is to cluster retrieved chunks and compress each cluster to a representative summary:

1. Retrieve top-N chunks (e.g., top 20)
2. Cluster by semantic similarity (embeddings)
3. Compress each cluster to N sentences
4. Return the compressed cluster representatives

This preserves diversity — you do not lose the signal from a minor but relevant point just because it appeared in a cluster with a dominant topic.

Layer 3: Dynamic Context Sizing

Not every request needs the same context size. Use a routing layer to size context dynamically:

def estimate_optimal_context_size(query: str, user_tier: str) -> int:
    """
    Estimate the optimal context window size based on query complexity.
    Simpler queries need less context; complex multi-hop questions need more.
    """
    # Simple heuristic — production would use a classifier model
    complexity_indicators = [
        "compare", "analyze", "difference between",
        "why did", "how does", "explain",
        "historical", "across multiple", "trend"
    ]

    score = sum(1 for indicator in complexity_indicators
                if indicator in query.lower())

    # Map complexity score to token budget
    if user_tier == "free":
        base = 4000
    elif user_tier == "pro":
        base = 32000
    else:
        base = 128000

    return min(base, 2000 + (score * 3000))  # 2000-14000 range

A production implementation would train a lightweight classifier on historical queries to predict context requirements. Track the "context saturation rate" — what percentage of the context window is actually used by model outputs — and use this to calibrate your sizing.

Layer 4: Semantic Caching at the Context Level

Traditional API caching caches by exact request match. Semantic caching caches by meaning — if your new query is semantically similar to a cached query, reuse the cached response:

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

class SemanticCache:
    def __init__(self, embedding_model: str = "text-embedding-3-small",
                 similarity_threshold: float = 0.92,
                 max_entries: int = 10000):
        self.client = OpenAI()
        self.embedding_model = embedding_model
        self.similarity_threshold = similarity_threshold
        self.cache = {}  # query_hash -> (embedding, response, tokens_used)
        self.access_order = []

    def _embed(self, text: str) -> np.ndarray:
        result = self.client.embeddings.create(
            model=self.embedding_model,
            input=text
        )
        return np.array(result.data[0].embedding)

    def get(self, query: str) -> tuple | None:
        """Check cache for a semantically similar response."""
        if len(self.cache) == 0:
            return None

        query_emb = self._embed(query)
        best_score = 0
        best_key = None

        for cached_hash, (cached_emb, _, _) in self.cache.items():
            score = cosine_similarity([query_emb], [cached_emb])[0][0]
            if score > best_score:
                best_score = score
                best_key = cached_hash

        if best_score >= self.similarity_threshold:
            self.access_order = [k for k in self.access_order
                                 if k != best_key] + [best_key]
            return self.cache[best_key][1]

        return None

    def set(self, query: str, response: str, tokens_used: int):
        """Add a new entry to the cache."""
        query_emb = self._embed(query)
        query_hash = hash(query)

        if len(self.cache) >= self.max_entries:
            oldest = self.access_order.pop(0)
            del self.cache[oldest]

        self.cache[query_hash] = (query_emb, response, tokens_used)
        self.access_order.append(query_hash)

Cache Hit Rate Benchmarks

Based on production data from several StackPulse readers:

Application Type	Cache Hit Rate	Token Savings
Customer support bots	35-45%	30-40%
Code explanation tools	25-35%	20-30%
Document Q&A	40-55%	35-50%
Multi-turn chat	15-25%	10-20%

Higher hit rates come from: shorter conversation history, deterministic document sets, and queries with common sub-patterns.

Layer 5: Context-Aware Architecture

The most powerful optimization is architectural — design your application to need less context in the first place.

A. Chunking Strategy for RAG

Standard chunking (512 tokens, 50-token overlap) wastes context budget on fragmented sentences and lost cross-chunk relationships. Better approaches:

Hierarchical chunking: Chunk at the document level, then section level, then paragraph level — retrieve at section level first, then fill context with subsection details.

Semantic chunking (recommended) splits by sentences, then merges adjacent sentences until hitting the token limit — preserves semantic coherence:

from langchain.text_splitter import RecursiveCharacterTextSplitter

def semantic_chunk(doc: str, max_tokens: int = 512,
                   similarity_threshold: float = 0.7) -> list[str]:
    """
    Split by sentences, then merge adjacent sentences
    until hitting token limit — preserves semantic coherence.
    """
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=max_tokens,
        chunk_overlap=int(max_tokens * 0.15),  # 15% overlap
        separators=["\n\n", "\n", ". ", " "]
    )
    return splitter.split_text(doc)

B. Query Decomposition

For complex queries that need large contexts, decompose into sub-queries:

Original: "Compare the Q3 and Q4 performance of our ML models, focusing on accuracy and latency, and explain any significant changes"

Decomposed:

1. "Q3 ML model accuracy and latency metrics"
2. "Q4 ML model accuracy and latency metrics"
3. "What changed between Q3 and Q4 in our ML pipeline"

Each sub-query retrieves a smaller context. Synthesize the final answer from three focused responses instead of one sprawling context window.

C. Hybrid Retrieval: Dense + Sparse

Pair semantic (embedding-based) retrieval with keyword/bm25 retrieval. This lets you be more precise about what goes into context — semantic retrieval gets relevance, bm25 retrieval gets exact keyword matches:

from rank_bm25 import BM25Okapi
import numpy as np

class HybridRetriever:
    def __init__(self, documents: list[dict]):
        self.documents = documents
        self.bm25 = BM25Okapi([doc["content"].lower().split()
                               for doc in documents])

    def retrieve(self, query: str, k: int = 10,
                 alpha: float = 0.5) -> list[dict]:
        """
        Hybrid retrieval combining semantic + keyword search.
        alpha=0.5: equal weight; alpha=0.8: prefer semantic
        """
        # Semantic scores (from embedding similarity)
        semantic_scores = self._semantic_scores(query)

        # BM25 scores
        bm25_scores = self.bm25.get_scores(query.lower().split())

        # Normalize and combine
        sem_norm = (semantic_scores - semantic_scores.min()) / \
                   (semantic_scores.max() - semantic_scores.min() + 1e-8)
        bm25_norm = (bm25_scores - bm25_scores.min()) / \
                    (bm25_scores.max() - bm25_scores.min() + 1e-8)

        combined = alpha * sem_norm + (1 - alpha) * bm25_norm

        top_indices = np.argsort(combined)[-k:][::-1]
        return [self.documents[i] for i in top_indices]

Measuring Your Context Efficiency

Track these metrics to quantify your optimization ROI:

Metric	Formula	Target
Context Utilization Rate	avg(output_tokens / input_tokens)	>0.25
Context Waste Rate	1 - utilization	<0.75
Effective Context Cost	cost_per_token × avg_input_tokens	Trending down
Cache Hit Rate	cached_responses / total_responses	>0.30

For Prometheus + Grafana users, instrument your context metrics with custom counters:

# Context utilization (requires custom instrumentation)
llm_context_tokens_in / (llm_context_tokens_in + llm_context_tokens_out)

# Cost per 1K queries
sum(rate(llm_tokens_total[1h])) * 0.003 / sum(rate(llm_requests_total[1h]))

# Cache hit rate (if using semantic cache)
sum(rate(semantic_cache_hits_total[5m])) /
sum(rate(semantic_cache_requests_total[5m]))

Common Pitfalls

Over-compression: Cutting context too aggressively degrades answer quality. A/B test compression ratios against answer accuracy on a golden dataset.

Ignoring output tokens: Input context costs dominate, but output token costs add up on long-form generation. Set max_output_tokens explicitly.

No observability: If you are not measuring context utilization per request type, you are flying blind. Instrument your context size distribution.

Forgetting conversation history: In multi-turn applications, cumulative context from conversation history can dwarf the retrieved document context. Prune or summarize old conversation turns.

The Optimization Stack

Here is the full stack, ordered by impact:

1. Semantic chunking — affects every RAG query, highest ROI
2. Context sizing guardrails — prevents runaway context on edge cases
3. Semantic caching — 20-50% of queries benefit immediately
4. LLM-based compression — cheap at scale, 40-60% context reduction
5. Query decomposition — for complex multi-hop use cases
6. Hybrid retrieval — precision + recall without the bloat

Run these in order — the first two are free and immediate; the later ones require more engineering investment.