AI context window management

The context window challenge

GPT-4o has a 128K token context window. Claude 3.5 Sonnet has 200K. Gemini 1.5 Pro has 1M. These windows are large but not infinite. Long-running agents, multi-turn conversations, and large document retrievals can still hit these limits.

When you hit the limit, you have three choices: truncate (lose information), compress (reduce size while preserving meaning), or split (process in chunks). Each has tradeoffs.

The context management stack

1. Prompt compression (tier 1) - Remove irrelevant context before it reaches the window. SuperCompress reduces size by ~65% with zero information loss for the query. This is the first line of defense.
2. Retrieval filtering (tier 2) - Retrieve more documents than you need, then filter by relevance score. Only the top-k most relevant documents enter the context window.
3. Sliding window (tier 3) - For conversation history, keep only the most recent N turns plus any turns that mention the current topic.
4. Summarization fallback (tier 4) - When compression and filtering are not enough, summarize the oldest context. This should be the last resort since summaries lose detail.

Implementing the stack

from supercompress import Compressor
comp = Compressor()    def manage_context(messages, query, max_tokens=32000):
        '''Keep context under max_tokens using tiered approach.'''
    # Step 1: Compress conversation history
    history = "
".join(m["content"] for m in messages[:-1])
    compressed = comp.compress(history, query)

    # Step 2: Check size
    import tiktoken
    enc = tiktoken.get_encoding("cl100k_base")
    tokens = len(enc.encode(compressed.compressed_text))

    if tokens <= max_tokens:
        return compressed.compressed_text

    # Step 3: Sliding window for overflow
    lines = compressed.compressed_text.split("
")
    while len(enc.encode("
".join(lines))) > max_tokens * 0.8:
        lines = lines[len(lines)//4:]  # Drop oldest quarter
    return "
".join(lines)

Frequently asked questions

What is the best context window strategy?

Start with compression (highest ROI), add retrieval filtering, then use sliding windows as a safety net. Avoid summarization unless absolutely necessary.

Does context management affect quality?

When done correctly with query-aware compression, quality is preserved. Blind truncation hurts quality. Smart compression does not.