A/B testing compression

A/B test protocol

1. Split traffic — 50% control (no compression), 50% treatment (with compression)
2. Run for sufficient samples — Minimum 1,000 queries per variant
3. Measure these metrics — Answer accuracy, user satisfaction, response latency, token cost
4. Statistical significance — Use a chi-squared test at p < 0.05

Implementation

import random
from supercompress import Compressor

comp = Compressor()

def handle_query(context, query, user_id):
    use_compression = hash(user_id) % 2 == 0
    if use_compression:
        result = comp.compress(context, query)
        context = result.compressed_text
    response = llm.generate(context, query)
    log_result(user_id, use_compression, response, context)
    return response

Frequently asked questions

How long should the A/B test run?

At least 1,000 queries per variant, or one week, whichever is longer.

What if quality drops?

Roll back compression for that use case and investigate which context types are being incorrectly removed.