Not all text is the same. A JSON config file, a Python traceback, and a product description all carry information differently. SuperCompress's domain preprocessors detect the content type automatically and apply specialized transformations before the compression engine runs — boosting savings from 65% to 70–90% on structured data without losing any signal.

Why Domain Preprocessors?

The general-purpose compression policy scores every line against the query. It's good at finding answer-critical content in narrative text. But structured data has different characteristics:

• JSON has repetitive structure (null fields, empty arrays, timestamps) that takes tokens but adds no signal
• Code has docstrings, comments, and verbose imports that aren't semantically relevant to the question
• Logs have repeated messages, long stack traces, and DEBUG noise that dwarfs the actual error signal

A line-level scorer treats every line equally. Domain preprocessors understand the structure and strip noise before the scorer runs. The result: the scorer sees a cleaner signal and makes better keep/drop decisions.

Content Router

The router samples the first 50 lines of context and classifies them using the existing line classifier:

The router runs automatically inside compressAdaptive() and compressContext(). You don't need to configure anything — the router samples, decides, and applies the right preprocessor.

JSON SmartCrusher

LLM APIs often return verbose JSON responses. SmartCrusher understands JSON structure and can compress it intelligently:

// Before (342 tokens):
{
  "users": [
    {"id": 1, "name": "Alice", "email": "alice@example.com", "status": "active", "created_at": "2026-01-15T08:30:00Z", "updated_at": "2026-07-07T12:00:00Z"},
    {"id": 2, "name": "Bob", "email": "bob@example.com", "status": "active", "created_at": "2026-02-20T09:15:00Z", "updated_at": "2026-07-07T12:00:00Z"},
    // ... 98 more users with identical structure
  ],
  "metadata": {
    "page": 1,
    "total_pages": 10,
    "total_count": 1000,
    "request_id": "req_abc123",
    "timestamp": "2026-07-07T12:00:00Z",
    "null_field": null,
    "empty_array": []
  },
  "logs": [
    {"level": "INFO", "message": "Request started", "timestamp": "..."},
    {"level": "DEBUG", "message": "Connecting to DB...", "timestamp": "..."},
    {"level": "ERROR", "message": "Connection timeout", "timestamp": "..."}
  ]
}

// After (87 tokens):
{
  "users": [{"id": 1, "name": "Alice", ...} _x98],
  "metadata": {
    "page": 1, "total_pages": 10, "total_count": 1000,
    "request_id": "req_abc123"
  },
  "logs": [
    {"level": "ERROR", "message": "Connection timeout"}
  ]
}

What it does:

• Drops null fields and empty arrays
• Samples homogeneous arrays — keeps first 3 items + count marker for arrays > 12
• Truncates long strings (> 200 chars) to first 100 chars + length note
• Removes timestamps from well-known keys (created_at, updated_at, timestamp, etc.)
• Safety gate — only replaces if crushed text is ≥ 15% smaller than original

Code AST Compressor

When agents include source code in context, most of the tokens are documentation and formatting:

// Before:
/**
 * Calculates the total price including tax and discounts.
 *
 * @param {number} basePrice - The base price before adjustments
 * @param {string} discountCode - The discount code to apply
 * @returns {number} The final calculated price
 * @throws {Error} If discount code is invalid
 */
function calculatePrice(basePrice, discountCode) {
  // Validate input
  if (typeof basePrice !== 'number') {
    throw new Error('basePrice must be a number');
  }

  const DISCOUNT_MAP = {
    'SAVE10': 0.10,
    'SAVE20': 0.20,
    'WELCOME5': 0.05,
  };

  const discount = DISCOUNT_MAP[discountCode] || 0;
  return basePrice * (1 - discount);
}

// After:
function calculatePrice(basePrice, discountCode) {
  const DISCOUNT_MAP = { 'SAVE10': 0.10, 'SAVE20': 0.20, 'WELCOME5': 0.05 };
  const discount = DISCOUNT_MAP[discountCode] || 0;
  return basePrice * (1 - discount);
}

What it strips:

• Docstrings (""", ''') and block comments (/* */)
• Line comments (//, #, ;) and JSDoc annotations
• Multi-line data literals collapsed to single lines
• Blank line runs — 3+ consecutive blanks → 1

What it preserves: function/class/interface definitions, decorators, imports/exports, return/yield/throw statements, and all executable code.

Log/Trace Compressor

Stack traces and debug logs are the worst offenders for token waste. A single crash can produce 100+ lines of trace, most of which are framework internals:

// Before (45 lines):
2026-07-07 10:15:23 ERROR [MainThread] Unhandled exception in request handler
Traceback (most recent call last):
  File "/app/handlers.py", line 142, in handle_request
    result = process_data(payload)
  File "/app/processors.py", line 87, in process_data
    validate(schema, data)
  File "/app/validators.py", line 34, in validate
    raise ValidationError("Missing required field: email")
  File "/app/validators.py", line 30, in validate
    check_field(field, value)
  File "/app/validators.py", line 22, in check_field
    if not value:
  File "/app/validators.py", line 18, in check_field
    return True
  [30+ internal frames collapsed]
2026-07-07 10:15:23 INFO [MainThread] Request completed
2026-07-07 10:15:23 DEBUG [DBPool] Connection returned to pool
2026-07-07 10:15:23 DEBUG [Cache] Invalidating key: user_42_profile

// After (6 lines):
ERROR [MainThread] Unhandled exception in request handler
  File "/app/validators.py", line 34 → ValidationError: Missing required field: email
  [... trace frames collapsed from 35 to 2]
INFO [MainThread] Request completed

What it does:

• Collapses stack traces — keeps first and last frame, drops internals
• Deduplicates repeated messages — fingerprints by normalizing timestamps/numbers
• Filters DEBUG/TRACE — dropped unless the question explicitly asks about debug info
• Concentrates errors — ERROR/WARN lines are kept even at high compression
• Final flush — ensures trace blocks at the end of input are collapsed correctly

Benchmarks

Usage

Preprocessors run automatically — no configuration needed. The output includes a preprocessor field so you can see which one was used:

curl -X POST https://supercompress.dev/api/v1/compress \
  -H "X-API-Key: sc_live_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"context": "{...}", "query": "What error occurred?"}'

# Response includes:
# {
#   "compressed_text": "...",
#   "preprocessor": "json",
#   "tokens_saved": 780,
#   "kv_savings_pct": 82.0
# }

Or in browser JS:

const result = SuperCompressEngine.compressAdaptive(context, query, model);
console.log(`Preprocessor: ${result.preprocessor}`); // "json", "code", "log", or "none"

What's Next

• YAML/TOML support — extend SmartCrusher to configuration formats
• SQL preprocessor — strip query plans, collapse large IN clauses
• Protocol buffer compression — understand protobuf wire format
• Custom preprocessor API — let users register their own domain-specific transformers

Domain preprocessors are available in SuperCompress v0.6+. Get your API key or try the playground — paste JSON, code, or logs and see the preprocessor in action.