- Python 94.2%
- HTML 5.7%
- Shell 0.1%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
The core insight: context compression was an open-loop system — the optimizer
squeezed context *forward* based on size alone, with no feedback from whether
the *last* response was actually useful. When quality collapsed (stubs,
refusals, missing code, hallucinations), compression continued on the same
aggressive course, deepening the collapse.
ACQG closes the loop with four mechanisms:
1. **Response-quality indicators** — Lightweight signals extracted from each
assistant response: response length ratio, code block density, repetition
score (n-gram overlap), hallucination/refusal markers ("I don't know",
"As an AI..."), and truncation detection. Each indicator feeds into an
aggregate quality score on [0, 1].
2. **Exponential moving average quality score** — A running EMA of response
quality (alpha=0.3, tunes per config). When quality degrades below a
threshold, compression aggressiveness is linearly backed off.
3. **Adaptive compression multiplier** — The guard returns a [0.0, 1.0]
multiplier that scales the effective budget thresholds. At healthy quality
(>=0.75) → 1.0x (full compression). At degraded quality (0.3-0.55) →
linear interpolation down to 0.0x. At critical quality (<0.3) → 0.0x
(compression paused entirely).
4. **Protected content zones** — Content referenced in responses (files read,
code touched) is registered with the guard's ContentProtection and
exempted from front-eviction for the next N turns.
When collapsed, the optimizer skips aggressive stages (code skeletonization,
proactive trimming, sliding-window trim) and forwards more context so the
model can recover. This creates a stable feedback loop: bad responses → less
compression → more context preserved → better responses → compression resumes.
Config: MOEPT_AGENTIC__QUALITY_GUARD_ENABLED (default true),
QUALITY_GUARD_ALPHA (0.3), QUALITY_GUARD_CRITICAL_THRESHOLD (0.3),
QUALITY_GUARD_DEGRADED_THRESHOLD (0.55), QUALITY_GUARD_HEALTHY_THRESHOLD (0.75).
Tests: 32 new tests covering QualityIndicators, ContentProtection,
AdaptiveQualityGuard, and integration scenarios (quality collapse/recovery).
Full suite: 459 passed, 2 skipped (6 pre-existing failures unrelated to this change).
Ruff: all checks passed.
|
||
| .kilo | ||
| scripts | ||
| src/moeptimizer | ||
| tests | ||
| .env.example | ||
| .gitignore | ||
| AGENTS.md | ||
| cache_preservation_guide.md | ||
| CHANGELOG.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| llm.txt | ||
| moe2.jpg | ||
| notes.md | ||
| pyproject.toml | ||
| README.md | ||
MOE-ptimizer
Transparent OpenAI API proxy that optimizes context for MoE + MTP models in multi-turns agentic tasks.
Features
MOE-ptimizer is a transparent OpenAI API proxy that optimizes context for MoE + MTP models in multi-turn agentic tasks — large token savings with byte-stable prefixes so the backend's native prefix cache is reused.
The full version-by-version feature history (v0.1.0 → v0.5.4, plus the review03.md §10 UX/operability fixes) lives in CHANGELOG.md.
Architecture
Client (OpenAI SDK) → moeptimizer:8080 → Lemonade Server:13305
│
├── SessionManager (per-session isolation)
│ └── Stable Anonymous Session Resolver
├── AgentContextOptimizer (cache-stability policy)
│ ├── Immutable Static Layer Guard
│ ├── Reasoning Content Preserver
│ ├── Stable Turn Structure Normalizer
│ ├── Top-Only Eviction Policy
│ ├── ToolOutputFilter (declarative regex filter for tool outputs)
│ ├── ToolOutputCompressor (boundary-compress large tool outputs)
│ └── OutputShaper (cache-safe terse instruction + turn-class max_tokens/reasoning_effort clamping)
├── AgentStateStore (KV graph)
├── ScratchpadCompactor
│ └── HierarchicalSummarizer (cache-stable rolling-summary compaction)
├── ThinkingPreserver (no-op; preserves <think> blocks verbatim)
├── StateBasedRAG
│ └── SymbolIndex (fuzzy symbol lookup)
├── LoopDetector
├── ProgressTracker
├── PromptTemplateManager (task classification)
│ └── ContextTemplateMatcher (template matching)
├── AttentionSinkManager (internal cache hint only; no model-visible markers)
├── ExpertRoutingCache (placeholder; fabricated expert masks)
├── CacheKeyRegistry (hit prediction)
│ └── HitPredictionModel (XGBoost early-exit)
├── KVSlotTracker (explicit cache control)
├── StaticPrefixKVCache (text memo; not real KV tensors)
├── ContextAligner (internal alignment; no prompt padding)
├── ContextCanonicalizer (newest-user-turn only)
├── SelectiveTruncator (no-op; returns unchanged)
├── PatternInjector (section markers; stripped before model input)
├── DependencyOrderer (no-op; import ordering not applied)
├── IncrementalUpdater (cache preservation)
├── CacheAwareChunker (aligned chunking)
├── ContextCompressor (newest-user-turn only)
├── CodeBlockOptimizer (tree-sitter code optimization)
├── ChunkFingerprintCache (SHA-256 chunk reuse)
├── DeltaEncoder (code delta compression)
├── TokenAwareTruncator (whole-message top-only fallback)
├── AsyncIOStage (async heavy stage offloading)
└── EmbeddingService (LanceDB + embeddings model)
Installation
pip install -e ".[dev]"
Configuration
Copy .env.example to .env and adjust:
cp .env.example .env
Environment variables use the MOEPT_ prefix with __ for nested config.
For example, server.url maps to MOEPT_SERVER__URL.
Proxy Server
| Variable | Default | Description |
|---|---|---|
MOEPT_PORT |
8080 |
Proxy listen port |
Lemonade Server
| Variable | Default | Description |
|---|---|---|
MOEPT_SERVER__URL |
http://localhost:13305/api/v1 |
Base URL of the Lemonade chat-completions server API |
MOEPT_SERVER__EMBED_URL |
http://localhost:13305/api/v1 |
Base URL of the OpenAI-compatible embedding server; set separately when embeddings are hosted elsewhere |
MOEPT_SERVER__LLM_MODEL |
Qwen3.6-35B-A3B-MTP-GGUF |
LLM model identifier for chat completions |
MOEPT_SERVER__EMBED_MODEL |
embed-gemma-300m-FLM |
Embedding model identifier |
MOEPT_SERVER__TOKENIZER |
auto |
Tokenizer for budget/counting. auto tries a local Qwen HF tokenizer, else falls back to tiktoken cl100k_base (GPT-4 BPE, approximate for Qwen). Set to a HuggingFace repo id or a local tokenizer.json path for exact Qwen counts. Runtime calibration from the backend's real prompt_tokens corrects the residual ratio each turn regardless. |
MOEPT_SERVER__TIMEOUT |
300.0 |
Request timeout in seconds for long context conversations |
Agentic Loop Tuning
| Variable | Default | Description |
|---|---|---|
MOEPT_AGENTIC__KEEP_FULL_STEPS |
3 |
Last N user-assistant pairs kept in full detail; older complete turns are evicted from the top |
MOEPT_AGENTIC__ARCHIVE_THRESHOLD |
3 |
Steps before this index are archived/compressed |
MOEPT_AGENTIC__MAX_OPTIMIZED_CHARS |
32000 |
Character fallback cap for optimized context. Should be a meaningful fraction of the backend context window, not ~1% of it: evicting/skeletonizing below this starves the model of task context and collapses quality. |
MOEPT_AGENTIC__MAX_OPTIMIZED_TOKENS |
8000 |
Token budget cap; takes precedence over character cap. Hard cap on the optimized context; a sane fraction (a few %) of the backend window so eviction only triggers when genuinely needed. |
MOEPT_AGENTIC__PROACTIVE_TRIM_RATIO |
0.45 |
Ratio of max tokens where proactive top-only trimming starts (base default; the balanced profile raises this to 0.6) |
MOEPT_AGENTIC__COMPACTION_TRIGGER_RATIO |
0.75 |
Ratio of max tokens where compaction/compression starts (base default; the balanced profile raises this to 0.85) |
MOEPT_AGENTIC__THINKING_PROTECT_RECENT |
2 |
Keep full thinking for last N steps |
MOEPT_AGENTIC__SESSION_TIMEOUT |
3600 |
Session inactivity timeout in seconds |
MOEPT_AGENTIC__MAX_SESSIONS |
256 |
Max concurrently tracked sessions; LRU eviction over the cap (0 disables) |
MOEPT_AGENTIC__USE_TOKEN_BUDGET |
true |
Use token-based budget enforcement |
MOEPT_AGENTIC__FAST_PATH_ENABLED |
true |
Bypass expensive transformations for contexts already under budget |
MOEPT_AGENTIC__RAG_ENABLED |
true |
Enable state-based RAG injection for long/over-budget sessions |
MOEPT_AGENTIC__OPTIMIZE_CODE_BLOCKS |
true |
Run tree-sitter code-block optimization (chunk dedup). Budget-gated: only fires when the context exceeds the proactive trim threshold, so lean contexts keep exact code and avoid proxy latency. |
MOEPT_AGENTIC__CODE_SKELETON_ENABLED |
false |
Compress large code blocks to skeletons under context pressure. Base default is true, but the balanced quality profile (the default) sets it false so the model keeps full code to edit/extend; only the aggressive profile enables skeletonization. |
MOEPT_AGENTIC__CODE_LEDGER_MAX_SIGS |
40 |
Max code-signature lines carried forward into the evicted-turn code ledger. When front-eviction drops a code-bearing turn, its function/class signatures are accumulated into a compact [Evicted-turn code index] system message appended to the protected tail, so the model keeps awareness of code that lived in dropped turns (fixes has_code_proxy=0 / code_block_loss). Capped to bound the ledger's own size. |
MOEPT_AGENTIC__PROMPT_TEMPLATE_ENABLED |
false |
Apply task-template specialization to rewrite the user prompt. OFF by default: for agentic coding the client's exact wording matters and template rewrites risk changing it (cache guide DONT #4). Gated behind the proactive threshold. |
MOEPT_AGENTIC__DELTA_ENCODE_INJECT |
true |
When a file is re-read after an edit, replace the full re-read file body with a compact unified diff against the prior snapshot (review §3.4). ON by default, but injection is decided dynamically per re-read: it only fires when the prior version of the file is already present in the context (verified by substring), so the model can apply the diff to a file it already sees. On a first read, or when the prior version was evicted/summarized out of context, the full current code is kept verbatim so edits stay correct. Set to false to always forward the full re-read body. |
MOEPT_AGENTIC__ATTENTION_SINKS_ENABLED |
false |
Inject model-visible attention-sink markers |
MOEPT_AGENTIC__STATIC_LAYER_ALIGNMENT_ENABLED |
false |
Pad static layer to cache-block boundaries |
MOEPT_AGENTIC__REASONING_PRESEED_ENABLED |
false |
Inject reasoning scaffolding into user messages |
MOEPT_AGENTIC__MTP_BOUNDARY_ALIGNMENT_ENABLED |
false |
Pad final context to an MTP prediction boundary |
MOEPT_AGENTIC__IMMUTABLE_PREFIX_ENABLED |
true |
Freeze the system prompt verbatim across turns so the backend's automatic prefix cache can reuse it. The first user message is not frozen (it is deterministically compressed and stays stable on its own). |
MOEPT_AGENTIC__MAX_STATE_STEPS |
200 |
Maximum steps retained in AgentStateStore per session; oldest archived steps are pruned beyond this cap |
MOEPT_AGENTIC__GOAL_RELEVANCE_THRESHOLD |
2.0 |
Minimum relevance score for a step to survive task-aware pruning (review §10). Steps scored below this threshold are evicted from the evictable body after goal decomposition. Set to 0.0 to disable. |
MOEPT_AGENTIC__LIVE_ZONE_COMPRESSION_ENABLED |
true |
Live-zone compression (review §5.1/§7.1). Tracks a content hash of the frozen stable prefix and only re-runs expensive stages (tree-sitter code optimization, tool-output filter/compression) on the live zone — messages new or changed since the previous turn. Keeps the prefix byte-identical across turns (guaranteeing backend prefix-cache reuse) and cuts per-turn CPU by avoiding redundant parsing of unchanged code blocks. |
MOEPT_AGENTIC__QUALITY_PROFILE |
balanced |
Optimization preset trading token savings against response fidelity (review03.md §10). One of quality (no summarization/RAG/anchor/code-skeleton; only lossless boundary compression — maximizes similarity to the direct baseline), balanced (defaults), aggressive (lower token cap, more top-only eviction, earlier compaction). Applied on top of any explicit field overrides, so individual fields can still be tuned. |
MOEPT_AGENTIC__EXPLAIN_MODE_ENABLED |
false |
Dry-run / explain mode (review03.md §10). When enabled, the proxy attaches the optimized prompt it would send to the backend as the X-MOEPT-Optimized-Messages response header (base64 JSON) so operators can inspect exactly what the proxy changed. A single request can also opt in via the X-MOEPT-Explain: true request header, which works regardless of this flag. Off by default to avoid leaking full context in response headers. |
Code Chunking
| Variable | Default | Description |
|---|---|---|
MOEPT_CODE_CHUNKING__CHUNK_MAX_CHARS |
1500 |
Maximum characters per code chunk |
MOEPT_CODE_CHUNKING__TOP_K_CHUNKS |
5 |
Number of top relevant chunks to retrieve |
MOEPT_CODE_CHUNKING__MIN_CHUNK_SCORE |
0.05 |
Minimum embedding similarity score |
MOEPT_CODE_CHUNKING__EMBEDDING_DIM |
384 |
Embedding vector dimension |
Cache
| Variable | Default | Description |
|---|---|---|
MOEPT_CACHE__EMBED_CACHE_MAX |
512 |
Maximum embeddings in memory cache |
MOEPT_CACHE__LANCEDB_PATH |
~/.moeptimizer/lancedb |
LanceDB vector database data directory |
Speculative Decoding
Client-proxy speculative decoding is non-functional (review03.md §2.1). The only effective path is a backend with native MTP support, enabled via
MOEPT_V050__NATIVE_MTP_PASSTHROUGH(auto-detected byMOEPT_V050__NATIVE_MTP_AUTODETECT). The oldMOEPT_SPECULATIVE__*variables were removed.
v0.5.x Optimizations
Auto-detection (default): with
MOEPT_V050__CAPABILITY_AUTODETECT=true, the proxy probes the live backend's own metadata on a TTL — the active device and per-modelbackend_urlfrom/health, native/slots, native/tokenize, and the model's checkpoint/labels — and enables slot pinning, native MTP passthrough, and the exact tokenizer only when the currently active device actually supports them. This is NPU↔GPU hot-swap aware: a session is never pinned to anid_sloton a device that has no slots. The manualSLOT_PINNING_ENABLED/NATIVE_MTP_PASSTHROUGHflags below are force-on overrides; leave themfalseto let auto-detection decide. All probes are best-effort and never block startup or requests.
| Variable | Default | Description |
|---|---|---|
MOEPT_V050__STATIC_PREFIX_KV_ENABLED |
true |
Enable static prefix memo — stores the prompt text (NOT real KV tensors; a client proxy cannot read backend KV). It only short-circuits the pipeline when the incoming prefix is byte-identical and already under budget; the backend's own prefix cache does the real KV reuse. |
MOEPT_V050__STATIC_PREFIX_KV_MAX_ENTRIES |
64 |
Max entries in static prefix KV-cache |
MOEPT_V050__TOKEN_AWARE_TRUNCATION_ENABLED |
true |
Enable token-aware truncation with tiktoken |
MOEPT_V050__CAPABILITY_AUTODETECT |
true |
Live, device-aware capability detection. Probes the backend's own metadata on a TTL and drives slot pinning, MTP passthrough, and tokenizer selection from what the active device (NPU vs GPU) actually supports. Best-effort; failures never block startup or requests. |
MOEPT_V050__CAPABILITY_PROBE_TTL_SECONDS |
30.0 |
How long a capability snapshot is trusted before a live re-check. Re-checks are cheap (cached between TTLs) and let device hot-swaps be picked up without restarting the proxy. |
MOEPT_V050__REMOTE_TOKENIZE_ENABLED |
true |
When the active backend exposes an exact native /tokenize (llama.cpp), prefer it for authoritative whole-prompt token counts (fingerprint-cached, never on the per-fragment hot path). Falls back to the local/tiktoken counter + backend prompt_tokens calibration otherwise (e.g. NPU device). Only takes effect with CAPABILITY_AUTODETECT on. |
MOEPT_V050__SLOT_PINNING_ENABLED |
false |
Force-on override for slot pinning: pin each session to a stable llama.cpp id_slot so the backend reuses the whole conversation prefix across turns instead of re-prefilling. With CAPABILITY_AUTODETECT on, this is enabled automatically on devices that expose /slots (e.g. GPU/llama.cpp) and skipped when the active device has none (e.g. NPU); set this true to force it unconditionally. id_slot is sent via extra_body, so non-llama.cpp backends ignore it. |
MOEPT_V050__CACHE_STABLE_MODE |
true |
Freeze a stable prefix block (system + first user + early turns) verbatim and never front-evict from it, so the backend reuses the KV cache across turns. Disable for backends without prefix caching to maximize token savings. |
MOEPT_V050__FROZEN_PREFIX_TURNS |
2 |
Number of early complete user-assistant turns (after the first user message) to freeze verbatim as part of the stable prefix when cache-stable mode is enabled. |
MOEPT_V050__CHUNK_FINGERPRINT_ENABLED |
true |
Enable chunk fingerprinting and reuse |
MOEPT_V050__CHUNK_FINGERPRINT_MAX_ENTRIES |
2048 |
Max entries in chunk fingerprint cache |
MOEPT_V050__EMBEDDING_BATCH_SIZE |
32 |
Batch size for embedding queries |
MOEPT_V050__HIT_PREDICTION_ENABLED |
true |
Enable lightweight hit-prediction model |
MOEPT_V050__HIT_PREDICTION_RETRAIN_THRESHOLD |
50 |
New samples before retraining |
MOEPT_V050__HIERARCHICAL_SUMMARY_ENABLED |
false |
Enable hierarchical summarization of old turns |
MOEPT_V050__HIERARCHICAL_SUMMARY_MAX_FULL_TURNS |
5 |
Max recent turns to keep in full |
MOEPT_V050__DELTA_ENCODING_ENABLED |
true |
Enable delta-encoding of code snapshots |
MOEPT_V050__DELTA_ENCODING_MAX_SNAPSHOTS |
100 |
Max code snapshots to keep |
MOEPT_V050__ENABLE_EXPERIMENTAL_BACKEND_HINTS |
false |
Send optional llama.cpp/MTP cache-control hints |
MOEPT_V050__NATIVE_MTP_PASSTHROUGH |
false |
Force-on override for MTP passthrough: forward MTP/speculative extra_body keys to the backend instead of stripping them, so a backend with native MTP speculative decoding (e.g. llama.cpp --speculative / Qwen3-MTP heads) uses the model's own 2–3× decode-speed feature. With CAPABILITY_AUTODETECT on, this is enabled automatically from backend metadata (a mtp label, a speculative slot, or --spec-type ...mtp launch args). |
MOEPT_V050__NATIVE_MTP_AUTODETECT |
true |
Last-resort chat-probe for native MTP when metadata is inconclusive. Best-effort and bounded by a timeout, so startup is never blocked. |
MOEPT_V050__ASYNC_IO_ENABLED |
true |
Enable async I/O for heavy pipeline stages. Offloads tree-sitter compression and embedding ranking to a thread pool (primary TTFT cost). |
MOEPT_V050__ASYNC_IO_MAX_THREAD_WORKERS |
4 |
Max thread workers for CPU-bound stages |
MOEPT_V050__ASYNC_IO_MAX_CONCURRENCY |
16 |
Max concurrent async tasks |
Usage
# Start the middleware
python -m moeptimizer
# or
./scripts/run.sh
# Test with curl
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3.6-35B-A3B-MTP-GGUF",
"messages": [
{"role": "user", "content": "Build a REST API with auth"}
],
"stream": true
}'
Conversation continuity is OpenAI-compatible by default. Clients can set the standard user field as a conversation/user key, and moeptimizer combines it with the first user message to reuse optimizer state across turns. If user is absent, the proxy fingerprints the messages history. Legacy _session_id and _session_state fields are still accepted for existing integrations, but they are stripped before forwarding requests to Lemonade.
API Endpoints
| Method | Path | Description |
|---|---|---|
POST |
/v1/chat/completions |
OpenAI-compatible chat completions (proxy + optimize) |
POST |
/v1/embeddings |
OpenAI-compatible embeddings (proxy to embeddings) |
GET |
/v1/models |
List available models |
GET |
/v1/health |
Health check |
POST |
/v1/agent/state |
Get agent session state |
POST |
/v1/agent/state/reset |
Reset agent session |
GET |
/v1/agent/sessions |
List active sessions |
GET |
/v1/agent/sessions/{id}/debug |
Per-session debug snapshot: live-zone boundary, real prefix-cache outcome + token savings, embedding circuit-breaker state, and per-session metrics (review §11.6) |
DELETE |
/v1/agent/session/{id} |
Delete a session |
POST |
/v1/cache/clear |
Clear caches |
GET |
/v1/metrics |
Proxy metrics: per-turn cached_tokens, prompt_tokens, saved_tokens, latency_ms, aggregate token savings / latency, a backend_errors count (turns the backend failed to serve, e.g. a truncated tool call), plus a per-session breakdown under sessions{} (review03.md §10) |
POST |
/v1/metrics/reset |
Reset the process-wide metrics counters |
Observability & Operations (review03.md §10)
Metrics endpoint
The proxy exposes a process-wide metrics aggregate (lock-protected) fed from
both the streaming and non-streaming completion paths. Each turn records
cached_tokens, prompt_tokens, saved_tokens, and latency_ms against the
backend. A per-session breakdown (bounded by an internal LRU) is also returned
under sessions{} so you can see cache-hit rate and prefix-cache reuse per
conversation, not just in aggregate.
curl -s http://127.0.0.1:8080/v1/metrics | jq
# -> { "requests": 30, "cache_hits": ..., "cache_misses": ..., "cache_hit_rate": ...,
# "total_cached_tokens": ..., "total_prompt_tokens": ..., "prefix_cache_reuse_ratio": ...,
# "total_saved_tokens": ..., "total_latency_ms": ..., "avg_latency_ms": ...,
# "backend_errors": ...,
# "sessions": { "<session_id>": { "requests": ..., "cache_hit_rate": ...,
# "total_cached_tokens": ..., "prefix_cache_reuse_ratio": ...,
# "total_saved_tokens": ..., "avg_latency_ms": ..., "backend_errors": ... }, ... } }
curl -s -X POST http://127.0.0.1:8080/v1/metrics/reset # reset counters
Quality profiles
Pick a preset that trades token savings against response fidelity to the
un-proxified baseline. Set MOEPT_AGENTIC__QUALITY_PROFILE (or
config.agentic.quality_profile):
quality— no middle-history mutation; only lossless boundary compression of oversized tool/assistant output. Maximizes similarity to the direct baseline (highest token cost).balanced(default) — the preset applied at app-build time: 8000-token / 32000-char cap, keep 4 full steps, RAG + cache-stable summary on, code skeletonization off. This is the effective default (the baseAgenticConfigfield defaults are overridden by this profile).aggressive— lower token cap, earlier compaction, more top-only eviction. Maximum token savings (lowest fidelity).
The preset is applied at app-build time and layered on top of any explicit
env/field overrides, so individual fields can still be tuned. An unknown
profile name falls back to balanced with a warning.
Dry-run / explain mode
Inspect exactly what the proxy would send to the backend without changing
behavior. Enable globally with MOEPT_AGENTIC__EXPLAIN_MODE_ENABLED=true, or
opt in per request with the X-MOEPT-Explain: true request header (works
regardless of the flag). The proxy attaches two response headers:
X-MOEPT-Explain: trueX-MOEPT-Optimized-Messages: <base64 JSON>— the optimized message list the proxy built for the backend.
The headers are set before the backend call, so they survive backend 500s.
Config sanity-check CLI
Validate the resolved configuration and surface risky / contradictory settings that would silently hurt prefix-cache reuse or response quality. Exits non-zero on any ERROR-level issue, so it can gate CI / deploy:
moeptimizer-config-check # console script
python -m moeptimizer --check-config
# [ERROR] bad_budget: max_optimized_tokens must be > 0.
# exit code 1
Severities: ERROR (blocks deploy), WARN (prefix-cache killers, e.g.
attention_sinks_enabled,
reasoning_preseed_enabled), INFO (legacy aliases, phantom / non-functional
subsystems and backend-compatibility notes).
Benchmarking
The benchmark script compares direct Lemonade vs moeptimizer proxy performance.
Every scenario runs as an OpenCode-style agentic harness by default: each
turn sends a real agent payload — the user task plus assistant tool_calls and
the corresponding tool results (file reads, test/lint/build logs) — and the
OpenAI tools schema is forwarded to the backend, exactly like a production
coding client. The proxy boundary-compresses large tool outputs (terminal logs,
file dumps) via ToolOutputCompressor before they enter the stable prefix, so
the benchmark exercises that path too. Pass --no-agentic to fall back to plain
user messages.
# Run with defaults (proxy on 8080, lemonade on localhost:13305)
python scripts/benchmark.py
# Real-life coding scenarios (all agentic / OpenCode-harness by default)
python scripts/benchmark.py --scenario debug --turns 15
python scripts/benchmark.py --scenario debug_long --turns 30
python scripts/benchmark.py --scenario refactor_long --turns 30
python scripts/benchmark.py --scenario feature_long --turns 30
python scripts/benchmark.py --scenario default_long --turns 30
python scripts/benchmark.py --scenario feature --turns 20
# OpenCode-harness replay of the real fixture project (user task + tool calls +
# real tool outputs read from scripts/fixtures/). The run_command log is >4k
# chars, so the proxy's ToolOutputCompressor fires on it.
python scripts/benchmark.py --scenario fixtures --turns 30
python scripts/benchmark.py --scenario opencode --turns 30
# Plain user messages instead of agent payloads
python scripts/benchmark.py --scenario debug_long --turns 30 --no-agentic
# Stress test with context eviction
python scripts/benchmark.py --turns 50 --budget 8000
# Aggressive token-savings profile (top-only eviction, 4000-token cap)
python scripts/benchmark.py --scenario refactor_long --turns 30 --profile aggressive --json > report.json
# Quality-fidelity profile (maximize similarity to the direct baseline)
python scripts/benchmark.py --scenario refactor_long --turns 30 --profile quality --json > report.json
# Regression gate: fail (exit 2) if mean semantic similarity drops below 0.85
python scripts/benchmark.py --scenario all --turns 10 --min-similarity 0.85
python scripts/benchmark.py --scenario refactor_long --turns 30 --min-similarity 0.80 --json > report.json
# JSON output for analysis
python scripts/benchmark.py --turns 20 --json > report.json 2> benchmark.log
# Dump full response pairs
python scripts/benchmark.py --turns 10 --dump-responses
# TTFT measurement and per-turn prefix-cache hit capture are ON by default
# (the benchmark streams responses via SSE but keeps each side's multi-turn
# conversation fully contiguous; only the transport changes). Per-round
# /v1/metrics snapshots are recorded for cache-reuse trend analysis.
# Disable with --no-measure-ttft if you must use the non-streaming path.
python scripts/benchmark.py --scenario refactor_long --turns 30 --no-measure-ttft
# Complete example
python scripts/benchmark.py --scenario opencode --json > benchmark_opencode_10_5.json 2> benchmark_opencode_10_5.log
Rounds default is 3 (was 1). Run at least 3 rounds so per-round variance is observable and the regression gate can use the robust round-mean-of-means instead of a single noisy sample. Override with
--rounds N.
Benchmark Parameters
| Parameter | Default | Description |
|---|---|---|
--turns |
10 |
Number of conversation turns per round. |
--rounds |
3 |
Number of full conversation rounds (run ≥3 so per-round variance is observable and the regression gate can use the robust round-mean-of-means). |
--max-tokens |
1024 |
Max tokens per response (realistic for agentic coding; 256 understates proxy savings). |
--port |
8080 |
Proxy server port. |
--json |
off | Output the report as JSON to stdout. |
--dump-responses |
off | Print direct vs proxy response pairs for quality inspection. |
--budget |
unset | Override MOEPT_AGENTIC__MAX_OPTIMIZED_CHARS (char budget); eviction triggers when context exceeds this. |
--profile |
balanced |
Context optimization profile: quality (max fidelity, no summarization/RAG), balanced (default), aggressive (max token savings, top-only eviction). |
--min-similarity |
unset | Regression gate (review03.md §10): exit 2 if the mean semantic similarity (proxy vs direct) falls below this threshold. Use in CI to block quality regressions. |
--scenario |
default |
Real-life coding scenario: debug, debug_long, refactor, refactor_long, feature, feature_long, default, default_long, fixtures, opencode, or all. |
--temperature |
0.0 |
Sampling temperature for both direct and proxy runs. 0 is deterministic so quality metrics are reproducible; raise only to stress-test nondeterminism. |
--no-agentic |
off (agentic on) | Send plain user messages instead of OpenCode-style agent payloads (user task + tool_calls + tool outputs). Agentic mode is the default for every scenario. |
--no-measure-ttft |
off (TTFT on) | Disable TTFT measurement and per-turn prefix-cache hit capture. By default the benchmark streams responses (SSE) to measure time-to-first-token and record per-round /v1/metrics snapshots; conversations stay contiguous, only the transport changes. |
You might need to run the benchmark as background task to avoid hitting command timeouts. Progress and per-turn information is dumped to stderr.
Benchmark Scenarios
| Scenario | Description |
|---|---|
debug |
Debugging session with error analysis and fix suggestions |
debug_long |
30-turn real-life debug conversation with evolving code blocks and context growth beyond 32k tokens |
refactor |
Code refactoring with performance optimization and type hints |
refactor_long |
30-turn real-life refactor conversation with evolving code blocks and context growth beyond 32k tokens |
feature |
Feature implementation with API design and testing |
feature_long |
30-turn real-life feature conversation with evolving code blocks and context growth beyond 32k tokens |
default |
General coding conversation (Fibonacci example) |
default_long |
30-turn general coding conversation with evolving code blocks and context growth beyond 32k tokens |
fixtures |
Alias of opencode — OpenCode-harness replay of the real scripts/fixtures/ project: each turn reads a real fixture file and runs a realistic test/lint/build log. |
opencode |
OpenCode-harness replay of the real scripts/fixtures/ project (user task + tool_calls + real tool outputs). Always agentic. fixtures is an alias of this scenario. |
All scenarios are agentic by default. --no-agentic sends plain user messages
instead of agent payloads. --profile selects the optimization preset
(quality / balanced / aggressive, default balanced); --min-similarity <float> makes the run exit 2 if the mean semantic similarity to the direct
baseline falls below the threshold (regression gate).
Metrics Collected
-
Latency: Direct vs proxy response times (mean, median, p95). TTFT (time to first token, ms) is captured per turn via the streaming path by default (disable with
--no-measure-ttft). -
Token Usage: Prompt tokens, cached tokens, token savings percentage
-
Context Window: Final utilization percentage
-
Prefix-cache reuse: The proxy's authoritative prefix-cache hit count per turn (
X-Prefix-Cache-Hit-Tokens, surfaced as a response header in non-streaming and an SSE comment when streaming). By default the benchmark records a per-round/v1/metricssnapshot (cache_hit_rate,prefix_cache_reuse_ratio,total_cached_tokens) for cache-reuse trend analysis (disable with--no-measure-ttft). -
Response Quality: Compares proxy-optimized responses against the direct Lemonade baseline. Similarity metrics are reported on a 0–1 scale, where
1.0is the perfect score; they are not percentages, so0.99means near-perfect alignment against the 1.0 top score. Exceptions are called out below.The quality block is split into headline (the non-redundant, robust signals the regression gate reasons about) and secondary (correlated / redundant overlap signals, kept for deep inspection only).
semantic_similarityis reported as its own informational key — theembed-gemma-300m-FLMembedder is weak on code, so it must not sit in the headline block.- Headline:
rouge_l_f1,token_jaccard,code_block_ratio,edit_similarity,length_ratio,code_syntax_validity. - Secondary:
trigram_overlap,markdown_structure_similarity,vocabulary_richness_delta,rouge_l_precision,rouge_l_recall,response_stability,code_structure_consistency,has_code_direct,has_code_proxy. - Semantic similarity (informational): Embedding cosine similarity. Higher is better; it means the proxy response preserves the same meaning even if wording differs. Weak on code, so it is not a headline/gate metric.
- Token Jaccard: Word-set overlap between proxy and baseline responses. Higher is better; it indicates shared vocabulary/content coverage.
- ROUGE-L F1: Longest common subsequence overlap between proxy and baseline text. Higher is better; it rewards shared wording and ordering.
- Edit similarity: Normalized longest-common-subsequence edit similarity. Higher is better; it means fewer insertions, deletions, or rewrites are needed to transform one response into the other.
- Code block ratio: Fraction of baseline code blocks preserved by the proxy response. Higher is better; 1.0 means all baseline code blocks were preserved or no baseline code blocks existed.
- Code syntax validity: Fraction of fenced
pythoncode blocks in the proxy response that parse withast.parse. Higher is better;1.0means no syntactically broken code was emitted as a side effect of optimization (boundary compression, summarization, eviction). This is a hard correctness signal the embedding/lexical metrics cannot catch. - Markdown structure similarity: Jaccard similarity of markdown structural elements such as headings, lists, code fences, and blockquotes. Higher is better.
- Length ratio: Proxy response length divided by baseline response length. Closer to 1.0 is better;
<0.5flags severe truncation and>2.0flags verbosity inflation. - Vocabulary richness delta: Absolute difference in type-token ratio between proxy and baseline. Lower is better; 0 means identical vocabulary diversity.
- MTP stability: Proxy preservation of baseline code-block and thinking-tag structure. Higher is better; it indicates stable MTP-style response structure.
- Syntax consistency: Similarity of code-structure keywords preserved in code blocks. Higher is better; it indicates the proxy kept similar code constructs when code was present.
- Headline:
-
MTP Stability: Code block preservation, syntax consistency
-
Eviction: Two distinct signals.
raw_exceeds_optimized_target(informational) counts turns whose raw input exceeded the proxy's post-optimization char budget (MOEPT_AGENTIC__MAX_OPTIMIZED_CHARS) — expected for long scenarios, not a failure. The real eviction signal iscompaction_triggered: turns where the proxy actually sent strictly fewer prompt tokens to the backend than the direct baseline (i.e. real compaction/optimization happened). The per-turn detail table flags eviction asproxy.prompt_tokens < direct.prompt_tokens. -
Long-horizon signals (cross-turn, reported under
long_horizonin the JSON report and as dedicated dashboard cards). These measure how the proxy behaves over a whole multi-turn conversation, not per-turn quality:- Context drift / fact recall: a small set of anchor facts is prepended to
Turn 1's user message and a recall probe is appended as the final turn of
every scenario (no separate scenario, no flag). The final-turn response is
graded against the facts via embedding cosine similarity (threshold
0.35);fact_recallis the fraction of facts preserved.Nonewhen the embedding model is unavailable. The anchor lands in user content, never the system prompt, so the frozen prefix is untouched. - Self-contradiction rate:
contradictionscounts turns whose assertions flip a prior assertion's polarity (conservative negation-flip heuristic, a lower bound — no extra backend calls). Lower is better. - Context-window wall:
context_window_wallis the first turn wherecode_block_ratio < 0.5ORsemantic_similarity < 0.3(the conversation "falls off the cliff"). Reported per side as{"proxy": int|None, "direct": int|None};nullmeans no wall was hit within the run. - Latency compounding: the dashboard also plots a per-turn TTFT growth curve (existing per-turn TTFT data, proxy vs direct) to show how time-to-first-token compounds across turns under each path.
- Context drift / fact recall: a small set of anchor facts is prepended to
Turn 1's user message and a recall probe is appended as the final turn of
every scenario (no separate scenario, no flag). The final-turn response is
graded against the facts via embedding cosine similarity (threshold
Development
./scripts/dev.sh # Install deps, run tests, lint, type-check
pytest tests/ -v # Run tests only
ruff check src/ tests/ # Lint
mypy src/moeptimizer/ # Type-check
License
MIT
