LLM Proxy v4: context overflow recovery and the stable gateway.

Cutting three major version numbers in under two months sounds like a churn signal. In llm-proxy-gateway, it marks the opposite: the deliberate consolidation of a foundational service. When v3 migrated the gateway from Node to .NET, it proved the raw throughput benefits of typed streaming pipelines. Over the subsequent weeks, dozens of capabilities accumulated: Anthropic to OpenAI translation, SSE mappers, model aliases, and rolling summarizers. Cutting v4.0.0 (v4.0.0) was the step that declared this accumulated architecture settled and production-grade.

The headline feature landing alongside the release was dynamic context overflow recovery (#200). Despite pre-flight token estimators, edge cases inevitably slip through: complex tool payloads, multi-modal prompt expansions, or tokenizer mismatches can cause an upstream engine (like vLLM or SGLang) to reject a request with an HTTP 400. In previous versions, this error surfaced directly to the agent as a fatal failure.

In v4, the proxy intercepts the rejection, uses BackendErrorClassifier to extract the exact token ceiling reported by the backend, and invokes ContextOverflowCompactor. The compactor prunes dead or repeated tool turns first, applies bounded eviction, and replays the request against the newly discovered window limit without the client agent ever knowing an overflow occurred.

Alongside overflow recovery came fixes for newer model behaviors. Reasoning engines frequently emit extensive chain-of-thought blocks while leaving the standard content property completely blank. When the turn summarizer processed these turns, it saw an empty string and dropped the turn entirely, discarding crucial conversational steps. PR #196 added explicit degradation to reasoning_content, preserving context across reasoning passes.

We also expanded cross-ecosystem translation, adding support for Codex tool definitions and native OpenAI Responses passthrough (#195), and scoped context eviction rules on a per-backend basis (#201) to protect local KV-cache prefix stability.

Lessons learned

• Parse backend overflow errors to recover oversized requests in flight. When an inference server rejects a request for exceeding context limits, its error payload usually contains the exact maximum token length it supports. Rather than failing the user call, parsing that error allows the proxy to compact prior turns to match the reported limit and retry transparently.
• Turn summarizers must inspect reasoning streams when content fields arrive empty. Thinking and reasoning models often treat content as optional during intermediate processing steps. Any pipeline performing prompt compaction, caching, or logging must check reasoning_content as an alternate payload to avoid silently losing context.
• Cache last-known-good model metadata across transient discovery blips. Dynamically polling model parameters over local HTTP endpoints is prone to occasional timeouts during heavy GPU prefill spikes. A failed poll must never clear registered context lengths; holding onto last-known-good metadata prevents unnecessary downstream validation errors.
• Tailor context eviction strategies to specific inference backends. A destructive eviction strategy that aggressively truncates early system turns might save dollars on commercial cloud endpoints, but on local SGLang or vLLM instances it destroys KV-cache prefix reuse and tanks inference speed. Gateway rules must respect backend characteristics.