Repo Intelligence Service closes in the gateway.

The repo intelligence service inside llm-proxy reached done this week, but the substantive change was architectural: the proxy grew out of its prototype shoes and turned into a modular gateway.

Up to this point, the gateway lived as an unwieldy single file of roughly 2,400 lines. That worked when all it did was proxy HTTP requests to a local llama.cpp instance. As soon as multiple backend targets, model translation layers, and fallback schemes entered the picture, every small fix risked touching shared state. PR #18 cleanly cut the monolith into discrete modules: gateway.js for HTTP lifecycle, routing.js for rules and fallback chains, translation.js for mapping between Anthropic and OpenAI schemas, and forward.js for SSE streaming and timeouts. Alongside that cut, machine-profile routing landed, isolating local machine capabilities from shared cluster endpoints.

With clean seams in place, routing grew actual judgment through worker-advisor escalation (#17). Instead of blindly sending every prompt to expensive hosted models or failing when a small local model ran out of depth, the gateway injects a synthetic call_advisor tool into local worker requests. When a local model encounters a query it cannot resolve with confidence, it invokes the tool; the proxy intercepts the call and forwards the original payload to a configured advisor chain without requiring client-side intervention.

The rest of the window tackled edge cases in local inference interoperability. Claude Code frequently interleaves role: "system" messages mid-array during tool execution loops. Frontier APIs accept this without issue, but local model runtimes running Jinja chat templates choke unless the system prompt is strictly first. PR #21 restored message sanitization specifically for local backends, hoisting mid-stream system prompts into the root turn while leaving native cloud payloads untouched.

A trickier failure mode cropped up during Ask Sage streaming (#33). When an upstream OpenAI-compatible endpoint returned an error page over an HTTP 200 SSE connection, the proxy’s parser swallowed the non-JSON text in a bare catch block, returning an empty-but-successful response to the client. The fix exposes unparseable chunks as actionable error events and adds a buffering fallback that re-streams responses over Anthropic-compatible SSE when an upstream target lacks native streaming support.

Lessons learned

• Never swallow upstream SSE parse failures into successful empty streams. Catching a JSON parse error on an SSE chunk and doing an early return turns an upstream backend crash into a successful empty turn, completely blinding the client and caller telemetry to the failure.
• Local inference templates enforce strict role ordering that frontier APIs tolerate. Anthropic’s managed endpoint permits late or repeated system turns; local Jinja chat templates fail catastrophically if role: "system" appears anywhere except turn zero. Target-specific sanitization belongs in the gateway layer.
• Bound backend discovery with explicit timeouts. If /v1/models queries three local machines in parallel, a single sleeping GPU workstation will block the entire discovery fan-out indefinitely unless bounded by a short discovery timeout.