Running large local models for daily agent work requires two things: an inference runtime that actually stays stable under heavy prompt loads, and a proxy gateway smart enough to trim bloated contexts before hitting physical VRAM limits. In this window, four different epics reached completion to make that pipeline reliable.
In LocalMoEManager, we integrated SGLang container profiles (#9) to serve
Qwen3.6-27B and 35B models on consumer hardware. Getting these hybrid
linear/full attention architectures stable took several tuning iterations. First,
auto-detection in the runtime identified the chat template but failed to
activate the output parsers, causing raw <think> blocks to leak downstream
into standard OpenAI payloads until explicit container flags were passed (#13).
Second, torch.compile warm-ups took over twenty minutes on every container
reboot when inductor caches were stored on host bind mounts; moving the cache to
a dedicated Docker volume dropped cold-boot times back down to seconds (#16).
Finally, an experiment with HiCache host-RAM radix spilling had to be reverted
(#21) after hybrid GDN layers deadlocked during generation.
On the gateway side, llm-proxy-gateway received its Epic-Prompt-Filtering
landing (#121). When automated workflows iterate across many turns, tool
results and JSON schemas rapidly saturate the context window. We implemented
dropTools to remove unused tool definitions via glob patterns (#125), added
round-based tool result aging so older outputs are pruned without breaking recent
turn context (#132), and introduced model-class aliases so callers can request
qwen and let the proxy route to whichever backend is currently live (#134).
The gateway also began advertising real backend limits on its Anthropic-compatible
endpoints (#126), reading max_model_len directly from the running engine.
Meanwhile, agent-skill-installer migrated local agent profiles to target the
new SGLang endpoints (#223) and finalized the native Go port of its execution
scripts (7dfafa7), and Project Hub cut its v0.4.0 UI release (v0.4.0)
alongside orchestration fixes to ensure work items deliver full content across
agent boundaries (#29).
Lessons learned
- Explicit flags beat auto-detection for inference runtime parsers. SGLang inspected the tokenizer template and recognized the Qwen format, but left the streaming parser disabled by default. Unless the container explicitly launches with reasoning and tool-call flags, chain-of-thought tokens bleed into standard content fields.
- Bind mounts kill kernel cache performance; use native container volumes for
compiler artifacts. Storing
torch.compileand Triton kernel artifacts over a virtiofs host bind caused 20-minute startup freezes and compilation loss on container teardown. A named container volume preserves compilation caches instantly. - Revert speculative decode or caching strategies when hybrid architectures deadlock. Techniques that work cleanly on standard dense Transformers (like HiCache host-RAM paging or NEXTN speculative drafting) can choke on hybrid recurrent/attention models. When state-space layers cause GPU stalls, dropping back to stable fp8 KV allocations is faster than fighting the driver.
- Drop unused tool schemas before context limits bite. Multi-agent environments tend to inject every registered tool schema into every turn. Glob-filtering tool definitions at the proxy layer frees up thousands of tokens before prompt filtering even touches dialogue history.