Nine days after shipping v2.0.0, llm-proxy published v3.0.1. Cutting two
major versions in less than a fortnight is usually a sign of churn; here, it was
a deliberate, clean-slate transition.
When v2.0.0 shipped, it carried both the established Node.js proxy and the
early scaffold of a compiled .NET port. Maintaining two runtimes in parallel
quickly generated friction: duplicate GitHub Actions / Gitea CI workflows,
parallel Dockerfile configurations, conflicting runner cache keys, and diverging
unit test suites. Instead of living with dual implementations through a prolonged
deprecation cycle, PR #60 executed a clean cut.
In a single pull request, nearly 14,000 lines of JavaScript were deleted. The
Node.js codebase (src/*.js) and its associated test harness were wiped, the
root Dockerfile was switched over to compile the .NET solution, and the .NET
runtime took over all production traffic. Because the internal architecture,
configuration schema, and container boundaries shifted completely, cutting a
major semantic version bump to v3.0.1 accurately reflected the break.
With the legacy runtime removed, the deployment pipeline received a much-needed
pruning (#61). Multi-architecture build matrixes and speculative edge lanes
were dismantled in favor of a single, hardened container publish job,
eliminating MTU packet drop issues that had plagued the self-hosted runners.
The newly crowned .NET gateway then received core observability upgrades. Under
PR #66 (IP-AAT-4), the gateway began minting a unique routing_decision_id
at the perimeter of every request. Rather than treating proxy routing as a black
box, this ID is stamped on response headers and passed downstream to telemetry
collectors, correlating incoming agent requests with exact model targets and
failover attempts.
Finally, PR #67 established an escalation measurement harness (IP-LLMP-9),
giving the platform empirical insight into how often worker models trigger the
call_advisor escape hatch versus handling turns directly on local hardware.
Lessons learned
- Rip out parallel legacy runtimes in a clean break rather than maintaining dual stacks. Carrying both Node.js and .NET implementations meant doubled CI runs, divergent tests, and split focus. Deleting the legacy codebase in one coordinated motion cleared substantial operational baggage.
- Mint and propagate routing decision IDs at the gateway perimeter. Attaching a unique decision ID to incoming requests before any routing logic runs allows traces, logs, and billing meters to link back to the exact routing branch chosen, even if the connection aborts mid-stream.
- Consolidate multi-architecture container build pipelines early. Maintaining complex multi-arch build lanes for a private cluster running on homogeneous x86-64 hardware wasted runner resources and generated recurring MTU network errors during image layer uploads.