KV- and Token-Aware Routing in Ray Serve LLM

Tue, 25 Aug 2026 00:00:00 +0000

I coauthored an Anyscale technical deep dive on integrating NVIDIA Dynamo’s KV-aware selection machinery into Ray Serve LLM. The collaboration is a useful example of a systems component becoming more valuable when it can be embedded outside the system where it started.

Multi-DC KV-Aware Routing with Gcore

Mon, 20 Jul 2026 00:00:00 +0000

I worked with Gcore’s AI team on the Dynamo side of a global inference-routing problem: when the same model is served in many data centers, how do you decide where each request should go without throwing away useful KV cache state?

Routing on Rudy Pei (裴彦儒)

KV- and Token-Aware Routing in Ray Serve LLM

Multi-DC KV-Aware Routing with Gcore