Routing

I coauthored an Anyscale technical deep dive on integrating NVIDIA Dynamo’s KV-aware selection machinery into Ray Serve LLM. The collaboration is a useful example of a systems component becoming more valuable when it can be embedded outside the system where it started. ...

I worked with Gcore’s AI team on the Dynamo side of a global inference-routing problem: when the same model is served in many data centers, how do you decide where each request should go without throwing away useful KV cache state? ...