KV- and Token-Aware Routing in Ray Serve LLM
I coauthored an Anyscale technical deep dive on integrating NVIDIA Dynamo’s KV-aware selection machinery into Ray Serve LLM. The collaboration is a useful example of a systems component becoming more valuable when it can be embedded outside the system where it started. ...