I worked with Gcore’s AI team on the Dynamo side of a global inference-routing problem: when the same model is served in many data centers, how do you decide where each request should go without throwing away useful KV cache state? ...
Simulating spin glasses at scale has always been a pain — Python is too slow, and C++ is too painful. Peapods is my attempt at a middle ground: a Monte Carlo package for Ising spin systems where the hot loops are written in Rust and everything else stays in Python, glued together with PyO3. You just pip install peapods and go. ...
I worked on the Flash Indexer, a high-throughput global KV-cache indexer for NVIDIA Dynamo. The problem is to track which inference workers hold which KV blocks, and then answer routing queries fast enough that the indexer itself does not become the bottleneck. ...
Dynamo is NVIDIA’s open-source datacenter-scale distributed inference serving framework for generative AI and reasoning models. Built in Rust for performance and Python for extensibility, it supports disaggregated prefill and decode, dynamic GPU scheduling, and LLM-aware request routing across multi-node multi-GPU topologies. It supports backends including TensorRT-LLM, vLLM, and SGLang. ...