We’ve been working on Live Migration of Tensorlake sandboxes, which lets us move sandboxes between machines with under 300ms of guest pause and stalled TCP connections. We retain all open TCP connections as sandboxes are migrated, with zero connection resets. Running stateful sandboxes at scale often means low cluster utilization as workload patterns shift throughout the day. Today, we don’t defragment the cluster to release underutilized nodes. Another problem with stateful sandboxes is that they can’t outgrow the available free resources (like RAM) on a tightly packed machine. Live Migration will help reduce operational expenses as we continue to scale, and unlock features like guaranteed vertical scaling: moving sandboxes around the cluster, and hot-plugging agent sandboxes with more compute resources when they need it. It’s been fun working on this problem. We went back to see what VMWare and other companies have published about live VM migration and incorporated it into our dataplane, VMM, and networking subsystems. Here is an example of moving a sandbox with 1GB RAM and 10GB disk from one node to another within a region in our production cluster!

Tensorlake’s sandbox scheduler uses a replicated state machine (RSM) to replicate cluster and sandbox metadata across multiple machines for high availability. Recently, we increased our throughput for scheduling sandboxes by 12x. RSMs linearize writes across nodes to provide strong consistency. As we scaled our sandbox infrastructure, we found that creating a burst of 100k sandboxes concurrently competed for the replication pipeline with scheduler updates that place sandboxes on dataplanes. Worse, dataplane updates which inform users that their sandboxes have started were jammed behind create and placement messages across nodes. To solve this problem, we borrowed ideas from massively distributed databases such as TiKV and CockroachDB and implemented Multi-Raft. Among other use cases, these databases use Multi-Raft to replicate databases independently of one another. We now have three replication lanes across scheduler nodes: 1. Intent to create/delete/update sandboxes, driven by users 2. Scheduler placement updates 3. Dataplane updates Multi-Raft concurrently transfers messages across these three lanes. Both (1) and (3) drive the scheduler, so the scheduler is never blocked waiting for more messages.