One Team, Full Stack: Fully Virtualized H100s, Zero Compromise

A behind-the-scenes look at how a small, vertically-integrated team turned Lightning.AI-owned H100 GPUs and raw AI software into one performant, AI-native cloud, and the details about systems work that made it possible.

When Lightning AI and Voltage Park merged, we didn't just combine a software company with an infrastructure company. We collapsed the boundary that usually separates them. The team that writes our VM scheduler sits next to the team that owns the actual H100 and B200 nodes those VMs run on. That proximity is the whole point and it's the reason a handful of engineers were able to stand up an entire end to end virtualization stack that is stable enough to run frontier training or customer production inference workloads.

Most clouds were built for websites. We're building one for GPUs. Here's how.

Why this couldn't have happened as two independent companies

Every cloud abstracts the hardware away from you. That's a feature for a web server and a liability for a training job, because the abstraction lies. It tells your guest a NUMA topology that doesn't exist, and your job runs at half speed with no error, no warning, nothing to grep for.

When you own both layers, you can do things like:

• Standardize the metal to a single known-good state. Same kernel, drivers, firmware, and topology on every node, so the scheduler can treat hundreds of H100 boxes as genuinely interchangeable.
• Present the guest a topology that matches the silicon, the real 8-node SNC4 layout, vCPUs pinned to their true HT siblings, each GPU paired with its rail-aligned NIC, so NCCL and PyTorch place everything correctly on their own.
• Schedule against the physical fabric, placing a multi-node job on machines that share a switch, because we know the network topology in advance.

None of that is configurable through someone else's API at the desired level. You have to own the metal and the hypervisor controller and the scheduler.

The rest of this post walks through how we built it.

Step 1: H100 Provisioning standardization

The first thing you learn operating a large GPU fleet is that a node you can't reason about is a node you can't trust with a training job. When you have hundreds of H100 boxes, the enemy isn't a single hardware failure, it's variance.

One node with hyper-threading off, one with the wrong NVIDIA driver, one where SNC4 didn't stick after a BIOS update, one with ACS misconfigured silently breaking peer-to-peer DMA between GPUs and NICs, or between nodes across the fabric, and suddenly your "identical" fleet produces wildly different NCCL benchmarks, and the customer running the job is the one who finds out first.

So before we run a single VM, every node goes through the same gate and comes out the same.

One base image, verified, washed at speed

Our provisioning is built around a single, stable base image with the entire installation flow baked in. We think of it like a washing machine: a node goes in dirty (bare hardware, vendor defaults, whatever the last tenant left behind), runs a fixed cycle, and comes out in a known-good state, same kernel, same drivers, same firmware, same kernel cmdline, every time. Baking the flow into the image instead of running it live on each node is what lets us re-provision a node in minutes rather than hours.

The result is that "node 47" and "node 412" are genuinely interchangeable. The scheduler can treat them as fungible because the provisioning guarantees they are.

The validation gate

A node isn't allowed to register with the platform until it's green across the board. We won't list the full checklist, but the categories give you the flavour of how deep this goes:

• Deterministic performance: CPU governor in performance mode across every core, so a vCPU never silently runs at a downclocked frequency mid-training
• Virtualization readiness: IOMMU and VT-x enabled, the prerequisites for safe VFIO passthrough.
• Exact software versions a pinned OS version, NVIDIA driver, and kernel, plus OFED/storage drivers compiled against that exact kernel.
• Topology guarantees hyperthreading on, SNC4 enabled with the expected per-NUMA CPU layout. This is the topology our hypervisor controller depends on when it pins vCPUs and pairs GPUs with their rail-aligned NICs.
• Memory layout: hugepages reserved through GRUB exactly once, with the right page sizes, NUMA balancing off, and transparent hugepages disabled.
• Storage shape: the right capacity mounted at the right paths with enough space left for our latest CPU & GPU VM disk images.
• Fabric health: InfiniBand ACS & ATS configured properly for multi-node, nvidia-fabricmanager enabled and active, PCIe and NVMe firmwares updated to the latest versions.

That last one is worth dwelling on, because it's a perfect example of why the gate exists.

Fabric manager will fail to start if its interface version doesn't match the GPU driver version. The failure mode is quiet, the box looks fine, GPUs enumerate, nvidia-smi is happy and right up until a multi-GPU job tries to use NVSwitch and falls over. A node like that is the kind of thing a checklist catches and a "looks healthy" heuristic doesn't.

Each line in our validation guide maps to a real failure we've seen in production. The checklist isn't bureaucracy; it's institutional memory encoded as a gate.

Step 2: Hardening the Hypervisor By Being Our Own Power Users

Standardizing the host is only half the story. The hypervisor itself has to be rock-solid, and during this work we hit several Cloud Hypervisor bugs that affected stability at scale.

Here's the thing about how we found them: At Lightning AI, every engineer develops the entire platform from a Lightning AI Studio. We are all power users of our own product. Usage and quality control are embedded in our DNA, which means we catch issues before a customer ever does.

Concretely, engineers run and deploy the full platform from a CPU Studio. This includes our React frontend and the entire backend Docker Compose stack: controlplane, postgres, clickhouse, redis, nats, etc..

We've optimized this whole setup into a single Base Studio. It cuts any new engineer onboarding down to 5 minutes to a fully deployed coding environment and, just as importantly, gives everyone a deterministic one. Same image, same stack, every time.

That dogfooding is exactly how the hypervisor bugs surfaced. While developing inside our Studios, especially under heavy docker-build load or prolonged LLM training and inference, we started noticing intermittent VM crashes. Because we live in these Studios all day, the pattern was impossible to ignore. It was a direct blocker to our own work.

After digging in, we traced the crashes to several core bugs inside Cloud Hypervisor and a few misconfigurations that fed back into our H100 provisioning standardisation.

Democratizing AI for everyone has always been Lightning AI's goal. We started with PyTorch Lightning and have steadily extended across the stack. Fixing the foundation we all build on, in the open, is part of that. So rather than carry private patches, we upstreamed four core fixes to Cloud Hypervisor:

• vmm: openapi: Fix typo in iommu_address_width field name (#8244)
• block: qcow: Preserve WRITE_ZEROES with backing files (#8245)
• block: qcow: decouple pointer table writes from cursor state (#8276)
• vmm: add memory reserve option to opt out of MAP_NORESERVE (#8350)

With a few more on the way.

Owning both the fleet and the contribution path means that when we find a sharp edge, we file the PR, get it reviewed, and ship the fix for everyone, instead of pinning a fork and hoping. That's the merged-team advantage showing up directly in the open-source commit log.

Step 3: Making the guest match the hardware

So instead of hiding the topology or flattening it, we faithfully present the real one to the guest. Some of what this involves:

• Preserving the true 8-node SNC4 layout. We don't collapse the host's NUMA nodes into fake sockets. The guest sees the same 8 NUMA nodes the hardware actually has, with the GPU and its rail-aligned NIC attributed to the correct node. When the topology the guest sees matches reality, the kernel, NCCL, and PyTorch all make the right placement decisions on their own.

• Memory zones per host NUMA node. With SNC4 each host node only has ~116 GB. If we asked for one large zone bound to a single host node, the OOM killer would fire. So we create one memory zone per host node and group them correctly under each guest node.

• CPU affinity that preserves HT siblings across different enumeration schemes. The host enumerates cores-first (CPU 0 ↔ CPU 104), while cloud-hypervisor enumerates paired-contiguous (vCPU 0 ↔ vCPU 1). We read thread_siblings_list from sysfs and pin every guest vCPU to the host hyperthread that makes the guest's view of HT siblings match the hardware's.

vCPU 0 → host CPU 0   (core 0, thread 0) ─┐ real HT siblings
vCPU 1 → host CPU 104 (core 0, thread 1) ─┘

• GPU+NIC PCI-segment pairing for GPUDirect RDMA. On bare metal, each GPU and its rail-aligned NIC share a PCIe switch, and NCCL detects this as PIX. In the guest, the equivalent is putting them on the same PCI segment. Split them onto different segments and you silently disable GDR: the job runs, just slowly, and the failure is invisible. So we pair them on a shared segment. A future post will cover this in more detail.

Does it work? The benchmarks.

Here's a host-to-device transfer bandwidth benchmark: a 1 GiB pinned-memory tensor copied from host to GPU over PCIe, 30 copies timed after warmup, measured three times per GPU in separate processes. Crucially, each measurement pins the process to the GPU's local CPUs first, so we're measuring the NUMA-local path the workload should actually take. We compare our tuned topology against an untuned guest, and against a GCP H100 node running the identical script.

Before (untuned guest topology): wildly uneven, and degrading badly on the NUMA-distant GPUs.

GPU0: 43.0, 41.9, 43.1   avg=42.7   spread=1.2 GB/s
GPU1: 40.1, 39.7, 40.0   avg=39.9   spread=0.4 GB/s
GPU2: 35.4, 35.4, 35.3   avg=35.4   spread=0.1 GB/s
GPU3: 29.3, 29.5, 29.5   avg=29.4   spread=0.2 GB/s   ← falling off a cliff
GPU4: 43.7, 43.4, 43.3   avg=43.5   spread=0.4 GB/s
GPU5: 38.1, 37.8, 35.6   avg=37.2   spread=2.5 GB/s
GPU6: 33.2, 33.3, 35.1   avg=33.9   spread=1.9 GB/s

Notice the problem: GPU0 gets 42.7 GB/s while GPU3 gets 29.4, a 45% spread across GPUs on the same node, plus high run-to-run jitter. That's the signature of a guest making bad placement decisions because the topology it sees doesn't match the silicon.

After (faithful SNC4 topology + HT-aware pinning + GPU/NIC segment pairing): tight, uniform, and high.

GPU0: 48.7, 48.5, 48.8   avg=48.7   spread=0.3 GB/s
GPU1: 48.8, 48.5, 48.3   avg=48.5   spread=0.5 GB/s
GPU2: 48.5, 48.5, 48.8   avg=48.6   spread=0.3 GB/s
GPU3: 49.1, 49.0, 49.0   avg=49.0   spread=0.1 GB/s
GPU4: 48.9, 48.9, 49.0   avg=48.9   spread=0.1 GB/s
GPU5: 48.1, 48.9, 49.2   avg=48.7   spread=1.1 GB/s
GPU6: 49.4, 49.6, 49.6   avg=49.5   spread=0.2 GB/s
GPU7: 48.8, 48.3, 48.2   avg=48.4   spread=0.6 GB/s

Every GPU now lands at ~48–49 GB/s with near-zero spread. The slow outliers are gone because every GPU is talking to memory on its own NUMA node, over correctly-pinned cores.

And for comparison, the same benchmark on a GCP H100 node:

GPU0: 26.4, 26.4, 26.4   avg=26.4   spread=0.0 GB/s
GPU1: 26.4, 26.3, 26.3   avg=26.3   spread=0.1 GB/s
GPU2: 26.4, 26.4, 26.4   avg=26.4   spread=0.0 GB/s
GPU3: 26.3, 26.3, 26.2   avg=26.3   spread=0.1 GB/s
GPU4: 26.4, 26.4, 26.4   avg=26.4   spread=0.0 GB/s
GPU5: 26.4, 26.4, 26.4   avg=26.4   spread=0.0 GB/s
GPU6: 26.4, 26.4, 26.4   avg=26.4   spread=0.0 GB/s
GPU7: 26.4, 26.4, 26.4   avg=26.4   spread=0.0 GB/s

GCP is beautifully consistent: 26.4 GB/s across the board, essentially zero spread. We pinned each process to its GPU's NUMA node, so this is GCP at its best, not a handicap. The consistency comes from flattening to two NUMA nodes, which removes the chance of a bad placement decision but also gives up the locality that the full SNC4 layout provides.

We get both. The same near-zero spread, and the higher bandwidth, because we expose the real topology and do the work in the hypervisor to make the guest place everything on it correctly. The payoff of owning the metal and the hypervisor: we don't have to choose between fast and predictable.

Step 4: A scheduler that's fast and topology-aware

A scheduler that places jobs on random free nodes is easy. A scheduler that places a multi-node training job on machines wired to the same network switch, while doing it under a thundering herd of concurrent requests, is not.

Because we own the datacenters, we know the entire network topology of every cluster in advance. We don't have to probe or guess which nodes are well-connected. The switch each node hangs off, which nodes share a rail, which cluster they live in: all of it is known data the scheduler can use to make the best placement decision up front.

Topology-aware placement

For multi-node jobs, the topology layer groups nodes into tiers of connectivity quality. Nodes on the same physical switch form the highest-quality group, since traffic between them never leaves the switch. Nodes in the same datacenter cluster form the next tier. The scheduler tries the best-connected group first and works its way down only if it has to.

The placement constraint that enforces this is simple but important. Once the first node of a batch is chosen, every subsequent node in that batch must land in the same region and fabric group as the ones already placed.

The effect is that an 8-node training run lands on machines that share a switch, and NCCL gets the fabric it needs. This is only possible because we own the switch topology and the scheduler together.

Protecting the big jobs

The harder problem is keeping large multi-node training jobs (MMT) feasible over time. Capacity gets fragmented as smaller jobs land here and there, and eventually there's no contiguous well-connected block left for the job that actually needs it.

We handle this with two policies.

Fill the small groups first.

When a request can fit in a smaller, more isolated group of nodes, the scheduler prefers that over carving into a large pristine block. Smaller workloads are steered toward smaller groups, which keeps big same-switch blocks intact for the jobs that genuinely require them.

Reclaim spot capacity for the jobs that need it.

When a large multi-node job needs room that's currently occupied by spot instances, the scheduler can reclaim that spot capacity to make the job feasible. The feasibility iterator proposes which servers to evict, and the eviction is committed (gracefully, then forcibly if a server overruns its drain window) before the new job starts. Spot users get the cheaper price in exchange for being preemptible; frontier training jobs get the contiguous fabric they need to run at all.

Fast under contention

All of this has to survive bursts of hundreds of concurrent placement requests. We took inspiration from Nomad's plan-apply model. The expensive read-and-compute work (listing machines, affinity filtering, building node groups, feasibility ranking) runs lock-free off a single database snapshot. The only serialized work is the commit, taken under a per-machine mutex, so schedulers aiming at different machines proceed fully in parallel.

At commit time, we re-verify capacity against fresh state. Two schedulers that both think they grabbed the last slot on a host can't both win, and the loser retries with a fresh snapshot. Combined with per-request candidate shuffling, so a burst of 500 requests doesn't all stampede the same top-ranked machine, this made our 500-parallel / 50-host benchmark an order of magnitude faster.

The payoff of owning the metal and the scheduler together: placement decisions are made with full knowledge of the physical fabric, not against an opaque cloud API that hides exactly the information that matters most.

What This Buys You

Put together, these layers mean that as a Lightning AI user you get:

• 36,000+ AI-Optimized GPUs (H100s, B200s, GB300s) that we own and operate, not rent through a chain of vendors.
• Bare-metal NCCL-level performance inside fully managed VMs. You don't trade away speed for convenience.
• VMs whose guest topology matches the silicon, so NCCL, PyTorch, and your training job make the right placement decisions without you tuning anything.
• A scheduler that puts multi-node jobs on the same switch fabric automatically, using network topology we know in advance because we own the datacenter.

A fleet that self-heals.

A failed VM creation never strands a GPU, a rebooted host reconstructs its own state, and admission pressure reroutes to another host instead of erroring out.

And it's not only our owned fleet. Lightning AI is a multi-cloud platform: alongside our own Voltage Park infrastructure, you can launch on AWS, GCP, Lambda, Nebius, Vultr, and more, all through the same unified software layer.

We track this from the user's point of view across every provider we run on: when you ask for compute, do you actually get it in a reasonable time? On that combined measure of stability and availability, our own bare-metal infrastructure sits in the top tier, ahead of every third-party cloud we benchmark, including GCP. Owning the stack end to end isn't just faster, it raises the odds your machine is there when you ask for it.

That's the whole point of owning the stack end to end: you write the model, and the Lightning AI platform handles the rest, correctly, down to the PCI segment within our own data centers.

The Team That Built This

A stable, vertically-integrated GPU cloud doesn't come from a large org chart. It comes from a small group of engineers who can hold the whole stack in their heads at once, from the PCIe segment up to the React frontend, and who own every layer in between.

David Weber — Network David owns the networking layer: the InfiniBand/UFM/OpenSM fabrics, routed inband VM networking, tenant isolation, and the high-performance fabric validation that makes multi-node RDMA actually work.

Mark Wagner — Node Provisioning Mark built the provisioning pipeline: the single base image, the washing-machine flow that turns a bare GPU box into a registered node in minutes, and the validation gate that guarantees every node comes out identical.

Thomas Chaton, Dmitry Frolov, Karolis Rusenas, Justus Perillieux, Ian Klemm — Hypervisor, Scheduler & Datacenter Awareness This group covered node validation, the maturation of the baremetal agent, Cloud Hypervisor stabilization, the scheduler and its datacenter-topology awareness, MMT support, and the upstream Cloud Hypervisor fixes behind this work. They're the ones who made the guest topology match the silicon, made VM creation leak-proof, and made the scheduler fast and switch-aware under load.

Ethan Harris— Director of Engineering Ethan coordinated the effort and made Lightning Studio run smoothly on our own compute, closing the dogfooding loop that lets every engineer develop the entire platform from a Studio.

Neil Bhatt— VP of Product Neil drove customer validation and made sure this team was building the thing customers actually needed.

This is a deliberately small, deeply senior team. Several of us are PyTorch Lightning core contributors who spent years building the software side of the stack, now working shoulder to shoulder with the infrastructure and networking engineers who own the metal and the fabric. That's exactly the combination the merger made possible, and it's why we could move this fast.

What surprised us most was that we didn't need to hire for this. Every piece of talent this project required was already inside the merged organization. The hard part wasn't finding the right people; it was discovering that the collective scope of what we knew, and could learn, was far larger than any of us had imagined going in. The sum of us turned out to be far greater than any of us individually.

For most of us, this meant going well beyond our comfort zone. PyTorch Lightning contributors found themselves reading kernel cmdline flags, reasoning about IOMMU address widths, writing in rust. Network engineers dug into hypervisor device config, contributed bug fixes to Studios. It was less like applying expertise we already had and more like becoming fluent in a new language under real conditions: no textbook, a production fleet as the classroom, and a working system as the only acceptable final exam. Sitting across the table from the people who owned the adjacent layer is what made that possible. When the person who understands the InfiniBand fabric is one desk away from the person writing the VM scheduler, you learn each other's language real fast.

We're hiring

This is systems engineering across the entire stack, from the datacenter floor to the coding environment humans and agents work in every day.

We design the physical fabric of our datacenters, provision the bare-metal H100 and B200 nodes that hang off it, build the virtualization layer that turns them into VMs without giving up bare-metal speed, and write the scheduler that places multi-thousand-GPU training jobs on the right switch. On top of all of it sits the Studio, where the work actually happens. We own every layer, and we build it as one system.

That vertical integration is the whole point. It's what lets a placement decision in the scheduler reflect a cabling decision in the datacenter, and what lets a topology choice in the hypervisor show up as faster work for the person, or the agent, at the keyboard. Most companies get to work on one slice of this. Here, the same team reasons about all of it at once.

And you don't have to walk in knowing all of it. None of us did. We learned the adjacent layers from the people who owned them, in production, with a real fleet as the classroom. What we look for isn't someone who has already mastered the whole stack, it's someone curious enough to want to, and unafraid to read a kernel flag, a scheduler, or an InfiniBand fabric they've never seen before. If that's you, the rest is learnable, and you'll have the people who built each layer sitting right next to you.

Small team, enormous leverage, real metal, all the way up to the best coding environment for humans and agents.

Come build the AI-native cloud with us →