Skip to content
Khalisa
All posts

6 min read

My confusing relationship with SR-IOV

A journey to understand SR-IOV and how it enables virtualization of PCIe devices like GPUs.


So couple of weeks ago, SR-IOV was briefly mentioned in my class lecture. Instinctively, I know what it is, basically a way to virtualize a physical PCIe device into multiple virtual ones. But I didn't know how it actually works underneath it, today I'm going to dive deep into it and consolidate my understanding of it.

PS: Not really a blog post, more like a note to myself :)

Background

How does cloud company profit more? By dividing one physical resource for multiple people or tenants, so that they can sell more to customers with less resources. Like buying 128 GB memory server and selling 32 GB memory VM to 4 different customers and they can even do overcommitment, sell it to 6 customers or more. Same goes for CPU.

But what about PCIe devices? Like GPU, NIC, etc. They are physical devices, but they can't be divided into multiple virtual devices as easily as CPU and memory.

About PCIe devices

Before moving on to SR-IOV, I need to understand how PCIe devices work first. It's a pretty complex topic, but I will only write SR-IOV related things here. You can read this PCIe article by Ripan Deuri for a very thorough but easily understandable explanation of PCIe.

A PCIe endpoint is a device function at the end of a PCIe connection that provides some functionality to the system. A NIC, for example, exposes functionality for sending and receiving network packets. An NVMe SSD exposes functionality for reading and writing storage.

The operating system discovers PCIe functions, configures them, and loads the appropriate device drivers.

That is essentially the challenge in trying to divide a physical PCIe device into multiple virtual devices. The operating system needs to be able to discover the functions. But what if the functions are not exposed to the operating system? Then the operating system can't discover them.

One important detail is that a physical device and a PCIe function are not necessarily the same thing. A single physical device can expose multiple functions. This distinction becomes important later when discussing SR-IOV.

How VMs access PCIe devices without SR-IOV

I'm using GPU here as an example because I want to explore more about it, but there are multiple examples with NIC and others that you can find online.

Now, imagine a cloud provider has a physical server with one GPU, and multiple customers want to use it for their workloads. How can the provider let multiple VMs access the same GPU? There are several approaches, each with different trade-offs.

1. Software managed

One approach is to provide a virtual GPU interface and let a software layer manage access to the physical GPU. Conceptually, it looks something like this:

Software managed GPU virtualization diagramSoftware managed GPU virtualization diagram

Each VM sees a GPU interface, but the virtualization layer coordinates access to the physical hardware. Depending on the implementation, GPU commands may be scheduled between workloads, and GPU memory may be managed or partitioned. The virtualization layer also needs to ensure that one VM cannot access another VM's resources.

This approach can provide flexibility, but it requires support from the GPU hardware, drivers, and virtualization software. It can also introduce overhead.

2. Passthrough

We can use PCI passthrough to assign the physical GPU to a VM. Essentially, the VM gets direct access to the physical GPU, bypassing the hypervisor. But then, other VMs cannot use the GPU at the same time, which is not ideal for a cloud provider. GPUs are expensive, and dedicating an entire GPU to a single customer, which may not be utilized 100% by the customer can be a profit loss.

3. Hardware supported sharing

Some GPUs support hardware partitioning or virtualization features that make this possible. For example, NVIDIA Multi-Instance GPU (MIG), on supported GPUs, partitions a physical GPU into multiple instances with dedicated compute and memory resources.

However, these approaches are not universally available across GPUs. The hardware and software must explicitly support the desired sharing model.

Now with SR-IOV

If PCI passthrough can provide direct access to a GPU, and hardware partitioning can let multiple workloads share one, what mechanism allows a compatible PCIe device to expose multiple hardware-backed functions?

SR-IOV provides a standardized PCIe mechanism for exposing multiple Virtual Functions (VFs) from a Physical Function (PF). Compatible devices can use these functions to let multiple VMs access device functionality more directly, without routing every operation through the host's ordinary software-mediated device path.

GPU partitioning and SR-IOV are not synonymous. MIG is an example of hardware partitioning, while SR-IOV defines how PCIe functions are exposed

With SR-IOV, let's say a physical GPU exposes 4 virtual functions. Each VM can be assigned one of these virtual functions. The operating system in each VM can discover the virtual function, load the appropriate driver, and interact with the GPU as if it were a dedicated device. The hypervisor and host manage the mapping between them.

We can think of function like this:

A function in this case is telling the OS "I'm GPU" So having 10 virtual functions means the OS can discover 10 GPUs, and each VM can use one of them.

It's not literally like that, but it's a good way to think about it.

SR-IOV is not a universal solution for GPU virtualization. A GPU must explicitly support SR-IOV, and its VFs must be supported by the relevant driver. But since I'm trying to understanding SR-IOV, I will focus on it.

Why SSD doesn't need SR-IOV?

This was one of the questions I had while learning about PCIe virtualization.

If an SSD is also a PCIe device, why don't we always need SR-IOV to share it between multiple VMs? The answer is that sharing a device and virtualizing its PCIe functions are two different things. Consider how storage is commonly virtualized. A hypervisor can create virtual disks backed by a physical SSD or a storage pool. Each VM sees its own virtual disk, while the host or storage layer manages access to the underlying device.

The VMs do not necessarily need direct access to the physical SSD. The host can mediate their read and write requests, enforce storage limits, and manage which parts of the storage are accessible to each VM. This is different from giving a VM direct access to an NVMe device or controller function.

So the important question is not whether the device uses PCIe. It is whether we need to expose multiple hardware-backed functions, or whether software can provide the required sharing and isolation.

Conclusion

Now I pretty much understand why SR-IOV is needed for processing heavy devices like GPU and NIC. I also understand why "dividing" them is quite a hassle, as normally it only exposes one function or a few functions, and the operating system needs to be able to discover them. So SR-IOV is a standard that allows a physical device to expose multiple virtual functions, which can be read by the VM OS.

In the future, I may write about a seperate blog on DMA and IOMMU's role in SR-IOV