How does a GPU get
into your virtual machine?
You add one line to a flavor — pci_passthrough:alias=GPU:1 — and Nova reaches across six services to hand a real piece of hardware to a guest. There are four distinct ways this can happen. This is the complete journey, traced from the real Nova source on branch nectar/2024.1.
The cast of characters
Before any code, the single most important thing to grasp: there are four different ways a device reaches a guest, and they share parts of the pipeline but are not the same path. Get these straight and nothing later will confuse you.
Four models, one pipeline
All four start with a request and end with a device in the guest, but each takes a different route through Nova. Keep this map in mind as you read.
The classic path. A flavor's pci_passthrough:alias turns into InstancePCIRequests, matched against per-host PciDeviceStats pools — no Placement involved.
New in 2024.1 (opt-in). The same alias flow, but each device is also modelled as a Placement resource provider with a resource class. Placement returns a device→RP mapping the filter then honours.
For NVIDIA/Intel virtual GPUs. The host slices a physical GPU into mdev devices reported under the VGPU resource class. libvirt gets a <hostdev type=mdev> identified by UUID, not a PCI address.
For accel:device_profile. A separate OpenStack service, Cyborg, owns the device. Nova creates and binds ARQs, then reads back a PCI address to pass through as a normal hostdev.
Models 1 and 2 are the same request path with Placement bolted on. Model 3 (vGPU) and model 4 (Cyborg) branch off at spawn time into their own device-creation logic. Watch which one a screen is describing.
Meet the services (click each one)
These are the components a GPU/PCI request passes through. Click any box to learn its job.
The journey in one breath
Here is the legacy/Placement path compressed into four phases. The vGPU and Cyborg branches join in at scheduling and spawn. The rest of the course unpacks each one.
nova-api turns the flavor alias (and any SR-IOV ports) into InstancePCIRequest objects and stores them on the instance and RequestSpec.
The PciPassthroughFilter keeps only hosts whose PCI pools can satisfy the request; the NUMATopologyFilter enforces device locality.
The PCI tracker picks concrete PciDevice objects from its pools, marks them CLAIMED, then ALLOCATED once the build succeeds.
The driver turns each device's PCI address (or mdev UUID, or ARQ attach handle) into a <hostdev> element in the guest XML.
Config & request parsing
The legacy alias flow. The operator whitelists assignable devices in nova/pci/whitelist.py and defines named aliases; nova-api translates the flavor's alias into requests.
device_spec: what is even assignable?
The operator whitelists assignable devices with [pci]device_spec (the legacy name was passthrough_whitelist). Each JSON entry becomes a PciDeviceSpec. A host device is only usable if it matches at least one spec.
def device_assignable(self, dev: ty.Dict[str, ty.Any]) -> bool:
"""Check if a device can be assigned to a guest.
:param dev: A dictionary describing the device properties
"""
for spec in self.specs:
if spec.match(dev):
return True
return False
nova/pci/whitelist.py · around line 85
alias → InstancePCIRequest
_get_alias_from_config() validates each [pci]alias against _ALIAS_SCHEMA and ORs together aliases with the same name. _translate_alias_to_requests then expands alias_name:count into request objects, each carrying the spec, the count, the alias name and the numa_policy.
pci_requests.append(objects.InstancePCIRequest(
count=int(count),
spec=spec,
alias_name=name,
numa_policy=policy,
request_id=uuidutils.generate_uuid(),
))
InstancePCIRequest with a unique request_id, a device count, and the matching device spec.
nova/pci/request.py · around line 189
Two sources, one list
get_pci_requests_from_flavor() reads the pci_passthrough:alias extra spec. But the flavor is not the only source: nova-api also calls network_api.create_resource_requests, which appends one SR-IOV InstancePCIRequest per Neutron port (VNIC types direct / direct-physical / vdpa).
pci_request_info = pci_request.get_pci_requests_from_flavor(
flavor, affinity_policy=pci_numa_affinity_policy)
result = self.network_api.create_resource_requests(
context, requested_networks, pci_request_info,
affinity_policy=pci_numa_affinity_policy)
InstancePCIRequests list stored on the instance and the RequestSpec.
nova/compute/api.py · around line 1118
SR-IOV ports produce InstancePCIRequests with source = NEUTRON_PORT. They share the claim machinery with the alias flow but are treated separately in PCI-in-Placement (skipped there) and tagged with a physical_network.
$ openstack flavor show $FLAVOR -c properties → pci_passthrough:alias='GPU:1'
compute$ grep -A5 '^\[pci\]' /etc/nova/nova.conf → device_spec (whitelist) + alias 'GPU' defined?
# the parsed pci_requests are frozen into the RequestSpec (API DB, JSON blob)
mysql> SELECT spec FROM nova_api.request_specs WHERE instance_uuid='$UUID' \G # grep for "pci_requests" and your alias name
# alias in flavor but not in nova.conf on the API node → 400 at boot time;
# empty pci_requests here means the API never translated the alias — fix config, reboot the request
PCI-in-Placement modelling
Optional, new in 2024.1. When [pci]report_in_placement is on, each assignable device also becomes a Placement resource provider — so Placement can do the matching, not just per-host stats.
When does this apply?
This is model 2, and it only runs when the operator sets [pci]report_in_placement. The compute node's update_provider_tree reports each device: standard / SRIOV_PF devices become a parent RP, while SRIOV_VF / VDPA devices become children of their PF's RP. Every RP gets the COMPUTE_MANAGED_PCI_DEVICE trait. Devices tagged physical_network (Neutron SR-IOV) are intentionally skipped.
The request parsing from Module 1 is unchanged. PCI-in-Placement adds a parallel accounting view in Placement so the scheduler gets a precise device→provider mapping instead of relying only on summarised host stats.
Resource class naming
Each device pool advertises inventory under a resource class. get_resource_class uses the operator-supplied resource_class tag if present, otherwise it synthesises CUSTOM_PCI_<vendor>_<product>.
def get_resource_class(
requested_name: ty.Optional[str], vendor_id: str, product_id: str
) -> str:
if requested_name:
rc = _normalize_resource_class(requested_name)
else:
rc = f"CUSTOM_PCI_{vendor_id}_{product_id}".upper()
return rc
CUSTOM_PCI_10DE_1EB8).
nova/compute/pci_placement_translator.py · around line 99
Building the provider tree
PciResourceProvider.update_provider_tree creates the child RP (with a Nova-generated UUID), sets inventory total and max_unit to the device count, applies traits, and records the RP UUID back onto each PciDevice.extra_info['rp_uuid'] so the claim can later honour the mapping.
provider_tree.update_inventory(
self.name,
{
self.resource_class: {
"total": len(self.devs),
"max_unit": len(self.devs),
}
},
)
PciDevice maps to via rp_uuid.
nova/compute/pci_placement_translator.py · around line 246
$ openstack resource provider list --in-tree $COMPUTE_RP_UUID
| qh2-rcc123 | qh2-rcc123_0000:3B:00.0 | … |
$ openstack resource provider inventory list $PCI_RP_UUID
| CUSTOM_PCI_10DE_1EB8 | total=1 | max_unit=1 |
$ openstack resource provider trait list $PCI_RP_UUID → COMPUTE_MANAGED_PCI_DEVICE
# the tracker's own ledger, in the cell DB — one row per whitelisted device
mysql> SELECT address, status, dev_type, vendor_id, product_id FROM nova.pci_devices WHERE compute_node_id=$CN_ID AND deleted=0;
# no RP for a device you whitelisted → the compute never reported it: check nova-compute's
# update_provider_tree log lines and that [pci]report_in_placement is on for that host
Scheduling: filtering & NUMA
The RequestSpec carries the pci_requests. The scheduler keeps only hosts that can actually provide the devices — and, when the guest has a NUMA topology, on the right NUMA node.
PCI requests → Placement RequestGroups
Only when [filter_scheduler]pci_in_placement is set, generate_request_groups_from_pci_requests turns each flavor-based request into one RequestGroup per requested device. (Neutron-port requests are skipped here — they are handled separately.) One group per device means VFs can be placed on different PFs.
for i in range(pci_request.count):
rg = objects.RequestGroup(
use_same_provider=True,
requester_id=f"{pci_request.request_id}-{i}",
resources={
self._rc_from_request(spec): 1
},
required_traits=self._traits_from_request(spec),
)
self.requested_resources.append(rg)
count=2 request becomes two single-device groups, so Placement can place each device on a possibly different physical provider.
nova/objects/request_spec.py · around line 572
The PciPassthroughFilter
For each surviving allocation candidate, the filter asks the host's pci_stats whether it can satisfy the requests under that candidate's RP mapping. Pass = keep the host.
good_candidates = self.filter_candidates(
host_state,
lambda candidate: host_state.pci_stats.support_requests(
pci_requests.requests, provider_mapping=candidate["mappings"]
),
)
nova/scheduler/filters/pci_passthrough_filter.py · around line 60
support_requests is a dry run
Crucially, filtering does not consume anything. support_requests deep-copies the stats and tries apply_requests on the copy; success simply means "schedulable". The real consumption happens later, during the claim on the chosen host.
stats = copy.deepcopy(self)
try:
stats.apply_requests(requests, provider_mapping, numa_cells)
except exception.PciDeviceRequestFailed:
return False
return True
consume_requests during the claim.
nova/pci/stats.py · around line 784
NUMA-aware fit
PCI devices have a numa_node. When the guest has a NUMA topology, the NUMATopologyFilter combines CPU/memory fitting with PCI placement in one call to hardware.numa_fit_instance_to_host, passing the pci_requests and host pci_stats together.
lambda candidate: hardware.numa_fit_instance_to_host(
...
pci_requests=pci_requests,
pci_stats=host_state.pci_stats,
)
request.numa_policy.
nova/scheduler/filters/numa_topology_filter.py · around line 105
$ openstack allocation candidate list --resource CUSTOM_PCI_10DE_1EB8=1
# the Nectar branch logs each filter's surviving hosts at INFO — read the story per instance
$ grep $UUID nova-scheduler.log | grep 'Filter '
Filter PciPassthroughFilter returned 0 hosts … # ← the culprit names itself
# then check the candidate host's device ledger: free devices on the right NUMA node?
mysql> SELECT address, status, numa_node, instance_uuid FROM nova.pci_devices WHERE compute_node_id=$CN_ID AND deleted=0;
# NoValidHost with NO filter lines at all → placement returned zero candidates;
# the inventory (module 2) is missing or fully consumed — filtering never even ran
The resource-tracker claim
A host was chosen. Now, on that compute node, the PCI tracker turns abstract requests into concrete devices and reserves them. This is where the device stops being a number and becomes a real BDF.
The claim call
Inside ResourceTracker.instance_claim, while holding the host's resource semaphore, Nova hands the instance's pci_requests and its claimed NUMA topology to the PCI tracker.
if self.pci_tracker:
self.pci_tracker.claim_instance(context, pci_requests,
instance_numa_topology)
nova/compute/resource_tracker.py · around line 201
consume_requests picks the devices
consume_requests filters the pools per request. With no Placement mapping it greedily consumes; with a mapping it follows the per-RP allocation exactly. If a request can't be met, it returns any partially-allocated devices to their pools and raises PciDeviceRequestFailed.
if not pools:
...
for d in range(len(alloc_devices)):
self.add_device(alloc_devices.pop())
raise exception.PciDeviceRequestFailed(requests=pci_requests)
nova/pci/stats.py · around line 287
claim → allocate lifecycle
Claimed devices live in self.claims[uuid]. Once the instance is actually built, allocate_instance moves them into self.allocations and calls dev.allocate(instance). The device's status walks through a fixed lifecycle.
def allocate_instance(self, instance: 'objects.Instance') -> None:
devs = self.claims.pop(instance['uuid'], [])
self._allocate_instance(instance, devs)
if devs:
self.allocations[instance['uuid']] += devs
free_instance returns the devices to their pools and they go back to AVAILABLE.
nova/pci/manager.py · around line 346
| 0000:3B:00.0 | allocated | $UUID | # available → claimed → allocated as the build lands
# the same walk, from the instance's side
mysql> SELECT address, status FROM nova.pci_devices WHERE instance_uuid='$UUID' AND deleted=0;
# device parked at 'claimed' but the boot failed long ago → the claim was never rolled back;
# nova-compute's update_available_resource periodic reconciles it on its next pass —
# if it doesn't, restart nova-compute on that host and re-check before touching the DB
libvirt: the hostdev
The device is claimed. Now the libvirt driver turns its PCI address into a <hostdev> element — the actual instruction QEMU and VFIO use to pass the hardware through.
address → hostdev
The driver iterates instance.get_pci_devices(source=FLAVOR_ALIAS). For each one, _get_guest_pci_device parses the stored BDF string into a 4-tuple via pci_utils.parse_address and builds the hostdev config.
def _get_guest_pci_device(self, pci_device):
dbsf = pci_utils.parse_address(pci_device.address)
dev = vconfig.LibvirtConfigGuestHostdevPCI()
dev.domain, dev.bus, dev.slot, dev.function = dbsf
self._set_managed_mode(dev)
return dev
<hostdev type=pci> element carrying its exact PCI address (domain, bus, slot, function).
nova/virt/libvirt/driver.py · around line 6012
The XML
LibvirtConfigGuestHostdevPCI.format_dom emits the <address> source element, ensuring each of domain / bus / slot / function is 0x-prefixed hexadecimal — the form libvirt and QEMU expect.
def format_dom(self):
dev = super(LibvirtConfigGuestHostdevPCI, self).format_dom()
address = etree.Element(
"address",
domain=self.domain if self.domain.startswith('0x')
else '0x' + self.domain,
bus=self.bus if self.bus.startswith('0x') else '0x' + self.bus,
...
nova/virt/libvirt/config.py · around line 2382
For models 1 and 2 the story ends here: the guest boots with the real device on its PCI bus. The next module covers the two paths that don't use a plain PCI address — vGPU and Cyborg.
compute$ virsh dumpxml instance-000xxxxx | grep -A4 'hostdev'
<address domain='0x0000' bus='0x3b' slot='0x00' function='0x0'/>
# on the host, the device must be bound to vfio-pci, not the vendor driver
compute$ lspci -k -s 3b:00.0 → Kernel driver in use: vfio-pci
# PCI-in-Placement: the instance allocation includes the device line
$ openstack resource provider allocation show $UUID → …, 'CUSTOM_PCI_10DE_1EB8': 1
# inside the guest: lspci | grep -i nvidia
The two other paths: vGPU & Cyborg
Models 3 and 4. Both end at a <hostdev>, but they reach it differently: vGPU through mediated devices and a UUID, Cyborg through a separate REST service and an ARQ.
vGPU / mdev — slicing a GPU
Operators set [devices]enabled_mdev_types (deprecated alias enabled_vgpu_types) plus per-type [mdev_<type>] sections. vGPUs report to Placement under the VGPU resource class on per-pGPU child RPs. At spawn, _vgpu_allocations filters the instance's allocations to mdev classes, _allocate_mdevs reuses an unassigned mdev or creates one, and _guest_add_mdevs adds the device by UUID.
def _guest_add_mdevs(self, guest, chosen_mdevs):
for chosen_mdev in chosen_mdevs:
mdev = vconfig.LibvirtConfigGuestHostdevMDEV()
mdev.uuid = chosen_mdev
guest.add_device(mdev)
VGPU allocation maps to a concrete mdev that is plugged into the guest as a <hostdev type=mdev> identified by UUID, not a PCI address.
nova/virt/libvirt/driver.py · around line 7536
A vGPU isn't a whole PCI device — it's a mediated slice. So there is no BDF to pass through; libvirt is given an mdev UUID and the model vfio-pci instead.
$ openstack resource provider list --in-tree $COMPUTE_RP_UUID | grep -i pgpu
$ openstack resource provider usage show $PGPU_RP_UUID → VGPU used/total
# the host: which mdevs exist, and which are attached to guests
compute$ virsh nodedev-list --cap mdev
compute$ mdevctl list → uuid, parent pGPU, type (e.g. nvidia-233)
# guest XML: hostdev type='mdev' with the UUID, not a BDF
compute$ virsh dumpxml instance-000xxxxx | grep -B2 -A3 mdev
# placement says free but boots fail with no candidates → enabled_mdev_types /
# [mdev_<type>] config drifted from what the driver actually created; compare mdevctl to nova.conf
Cyborg — device profile → request groups
For accel:device_profile=<name>, the conductor calls cyborg.get_device_profile_request_groups to turn the profile into Placement RequestGroups before scheduling.
def get_device_profile_request_groups(context, dp_name, owner=None):
cyclient = get_client(context)
dp_groups = cyclient.get_device_profile_groups(dp_name)
return cyclient.get_device_request_groups(dp_groups, owner)
nova/accelerator/cyborg.py · around line 90
Cyborg — ARQ attach handle → hostdev
After scheduling, the conductor calls create_arqs_and_match_resource_providers (POST) and bind_arqs (PATCH). Compute waits for accelerator-request-bound events, reads the resolved ARQs via get_arqs_for_instance, and passes accel_info to driver.spawn. The bound ARQ's attach_handle_info PCI address becomes a hostdev.
def _guest_add_accel_pci_devices(self, guest, accel_info):
for arq in accel_info:
dev = vconfig.LibvirtConfigGuestHostdevPCI()
pci_addr = arq['attach_handle_info']
dev.domain, dev.bus, dev.slot, dev.function = (
pci_addr['domain'], pci_addr['bus'],
pci_addr['device'], pci_addr['function'])
nova/virt/libvirt/driver.py · around line 7598
Unlike models 1–3, Nova does not own the device here. Cyborg does. Nova asks for it (ARQ), binds it, reads back where it landed, and passes it through. On delete, delete_arqs_if_needed calls delete_arqs_for_instance (DELETE) to give it back.
The full journey, animated
Press Next step to watch a request travel the entire path. Watch the labels for CAST, CALL, REST and LIBRARY — and notice where the vGPU and Cyborg branches join in.
CAST = fire-and-forget RPC on the queue. CALL = an in-process or blocking RPC call. REST = HTTP to another project (Placement, Cyborg, Neutron). LIBRARY = a local call into libvirt/VFIO. The Cyborg (CY) and vGPU (libvirt) branches only fire for those two models.
When it goes wrong
PCI passthrough has its own family of failures, spread across config validation, scheduling, the claim, and the Placement report.
Misconfigured aliases
Two failures fire during request parsing, before any host is even considered.
The flavor references an alias name that does not exist in [pci]alias. Typo in the flavor, or alias not configured on the API node.
Two same-named aliases disagree on device_type or numa_policy, or an alias fails the JSON schema.
No device that fits
The most common runtime failure. PciDeviceRequestFailed is raised when consume_requests / apply_requests cannot find enough matching, NUMA-affined, or RP-matched devices.
At filter time this makes support_requests return False and the host is rejected. At claim time it aborts the build on the chosen host. Same exception, two very different moments. (nova/pci/stats.py L296, L317)
NUMA affinity not satisfiable
If the instance has a NUMA topology but a device with numa_node is None is assigned, the PCI tracker logs a warning. Under the legacy policy this is not fatal — but with required policy it becomes a hard failure during fitting.
Logged from nova/pci/manager.py L318. The numa_policy on the request (legacy / preferred / required / socket) decides whether a missing-affinity device is tolerated or rejected.
The Placement report rejects the config
Only in PCI-in-Placement mode. If VFs of one PF disagree on resource_class or traits, or a devname is used while report_in_placement is on, the compute node's report fails outright.
Raised from nova/compute/pci_placement_translator.py. The fix is operator-side: make the device_spec entries for one PF's VFs consistent, and avoid devname matching in Placement mode.
Check yourself
Three questions and a matching exercise. If these click, you can tell the four models apart and trace any one of them.
Quiz
[pci]report_in_placement enabled, what resource class does a device get if the operator did NOT set a resource_class tag?PciDeviceStats.support_requests actually do?Match the code to its job
Drag each Nova symbol onto the job it performs in the GPU/PCI flow.
From a flavor alias to a <hostdev> in the guest XML: parsing in the API, optional modelling in Placement, filtering in the scheduler, a concrete claim by the PCI tracker, and a libvirt spawn — plus the vGPU and Cyborg branches. Four models, one pipeline.