How does the scheduler
pick a host?
The conductor asks nova-scheduler to find a home for your instance. The scheduler mutates the request, queries Placement for candidates, builds in-memory host objects, runs filters then weighers, claims the resources on the winner, and hands back one or more Selection objects. This is that procedure, traced from real Nova source.
The cast of characters
Picking a host is a pipeline. Before we trace it, meet the four core ideas and the players that carry them out.
Four ideas that run the whole thing
The entire procedure is built on four concepts. Hold these in your head and the rest is just detail.
Placement is the resource accountant. Given a list of needed resources, it returns every allocation candidate — a provider tree with room — plus a provider summary for each.
Each filter answers yes or no for every host. A host that fails any filter is dropped. What survives is the set of acceptable hosts. No scores, just gates.
Each weigher gives every surviving host a number. The numbers are normalized, multiplied by a multiplier and summed, then hosts sort best-to-worst. This is ranking, not gating.
Ranking is not enough — two requests might like the same host. The scheduler claims resources in Placement (PUT /allocations) on the best host. The first claim to succeed wins; a generation conflict sends the loser to the next host.
The canonical order never changes: request_filters → Placement candidates → host states → filters → weighers → claim → alternates. Most confusion comes from mixing up the request filters (before Placement) with the in-process filters (after Placement).
Meet the players (click each one)
These are the components your scheduling request flows through. Click any box to learn its job.
The procedure in one breath
Here is the canonical order, compressed. The rest of the course unpacks each stage.
Before Placement, process_reqspec tightens the RequestSpec with required/forbidden traits and aggregates.
GET /allocation_candidates returns every provider tree with room, plus a provider summary for each.
report client.get_allocation_candidatesOnly the candidate provider UUIDs are loaded into in-memory HostState objects across enabled cells.
host_manager.get_host_states_by_uuidsEach enabled filter is a pass/fail gate; the survivor list shrinks filter by filter.
host_manager.get_filtered_hostsSurvivors are scored, normalized, summed and sorted best-to-worst; the top subset is shuffled.
host_manager.get_weighed_hostsThe first host whose PUT /allocations succeeds wins and is consumed locally for the next instance.
utils.claim_resources_get_alternate_hosts wraps each winner plus same-cell backups into Selection objects.
Phase 1 — The conductor hands off
The conductor does not pick hosts itself. It prepares the affinity state, then makes the single synchronous RPC call in the whole procedure.
The conductor calls the query client
_schedule_instances sets up the instance group, then calls select_destinations requesting objects and alternates. The result is a list of host lists, one per instance.
def _schedule_instances(self, context, request_spec,
instance_uuids=None, return_alternates=False):
scheduler_utils.setup_instance_group(context, request_spec)
with timeutils.StopWatch() as timer:
host_lists = self.query_client.select_destinations(
context, request_spec, instance_uuids, return_objects=True,
return_alternates=return_alternates)
return_objects=True asks for full Selection objects and return_alternates asks for backup hosts too. nova/conductor/manager.py ~938RPC version negotiation for select_destinations
The RPC client sends version 4.5 (Selection objects + alternates). If the deployment can only do older versions it downgrades and drops parameters. It CALLs select_destinations with a long timeout and waits for the reply.
def select_destinations(self, ctxt, spec_obj, instance_uuids,
return_objects=False, return_alternates=False):
# Modify the parameters if an older version is requested
version = '4.5'
msg_args = {'instance_uuids': instance_uuids,
'spec_obj': spec_obj,
'return_objects': return_objects,
'return_alternates': return_alternates}
SchedulerQueryClient (nova/scheduler/client/query.py) forwards to this RPC client. It defaults to the newest contract — Selection objects with alternates. nova/scheduler/rpcapi.py ~132This conductor→scheduler hop is the only blocking RPC CALL in the procedure. Everything the scheduler then does is in-process Python plus REST to Placement. The conductor simply waits for the list of Selections.
mysql> SELECT created_at, spec FROM nova_api.request_specs WHERE instance_uuid='$UUID' \G
# is a scheduler alive to answer the CALL at all?
$ openstack compute service list --service nova-scheduler → state up
# if the call never returns: MessagingTimeout in nova-conductor's log after long_rpc_timeout,
# then 'Failed to schedule instances' and the boot is buried in cell0 — start the diagnosis there
$ grep $UUID nova-conductor.log | grep -i 'schedul'
Phase 2 — Request filters mutate the spec
On the scheduler side, before talking to Placement, the manager runs the pre-placement request filters. They do not drop hosts — they tighten the RequestSpec.
Request filters run, then build the Placement query
select_destinations calls process_reqspec (only when this is not a rebuild), then resources_from_request_spec, then the Placement GET. A RequestFilterFailed becomes NoValidHost.
if not is_rebuild:
try:
request_filter.process_reqspec(context, spec_obj)
except exception.RequestFilterFailed as e:
raise exception.NoValidHost(reason=e.message)
resources = utils.resources_from_request_spec(
context, spec_obj, self.host_manager,
enable_pinning_translate=True)
res = self.placement_client.get_allocation_candidates(
context, resources)
The ordered request-filter chain
ALL_REQUEST_FILTERS is run in order by process_reqspec. Each is a small function that optionally tightens the request before it ever reaches Placement.
ALL_REQUEST_FILTERS = [
require_tenant_aggregate,
map_az_to_placement_aggregate,
require_image_type_support,
compute_status_filter,
isolate_aggregates,
transform_image_metadata,
accelerators_filter,
require_tenant_aggregate (tenant isolation), map_az_to_placement_aggregate (AZ → aggregate), and compute_status_filter (always forbids disabled computes). nova/scheduler/request_filter.py ~447compute_status_filter is mandatory
This filter unconditionally adds COMPUTE_STATUS_DISABLED to the spec's forbidden traits, so disabled compute services are excluded by Placement — mirroring the in-process ComputeFilter.
def compute_status_filter(ctxt, request_spec):
trait_name = os_traits.COMPUTE_STATUS_DISABLED
request_spec.root_forbidden.add(trait_name)
LOG.debug('compute_status_filter request filter added forbidden '
'trait %s', trait_name)
return True
Request filters add traits and aggregates to the spec so Placement does the filtering. The in-process filters (Phase 6) drop HostState objects after Placement answers. Same idea — applied at two different points.
Build the Placement query — ResourceRequest
resources_from_request_spec turns the mutated RequestSpec into a ResourceRequest: resource amounts, required/forbidden traits, and aggregate member_of constraints. It honors a forced host by adding an in_tree constraint.
if target_host or target_node:
nodes = host_manager.get_compute_nodes_by_host_or_node(
ctxt, target_host, target_node, cell=target_cell)
if not nodes:
reason = (_('No such host - host: %(host)s node: %(node)s ') %
{'host': target_host, 'node': target_node})
raise exception.NoValidHost(reason=reason)
in_tree constraint so Placement only returns candidates rooted there. nova/scheduler/utils.py ~715Render the GET /allocation_candidates querystring
to_querystring builds limit, group_policy, root_required (required traits plus !forbidden), same_subtree, and per-group resources[]/required[]/member_of[] params.
if self._root_required or self._root_forbidden:
vals = sorted(self._root_required) + ['!' + t for t in
sorted(self._root_forbidden)]
qparams.append(('root_required', ','.join(vals)))
for group_suffixes in self._same_subtree:
qparams.append(('same_subtree', ','.join(sorted(group_suffixes))))
!. This is the literal querystring sent to Placement. nova/scheduler/utils.py ~541$ grep $REQ_ID nova-scheduler.log | grep request_filter
require_tenant_aggregate … / compute_status_filter … added forbidden trait COMPUTE_STATUS_DISABLED
# the ground truth those filters rely on — nova aggregates mirrored into placement
$ openstack aggregate show $AGG -c properties → filter_tenant_id / availability_zone metadata
$ openstack resource provider aggregate list $COMPUTE_RP_UUID → is the host in the placement aggregate?
$ openstack resource provider trait list $COMPUTE_RP_UUID | grep COMPUTE_STATUS → disabled host?
# classic trap: filter_tenant_id set on the nova aggregate but the placement mirror is stale →
# the tenant's boots die with NoValidHost BEFORE any host filter ever runs
Phase 4 — Query Placement for candidates
The report client issues a single REST GET /allocation_candidates. No candidates means NoValidHost immediately, before any filtering.
GET /allocation_candidates
The client builds the URL from the querystring and parses the 200 body into two structures plus the microversion used: the claimable allocation_requests and the provider_summaries.
version = SAME_SUBTREE_VERSION
qparams = resources.to_querystring()
url = "/allocation_candidates?%s" % qparams
resp = self.get(url, version=version,
global_request_id=context.global_id)
if resp.status_code == 200:
data = resp.json()
return (data['allocation_requests'], data['provider_summaries'],
version)
Index by provider UUID; fail fast if empty
After merging any PCPU/VCPU fallback, the manager bails to NoValidHost if there are zero allocation requests, then builds a dict from RP UUID to its allocation requests for fast lookup at claim time.
if not alloc_reqs:
LOG.info(
"Got no allocation candidates from the Placement API. ...")
raise exception.NoValidHost(reason="")
alloc_reqs_by_rp_uuid = collections.defaultdict(list)
for ar in alloc_reqs:
for rp_uuid in ar['allocations']:
alloc_reqs_by_rp_uuid[rp_uuid].append(ar)
$ openstack allocation candidate list --resource VCPU=4 --resource MEMORY_MB=16384 --resource DISK_GB=30
(add the flavor's resources:* extra specs — e.g. --resource CUSTOM_RESERVATION_…=1)
# the fingerprint of an empty answer, in nova-scheduler.log:
"Got no allocation candidates from the Placement API" → capacity/traits/aggregates, NOT filters
# how much room a suspect host really has (mind allocation_ratio and reserved)
$ openstack resource provider usage show $COMPUTE_RP_UUID
$ openstack resource provider inventory list $COMPUTE_RP_UUID
Phase 5 — Build host states from candidates
The candidate provider UUIDs restrict which compute nodes are loaded into mutable HostState objects.
Restrict host states to candidate providers
Only providers Placement returned are loaded. The keys of provider_summaries are the candidate compute-node UUIDs; an empty set yields no hosts (eventually NoValidHost).
def _get_all_host_states(self, context, spec_obj, provider_summaries):
compute_uuids = None
if provider_summaries is not None:
compute_uuids = list(provider_summaries.keys())
return self.host_manager.get_host_states_by_uuids(
context, compute_uuids, spec_obj)
Attach allocation candidates to each host
_schedule wraps the HostState generator so each host gets a deep copy of its allocation requests (keyed earlier by RP UUID). This is what filters can inspect and what the claim step consumes.
def hosts_with_alloc_reqs(hosts_gen):
for host in hosts_gen:
host.allocation_candidates = copy.deepcopy(
alloc_reqs_by_rp_uuid[host.uuid])
yield host
hosts = self._get_all_host_states(
elevated, spec_obj, provider_summaries)
Phases 6 & 7 — Filter, then weigh
Per instance, _get_sorted_hosts first gates the hosts through the ordered filters, then scores and ranks the survivors with the weighers.
The pipeline order, so far
Two stages, in this fixed order. Filters first (cheap pass/fail), then weighers (scoring) on whatever survives.
Filter chain — each filter narrows the list
get_filtered_objects runs filters in order; filter_all yields the passing hosts; if a filter returns zero hosts it breaks and the request fails. Hosts pass through each filter like a sieve.
objs = filter_.filter_all(list_objs, spec_obj)
if objs is None:
LOG.debug("Filter %s says to stop filtering", cls_name)
return
list_objs = list(objs)
end_count = len(list_objs)
...
else:
LOG.info("Filter %s returned 0 hosts", cls_name)
get_filtered_hosts also handles ignore_hosts/force_hosts/force_nodes/requested_destination. nova/filters.py ~89The default enabled_filters
The shipped default ordered list. The advice is to place the most restrictive filters first for efficiency.
default=[
"ComputeFilter",
"ComputeCapabilitiesFilter",
"ImagePropertiesFilter",
"ServerGroupAntiAffinityFilter",
"ServerGroupAffinityFilter",
],
Weigh: normalize, multiply, sum
get_weighed_objects normalizes each weigher's raw weights, multiplies by the weigher multiplier, accumulates onto each object, and returns descending-sorted WeighedObjects.
for i, weight in enumerate(weights):
obj = weighed_objs[i]
multiplier = weigher.weight_multiplier(obj.obj)
weigher_score = multiplier * weight
obj.weight += weigher_score
...
return sorted(weighed_objs, key=lambda x: x.weight, reverse=True)
host_subset_size randomization
After sorting, the top host_subset_size hosts are shuffled so the same host is not always chosen for identical requests.
host_subset_size = CONF.filter_scheduler.host_subset_size
if host_subset_size < len(weighed_hosts):
weighed_subset = weighed_hosts[0:host_subset_size]
else:
weighed_subset = weighed_hosts
random.shuffle(weighed_subset)
$ grep $UUID nova-scheduler.log | grep 'Filter '
Filter ComputeFilter returned 5 hosts […]
Filter AggregateInstanceExtraSpecsFilter returned 2 hosts […]
Filter NUMATopologyFilter returned 0 hosts # ← the culprit names itself
# the final verdict line, plus the weighing order at DEBUG
"Filtering removed all hosts …" / "Weighed [WeighedHost …]"
# filter lines present = placement DID return candidates; the failure is host-side —
# no filter lines at all = go back one module: placement returned nothing
Phase 8 — Claim, then build Selections
Ranking is not commitment. The scheduler walks the sorted hosts and claims resources in Placement; the first claim that succeeds wins the host.
Claim the first candidate that succeeds
For each sorted host, the manager tries the host's first allocation request. The first successful PUT /allocations wins and becomes claimed_host.
alloc_req = host.allocation_candidates[0]
if utils.claim_resources(
elevated, self.placement_client, spec_obj, instance_uuid,
alloc_req,
allocation_request_version=allocation_request_version,
):
claimed_host = host
break
PUT /allocations with generation-conflict handling
The report client PUTs the allocation. A 204 means success. A placement.concurrent_update error is either a consumer-generation conflict (a hard AllocationUpdateFailed) or a provider-generation race (a Retry handled locally).
if r.status_code != 204:
err = r.json()['errors'][0]
if err['code'] == 'placement.concurrent_update':
if 'consumer generation conflict' in err['detail']:
...
raise exception.AllocationUpdateFailed(
consumer_uuid=consumer_uuid, error=reason)
...
raise Retry('claim_resources', reason)
Consume locally so the next instance sees the change
After a successful claim, _consume_selected_host subtracts the instance's resources from the HostState and appends the host to the server group, so subsequent instances in a multi-create are scheduled against updated state.
@staticmethod
def _consume_selected_host(selected_host, spec_obj, instance_uuid=None):
LOG.debug("Selected host: %(host)s", {'host': selected_host},
instance_uuid=instance_uuid)
selected_host.consume_from_request(spec_obj)
if spec_obj.instance_group is not None:
spec_obj.instance_group.hosts.append(selected_host.host)
Build Selection objects — chosen + alternates
After all instances are claimed (and _ensure_sufficient_hosts passes), _get_alternate_hosts turns each claimed host into a primary Selection and adds same-cell, unclaimed hosts as alternates, up to max_attempts - 1.
selection = objects.Selection.from_host_state(
selected_host, allocation_request=selected_alloc_req,
allocation_request_version=allocation_request_version)
selected_plus_alts = [selection]
cell_uuid = selected_host.cell_uuid
$ openstack resource provider allocation show $UUID
| $RP_UUID (the winning node) | {'VCPU': 4, 'MEMORY_MB': 16384, 'DISK_GB': 30} |
# claim races leave 'placement.concurrent_update' in the placement API log —
# provider-generation races are retried; a consumer conflict is fatal
# (AllocationUpdateFailed in nova-scheduler.log)
# instance ended ERROR but the allocation is still there → the build failed after the claim;
# find and fix leaks:
$ nova-manage placement audit --verbose
Filter & weigher reference
The default gates and scorers, each with what it checks or optimizes. Filters decide whether a host qualifies; weighers decide how good it is.
Default enabled filters (run in order)
These ship enabled out of the box, run in this order, in nova/scheduler/filters/.
Passes only hosts whose compute service is enabled and up (alive). RUN_ON_REBUILD = False.
Host capabilities must satisfy the flavor's capabilities:-namespaced extra specs (_satisfies_extra_specs).
Host's supported_instances (architecture, hypervisor type, vm_mode) must match the image's hw_architecture/img_hv_type/hw_vm_mode.
When the group policy is anti-affinity, rejects hosts already running a group member (honoring max_server_per_host).
When the group policy is affinity, keeps only hosts already hosting group members.
NUMATopologyFilter, PciPassthroughFilter, AggregateImagePropertiesIsolation, AggregateInstanceExtraSpecsFilter, NectarAggregateMultiTenancyIsolation, NumInstancesFilter, IoOpsFilter, IsolatedHostsFilter, JsonFilter, MetricsFilter, AggregateTypeAffinityFilter, AllHostsFilter, ProjectTagsFilter, RestrictedZoneFilter.
Default weighers (all loaded)
Each computes a raw weight per host; values are normalized to 0..1 across survivors, scaled by the weigher's multiplier, and summed. From nova/scheduler/weights/.
Returns free_ram_mb; positive multiplier means spread (default 1.0). ram.py
Based on free vCPUs; positive multiplier means spread. cpu.py
Based on free disk; positive multiplier means spread. disk.py
Weighs by num_io_ops; default multiplier is negative to avoid busy hosts. io_ops.py
Returns failed_builds; multiplier is negated, so hosts with recent build failures are penalized. compute.py
Prefers higher hypervisor versions. hypervisor_version.py
Biases PCI-device requests toward or away from PCI-heavy hosts. pci.py
Soft, best-effort (anti-)affinity preferences. affinity.py
Prefers (or avoids) hosts in the instance's current cell for moves. cross_cell.py
Weighs by configured compute metrics. metrics.py
Weighs by the instance count on the host. num_instances.py
In nova/weights.py normalize, each weigher's raw values map to 0..1 across the surviving set (all-equal ⇒ all 0), then score = multiplier * normalized_weight is summed across weighers.
Aggregate isolation — who is confined to whom?
Filters and weighers decide which surviving host wins. But several aggregate-driven filters decide which hosts are even eligible for a given flavor, image, or project. Each enforces only one direction, and the direction depends entirely on which collection the filter loops over.
Read every relationship as a single arrow: A → B means "A may only be placed on B". A "reserved" host is really two arrows pointing at each other.
If it loops over the request (flavor extra_specs / image traits), the request becomes a demand — that confines the workload to matching hosts. If it loops over the host's aggregate metadata, the host becomes the gate — that reserves the host. Same data, opposite direction.
The routing board — who lands where?
Pick a filter below. Green solid = the boot is allowed there; red dashed = blocked. The matrix gives the exact reason per cell.
| boot ↓ target → | Aggregate A (reserved) | Aggregate B (allow-all) |
|---|---|---|
| Subject A | ||
| Subject C | ||
| Subject B |
Flavor and Project each have a matched pair of opposite-direction filters — combine them for a two-way lock with plain filters (try "Both combined" on the board). Image has no such pair: its only plain filter is conflict-only, enforcing nothing hard. That gap is why image isolation needs traits (bug 1677217).
Why the image filter does not mirror the flavor filter
The flavor filter loops the flavor, so each spec is a demand — a host lacking the key fails, and the flavor is confined.
The image filter loops the host's metadata, so the image property is passive: it only fails on an active value conflict. A missing property never rejects, and an untagged host has nothing to check — so the image is not confined. That is the asymmetry.
Decision table
| Goal | Use | Image? | [scheduler] toggle? |
|---|---|---|---|
| flavor + image ↔ host (bidirectional) | isolated_aggregate_filtering (traits) | Yes | Yes — enable_isolated_aggregate_filtering |
| flavor ↔ host, no toggle | ExtraSpecs + TypeAffinity | No | No (both in enabled_filters) |
| project ↔ host | require_tenant_aggregate + NectarAggregateMultiTenancyIsolation | n/a | Yes — limit_tenants_to_placement_aggregate |
| image ↔ host (beyond conflict) | only isolated_aggregate_filtering | Yes | Yes |
The full journey, animated
Press Next step to watch the scheduling procedure run end to end. Notice the single blocking RPC CALL (conductor → scheduler); everything else is in-process Python or REST to Placement.
There is exactly one blocking RPC CALL in the procedure: conductor → scheduler (steps 1–2), with its reply at step 23. Between them the scheduler runs in-process Python plus a handful of REST round-trips to Placement. That's what keeps scheduling fast and scalable.
When it goes wrong
The procedure has a clear contract for failure: either there is no host (NoValidHost), or a claim races (retry or hard fail), and any partial claims are rolled back.
NoValidHost — no candidates
Placement returns zero allocation_requests, or a request filter raised RequestFilterFailed. The manager raises NoValidHost before any filtering or claiming.
A RequestFilterFailed is converted straight to NoValidHost before the Placement query.
If alloc_reqs is empty, NoValidHost is raised immediately — no host could fit the resources/traits.
NoValidHost — filters or claims exhausted
Filters remove every host, or no host's claim succeeds for an instance. _ensure_sufficient_hosts finds fewer claimed hosts than required, calls _cleanup_allocations to roll back prior claims, and raises NoValidHost.
If a multi-create gets partway — say two of three instances claimed a host — and then runs out of hosts, the scheduler does not leave dangling reservations. _cleanup_allocations releases the already-claimed allocations before raising. nova/scheduler/manager.py ~502
Claim conflict — retry vs hard fail
When the PUT /allocations returns placement.concurrent_update, the report client splits the two cases.
Two requests raced for the same provider; the loser's generation changed. The client raises Retry and the claim is retried locally against fresh state.
The consumer's generation changed between the GET and the PUT. The instance is in an inconsistent state, so AllocationUpdateFailed is treated as a hard error and propagated.
A provider-generation race just means someone else changed that provider's inventory; re-reading and re-claiming is harmless. A consumer-generation conflict means this instance's own allocations changed unexpectedly — that's not safe to silently retry.
Check yourself
Three questions and a matching exercise. If these click, you understand how the scheduler picks a host.
Quiz
_consume_selected_host accomplish during a multi-create request?Match the function to what it does
Drag each function onto the job it performs in the scheduling procedure.
From the conductor's single RPC call to a returned Selection: request filters tighten the spec, Placement lists candidates, the HostManager builds host states, filters gate and weighers rank them, a claim wins the host, and alternates are wrapped alongside it. That's how Nova picks a host.