What really happens
when you boot a server?
You type one command. Behind the curtain, six services pass messages back and forth across two different communication channels to turn your request into a running virtual machine. This is the complete journey, traced from the real Nova source code.
The cast of characters
Booting a server is a team effort. Before we trace the messages, meet the players and the two ways they talk to each other.
Two ways to send a message
Every interaction in this story travels over one of two channels. Knowing which is which is the single most useful idea in the whole walkthrough.
Nova's own services (API, conductor, scheduler, compute) talk to each other through a message broker like RabbitMQ. A cast is fire-and-forget; a call waits for an answer.
When Nova needs another OpenStack project — Keystone, Glance, Neutron, Cinder, Placement — it makes an ordinary HTTP REST request, just like a web app calling an API.
If both ends are Nova services, it's RPC on the queue. If the other end is a different OpenStack project, it's REST over HTTP. Watch for this split throughout.
Meet the services (click each one)
These are the components your boot request will pass through. Click any box to learn its job.
The journey in one breath
Here is the whole thing compressed into four phases. The rest of the course unpacks each one.
nova-api checks your request against Keystone, Glance, Neutron and Cinder, writes a BuildRequest to the database, and casts to the conductor.
The conductor asks the scheduler, which queries Placement, ranks hosts, and claims resources on the winner.
The conductor creates the real instance record in the cell database and casts the build to the chosen compute host.
nova-compute claims resources, wires networking via Neutron, attaches storage via Cinder, downloads the image from Glance, and starts the domain through libvirt.
The API phase
Your request lands at nova/api/openstack/compute/servers.py. Nothing gets built yet — this phase is all about validation and recording intent.
The front door opens
A POST /servers request arrives. The ServersController.create() method normalises the input, then immediately checks whether you are even allowed to do this — a policy check backed by your Keystone token.
# context.can(...) consults policy,
# which is gated by your Keystone token
context.can(server_policies.SERVERS %
'create', target={})
# ... then hand off to the compute API
self.compute_api.create(context,
flavor, image_uuid, ...)
"Does everything you asked for actually exist?"
Inside compute_api.create() → _create_instance() → _validate_and_build_base_options(), Nova makes a flurry of REST calls to the other projects. It is checking your request against reality before committing to anything expensive.
"Does this image exist, and what are its properties?" A REST GET to Glance. Kernel and ramdisk images are checked too, if present.
image_api.get() · compute/api.py"Are these networks, ports and security groups valid and available?" REST calls validate them and reserve port quota.
network_api.validate_networks() · compute/api.pyIf booting from volume, "do these volumes exist, are they available, and are they in a compatible availability zone?" REST calls to Cinder.
volume_api.get() · compute/api.pyLocal quota checks plus, optionally, unified limits enforced via Placement. "Are you under your instance and resource quota?"
check_num_instances_quota() · compute/api.pyCatching a bad image or a missing network here means a fast, clean HTTP error back to you — instead of a half-built instance failing deep inside a compute host minutes later.
Recording intent: three database objects
Once everything checks out, _provision_instances() writes three records in a single transaction to the API database. These say "we intend to build this" — the instance does not exist in a cell yet.
Everything the scheduler needs to place the instance: flavor, image, networks, NUMA topology, PCI needs.
The "to-build" record. It stands in for the instance until a real cell record is created.
A pointer that will later say which cell the instance lives in. For now its cell is empty.
The hand-off: a cast to the conductor
The API's last act is to drop a message on the queue and walk away. This is a cast — fire-and-forget. The API then returns 202 Accepted to you almost immediately, long before the VM exists.
cctxt = self.client.prepare(version=version)
cctxt.cast(context,
'schedule_and_build_instances',
**kw)
conductor queue. Because it's a cast, the API gets no reply and your HTTP request returns right away with the new server's ID and a status of BUILD.
The 202 means "accepted, working on it" — not "done". Everything from here on happens asynchronously while you poll the server status.
$ openstack server show $UUID -c status -c OS-EXT-STS:task_state -c OS-EXT-SRV-ATTR:host
| BUILD | scheduling | None |
# API DB: the three intent records — the instance is NOT in any cell yet
mysql> SELECT count(*) FROM nova_api.build_requests WHERE instance_uuid='$UUID'; → 1
mysql> SELECT cell_id FROM nova_api.instance_mappings WHERE instance_uuid='$UUID'; → NULL
mysql> SELECT count(*) FROM nova_api.request_specs WHERE instance_uuid='$UUID'; → 1
# stuck like this forever? the cast to the conductor was lost —
# check nova-conductor is up and the 'conductor' queue is being consumed
$ rabbitmqctl list_queues name messages consumers | grep conductor
Scheduling: finding a home
The conductor picks up the cast and orchestrates the search for a host — the one moment in the whole flow that uses a blocking call.
The conductor asks the scheduler — and waits
schedule_and_build_instances() in the conductor calls _schedule_instances(), which reaches the scheduler through select_destinations(). Unlike the API's cast, this is an RPC call: the conductor blocks until the scheduler answers.
instance_uuids = [spec.instance_uuid
for spec in request_specs]
host_lists = self._schedule_instances(
context, request_specs[0],
instance_uuids,
return_alternates=True)
return_alternates=True asks for backup hosts too, so compute can retry elsewhere if the first choice fails. The reply is a list of chosen hosts.
Step 1 — "Which hosts even fit?" (Placement)
The scheduler turns the request into a set of required resources and asks Placement, over REST, for allocation candidates: every host that currently has room.
resources = utils.resources_from_request_spec(
context, spec_obj, self.host_manager,
enable_pinning_translate=True)
res = self.placement_client.\
get_allocation_candidates(
context, resources)
GET /allocation_candidates. Placement is the single source of truth for what is free, so the scheduler never has to poll every host directly.
Step 2 — Filter, then weigh
The candidates pass through two stages. Filters are pass/fail gates ("must be in this availability zone", "must have enough PCI devices"). Weighers then score the survivors so the best host floats to the top.
Each filter answers yes/no for every host. A host that fails any filter is discarded. What remains is the set of acceptable hosts.
host_manager.get_filtered_hosts()Each surviving host gets a score (e.g. most free RAM first). The list is sorted best-to-worst.
host_manager.get_weighed_hosts()Step 3 — Claim the resources (first one wins)
Ranking isn't enough — two simultaneous requests might both like the same host. So the scheduler claims the resources in Placement with a REST PUT /allocations. The first claim to succeed wins the host; a conflict means try the next one down the list.
Because the winning host's resources are reserved in Placement before the answer is returned, two parallel boots can't both grab the last slot on one host. Placement's generation check rejects the loser.
Back in the conductor: commit to a cell
The scheduler returns the chosen host (plus alternates). Now the conductor does the work only it is allowed to do — touching the cell database directly. It creates the real instance record, points the InstanceMapping at the cell, and writes the block device mappings, tags and an instance action.
The chosen host's HostMapping tells the conductor which cell to write into.
The BuildRequest becomes a real Instance row in that cell's database. The InstanceMapping is updated to point at the cell.
Finally, the conductor casts build_and_run_instance to the specific chosen host.
with obj_target_cell(instance, cell) as cctxt:
self.compute_rpcapi.build_and_run_instance(
cctxt, instance=instance, image=image,
request_spec=request_spec,
host=host.service_host,
node=host.nodename,
limits=host.limits,
host_list=host_list, ...)
cast — fire-and-forget again — but this time addressed to one specific host (host=host.service_host). The host_list carries the alternates so compute can reschedule itself if the build fails.
mysql> SELECT cm.name FROM nova_api.instance_mappings im JOIN nova_api.cell_mappings cm ON im.cell_id=cm.id WHERE im.instance_uuid='$UUID'; → cell name (row with NULL = still unscheduled)
mysql> SELECT count(*) FROM nova_api.build_requests WHERE instance_uuid='$UUID'; → 0 (deleted on commit)
# the real instance row, in THAT cell's DB (find it: nova-manage cell_v2 list_cells)
mysql> SELECT vm_state, task_state, host, node FROM nova.instances WHERE uuid='$UUID';
| building | scheduling | qh2-rcc123 | qh2-rcc123 | # task_state then walks the module-3 ladder
# the scheduler's claim is already visible in placement
$ openstack resource provider allocation show $UUID
| $RP_UUID (compute node) | {'VCPU': 4, 'MEMORY_MB': 16384, 'DISK_GB': 30} |
# host assigned but task_state frozen and nothing in that host's logs?
# the build_and_run_instance cast was lost — check nova-compute on that host
The build on the compute host
This is where the VM is actually born. nova-compute claims resources, wires up networking and storage, fetches the image, and asks libvirt to start the domain.
A locked, background build
build_and_run_instance() doesn't block the RPC worker. It grabs a per-instance lock and runs the real work in a greenthread, so the compute service stays responsive.
def build_and_run_instance(self, context,
instance, image, request_spec, ...):
@utils.synchronized(instance.uuid)
def _locked_do_build_and_run_instance(...):
with self._build_semaphore:
result = self.\
_do_build_and_run_instance(...)
synchronized(instance.uuid) lock makes sure nothing else touches this instance mid-build. A semaphore caps how many builds run at once so one host isn't overwhelmed.
Claim it locally, then build the resources
First the resource tracker makes a local claim — checking NUMA topology and PCI devices actually fit. Then a chain of steps wires up the instance, each one moving the task_state forward so you can watch progress.
Networking — asked early, awaited late
Network allocation is kicked off asynchronously. Compute asks Neutron to create and bind the ports, but doesn't sit and wait — it lets the request run in the background while it gets on with storage and the image.
instance.vm_state = vm_states.BUILDING
instance.task_state = task_states.NETWORKING
instance.save(expected_task_state=[None])
return network_model.NetworkInfoAsyncWrapper(
self._allocate_network_async,
context, instance, requested_networks, ...)
NetworkInfoAsyncWrapper runs allocate_for_instance() against Neutron in the background. Neutron creates the ports, binds them to this host, and queues a network-vif-plugged event for later. Compute will collect the result just before it needs it.Storage and image, then hand to libvirt
With networking in flight, compute attaches the volumes (Cinder + os-brick), then downloads the image from Glance into the local cache. Now it flips to SPAWNING and calls the virt driver.
instance.vm_state = vm_states.BUILDING
instance.task_state = task_states.SPAWNING
instance.save(expected_task_state=
task_states.BLOCK_DEVICE_MAPPING)
self.driver.spawn(context, instance,
image_meta, injected_files,
admin_password, allocs,
network_info=network_info,
block_device_info=block_device_info, ...)
The clever part: wait for the network to be plugged
Here networking and compute finally rendezvous. Before powering on, the libvirt driver pauses and waits for Neutron to confirm each virtual NIC is actually wired into the network — the network-vif-plugged events from earlier. Only then does it start the domain.
with self.virtapi.wait_for_instance_event(
instance, events, deadline=timeout,
error_callback=self._neutron_failed_callback,
):
self.plug_vifs(instance, network_info)
guest = self._create_guest(
context, xml, instance,
pause=pause, power_on=power_on, ...)
wait_for_instance_event until the network-vif-plugged events arrive. Once they do (or a timeout fires), the domain is resumed and powered on.
If Neutron never sends the event within the deadline and vif_plugging_is_fatal is on, the build fails. This is one of the most common real-world boot failures — the network back-end didn't confirm the port in time.
Powered on → ACTIVE
Compute polls the hypervisor until the power state is RUNNING, sets vm_state = ACTIVE and task_state = None, stamps launched_at, and emits an instance.create.end notification. Your server is booted. Because the original message was a cast, no reply travels back up — the database state is the result.
Next time you poll openstack server show, the status reads ACTIVE — the database row compute just saved.
$ openstack server show $UUID -c OS-EXT-STS:vm_state -c OS-EXT-STS:task_state -c OS-EXT-STS:power_state
| active | None | Running | # stuck at spawning → libvirt/glance; networking → neutron
# every phase is journaled as an instance-action event, with per-host results
$ openstack server event list $UUID
$ openstack server event show $UUID $REQUEST_ID # includes the traceback if an event failed
mysql> SELECT e.event, e.result, e.host FROM nova.instance_actions_events e JOIN nova.instance_actions a ON e.action_id=a.id WHERE a.instance_uuid='$UUID';
# and on the compute host / neutron side
$ virsh list --all | grep $INSTANCE_NAME # instance-000xxxxx running?
$ openstack port list --device-id $UUID # ports created and ACTIVE, or stuck DOWN?
The full journey, animated
Press Next step to watch a single message packet travel the entire path. Notice when it stays inside Nova (RPC) versus when it leaves for another project (REST).
Spot the asymmetry: there is exactly one blocking RPC call in the whole flow (conductor → scheduler, step 7). Everything else between Nova services is a non-blocking cast. That's what keeps the system scalable.
When it goes wrong
A robust boot flow needs answers for failure. Nova has two main recovery paths — one for "nowhere to put it" and one for "the build broke".
No valid host → bury in cell0
If the scheduler can't find or claim any host, it raises NoValidHost. The conductor catches it and buries the instance in cell0 — a graveyard cell where the instance is recorded in ERROR state so you can see why it failed.
try:
host_lists = self._schedule_instances(
context, request_specs[0], ...)
except Exception as exc:
LOG.exception('Failed to schedule instances')
self._bury_in_cell0(context,
request_specs[0], exc, ...)
return
vm_state = ERROR. That's why a failed boot still shows up in openstack server list with a fault message, rather than silently vanishing.
$ openstack server show $UUID -c status -c fault
| ERROR | {'code': 500, 'message': 'No valid host was found. …'} |
# which cell did it land in? cell0 = it was never scheduled at all
mysql> SELECT cm.name FROM nova_api.instance_mappings im JOIN nova_api.cell_mappings cm ON im.cell_id=cm.id WHERE im.instance_uuid='$UUID'; → cell0
# the full traceback lives in cell0's own fault table
mysql> SELECT code, message, details FROM nova_cell0.instance_faults WHERE instance_uuid='$UUID' ORDER BY created_at DESC LIMIT 1;
# a rescheduled build leaves a trail — one build event per attempted host
$ openstack server event list $UUID # then event show: compute_build_and_run_instance per host
Build failed on the host → reschedule
If the build breaks after a host was chosen (a transient error, a flaky network bind), compute raises a RescheduledException. It releases its Placement allocation and casts back to the conductor to try the next host on the host_list of alternates.
It unplugs networks and deletes its Placement allocation so the failed host's resources are freed.
Compute calls build_instances on the conductor with the remaining alternates.
The conductor dispatches the build to the next alternate. This repeats until success or the alternates run out.
Remember return_alternates=True from the scheduling call? Those backup hosts are what makes reschedule cheap — compute doesn't have to go all the way back through the scheduler for every retry.
Build aborted → no retry
Some failures aren't worth retrying — an unsupported flavor/image combination, device tagging the host can't do, or certificate validation failing. These raise BuildAbortException: the instance goes straight to ERROR with no reschedule, because trying another host would fail identically.
Check yourself
Three questions and a matching exercise. If these click, you understand the boot flow.
Quiz
server create?Match the service to its job
Drag each service onto the task it performs during a boot.
From POST /servers to vm_state = ACTIVE: validation in the API, claim-based scheduling through Placement, a commit in the conductor, and a libvirt spawn that waits for the network to come up. That's the whole journey.