Moving a running
machine without
turning it off
Live migration relocates a running instance from one compute host to another with almost no downtime. The conductor orchestrates with blocking RPC calls, the source and destination negotiate a handshake, libvirt streams the live guest memory across, and only once it lands on the destination do the network, storage and resource accounting follow. Traced from the real Nova source.
The cast of characters
Live migration is choreography. Before we trace the messages, meet the players, the two compute hosts that do the dance, and the three ways they talk.
Three ways to send a message
Every interaction in this story travels over one of three channels. In live migration the distinction between a call and a cast is not a footnote — it is the plot.
The sender drops a message on the queue and waits for a reply. Conductor's "can you take this guest?" checks are calls — it needs the migrate_data back.
The sender drops the message and moves on without waiting. The actual live_migration command is a cast — conductor cannot block for the minutes a transfer may take.
When Nova talks to another OpenStack project — Placement, Neutron, Cinder — it makes an ordinary HTTP REST request, just like a web app calling an API.
Conductor calls the destination compute, which in turn calls the source compute back. Two synchronous round trips nested inside each other, producing the migrate_data that drives the whole move. Nothing else in Nova looks quite like this.
Meet the services (click each one)
These are the components a live migration passes through. Note that two compute services take part: the source you are leaving and the destination you are heading to. Click any box to learn its job.
The journey in one breath
Here is the whole move compressed into five beats. The rest of the course unpacks each one.
nova-api sets task_state=MIGRATING (the VM keeps running), loads the RequestSpec, and casts to the conductor.
It opens a Migration record, parks the source allocation on it, and picks/validates a destination.
Conductor CALLS the destination, which CALLS the source back, producing migrate_data. Conductor pre-binds Neutron ports, then CASTS the migration to the source.
The source asks the destination to prepare storage and networking, then libvirt copies the live RAM/CPU across while a monitor watches.
Once the guest runs on the destination, the network and volumes switch over, the instance's home is set to the destination, and the source allocation is freed.
The API phase
Your request lands at nova/compute/api.py live_migrate(). Nothing moves yet — this phase validates the target, loads the original scheduling request, and flips one flag.
task_state goes to MIGRATING, RequestSpec loaded
The instance is moved to MIGRATING only after the optional host validation, so a bad host name fails before the instance state changes. vm_state is never changed here — the VM is live the whole time. The original RequestSpec is reloaded so the scheduler can run again if needed.
if host_name:
# Validate the specified host before changing
# the instance task state.
nodes = objects.ComputeNodeList.get_all_by_host(
context, host_name)
request_spec = objects.RequestSpec.get_by_instance_uuid(
context, instance.uuid)
instance.task_state = task_states.MIGRATING
instance.save(expected_task_state=[None])
expected_task_state=[None] guard means we only proceed if the instance isn't already busy with another task. The guest stays ACTIVE and running throughout.
force=False unsets the host so the scheduler still runs
This is the subtle "forced host" logic. With the newer microversion, force is an explicit boolean. If force is False but a host was named, the host becomes a scheduler hint (requested_destination) rather than a hard target, so the scheduler filters still run against it.
if force is False and host_name:
# Unset the host to make sure we call the
# scheduler from the conductor
# LiveMigrationTask. Yes this is tightly-
# coupled to behavior in conductor.
host_name = None
...
request_spec.requested_destination = destination
Forcing a host bypasses the scheduler's filters — useful but risky, because nothing checks the destination is sensible beyond a few hard guards. The default is to let the scheduler vet your choice.
| MIGRATING | migrating | active | qh2-rcc001 | # vm_state stays active — the guest is still running
# the migration ledger row is created by the CONDUCTOR — its existence proves the cast arrived
$ openstack server migration list --server $UUID
mysql> SELECT id, status FROM nova.migrations WHERE instance_uuid='$UUID' AND migration_type='live-migration' ORDER BY id DESC LIMIT 1; (cell DB)
# task_state=migrating but NO migration row → the cast to nova-conductor was lost
Conductor opens the ledger
The conductor picks up the cast and builds the spine of the whole flow: a Migration record. Then it performs the trick at the heart of move-based allocations.
A Migration record is created with status 'accepted'
ComputeTaskManager._live_migrate creates a Migration object (type live-migration, status accepted, source = current host) and wraps everything in a LiveMigrationTask. This object is the spine: allocations, status reporting, and post/rollback cleanup all key off it.
migration = objects.Migration(
context=context.elevated())
migration.dest_compute = destination
migration.status = 'accepted'
migration.instance_uuid = instance.uuid
migration.source_compute = instance.host
migration.migration_type = \
fields.MigrationType.LIVE_MIGRATION
preparing, running, completed or error.
Source allocation is parked on the migration UUID
LiveMigrationTask._execute() first runs sanity checks (instance active, no old NUMA, source host up), then calls replace_allocation_with_migration. This is the heart of move-based allocations: move_allocations re-keys the existing Placement allocation from instance.uuid to migration.uuid, so the source is "held by the migration" while the instance becomes an empty consumer ready to receive the destination allocation.
success = reportclient.move_allocations(
context, instance.uuid, migration.uuid)
if not success:
...
raise exception.NoValidHost(
reason=_('Unable to replace instance claim on source'))
...
return source_cn, orig_alloc
A Placement consumer (the instance) can only hold one allocation per resource provider. If the instance kept the source claim, claiming the destination against the same consumer would conflict. Parking the source claim on the migration record frees the instance to receive the destination claim.
Scheduler path vs forced-host path
If no destination was forced, _find_destination calls the scheduler, which creates the destination allocation against the instance. If a host was forced, the scheduler is skipped and claim_resources_on_destination reproduces the source allocation on the destination node directly — with consumer_generation=None because the instance is now an empty consumer.
if not self.destination:
self.destination, dest_node, self.limits = \
self._find_destination()
else:
self._check_destination_is_not_source()
self._check_host_is_up(self.destination)
self._check_destination_has_enough_memory()
source_node, dest_node = (
self._check_compatible_with_source_hypervisor(
self.destination))
scheduler_utils.claim_resources_on_destination(...)
| 87 | qh2-rcc001 | qh2-rcc002 | accepted | live-migration |
mysql> SELECT uuid, status, source_compute, dest_compute FROM nova.migrations WHERE instance_uuid='$UUID' ORDER BY id DESC LIMIT 1;
# placement: the trick from this module, visible live
$ openstack resource provider allocation show $MIGRATION_UUID → source resources, parked on the migration
$ openstack resource provider allocation show $UUID → fresh claim on the destination
# migration status 'error' this early → pre-checks or scheduling failed;
# the exception is in nova-conductor's log, not on either compute host
The nested handshake
This is the signature pattern of live migration. Conductor issues a synchronous RPC call to the destination, which issues another synchronous call back to the source. The result is the migrate_data object that carries everything forward.
Conductor calls the destination, which calls the source
Note cctxt.call (not cast): conductor blocks waiting for migrate_data. The destination manager in turn blocks on a call to the source. Two synchronous round trips nested inside each other.
try:
self.migrate_data = self.compute_rpcapi.\
check_can_live_migrate_destination(
self.context, self.instance, destination,
self.block_migration, self.disk_over_commit,
self.migration, self.limits)
except messaging.MessagingTimeout:
raise exception.MigrationPreCheckError(msg)
call — blocking. If the messaging layer times out, Nova treats it as a precheck failure and aborts before anything moves.
Destination compute calls source compute back
Inside the destination manager, after the driver builds its check data, it makes a blocking RPC call to the source host, which builds and returns the source-side migrate_data. This is the inner half of the nested handshake.
try:
migrate_data = (
self.compute_rpcapi.check_can_live_migrate_source(
ctxt, instance, dest_check_data)
)
except Exception as ex:
...
raise exception.MigrationPreCheckError(msg)
migrate_data that the rest of the flow relies on.
Both check RPCs are synchronous calls
In nova/compute/rpcapi.py, both check_can_live_migrate_destination and check_can_live_migrate_source use cctxt.call (with long_rpc_timeout for the destination), unlike the actual live_migration which is a fire-and-forget cast.
cctxt = client.prepare(server=destination,
version=version,
call_monitor_timeout=CONF.rpc_response_timeout,
timeout=CONF.long_rpc_timeout)
return cctxt.call(ctxt,
'check_can_live_migrate_destination', **kwargs)
bind_ports_to_host creates dest bindings before the move
Still in the precheck path (_call_livem_checks_on_host), if Neutron supports the binding-extended API, conductor creates inactive port bindings on the destination via bind_ports_to_host and folds the resulting vif details into migrate_data.vifs. The destination bindings exist but are inactive; the active binding is still the source.
bindings = self.network_api.bind_ports_to_host(
context=self.context, instance=self.instance,
host=destination, vnic_types=None,
port_profiles=ports_profile)
except exception.PortBindingFailed as e:
raise exception.MigrationPreCheckError(
reason=e.format_message())
return bindings
$ openstack port show $PORT_ID -c binding_host_id -c status
| qh2-rcc001 | ACTIVE |
# but neutron's DB now holds TWO bindings for the port: source ACTIVE, dest INACTIVE
mysql> SELECT host, status, vif_type FROM neutron.ml2_port_bindings WHERE port_id='$PORT_ID';
| qh2-rcc001 | ACTIVE | ovs |
| qh2-rcc002 | INACTIVE | ovs |
# migration died during the pre-checks? status goes 'error' and the exception names the
# failing side: MigrationPreCheckError from dest or source appears in nova-conductor's log
The transfer
Now the pivot: conductor stops orchestrating and casts the migration to the source. The source prepares the destination, then libvirt streams the live guest across while a monitor watches.
The actual migration command is a cast, not a call
This is the pivot from synchronous orchestration to asynchronous execution. Once a destination is settled and validated, the task fills in migration.dest_node/dest_compute, saves it, and issues compute_rpcapi.live_migration — a cast to the source host. Conductor does not wait for the migration to finish.
return self.compute_rpcapi.live_migration(self.context,
host=self.source,
instance=self.instance,
dest=self.destination,
block_migration=self.block_migration,
migration=self.migration,
migrate_data=self.migrate_data)
Source compute submits the job to a thread pool
live_migration sets migration status queued, registers the instance in _waiting_live_migrations (so it can be aborted), and submits _do_live_migration to a green-thread executor. The RPC worker returns immediately; the heavy lifting happens in the background.
self._set_migration_status(migration, 'queued')
self._waiting_live_migrations[instance.uuid] = (None, None)
try:
future = nova.utils.pass_context(
self._live_migration_executor.submit,
self._do_live_migration, context, dest, instance,
block_migration, migration, migrate_data)
self._waiting_live_migrations[instance.uuid] = (migration, future)
_waiting_live_migrations is what makes openstack server migration abort possible while the job is still queued.
New Cinder attachment created on the destination
_do_live_migration sets status preparing and calls the destination's pre_live_migration (a synchronous call, wrapped in wait_for_instance_event for network-vif-plugged). For each volume BDM using the cinder v3.44 attachment API, the destination creates a brand-new attachment and records the old attachment id in migrate_data.old_vol_attachment_ids.
attach_ref = self.volume_api.attachment_create(
context, bdm.volume_id, bdm.instance_uuid,
connector=connector, mountpoint=bdm.device_name)
...
migrate_data.old_vol_attachment_ids[bdm.volume_id] = \
bdm.attachment_id
bdm.attachment_id = attach_ref['id']
bdm.save()
Driver plugs vifs; wait_for_vif_plugged is set
The destination driver's pre_live_migration plugs the vifs and connects volumes. If multiple port bindings are in use and live_migration_wait_for_vif_plug is on, the source will wait for a network-vif-plugged event before starting the guest transfer.
migrate_data = self.driver.pre_live_migration(
context, instance, block_device_info,
network_info, disk, migrate_data)
...
migrate_data.wait_for_vif_plugged = (
CONF.compute.live_migration_wait_for_vif_plug and
using_multiple_port_bindings
)
Source binds post/rollback callbacks then calls the driver
Back on the source, _do_live_migration sets status running and calls driver.live_migration with two callbacks. The virt driver knows nothing about RPC or Placement; it just calls post_method on success or recover_method on failure. This decoupling is why conductor's rollback comments note it can't roll back the compute call.
post_live_migration = functools.partial(
self._post_live_migration_update_host,
source_bdms=source_bdms)
rollback_live_migration = functools.partial(
self._rollback_live_migration, source_bdms=source_bdms)
...
self.driver.live_migration(context, instance, dest,
post_live_migration, rollback_live_migration,
block_migration, migrate_data)
The actual libvirt domain transfer
_live_migration_operation calls guest.migrate(...) (libvirt's migrateToURI3 under the hood) with the destination URI and the rewritten destination XML. This is the call that copies live RAM and CPU state across the wire.
guest.migrate(self._live_migration_uri(dest),
migrate_uri=migrate_uri,
flags=migration_flags,
migrate_disks=device_names,
destination_xml=new_xml_str,
bandwidth=CONF.libvirt.live_migration_bandwidth)
LOG.debug("Migrate API has completed", instance=instance)
Monitor dispatches to post_method or recover_method
_live_migration_monitor polls the libvirt job. On VIR_DOMAIN_JOB_COMPLETED it calls post_method; on JOB_FAILED or JOB_CANCELLED it runs recover tasks then recover_method.
elif info.type == libvirt.VIR_DOMAIN_JOB_COMPLETED:
LOG.info("Migration operation has completed", ...)
post_method(context, instance, dest,
block_migration, migrate_data)
break
elif info.type == libvirt.VIR_DOMAIN_JOB_FAILED:
...
recover_method(context, instance, dest, migrate_data)
JOB_COMPLETED is the point of no return: the guest is now on the destination.
Up to this moment, any failure rolls back cleanly — the guest is still running on the source. After JOB_COMPLETED, the guest is on the destination and there is no going back; later failures force the instance to ERROR rather than rolling back.
$ openstack server migration show $UUID $MIGRATION_ID
memory_processed_bytes / memory_remaining_bytes / disk_* — refreshed as the monitor polls
# the same numbers straight from libvirt, on the SOURCE host
source$ virsh domjobinfo instance-000xxxxx
# where is it stuck? queued = too many concurrent migrations on the source
# (max_concurrent_live_migrations); preparing = pre_live_migration on the DEST
# (vif plug, cinder attachment_create); running forever = memory dirtying faster than copy
# operator levers while it runs
$ openstack server migration force complete $UUID $MIGRATION_ID # pause/post-copy to converge
$ openstack server migration abort $UUID $MIGRATION_ID # cancel; guest keeps running on the source
The cutover and cleanup
The guest is now running on the destination. The source runs _post_live_migration to switch the network over, drop its volume attachment, then calls the destination to define the domain and claim the instance as its own.
migrate_instance_start activates destination port bindings
On success, _post_live_migration runs on the source. It activates the destination port bindings first, in a try, to minimise downtime — this atomically flips the destination binding to ACTIVE and the source to inactive. The network cutover.
migration = objects.Migration(
source_compute=self.host, dest_compute=dest,
)
# For neutron, migrate_instance_start will
# activate the destination host port bindings,
# if there are any created by conductor before
# live migration started.
self.network_api.migrate_instance_start(
ctxt, instance, migration)
finally, so it runs even if this step failed.
activate_port_binding is a POST-like action in Neutron
Internally migrate_instance_start calls client.activate_port_binding(vif['id'], dest_host) per vif — a dedicated action rather than a normal status PUT.
try:
# This is a bit weird in that we don't PUT and
# update the status to ACTIVE, it's more like a
# POST action method in the compute API.
client.activate_port_binding(vif['id'], dest_host)
Source volume attachments deleted; source cleaned up
_post_live_migration_remove_source_vol_connections deletes the old source-host Cinder attachment (v3.44) or terminates the legacy connection, all inside the finally so it runs even if vif activation failed. Source vifs are unplugged and the source domain destroyed.
else:
# cinder v3.44 api flow - delete the old
# attachment for the source host
self.volume_api.attachment_delete(context,
bdm.attachment_id)
migrate_instance_finish + instance host updated to destination
The source calls (synchronous RPC) post_live_migration_at_destination. The destination calls migrate_instance_finish (updates each port's binding:host_id to the destination), allocates claimed PCI, defines the persistent domain via the driver, then sets instance.host/node/compute_id to itself and task_state=None. This is where the instance officially belongs to the destination in the database.
instance.apply_migration_context()
instance.drop_migration_context()
instance.host = self.host
instance.power_state = current_power_state
instance.task_state = None
instance.node = node_name
instance.compute_id = compute_node and compute_node.id or None
instance.progress = 0
instance.save(expected_task_state=task_states.MIGRATING)
expected_task_state=MIGRATING guard protects against races with other operations on the instance.
Migration marked completed and source allocation deleted
Right at the end, after post_live_migration_at_destination succeeds, the migration is set completed and _delete_allocation_after_move deletes the source allocation that the migration record was holding in Placement. The instance keeps only its destination allocation.
if migrate_data and migrate_data.obj_attr_is_set('migration'):
migrate_data.migration.status = 'completed'
migrate_data.migration.save()
self._delete_allocation_after_move(ctxt,
instance, migrate_data.migration)
migration.status = completed.
The guest never stopped running. vm_state stayed ACTIVE the entire time; only task_state moved through MIGRATING and back to None. openstack server show now reports the destination host.
| ACTIVE | None | qh2-rcc002 |
$ openstack server migration list --server $UUID → status completed
mysql> SELECT status FROM nova.migrations WHERE uuid='$MIGRATION_UUID'; → completed
# the parked source allocation must be GONE, network and storage must point at the dest
$ openstack resource provider allocation show $MIGRATION_UUID → (empty ✓)
$ openstack port show $PORT_ID -c binding_host_id → qh2-rcc002
$ openstack --os-volume-api-version 3.27 volume attachment list --all-projects | grep $UUID → one row, dest host
# instance ACTIVE on the dest but migration never 'completed' + allocation leak →
# the post steps died half-way; reconcile with: nova-manage placement audit --verbose
The full journey, animated
Press Next step to watch a single message packet travel the entire move. Watch for the rhythm: blocking calls for the handshake, a cast for the transfer, and REST whenever Nova talks to another project.
Steps 5, 6, 7, 11 and 20 are blocking calls; step 9 is the one cast that hands control to the source so conductor never blocks on the long transfer. The nested call at step 7 — destination calling the source back — is the signature you should remember.
When it goes wrong
Live migration is reversible — right up until the guest lands on the destination. The recovery path you get depends entirely on when the failure happens relative to JOB_COMPLETED.
Before the transfer: precheck failure
If a precheck fails in conductor — NoValidHost, MigrationPreCheckError, hypervisor too old, not enough memory, port binding failed — LiveMigrationTask.rollback calls revert_allocation_for_migration (moving the allocation back from migration.uuid to instance.uuid) and resets the instance to its prior vm_state; migration.status is set to error. The libvirt transfer never starts.
The source claim parked on the migration record moves back to the instance. Placement is as it was.
revert_allocation_for_migration · tasks/migrate.py_set_vm_state puts the instance back to its prior vm_state; the guest never moved.
During prepare: pre_live_migration fails on the destination
If pre_live_migration fails, _cleanup_pre_live_migration sets migration.status = error and calls _rollback_live_migration(..., pre_live_migration=True). The new destination Cinder attachments are deleted and BDMs restored to the source; destination port bindings are torn down; the destination allocation is reverted. The source guest was never touched.
Because every destination-side change (attachment, binding, allocation) was recorded, each can be precisely undone. The remembered old_vol_attachment_ids let the BDMs point back at the source.
During transfer: libvirt fails or is cancelled
The monitor calls recover_method = _rollback_live_migration. It reverts the destination allocation, deletes destination volume connections/attachments, tears down destination networking, optionally drops the NUMA move-claim on the destination, and resets task_state to None. Crucially, vm_state stays ACTIVE — the guest is still running on the source.
After JOB_COMPLETED: there is no rollback
Once libvirt reports VIR_DOMAIN_JOB_COMPLETED, the guest is running on the destination and cannot be moved back. If post_live_migration_at_destination then fails (for example the driver can't define the domain), the instance is forced to ERROR via _set_instance_obj_error_state — but _post_live_migration_update_host still sets instance.host=dest, because that is now the truth.
After the guest's memory and CPU state live on the destination, "rolling back" would mean discarding the running machine. Nova won't do that. Post-point-of-no-return failures land the instance in ERROR on the destination, where an operator can investigate — never silently lost.
Check yourself
Three questions and a matching exercise. If these click, you understand the live migration flow.
Quiz
live_migration command to the source compute, and how does that differ from the can-migrate checks?Match the function to its job
Drag each function onto the task it performs during a live migration.
From os-migrateLive to migration.status = completed: a Migration ledger, a moved Placement allocation, the nested call-back handshake, a cast that frees conductor, a libvirt stream of live memory, and a careful cutover of network and storage — all with the guest never leaving ACTIVE.