What really happens
when you attach a volume?
You run one command to plug a Cinder disk into a running server. Behind the curtain, nova-api, nova-compute and Cinder perform a careful three-step handshake — reserve, connect, confirm — while os-brick and libvirt wire the disk into the live virtual machine. This is the complete journey, attach then detach, traced from the real Nova source code.
The cast of characters
Attaching a volume is a coordinated dance between three services. Before we trace the messages, meet the players and the key idea that runs through the whole flow: the modern Cinder attachment handshake.
The big idea: a three-step handshake
Modern Nova (Queens and later, Cinder microversion 3.44+) uses a tidy three-call handshake to attach a volume. Each call moves the volume one step closer to being usable, and the attachment_id threads them together.
In nova-api. Reserves the volume in Cinder with no connector. The volume goes from available to reserved and Cinder hands back an attachment_id.
On the compute host. Sends the host connector so Cinder maps/exports the volume, and returns the connection_info needed to wire the disk in.
On the compute host. Once the disk is attached and the connection details are saved, this marks the volume in-use. The attachment is now fully connected.
The older path used reserve_volume then initialize_connection then terminate_connection. Nova still falls back to it when a BDM has no attachment_id, but the attachment API is the default. Watch for the if not self['attachment_id'] branch later.
Two ways to send a message
Just like a server boot, every hop here travels over one of two channels. Knowing which is which tells you who is talking to whom.
Nova's own services (api, compute) talk through a message broker like RabbitMQ. A cast is fire-and-forget; a call waits for a reply.
When Nova talks to Cinder it makes ordinary HTTP REST requests through its Cinder client. The attachment handshake calls are all REST.
Meet the players (click each one)
These are the components a volume attach passes through. Click any box to learn its job.
The journey in one breath
Here is the whole attach compressed into four phases. The rest of the course unpacks each one, then walks the detach.
nova-api fetches the volume from Cinder, asks the compute host to create the BDM with a free device name, then calls attachment_create to reserve the volume.
nova-api records the action and casts attach_volume to the instance's compute host, then the request is done from the API's point of view.
DriverVolumeBlockDevice builds the connector, calls attachment_update for connection_info, asks libvirt to attach the device, saves connection_info, then calls attachment_complete.
os-brick connects the volume locally; libvirt builds the disk XML and hot-plugs it into the running guest and its saved config.
Attach · validation and the BDM
The request lands in nova/compute/api.py. Nothing is wired into the VM yet — this phase is about validating, reserving a device name, and reserving the volume in Cinder. All of this runs in nova-api.
Reserve a device name and create the BDM on the compute host
For a live instance the BDM is created via an RPC call to the compute host, so two simultaneous attaches don't grab the same /dev/vdX. The multiattach flag from the Cinder volume is passed along.
volume_bdm = self.compute_rpcapi.reserve_block_device_name(
context, instance, device, volume_id, disk_bus=disk_bus,
device_type=device_type, tag=tag,
multiattach=volume['multiattach'])
volume_bdm.delete_on_termination = delete_on_termination
volume_bdm.save()
return volume_bdm
The compute host knows which device names are already in use on that instance. Letting it choose prevents two parallel attaches from both claiming /dev/vdb.
attachment_create reserves the volume and yields attachment_id
_check_attach_and_reserve_volume() validates the availability zone and the multiattach microversion, then calls attachment_create with no connector. The returned id becomes bdm.attachment_id. This is the modern flow that replaced reserve_volume.
attachment_id = self.volume_api.attachment_create(
context, volume_id, instance.uuid)['id']
bdm.attachment_id = attachment_id
...
if bdm.obj_attr_is_set('id'):
bdm.save()
available to reserved. No connector is sent yet, so it is reserved but not connected.
attachment_create on the Cinder client (microversion 3.44)
Nova's Cinder client requires microversion 3.44 so that attachment_complete is available. With a null connector the attachment is reserved but not connected.
try:
attachment_ref = cinderclient(context, '3.44').attachments.create(
volume_id, _connector, instance_id)
return _translate_attachment_ref(attachment_ref)
attachment_complete call exists.
API casts attach_volume to the instance's host
With the BDM created and the attachment reserved, _attach_volume() records the ATTACH_VOLUME action and fires an asynchronous RPC cast. If anything before the cast failed, the BDM is destroyed so no orphan record is left.
self._record_action_start(
context, instance, instance_actions.ATTACH_VOLUME)
self.compute_rpcapi.attach_volume(context, instance, volume_bdm)
except Exception:
with excutils.save_and_reraise_exception():
volume_bdm.destroy()
cast, the API returns to the user immediately; the real attach happens asynchronously on the host.
mysql> SELECT device_name, attachment_id, connection_info IS NULL AS not_wired FROM nova.block_device_mapping WHERE instance_uuid='$UUID' AND volume_id='$VOL_ID' AND deleted=0;
| /dev/vdb | 9c1f… | 1 | # (cell DB)
# cinder's ledger: volume reserved, attachment exists without a connector
$ openstack volume show $VOL_ID -c status → reserved
$ openstack --os-volume-api-version 3.44 volume attachment list --volume-id $VOL_ID --all-projects
# frozen exactly like this → the attach_volume cast never reached the host;
# check nova-compute on the instance's host, then the instance action journal:
$ openstack server event list $UUID → action attach_volume, result …
Attach · the handshake on the host
The compute manager picks up the cast under a per-instance lock and hands off to DriverVolumeBlockDevice. This is the heart of the flow: connector, attachment_update, driver attach, persist, attachment_complete. Everything here runs on the compute host.
Compute manager runs under a per-instance lock
attach_volume() converts the BDM into a DriverVolumeBlockDevice and calls _attach_volume() inside @utils.synchronized(instance.uuid); on failure the BDM is destroyed.
driver_bdm = driver_block_device.convert_volume(bdm)
@utils.synchronized(instance.uuid)
def do_attach_volume(context, instance, driver_bdm):
try:
return self._attach_volume(context, instance, driver_bdm)
synchronized(instance.uuid) lock makes sure two volume operations on the same instance don't interleave.
Get the host connector and choose the new-style path
_do_attach() gets the os-brick connector from the virt driver. If the BDM has an attachment_id it uses _volume_attach() (new flow); otherwise it falls back to _legacy_volume_attach().
context = context.elevated()
connector = virt_driver.get_volume_connector(instance)
if not self['attachment_id']:
self._legacy_volume_attach(context, volume, connector, instance,
volume_api, virt_driver, do_driver_attach)
else:
self._volume_attach(context, volume, connector, instance,
volume_api, virt_driver,
self['attachment_id'], do_driver_attach)
attachment_id on the BDM is the switch between the modern and legacy flows.
attachment_update sends the connector and returns connection_info
The previously-reserved attachment is now updated with the host connector and mountpoint. Cinder exports/maps the volume and returns connection_info. For multiattach volumes a flag is stashed in connection_info.
connection_info = volume_api.attachment_update(
context, attachment_id, connector,
self['mount_device'])['connection_info']
if 'serial' not in connection_info:
connection_info['serial'] = self.volume_id
self._preserve_multipath_id(connection_info)
if vol_multiattach:
connection_info['multiattach'] = True
attachment_update is the call that actually produces connection_info. Nova stamps in the volume id as serial, preserves any multipath id, and tags multiattach volumes.
Drive the hypervisor attach, then persist connection_info
With do_driver_attach=True, the libvirt driver's attach_volume() is called. After it succeeds, connection_info is written into the BDM and saved before the volume is marked in-use, because detach later needs it.
virt_driver.attach_volume(
context, connection_info, instance,
self['mount_device'], disk_bus=self['disk_bus'],
device_type=self['device_type'], encryption=encryption)
...
self['connection_info'] = connection_info
self.save()
attachment_complete marks the volume in-use
Once the device is attached and connection_info is persisted, attachment_complete tells Cinder the attachment is fully connected. On failure the driver detaches and deletes the attachment to free the volume.
try:
# This marks the volume as "in-use".
volume_api.attachment_complete(context, attachment_id)
except Exception:
with excutils.save_and_reraise_exception():
if do_driver_attach:
...
volume_api.attachment_delete(context, self['attachment_id'])
attachment_delete, returning the volume to available.
create (reserve) → update (connect, get connection_info) → driver attach → save connection_info → complete (mark in-use). The middle call is the only one that returns connection_info.
$ openstack volume show $VOL_ID -c status → reserved … then in-use
# the fine-grained milestone: connection_info lands on the BDM after attachment_update
mysql> SELECT device_name, LENGTH(connection_info) FROM nova.block_device_mapping WHERE instance_uuid='$UUID' AND volume_id='$VOL_ID' AND deleted=0;
| /dev/vdb | 1874 | # NULL/0 = attachment_update hasn't returned yet
# stuck 'reserved' with connection_info present → the driver attach or attachment_complete
# died — read nova-compute's log on the host; the rollback deletes the attachment,
# so a LINGERING attachment with no in-use volume is itself a finding
Attach · os-brick and libvirt wiring
Drilling into the driver attach from the previous module: the libvirt driver builds the host connector, connects the volume locally via os-brick, builds the disk XML and hot-plugs it into the running guest. All on the compute host.
get_volume_connector builds host properties via os-brick
os-brick's get_connector_properties() collects this host's initiator info (IQN, IP, multipath capability, hostname) — the connector Cinder uses to export the volume to the right host.
def get_volume_connector(self, instance):
root_helper = utils.get_root_helper()
return connector.get_connector_properties(
root_helper, CONF.my_block_storage_ip,
CONF.libvirt.volume_use_multipath,
enforce_multipath=True,
host=CONF.host)
attachment_update sends to Cinder so the volume is exported to exactly this machine.
Connect locally, build disk config, hot-plug the device
attach_volume() calls _connect_volume() (os-brick local connect), computes disk_info, builds the libvirt conf, then attaches it live with guest.attach_device(persistent=True, live=live) and rebuilds device metadata.
self._connect_volume(context, connection_info, instance,
encryption=encryption)
disk_info = blockinfo.get_info_from_bdm(
instance, CONF.libvirt.virt_type, instance.image_meta, bdm)
...
conf = self._get_volume_config(instance, connection_info, disk_info)
...
guest.attach_device(conf, persistent=True, live=live)
persistent=True, live=live means the change applies both to the live guest and the persistent domain definition.
The guest now sees a new block device at the chosen device name, and Cinder reports the volume in-use. No reboot required.
$ openstack server volume list $UUID → | /dev/vdb | $VOL_ID | …
# cinder's view
$ openstack volume show $VOL_ID -c status -c attachments → in-use, attached to $UUID
# the hypervisor's view — the disk is really in the domain
compute$ virsh domblklist instance-000xxxxx
vdb <rbd/iscsi path for the volume>
# inside the guest: lsblk shows the new device
# cinder says in-use but virsh doesn't list it (or vice versa) → the layers diverged;
# the BDM's connection_info tells you what nova THINKS is wired — trust it over memory
Detach · unwinding the attachment
Detach reverses the handshake. nova-api marks the volume detaching and casts to compute; the host unplugs the device, disconnects via os-brick, deletes the Cinder attachment, and destroys the BDM. We mark which steps run in nova-api vs the compute host.
begin_detaching then cast detach_volume (in nova-api)
The API marks the volume detaching in Cinder, looks up the per-instance attachment_id from the volume's attachments, records the action, and casts to compute with volume_id and attachment_id.
try:
self.volume_api.begin_detaching(context, volume['id'])
except exception.InvalidInput as exc:
raise exception.InvalidVolume(reason=exc.format_message())
attachments = volume.get('attachments', {})
attachment_id = None
if attachments and instance.uuid in attachments:
attachment_id = attachments[instance.uuid]['attachment_id']
...
self.compute_rpcapi.detach_volume(context, instance=instance,
volume_id=volume['id'], attachment_id=attachment_id)
begin_detaching moves the volume to detaching; the cast does the rest asynchronously.
Manager converts the BDM and calls driver detach (on the host)
The compute manager's detach_volume() reloads the BDM, then _detach_volume() converts it and calls DriverVolumeBlockDevice.detach(), passing the attachment_id and whether to destroy the BDM. The BDM is destroyed last.
driver_bdm = driver_block_device.convert_volume(bdm)
driver_bdm.detach(context, instance, self.volume_api, self.driver,
attachment_id=attachment_id, destroy_bdm=destroy_bdm)
...
if destroy_bdm:
bdm.destroy()
driver_detach unplugs from the hypervisor (on the host)
driver_detach() reads the stored connection_info and mountpoint and calls the virt driver's detach_volume(). A guest refusal triggers roll_detaching on Cinder.
encryption = encryptors.get_encryption_metadata(context,
volume_api, volume_id, connection_info)
virt_driver.detach_volume(context, connection_info, instance, mp,
encryption=encryption)
...
except exception.DeviceDetachFailed:
with excutils.save_and_reraise_exception():
...
volume_api.roll_detaching(context, volume_id)
roll_detaching returns the volume from detaching to in-use.
New-style detach deletes the Cinder attachment (on the host)
Because the BDM carries an attachment_id, _do_detach() finishes by deleting the attachment in Cinder, which frees the volume — no separate terminate_connection needed. A missing attachment is tolerated.
else:
try:
volume_api.attachment_delete(context, self['attachment_id'])
except exception.VolumeAttachmentNotFound:
LOG.info(
"Ignoring a volume attachment deletion failure as the "
...)
in-use back to available. A VolumeAttachmentNotFound is logged and ignored — the goal is already achieved.
Detach the device with retry, then disconnect locally (libvirt + os-brick)
detach_volume() resolves the guest disk, runs _detach_with_retry() (handles live + persistent domains and waits for libvirt device-removed events), then _disconnect_volume() tears down the host-side connection via os-brick.
self._detach_with_retry(
guest,
instance.uuid,
get_dev,
device_name=disk_dev,
)
...
self._disconnect_volume(context, connection_info, instance,
encryption=encryption)
_disconnect_volume avoids cutting a connection another instance still needs.
mysql> SELECT count(*) FROM nova.block_device_mapping WHERE instance_uuid='$UUID' AND volume_id='$VOL_ID' AND deleted=0; → 0 ✓
compute$ virsh domblklist instance-000xxxxx → device gone ✓
# stuck 'detaching'? the guest refused or the host timed out — the BDM survives so a
# retry is safe; the reason is in nova-compute's log (DeviceDetachFailed)
# orphaned attachment (instance long gone, volume still held) — list, then release it
$ openstack --os-volume-api-version 3.44 volume attachment list --volume-id $VOL_ID --all-projects
$ openstack --os-volume-api-version 3.44 volume attachment delete $ATTACHMENT_ID
The full journey, animated
Press Next step to watch one message packet travel the entire path — first the attach, then the detach. Each label notes whether the hop is a CAST, a CALL, a REST request, or a local LIBRARY (os-brick) call.
Anything to or from Cinder is REST. os-brick is a local LIBRARY call on the host. The api↔compute hops are RPC — a CALL for reserve_block_device_name, but a CAST for the actual attach_volume / detach_volume.
When it goes wrong
The handshake is built so that a failure at any point leaves the volume in a clean, recoverable state — never half-attached. Here are the four recovery paths.
Driver attach fails after Cinder mapped the volume
If virt_driver.attach_volume() raises after Cinder already exported the volume, _volume_attach() catches it, calls attachment_delete (and virt_driver.detach_volume if needed) to return the volume to available, and re-raises. The manager also destroys the BDM.
The volume never gets stuck in reserved or attaching: deleting the attachment frees it, and the orphan BDM is removed.
attachment_complete fails
The BDM connection_info is already saved, so the driver detaches the device and deletes the attachment, rolling the volume back to available. This is exactly why bdm.save() runs before attachment_complete.
RPC MessagingTimeout during BDM creation
If the reserve_block_device_name call times out, _attach_volume() in the API destroys any orphan BDM so a dangling reserved attachment is not left unowned.
The compute host didn't answer the device-name RPC in time.
The API's exception handler calls volume_bdm.destroy() so no stray BDM survives.
Guest refuses to release the disk on detach
A DeviceDetachFailed (or a live-detach timeout that exhausts the retries in _detach_with_retry) means the guest won't let go. driver_detach() calls roll_detaching to put the Cinder volume back to in-use and re-raises, leaving the BDM intact so you can try again.
Because the BDM survives and the volume is restored to in-use, the disk is still safely attached to the guest — Nova would rather fail loudly than leave a half-detached device.
Check yourself
Three questions and a matching exercise. If these click, you understand the attachment handshake.
Quiz
connection_info used to wire the volume into the guest?bdm.save() called with connection_info before attachment_complete?available?Match the call to what it does
Drag each call onto the job it performs during attach or detach.
From attachment_create reserving the volume, through attachment_update returning connection_info and libvirt hot-plugging the disk, to attachment_delete releasing it on detach. That's the whole modern Cinder attachment handshake.