What really happens
when you reserve a GPU?
A researcher books four GPU nodes for next Tuesday. Between that click and the moment their instances boot on exactly the right hypervisors sit three services, two databases you rarely look at, and a private flavor with two magic extra specs. This course traces the whole lifecycle from the source — and at every step shows the CLI to prove the system did what it should.
Traced from warre @ master, blazar @ 2025.1, nova @ nectar/2024.1. All code excerpts are exact quotes.
The cast: a booking system in three layers
Think of an airline. There is the travel agent who sells you the ticket, the airline's reservation system that assigns you an aircraft and a seat, and the airport operations that actually fly the plane. Nectar's reservation stack has exactly those three layers.
Why reservations exist at all
GPUs are scarce. On-demand boots fail with NoValidHost the moment capacity runs out, and a researcher who needs eight GPUs for a workshop on Tuesday cannot gamble on luck. A reservation guarantees capacity for a time window — by fencing real hypervisor capacity off ahead of time.
Nectar's own booking desk (only some sites use it). Owns the flavor catalogue, quotas, the availability calendar, maintenance windows and user emails. Never touches nova itself — it delegates every booking to blazar.
The upstream OpenStack reservation service. Owns leases, picks the backing hosts, schedules start/end events, and creates all the nova and placement plumbing each reservation needs.
nova boots the instances; placement is the seat map — the ledger that decides which requests fit on which hosts. Blazar manipulates that ledger so reserved capacity is only visible to the reservation holder.
Sites without warre drive blazar directly with openstack reservation lease create. Everything from Module 2 onwards applies to them unchanged — warre is a layer on top, not a replacement.
The map — click each service
The whole lifecycle, as a group chat
Before the deep dive, here is the entire story in one conversation. Press play.
warre books the slot, blazar fences the capacity, and nova/placement only ever see ordinary flavors, aggregates and inventory — reservations are implemented entirely with standard nova/placement primitives. That is why every step is checkable with the standard CLI.
The booking: warre says yes or no
A reservation begins as POST /v1/reservations with just four fields: flavor_id, start, end and an optional instance_count. Before warre says yes, the request runs a gauntlet.
The gauntlet, in order
keystonemiddleware validates the token; policy decides who may create, read, extend, delete.
warre/common/keystone.pyTwo limits, both registered in keystone: reservation (how many) and hours (instance_count × duration). Breach → HTTP 413. Only PENDING_CREATE / ALLOCATED / ACTIVE rows count — ERROR and COMPLETE rows consume no quota.
Flavor must be active; private flavors need a FlavorProject grant; duration ≤ max_length_hours; the window must fit inside the flavor's own start/end; no overlap with a maintenance window (admins can bypass).
A sweep-line pass over every overlapping reservation: each flavor has slots concurrent bookings; multi-instance reservations occupy one slot per instance, maintenance windows occupy all of them.
"No capacity" — the exact moment it is decided
free_slots = self.flavor_free_slots(
context,
flavor,
reservation.start,
reservation.end,
reservation,
include_maintenance=not bypass_maintenance,
)
if free_slots:
f_start = free_slots[0].get("start")
f_end = free_slots[0].get("end")
if f_start != reservation.start or f_end < reservation.end:
raise exceptions.InvalidReservation("No capacity")
else:
raise exceptions.InvalidReservation("No capacity")
If a user reports "it says No capacity but the calendar looks free", the states that block a slot are PENDING_CREATE, ALLOCATED and ACTIVE — including someone's stuck PENDING_CREATE row. ERROR and COMPLETE rows never block. There is also a 60-second guard gap inserted between adjacent bookings.
Accepted → a row and a cast
On success warre commits the row with status=PENDING_CREATE, then hands the real work to its worker with a cast — and immediately returns success to the user.
def create_lease(self, ctxt, reservation_id):
cctxt = self._client.prepare(version="1.0")
cctxt.cast(ctxt, "create_lease",
reservation_id=reservation_id)
The API returns as soon as the message is on RabbitMQ topic warre-worker. If the message is lost or no worker is running, nothing retries it. The row stays PENDING_CREATE forever — while still consuming quota and blocking the calendar slot.
# CLI = python-warreclient, an `openstack` plugin: every command starts with `openstack warre`
$ openstack warre reservation show $RID
| status | PENDING_CREATE | | lease_id | None | | compute_flavor | None | | status_reason | None |
$ openstack warre reservation list --all-projects --flavor $FLAVOR # admin: hunt across projects
# or straight at the warre DB (columns worth knowing by heart)
mysql> SELECT status, status_reason, lease_id, compute_flavor FROM warre.reservation WHERE id='$RID';
# the calendar the capacity check ran against
$ openstack warre flavor free-slots $FLAVOR --start 2026-08-12 --end 2026-09-12
| Start | End | # gaps where new bookings fit
That means the cast was never consumed: warre-worker is down, or RabbitMQ dropped it. There is no periodic retry for this state. Check the worker service and its logs, then delete and re-create the reservation — remember the stuck row is still blocking the slot for everyone else.
The hand-off: a bot books a lease in your project
warre never talks to nova. Its worker impersonates a bot user inside the customer's project and asks blazar for a lease. Everything blazar creates therefore belongs to the user's project — not to warre.
Step one: become the bot
The worker grants bot_user_id the role bot_role_id on the customer's project via keystone admin.
Password auth as the bot, scoped to reservation.project_id. From keystone's point of view, the bot is a member of the project now.
The blazar client is built with service_type="reservation" — no hard-coded endpoint anywhere in warre.
Looking for the lease? It is in the user's project, created by the bot user. openstack reservation lease list as admin shows nothing unless you filter or scope to that project. Bot credential problems (expired password, missing role grants) surface as warre reservations in ERROR with a keystone auth message in status_reason.
The exact payload on the wire
def create_lease(self, reservation):
reservation_info = {
"resource_type": "virtual:instance",
"amount": reservation.instance_count,
"vcpus": reservation.flavor.vcpu,
"memory_mb": reservation.flavor.memory_mb,
"disk_gb": reservation.flavor.disk_gb,
"ephemeral_gb": reservation.flavor.ephemeral_gb,
"affinity": None,
"resource_properties": reservation.flavor.properties,
"extra_specs": reservation.flavor.extra_specs,
}
name = f"Reservation {reservation.id}"
start = reservation.start.strftime(LEASE_DATE_FORMAT)
end = reservation.end.strftime(LEASE_DATE_FORMAT)
lease = self.client.lease.create(
name=name,
start=start,
end=end,
reservations=[reservation_info],
events=[],
)
virtual:instance reservation."
resource_properties is blazar's host filter — at Nectar it matches extra capabilities like warre_id on the blazar hosts backing this warre flavor.
extra_specs and ephemeral_gb are Nectar extensions: upstream blazar 2025.1 only accepts vcpus, memory_mb, disk_gb, amount, affinity, resource_properties. Whether GPU extra specs reach the reserved flavor depends on the site's blazar fork — always verify with openstack flavor show.
Inside blazar: lease, reservation, three alarm clocks
blazar-api wraps the request in a keystone trust, then makes a blocking RPC call to blazar-manager. The manager computes candidate hosts, checks usage enforcement, and writes the lease atomically — including its future.
events.append({'event_type': 'start_lease',
'time': start_date,
'status': status.event.UNDONE})
events.append({'event_type': 'end_lease',
'time': end_date,
'status': status.event.UNDONE})
events table, waiting for their time to come."
A third row, before_end_lease, defaults to 60 minutes before the end (minutes_before_end_lease). The lease goes CREATING → PENDING; the blazar reservation inside it is pending; all three events are UNDONE. These event rows are the schedule — no cron, no timers, nothing else.
The reply lands back in warre
else:
reservation.lease_id = lease["id"]
reservation.compute_flavor = lease.get("reservations")[0].get(
"flavor_id"
)
reservation.status = models.Reservation.ALLOCATED
LOG.info("Created Blazar lease with ID %s", reservation.lease_id)
compute_flavor is the flavor users will boot with. On failure the except branch above this sets status=ERROR and copies the exception into status_reason — which is why blazar errors like NotEnoughHostsAvailable appear verbatim in warre.
Four IDs, easy to confuse: the warre reservation id (the lease is named Reservation <id>) · the blazar lease id (warre's lease_id) · the blazar reservation id inside the lease · and the nova flavor id. The last two are the same string — blazar creates the flavor with flavorid=reservation_id, and warre stores it as compute_flavor.
$ openstack warre reservation show $RID -c status -c lease_id -c compute_flavor
| ALLOCATED | 42a7…lease | 9f2a…rsv |
mysql> SELECT status, lease_id, compute_flavor FROM warre.reservation WHERE id='$RID'; # same, from the DB
# blazar side: the lease exists in the USER's project
$ openstack reservation lease list --project $PROJECT_ID
$ openstack reservation lease show "Reservation $RID"
status: PENDING degraded: False
reservations: [{"status": "pending", "resource_type": "virtual:instance", ...}]
events: start_lease/end_lease/before_end_lease — all UNDONE
Ground prepared early: four artefacts appear at PENDING
Here is the fact that saves you troubleshooting time: blazar builds almost all the nova/placement plumbing when the lease is created — days before it starts. Only the keys to use it are withheld until start time.
One reservation, four artefacts
Named reservation:<rsv-id>, with flavorid = <rsv-id> and is_public=False. Nobody — not even the project — can use it yet.
Named <rsv-id>, metadata reservation=<rsv-id>, filter_tenant_id=<project>, blazar:owner. Zero hosts until the lease starts.
CUSTOM_RESERVATION_<RSV_ID> (UUID upper-cased, dashes → underscores). Registered, but with zero inventory anywhere — worth nothing yet.
computehost_allocations: one row per instance slot, linking the reservation to concrete hypervisors. The ground truth of which hosts back the booking.
The two extra specs that make everything work
# Set extra specs to the flavor
rsv_id_rc_format = reservation_id.upper().replace("-", "_")
reservation_rc = "resources:CUSTOM_RESERVATION_" + rsv_id_rc_format
extra_specs = {
FLAVOR_EXTRA_SPEC: reservation_id,
reservation_rc: "1"
}
if group_id is not None:
extra_specs["affinity_id"] = group_id
reserved_flavor.set_keys(extra_specs)
FLAVOR_EXTRA_SPEC is the constant aggregate_instance_extra_specs:reservation — it must match the aggregate's metadata (module 5).
resources:CUSTOM_RESERVATION_…=1 tells nova to demand one unit of the custom resource class from placement. Together they are the entire enforcement mechanism.
How blazar picked the hosts (not how you think)
query_available_hosts filters blazar's own computehosts table by the request's resource_properties (extra capabilities like warre_id).
For every candidate host it walks all overlapping leases' events and computes the vcpu/memory/disk high-water mark — blazar does its own capacity math from its own DB, not from placement.
instance_plugin.py:94-143 · max_usages()Each instance slot that fits becomes a computehost_allocations row. Not enough slots → NotEnoughHostsAvailable — which lands verbatim in warre's status_reason.
warre uses virtual:instance, which reckons capacity from blazar's computehosts columns (vcpus/memory/disk only). The newer flavor:instance plugin instead schedules against a cached copy of each host's placement inventory (computehost_resource_inventory) and understands resources:* extra specs such as GPU resource classes — but not traits. If a host's cached inventory is stale, its candidate maths is silently wrong.
Verify the ground work
$ openstack flavor show $RSV_ID -c name -c properties
name: reservation:$RSV_ID
properties: aggregate_instance_extra_specs:reservation='$RSV_ID', resources:CUSTOM_RESERVATION_…='1'
$ openstack flavor access list --flavor $RSV_ID
(empty — access is granted at lease start)
# 2. the aggregate — exists, has metadata, has NO hosts yet
$ openstack aggregate show $RSV_ID
hosts: [] properties: reservation='$RSV_ID', filter_tenant_id='…'
# 3. the resource class
$ openstack resource class list | grep CUSTOM_RESERVATION
# 4. host pinning — blazar's allocations API (admin), or its DB
$ openstack reservation allocation list host --reservation-id $RSV_ID
mysql> SELECT compute_host_id FROM blazar.computehost_allocations WHERE reservation_id='$RSV_ID';
openstack aggregate show <rsv-id> and see zero hosts. Is something broken?The clock fires: start_lease
There is no cron job. blazar-manager polls its own events table every 10 seconds — a hard-coded constant — and executes whatever is due. Understanding this loop is understanding every "my lease didn't start" ticket.
The event poller — blazar's only heartbeat
events = db_api.event_get_all_sorted_by_filters(
sort_key='time',
sort_dir='asc',
filters={'status': status.event.UNDONE,
'time': {'op': 'le',
'border': timeutils.utcnow()}}
)
Due events are batched by type in priority order before_end_lease → end_lease → start_lease, so back-to-back leases on the same hosts tear down before the next one starts. Each event is marked IN_PROGRESS and executed in a green thread.
UNDONE past its time → blazar-manager is not running (or the poller is wedged). IN_PROGRESS forever → the executor thread died mid-flight; the event will never be retried automatically. ERROR → the plugin action raised; check blazar-manager logs for the traceback.
on_start: three keys are handed over
The lease transitions PENDING → STARTING, and the plugin's on_start hands the project its keys: flavor access, aggregate membership, and — the real enforcement — placement inventory.
for host_id, num in allocation_map.items():
host = db_api.host_get(host_id)
pool.add_computehost(instance_reservation['aggregate_id'],
host['service_name'], stay_in=True)
self.placement_client.update_reservation_inventory(
host['hypervisor_hostname'], reservation_id, num)
stay_in=True means the host also stays in the freepool — instance reservations share hosts; whole-host reservations remove them.
update_reservation_inventory lazily creates a child resource provider named blazar_<hypervisor> under the compute node, then sets inventory CUSTOM_RESERVATION_<ID>: total=num, max_unit=1, allocation_ratio=1.0.
Just before that loop, the project finally gets to see the flavor: self.nova.flavor_access.add_tenant_access(reservation_id, ctx.project_id) instance_plugin.py:595-597. Then the lease settles at ACTIVE, and blazar emits the notification lease.event.start_lease.
The whisper back to warre
Publisher blazar.lease, event_type lease.event.start_lease, payload: lease_id, user, project, dates.
The site's telemetry pipeline forwards it as a ceilometer-style event onto exchange ceilometer, topic warre — where warre-notification listens.
Looks up the reservation by the lease_id trait, sets ACTIVE, emits warre.reservation.start, emails the user.
status: ACTIVE events: start_lease → DONE
# the alarm-clock rows themselves (what blazar-manager polls every 10 s)
mysql> SELECT event_type, time, status FROM blazar.events WHERE lease_id='$LEASE_ID';
$ openstack flavor access list --flavor $RSV_ID
| $RSV_ID | $PROJECT_ID | # project can now see the flavor
$ openstack aggregate show $RSV_ID -c hosts
hosts: ['qh2-rcc123', …] # reserved hypervisors joined
# the placement seats: child RP under each reserved compute node
$ openstack resource provider list --name blazar_qh2-rcc123
$ openstack resource provider inventory list $BLAZAR_RP_UUID
| CUSTOM_RESERVATION_9F2A… | total=4 | max_unit=1 | allocation_ratio=1.0 |
# warre agrees (or the notification pipeline is broken)
$ openstack warre reservation show $RID -c status → ACTIVE
mysql> SELECT status FROM warre.reservation WHERE id='$RID'; → ACTIVE
The lease started but the notification never arrived: check blazar's notification transport, the telemetry relay onto exchange ceilometer / topic warre, and the warre-notification daemon. The safety net is warre-worker's 30-minute periodic, which only fixes the end case ("has ended but still active" warning) — there is no equivalent for a missed start. The user never gets their "your reservation is ready" email.
Booting: two extra specs steer nova
The user runs openstack server create --flavor <rsv-id>. From here on it is a completely ordinary nova boot — except the flavor's two extra specs quietly bend scheduling toward the reserved hosts.
Spec one goes to placement
# extra_specs-specific consts
XS_RES_PREFIX = 'resources'
XS_TRAIT_PREFIX = 'trait'
# Regex patterns for suffixed or unsuffixed resources/trait keys
XS_KEYPAT = re.compile(r"^(%s)([a-zA-Z0-9_-]{1,64})?:(.*)$" %
'|'.join((XS_RES_PREFIX, XS_TRAIT_PREFIX)))
resources: or trait: becomes part of the placement query."
So resources:CUSTOM_RESERVATION_…=1 joins VCPU/MEMORY_MB/DISK_GB in the request: GET /allocation_candidates?resources=CUSTOM_RESERVATION_…:1,DISK_GB:30,MEMORY_MB:…,VCPU:…
Note what doesn't match: aggregate_instance_extra_specs:… never reaches placement. It is handled later, by a host filter.
Only hosts whose blazar_<host> child provider carries that inventory can answer. The allocation that comes back — and is claimed with PUT /allocations/<instance> — spans two providers:
| $ROOT_RP (compute node) | {'VCPU': 4, 'MEMORY_MB': 16384, 'DISK_GB': 30} |
| $CHILD_RP (blazar_qh2-…) | {'CUSTOM_RESERVATION_9F2A…': 1} |
# dry-run scheduling: would placement even return candidates?
$ openstack allocation candidate list --resource CUSTOM_RESERVATION_9F2A…=1
# seats taken vs total (total=amount, max_unit=1 caps concurrency)
$ openstack resource provider usage show $CHILD_RP
Spec two is checked by a filter
aggregate_vals = metadata.get(key, None)
if not aggregate_vals:
LOG.debug(
"%(host_state)s fails flavor extra_specs requirements. "
"Extra_spec %(key)s is not in aggregate.",
{'host_state': host_state, 'key': key})
return False
AggregateInstanceExtraSpecsFilter matches the flavor's aggregate_instance_extra_specs:reservation=<id> against the reservation aggregate's reservation=<id> metadata from module 3.
Config gotcha: this filter is not in nova's default enabled_filters. If it is missing from nova.conf, the aggregate half of the confinement silently does nothing — placement inventory becomes the only fence.
The nectar/2024.1 branch carries a Nectarism ("Add more host info in scheduler log", nova/filters.py:86-104) that logs each filter's surviving host list at INFO — upstream only logs counts at DEBUG. In nova-scheduler logs you can literally watch Filter AggregateInstanceExtraSpecsFilter returned 2 hosts [...] for the instance UUID and see exactly where a reserved boot lost its hosts.
Triage: three failures, three fingerprints
The flavor is private and access is only granted at start_lease. Before that, nova-api answers 400 (FlavorNotFound); no instance record, nothing in scheduler logs.
Inventory missing (start never ran?) or all seats taken. The tell: NoValidHost with no filter lines at all in the scheduler log — it never reached filtering.
nova/scheduler/manager.py:237-243Placement returned candidates but AggregateInstanceExtraSpecsFilter rejected them: aggregate metadata missing or hosts never joined. The Nectar INFO logs show which filter zeroed the list.
Got no allocation candidates from the Placement API and no filter lines at all. Where do you look first?The end: teardown is not gentle
Sixty minutes before the end, before_end_lease fires — for instance reservations it changes nothing, it only triggers warre's warning email. Then end_lease arrives, and it means it.
end_lease, step by step
No new instances can be created with the reservation flavor.
instance_plugin.py:627-631computehost_allocations rows destroyed.
See the code below. This is the step users must be warned about.
instance_plugin.py:641-651Polls every 5 s, up to 10 minutes. Timeout → ServerDeletionTimeout → lease ERROR, teardown incomplete.
cleanup_resources() — the aggregate delete moves any hosts back toward the freepool first.
Inventory removed from each blazar_<host>, then the resource class deleted. The child RP itself survives — it belongs to the host, not the lease.
The sharp edge, verbatim
for server in self.nova.servers.list(search_opts={
'flavor': reservation_id,
'all_tenants': 1}, detailed=False):
try:
self.nova.servers.delete(server=server)
except nova_exceptions.NotFound:
LOG.info("Could not find server '%s', may have been deleted "
"concurrently.", server.id)
except Exception as e:
LOG.exception("Failed to delete server '%s': %s.", server.id,
str(e))
Not shelved. Not stopped. Deleted. This is why warre's before_end email says instances "will be DELETED" — and why researchers must snapshot or copy results off before the lease ends.
$ openstack flavor show $RSV_ID → No flavor with a name or ID … (404 ✓)
$ openstack aggregate list | grep $RSV_ID → (empty ✓)
$ openstack resource class list | grep 9F2A → (empty ✓)
$ openstack server list --all-projects --flavor $RSV_ID → (empty ✓)
# the child RP blazar_<host> still exists — that is normal; it is per-host, not per-lease
# warre: COMPLETE now, row auto-purged 7 days after end
$ openstack warre reservation show $RID -c status → COMPLETE
mysql> SELECT status FROM warre.reservation WHERE id='$RID'; → COMPLETE
If nova cannot delete an instance within 10 minutes (host down, deletion stuck in deleting), blazar raises ServerDeletionTimeout: the lease lands in ERROR with teardown half-done — flavor and aggregate possibly still present, inventory still in placement. Fix the stuck instance, then delete the lease (openstack reservation lease delete re-runs on_end for reservations not yet deleted).
The full journey, animated
Twenty-one steps, four services, one booking. Watch where the packet leaves one project and enters the next — every hop is a place where things can silently stall.
The troubleshooting playbook
Six symptoms cover almost every reservation ticket. For each: the fingerprint, and the first place to look.
Symptom → first look
The cast to warre-worker was never consumed. Check the worker service and RabbitMQ. Nothing retries this — and the row keeps blocking quota and the calendar slot until deleted.
Read status_reason — it is the blazar exception verbatim. NotEnoughHostsAvailable → blazar host capacity; keystone auth errors → the bot user's credentials or role grants.
lease show → look at the events. UNDONE past its time → blazar-manager down. IN_PROGRESS forever → executor thread died. ERROR → plugin raised; read blazar-manager logs.
Lease ACTIVE/TERMINATED but warre still ALLOCATED/ACTIVE → the notification relay (blazar → exchange ceilometer, topic warre → warre-notification) is broken. Watch for warre-worker's "has ended but still active" warnings.
A reserved host failed. Blazar marked it reservable=0, tried to heal onto a replacement, and set reservation flags missing_resources / resources_changed. All three surface in lease show.
warre-worker crashed after creating the lease but before committing → a lease warre doesn't know about (warre-notification logs "unknown lease" warnings). Failed teardown → leftover flavor/aggregate/inventory. Compare warre rows against lease list per project.
Where the truth lives
When two layers disagree, this is the pecking order of evidence — each layer only knows about its own objects.
For any reservation ticket, walk the chain in order: warre status → lease status + events → flavor + access → aggregate hosts + metadata → child RP inventory + usage. The first mismatch between two adjacent layers is where the fault lives.
Scenario quiz
openstack reservation lease show says TERMINATED. warre-worker logs repeat "has ended but still active, marking as COMPLETE". What actually broke?server create. Fastest next step?Match the question to the command
Drag each command onto the question it answers.
warre books the slot and holds the quota; blazar schedules events and fences capacity; nova and placement enforce it with a private flavor, an aggregate, and a custom resource class on a child provider. Every hand-off has a CLI check — and the first mismatch between layers is always your fault line.