An interactive course · OpenStack Neutron + OVN
Reaching Out
Project Networks, Routers and Floating IPs with OVN
You have created a network, a router and a floating IP a hundred times in the dashboard. This course shows what each of those clicks actually writes into OVN, and how a packet from your laptop finds one VM among thousands.
Sibling course: Plugged In follows how a VM boots and gets its port wired. Here we assume the VM boots fine, and study the shape of the network around it.
SCROLL TO SET SAIL ↓
The Island Story
The whole course in one story: an island, a causeway and a letterbox, told in five commands.
One island chain, four ideas
Every concept in this course is a spot on the map above. Four ideas, four OVN objects; keep the map in your head and the database tables will follow.
The private island
A project network: every VM on it can chat freely, nothing can leave the shore. In OVN it is exactly one Logical_Switch.
The causeway
A router: the bridge you build towards the mainland, the provider network. In OVN it is one Logical_Router.
The shared postage address
Outgoing mail gets stamped with one public return address shared by the whole island. That is SNAT, and it works one way: outward.
Your beachfront letterbox
A floating IP with your name on it, so the outside world can find one specific house. In OVN it is a single dnat_and_snat rule (DNAT in, SNAT back out).
Watch the whole story happen
Press play: five commands from you, and Neutron narrates the rows each one writes. In the room too: the NB database (OVN's blueprint registry) and ovn-controller (the muscle on every node). Every later module zooms into one exchange from this conversation.
Networks are just named rows
"Create network" patches no cable and starts no process. It ends here, as one row in the NB database: a Logical_Switch whose name is simply neutron-<your-network-uuid>.
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 2097-2102
def create_network(self, context, network): # Create a logical switch with a name equal to the Neutron network # UUID. This provides an easy way to refer to the logical switch # without having to track what UUID OVN assigned to it. lswitch_params = self._gen_network_parameters(network) lswitch_name = utils.ovn_name(network['id'])When you click "create network", Neutron's OVN driver lands here with your network in hand.
The comment says it all: the logical switch will be named after your network...
...so anything can find it later just by knowing the Neutron ID...
...with no lookup table of OVN-assigned IDs needed, ever.
Bundle up the network's settings (name, MTU and friends) ready for the new row.
And the name is literally neutron-<your-network-uuid>. Remember this trick; the whole course leans on it.
In the sibling course, Neutron wrote port rows and ovn-controller made them real. Routing is the same architecture with different tables: Logical_Router, NAT rules and static routes instead of ports. No agents, no namespaces on network nodes; routing is rows that become flows.
Seven steps to the open sea
The whole scenario is Neutron translating five API calls into NB rows. Each card is one step; the tag says which module unpacks it.
openstack network create
A private island rises from the sea: one Logical_Switch row.
openstack subnet create
Street addresses for the island: a CIDR range plus DHCP data.
openstack router create
A causeway with no far end yet: a Logical_Router with nothing attached.
openstack router add subnet
The causeway touches your island: a Logical_Router_Port on the switch.
openstack router set --external-gateway
The causeway reaches the mainland: a default route and SNAT rules appear.
openstack floating ip create + server add floating ip
Your own letterbox: one dnat_and_snat rule with your VM's address on it.
The payoff: we trace one inbound packet all the way from the mainland to the front door.
"My VM has no internet" and "the floating IP does not answer" are the two classic tickets. After this course you can bisect them like a boot hang: which row is missing, and which chassis should be doing the work?
Check your bearings
Three scenarios, no memorising. Steer by the island map.
A brand-new project network has no router at all, yet two VMs on the same subnet can reach each other. Why?
Working through the seven steps, at which moment could the outside world first reach your VM?
You run openstack router set r1 --external-gateway public. What does Neutron actually write?
Next stop: Your Private Island, where we watch network create and subnet create build the island row by row.
Your Private Island
Two clicks, one database row, and a new island rises from the sea. This is how networks and subnets are born.
Two clicks, and an island appears
In The Island Story you sailed the whole seven-step journey; this module drops anchor at steps 1 and 2, where the island itself is born.
Two dashboard clicks, and every hypervisor in the region can suddenly deliver packets for a network that did not exist ten seconds ago. Nothing was installed anywhere. What changed?
Think of it as founding a new island town, in three parts:
The island
network create raises it: one Logical_Switch row in OVN's Northbound database, named neutron-<network-uuid>.
The street grid
subnet create draws it: a CIDR, an allocation pool and DHCP data. House numbering for every future resident.
The ferry terminal
Only provider networks get one: a localnet port docking the island to a real physical wharf. Project networks are reachable only by tunnel.
Step 1: raising the island
When you click Create Network, Neutron hands the request to its OVN driver, and this function runs. The heart of it is ls_add: one new row in the Northbound database, and the island exists everywhere at once.
def create_network(self, context, network): # Create a logical switch with a name equal to the Neutron network # UUID. This provides an easy way to refer to the logical switch # without having to track what UUID OVN assigned to it. lswitch_params = self._gen_network_parameters(network) lswitch_name = utils.ovn_name(network['id']) # NOTE(mjozefcz): Remove this workaround when bug # 1869877 will be fixed. segments = segments_db.get_network_segments( context, network['id']) with self._nb_idl.transaction(check_error=True) as txn: txn.add(self._nb_idl.ls_add(lswitch_name, **lswitch_params, may_exist=True)) for segment in segments: if segment.get(segment_def.PHYSICAL_NETWORK): self.create_provnet_port(network['id'], segment, txn=txn) db_rev.bump_revision(context, network, ovn_const.TYPE_NETWORKS) self.create_metadata_port(context, network) return networkThe function Neutron runs for every new network, project and provider alike.
The naming trick, in the authors' own words: name the switch after the network UUID, so the two worlds never need a lookup table to find each other.
Pack the island's founding charter: the settings this network was created with.
Build the name: neutron- followed by the network's UUID.
A developer's sticky note about a workaround. Real code keeps its honest clutter.
Ask how this network is carried: its segments reveal whether it touches anything physical.
Open one all-or-nothing transaction against the Northbound database.
Raise the island. This single row IS the network, and may_exist=True makes it safe to run twice.
Only segments that name a physical network earn a ferry terminal. Project networks sail straight past this loop.
Stamp a revision number so the sync machinery can tell fresh data from stale.
Every island gets a concierge desk: the metadata port, built before a single VM arrives.
Done. Hand the finished network back.
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py, lines 2097-2115
Two kinds of island
Every network you will ever create is one of these two. In the database, the difference between them is exactly one row.
Project network: the tunnelled island
🚇 Reachable only by Geneve tunnel: traffic between hypervisors rides inside ordinary IP packets.
🗺️ Exists on every chassis at once, because every chassis reads the same database.
🆓 Costs nothing physical: no VLAN consumed, no switch configured. Create fifty of them before lunch.
🧾 In the Northbound database: one Logical_Switch row. That is all.
Provider network: the island with a ferry terminal
🧾 Everything the project island has, plus exactly one extra row: a localnet port.
🏷️ That port carries a VLAN tag, and its options name a physnet.
⚓ On each chassis, ovn-controller docks the ferry only if its own bridge mapping lists that physnet.
⚠️ No mapping on a chassis means no physical reach from that chassis.
Here is the ferry terminal being built, all eight lines of it.
cmd = self._nb_idl.create_lswitch_port( lport_name=utils.ovn_provnet_port_name(segment['id']), lswitch_name=utils.ovn_name(network_id), addresses=[ovn_const.UNKNOWN_ADDR], external_ids={}, type=ovn_const.LSP_TYPE_LOCALNET, tag=tag, options=options)Add one more port to the island, in the same database.
Name it after the segment, so anyone browsing the database can spot ferry terminals at a glance.
Attach it to this network's switch, found via the UUID naming trick.
Address: unknown. The ferry carries traffic for anyone, not for one fixed machine.
No Neutron bookkeeping notes attached.
The magic word. Type localnet declares: this port is a doorway to a physical network.
The VLAN tag this traffic will wear on the physical wire.
And the physnet name, which every chassis will compare against its own bridge mappings.
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py, lines 2053-2060
The Logical_Switch row and its localnet port are identical for every chassis, because they all read the same database. So a provider network that works on some hypervisors and not others is almost always a bridge mapping problem on the broken ones.
Step 2: drawing the street grid
A subnet gives the island its addressing plan: the CIDR, the allocation pool, the DHCP answers. Watch how little OVN is asked to do.
def create_subnet(self, context, subnet, network): if subnet['enable_dhcp']: mport_updated = False if subnet['ip_version'] == const.IP_VERSION_4: mport_updated = self.update_metadata_port( context, network, subnet=subnet) if subnet['ip_version'] == const.IP_VERSION_6 or not mport_updated: # NOTE(ralonsoh): if IPv4 but the metadata port has not been # updated, the DHPC options register has not been created. self._add_subnet_dhcp_options(subnet, network) db_rev.bump_revision(context, subnet, ovn_const.TYPE_SUBNETS)Step 2 begins. Notice what is missing: no switch, no tunnel, not even a new port.
If DHCP is off, OVN is barely involved at all.
Keep score: has the concierge desk been updated yet?
For IPv4, give the metadata port an address on the new street. That update creates the DHCP options along the way.
For IPv6, or when that shortcut did not fire...
...the comment explains why (typo and all: we promised you the unedited code)...
...write the DHCP_Options row directly: the answers VMs will hear when they ask for an address.
Finally, stamp the revision, exactly as create_network did.
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py, lines 2502-2512
So where did the street numbers actually get reserved? Not here. Two ledgers, two jobs:
The IPAM ledger and the allocation pool live here, in ordinary SQL. Every next-free-IP decision is made on this side.
It stores outcomes only: addresses on ports, DHCP_Options rows. OVN never chooses an IP.
create_network builds the very metadata port the sibling course's metadata module runs on, before any VM exists. Each new subnet merely adds its street to that port's address list; nothing is ever rebuilt.
Watch the two clicks travel
Two commands enter through the Neutron API; everything after that is database writes. Press Next Step and notice who never gets a message.
$ openstack network show blue-island -c id -c status → $NETWORK_ID | ACTIVE
$ ovn-nbctl ls-list | grep neutron-$NETWORK_ID
# the concierge desk is already there, before any VM: the metadata port
$ openstack port list --network blue-island --device-owner network:distributed
# the street grid: DHCP answers waiting for the first resident
$ ovn-nbctl --columns=cidr,options find DHCP_Options 'external_ids:"neutron:subnet_id"=$SUBNET_ID'
| cidr: 10.0.0.0/24 | options: {server_id=…, router=…, mtu=…} |
# provider island? the ferry terminal must exist, typed localnet, naming its physnet
$ ovn-nbctl lsp-get-type provnet-$SEGMENT_ID → localnet
$ ovn-nbctl lsp-get-options provnet-$SEGMENT_ID → network_name=physnet1
# works on some hypervisors and not others? the field note says bridge mappings:
$ ovn-sbctl --columns=hostname,other_config find Chassis hostname=$BROKEN_HOST
Prove it to yourself
Three situations you could meet next week. Reason them out with the island, the ferry terminal and the two ledgers.
Your team spins up 50 project networks for a test suite. The datacentre has only about 4000 VLANs in total. Should you warn the network engineers?
A provider network gives VMs perfect connectivity on hypervisors A and B, but VMs on hypervisor C get nothing. Where do you look first?
A new VM boots and receives 10.0.0.57. Which component decided that 57 was the next free number?
The island is built; next, in The Bridge, we raise the router that connects it to other islands and to the mainland.
The Bridge
Your router is not a machine somewhere. It is a causeway that every hypervisor carries a copy of.
Where is my router?
Module 2 left you owning Your Private Island: a network with a subnet, isolated by design. Time to build a bridge to the other islands.
In the old ML2/OVS world a router was a Linux namespace on a network node: a real place that could be full, slow or dead. With OVN, ask where your router is and the honest answer is: everywhere.
Then · ML2/OVS router
namespace
a real place on one network node
L3 agent
a L3 agent process that must stay alive
choke point
cross-subnet traffic detours through one box
failover
HA via VRRP standbys, fingers crossed
Now · OVN router
rows
one Logical_Router record in the northbound database
no process
nothing to restart, nothing to die
routed locally
east-west traffic never leaves the hypervisor
nothing to fail over
every chassis already carries the router as flows
A different department builds bridges
Networks came from the ML2 mechanism driver. Routers come from somewhere else: a service plugin called OVNL3RouterPlugin, living in neutron/services/ovn_l3/plugin.py.
Separate department
Routers are an add-on API, served by their own plugin, not by the core networking machinery.
Borrowed filing cabinets
It reuses Neutron standard L3 database layer, so ownership, quotas and the API behave exactly as always.
Same pen
All OVN writes go through the same OVNClient the ML2 driver uses. One author, one wish list.
self.scheduler = l3_ovn_scheduler.get_scheduler()
line 95 of plugin.py: the gateway scheduler. Remember this one, it is the star of module 4.
Prefabricating the causeway
Think of router create as prefabricating a causeway in a yard: the whole span is built, but nothing is attached to any island yet. Here is the entire ceremony.
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 1491-1504
def create_router(self, context, router, add_external_gateway=True):
"""Create a logical router."""
external_ids = self._gen_router_ext_ids(router)
enabled = router.get('admin_state_up')
lrouter_name = utils.ovn_name(router['id'])
added_gw_ports = []
options = {'always_learn_from_arp_request': 'false',
'dynamic_neigh_routers': 'true',
ovn_const.LR_OPTIONS_MAC_AGE_LIMIT:
ovn_conf.get_ovn_mac_binding_age_threshold()}
with self._nb_idl.transaction(check_error=True) as txn:
txn.add(self._nb_idl.lr_add(router=lrouter_name, may_exist=True,
external_ids=external_ids,
enabled=enabled, options=options))
When you ask for a router, this one function answers. Notice how little it does.
The code admits it up front: create a LOGICAL router. Written down, not built.
Prepare sticky labels tying the new row back to your Neutron router.
Note whether you created the router switched on or off.
Same naming trick as your island: the row is named neutron- plus the router UUID.
An empty list reserved for gateway ports. Ours stays empty until module 4.
Now some manners: do not memorise a neighbour from every ARP request you overhear...
...but do actively keep the neighbour entries you hold fresh...
...and forget a learned MAC address after a configurable age. Hygiene tuning, not plumbing.
Open one all-or-nothing transaction against the northbound database...
...and write a single Logical_Router row. That is the whole router. No process started, anywhere.
Landing the causeway on your island
router add subnet anchors one end of the causeway. That takes two port-ish rows: a ramp on the router, called a LRP, and the island-side switch port flipped to pair with it, wired together like a patch port.
Bolt a ramp (LRP) onto the router
Flip the island switch port to type router
Pair the two: traffic can now cross
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 1783-1807
commands = [
self._nb_idl.add_lrouter_port(
name=lrouter_port_name,
lrouter=lrouter,
mac=port['mac_address'],
networks=networks,
may_exist=True,
external_ids=self._gen_router_port_ext_ids(port, router['id']),
**columns)
]
if is_gw_port:
port_net = self._plugin.get_network(
n_context.get_admin_context(), port['network_id'])
physnet = self._get_physnet(port_net)
az_hints = common_utils.get_az_hints(router)
commands.append(
self._nb_idl.schedule_new_gateway(lrouter_port_name,
self._sb_idl,
lrouter, self._l3_plugin,
physnet, az_hints))
commands.append(
self._nb_idl.set_lrouter_port_in_lswitch_port(
port['id'], lrouter_port_name, is_gw_port=is_gw_port,
lsp_address=lsp_address))
Start a shopping list of database writes.
First item: the landing ramp, a Logical Router Port bolted onto the router...
...named with the usual convention and attached to your router row...
...carrying the MAC address the ramp answers on...
...and the IP networks it serves for this island.
No drama if the ramp already exists (may_exist)...
...plus sticky labels tying it back to the Neutron port, and any extra columns.
Ramp done. Now, a fork in the road.
Is this port the way OUT to the wider world? Then extra paperwork applies:
...look up the network the port sits on...
...find which physical network the outside world lives on...
...respect any availability zone preferences on the router...
...and hold a little election: schedule one chassis for gateway duty. Hold that thought.
Finally, every time, gateway or not:
...flip the island-side switch port to type router...
...and pair it with the ramp by name. Two anchored ends, one causeway.
Watch it appear everywhere at once
The cast: you, the L3 plugin, the northbound database, ovn-northd, and ovn-controller on two compute nodes. Two VMs on different subnets, both on compute-1. Click through, and keep your eye on the final step.
ML2/OVS needed DVR, a whole configuration maze, to spread routing across compute nodes. In OVN, east-west distribution is not a feature you enable. It is the only mode there is.
$ openstack router show causeway-1 -c id -c status → $ROUTER_ID | ACTIVE
$ ovn-nbctl lr-list | grep neutron-$ROUTER_ID
# the ramps: one LRP per attached subnet, carrying the router's MAC and IP
$ ovn-nbctl lrp-list neutron-$ROUTER_ID → lrp-$PORT_ID (one per subnet)
# the island end: the switch port flipped to type router and paired by name
$ ovn-nbctl lsp-get-type $PORT_ID → router
$ ovn-nbctl lsp-get-options $PORT_ID → router-port=lrp-$PORT_ID
# both ramps present but cross-subnet ping still fails? then it is not routing:
# check both VM ports are ACTIVE and the security groups allow the traffic
Check your bridge-building instincts
Two VMs sit on different subnets of the same router, on the same hypervisor. What path does a packet take between them?
Routing misbehaves. In ML2/OVS days you might restart the router. What is the OVN equivalent?
Architecture puzzle: why does attaching a subnet write TWO port-ish rows?
Snippet B hid a mystery: if is_gw_port: runs only when the port is the router door to the outside world, and schedule_new_gateway quietly holds an election for which chassis stands guard. Module 4, The Way Out, counts the votes: the gateway chassis, SNAT, and the way off your island.
The Way Out
Every island posts its mail through one elected port town. This is SNAT, the gateway chassis, and the road to the internet.
The branch we tiptoed past, resolved
Module 3 ended on a mystery: if is_gw_port:, a branch that quietly calls schedule_new_gateway. It fires the moment you run openstack router set --external-gateway ext-net, and it exists because of an awkward truth.
East-west routing lives everywhere, but everywhere cannot own one public IP. The moment your traffic heads for the internet, someone specific has to do the job.
So picture the island post office. Outgoing mail from any house is stamped with one shared return address, and all of it ships through a single designated port town.
SNAT rewrites the source of every outbound packet to the router external IP. The houses never write their own.
The gateway LRP is pinned to an elected gateway chassis. All north-south traffic funnels through it.
The scheduler ranks up to five port towns. If the port floods, the next takes over automatically.
One command, three writes
Setting the external gateway lands in _add_router_ext_gw, and the code numbers its own to-do list 1, 2, 3. Here are jobs one and three, with job two summarised in between.
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 1332-1338 and 1357-1366
# 1. Add the external gateway router port.
admin_context = context.elevated()
added_ports = []
for gw_port in self._get_router_gw_ports(admin_context, router['id']):
port = self._plugin.get_port(admin_context, gw_port['id'])
self._create_lrouter_port(admin_context, router, port, txn=txn)
added_ports.append(port)
⋯ lines 1339 to 1356 prepare job 2, the route for off-island mail ⋯
txn.add(self._nb_idl.add_static_route(
lrouter_name, ip_prefix=gw_info.ip_prefix,
nexthop=gw_info.gateway_ip,
maintain_bfd=router_default_route_bfd_enabled,
**columns))
# 3. Add necessary snat rule(s) in lrouter if snat is enabled
if utils.is_snat_enabled(router):
self.update_nat_rules(router['id'], enable_snat=True, txn=txn)
return added_ports
The code numbers its own to-do list. Job one: hang the door.
Put on the admin hat: the provider network belongs to the operator, not to your project.
A list to remember every door we hang.
Fetch each gateway port Neutron recorded for this router (usually exactly one)...
...load its full details, MAC and IP included...
...and bolt it on with the very same ramp-builder from module 3. A gateway is just an LRP with a fancy job, written in the shared transaction.
Remember it for the caller.
(the skipped lines work out gw_info: which prefix and which gateway address the provider network offers)
Job two lands: write one static route on the router...
...matching the catch-all default prefix, meaning anywhere not local...
...and forward such packets to the nexthop, the provider network gateway.
Optionally watch that nexthop with BFD, a fast are-you-alive heartbeat.
Any extra columns, and the route is written.
Job three announces itself in its own comment.
Only if SNAT is switched on for this router (it is, by default)...
...fire the stamp machine. That is our next screen.
Done: hand back the doors we hung.
Three writes, three distinct failure modes. Quiz yourself before peeking: what breaks if each one goes missing?
1 · Gateway port
The router door onto the provider network, an LRP pinned to elected chassis. If missing: there is no door at all, and nothing north-south moves.
2 · Default route
One rule: anything not local, hand to the provider gateway. If missing: the router shrugs at off-island mail and drops it, door or no door.
3 · SNAT rules
One per internal CIDR, all stamped with the router external IP. If missing: packets leave with private return addresses, so replies never come home.
One stamp for every street
Job three opens into update_nat_rules, the stamp machine. It writes one snat row for every internal street, and every row carries the same shared stamp.
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 2018-2035
def update_nat_rules(self, router_id, enable_snat, cidrs=None, txn=None):
if enable_snat:
idl_func = self._nb_idl.add_nat_rule_in_lrouter
else:
idl_func = self._nb_idl.delete_nat_rule_in_lrouter
func = functools.partial(
idl_func, utils.ovn_name(router_id), type='snat')
context = n_context.get_admin_context()
cidrs = (
cidrs or
self._get_snat_cidrs_for_external_router(context, router_id)
)
commands = [
func(logical_ip=cidr, external_ip=router_ip)
for router_ip in self._iter_ipv4_gw_addrs(context, router_id)
for cidr in cidrs
]
One machine both stamps and un-stamps, depending on enable_snat.
Turning SNAT on?
Then every command will add a nat rule inside the logical router.
Turning it off?
Exactly the same shape, delete instead of add.
Pre-fill the paperwork shared by every rule:
this router, and always type snat.
Admin hat on again.
Which streets get stamped?
Either the exact list the caller handed over, or...
...every internal CIDR currently plugged into this router.
Now the mail run, one rule per pairing:
the internal street as logical_ip, the router address as the external_ip stamp...
...for each IPv4 address the router holds on the outside (usually just one)...
...crossed with every internal street.
Every street, one shared stamp.
What lands in the logical router, for a router with two subnets:
snat 192.168.10.0/24 → 203.0.113.50
street one, stamped
snat 192.168.20.0/24 → 203.0.113.50
street two, same stamp
Notice what the stamp implies: to un-stamp the replies, something must remember every conversation. That memory is connection state, and it is why this job cannot live everywhere.
Electing the port town
Because SNAT needs one owner, the gateway port is pinned to specific chassis, chosen by election. First, the electoral roll: who may even stand?
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 1644-1650
candidates = set()
for chassis, physnets in chassis_physnets.items():
if (physnet and
physnet in physnets and
chassis in cms):
candidates.add(chassis)
candidates = list(candidates)
Start an empty electoral roll.
Walk every chassis in the cloud, with the physnets each one carries...
A name goes on the roll only if we know which external physnet we need...
...this chassis actually carries it, via a bridge mapping (a real cable to the outside)...
...and the chassis volunteered: cms lists every chassis that registered enable-chassis-as-gw.
Both boxes ticked? Onto the roll.
Roll closed. No ranking yet, just eligibility.
Then the count. The scheduler seats up to five winners, ranked by priority, so the port town always has understudies.
neutron/scheduler/l3_ovn_scheduler.py · lines 73-92
if not candidates:
LOG.warning('Gateway %s was not scheduled on any chassis, no '
'candidates are available', gateway_name)
return
chassis_count = min(
ovn_const.MAX_GW_CHASSIS - len(existing_chassis),
len(candidates)
)
# The actual binding of the gateway to a chassis via the options
# column or gateway_chassis column in the OVN_Northbound is done
# by the caller
chassis = self._select_gateway_chassis(
nb_idl, sb_idl, candidates, 1, chassis_count, target_lrouter
)[:chassis_count]
# priority of existing chassis is higher than candidates
chassis = existing_chassis + chassis
LOG.debug("Gateway %s scheduled on chassis %s",
gateway_name, chassis)
return chassis
Nobody on the roll?
Then this exact warning lands in the log. Word for word, it is the line you grep for...
...when a router silently has no internet.
And scheduling gives up. No gateway, no way out.
How many seats to fill?
Up to MAX_GW_CHASSIS (that is 5), minus seats already taken...
...and never more than the roll can supply.
The scheduler is careful to say what it does not do:
it only picks names. The caller writes the actual Gateway_Chassis rows...
...into the northbound database.
Run the election proper. The default policy seats the least-loaded chassis first...
...(the other policy, chance, is literally a random shuffle)...
...then trim to the number of open seats.
Incumbents outrank newcomers, so...
...a reshuffle never demotes a healthy sitting gateway.
Publish the results...
...and return them ranked. Top priority carries the traffic today.
The result, written as Gateway_Chassis rows on the port:
priority 5
gw-host-3: elected, carries the traffic today
priority 4
gw-host-1: first understudy, standing by
priority 3, 2, 1
further understudies, ranked and ready
Neutron schedules: it picks names and priorities and writes the rows, once. OVN fails over: chassis watch each other with BFD and the next rank takes over in moments, no Neutron in the loop. The slow control plane makes the decision; the fast data plane handles the emergency.
$ openstack router show causeway-1 -c external_gateway_info → network_id + the external IP
# write 2, the route: anything not local goes to the provider gateway
$ ovn-nbctl lr-route-list neutron-$ROUTER_ID → 0.0.0.0/0 203.0.113.1 dst-ip
# write 3, the stamps: one snat row per internal street, one shared stamp
$ ovn-nbctl lr-nat-list neutron-$ROUTER_ID
| snat | 203.0.113.50 | 192.168.10.0/24 |
| snat | 203.0.113.50 | 192.168.20.0/24 |
# the election result: ranked winners on the gateway port, priority 5 down to 1
$ ovn-nbctl lrp-get-gateway-chassis lrp-$GW_PORT_ID
# who is carrying it right now: BFD may already have promoted an understudy
$ ovn-sbctl --columns=chassis find Port_Binding logical_port=cr-lrp-$GW_PORT_ID
# empty election? grep the neutron log for 'was not scheduled on any chassis',
# then inspect the volunteers: enable-chassis-as-gw and the physnet mapping
$ ovn-sbctl --columns=hostname,other_config find Chassis hostname=gw-host-3
| other_config: {ovn-bridge-mappings="physnet1:br-ex", ovn-cms-options=enable-chassis-as-gw} |
Follow a packet off the island
The cast: a VM at 192.168.10.5 on compute-1, whose OVS carries the router as flows, plus the elected gateway chassis and the wider world. Between chassis, traffic rides a Geneve tunnel.
Click through both legs of the trip, and watch where the reply is forced to land.
No internet? You now have an algorithm
"VMs cannot reach the internet" stops being a mystery and becomes a checklist, run top to bottom. It also answers the audit question "which node carries our internet traffic?" precisely.
Check the router for an external gateway. No door, no way out: write number one never happened.
Grep the logs for: was not scheduled on any chassis. An empty roll means no volunteer carries the external physnet.
Find the top-priority Gateway_Chassis entry and check that host and its uplink. BFD may already have promoted an understudy.
One snat row per internal CIDR on the logical router, all stamped with the router external IP.
VMs on all networks behind router r1 lost internet access at the same moment, yet east-west traffic is fine. What is your first suspect?
The log says: Gateway ... was not scheduled on any chassis, no candidates are available. What two properties must a chassis have to make the roll?
Architecture puzzle: routing is distributed to every chassis, so why is SNAT centralised on a gateway chassis?
What limits how many chassis back up one gateway port, and how does takeover actually happen?
SNAT sends everyone out under one shared address. Module 5, The Way In, hands the world a key to one specific VM: the floating IP.
The Way In
A floating IP is one database row that gives a single VM a public name, and, optionally, its own private path.
A letterbox on the beachfront
Module 4 built the way out: every VM on the island shares one outgoing stamp, SNAT at the elected gateway chassis. Fine for browsing out, useless for being found.
A floating IP is a personal, publicly listed letterbox bolted to the beachfront, forwarding both ways to one specific house.
The VM believes its address is
192.168.10.5
Your monitoring pings
203.0.113.7
Neither is lying. One database row makes both true.
The shared stamp
Module 4 recap: outgoing mail is re-stamped with one island-wide return address. Nobody outside can write to you.
The personal letterbox
Mail addressed to it is walked to one specific house (DNAT), and that house has its outgoing mail re-stamped with the letterbox address instead of the shared one.
Unscrewable
The letterbox can be unscrewed and moved to another house in seconds. The house never knows it exists.
Born DOWN, and that is fine
Ordering the letterbox does not install it. When you run openstack floating ip create, the OVN L3 plugin writes a ledger entry, nothing more.
neutron/services/ovn_l3/plugin.py · lines 188-196
def create_floatingip(self, context, floatingip,
initial_status=n_const.FLOATINGIP_STATUS_DOWN):
# The OVN L3 plugin creates floating IPs in down status by default,
# whereas the L3 DB layer creates them in active status. So we keep
# this method to create the floating IP in the DB with status down,
# while the flavor drivers are responsible for calling the correct
# backend to instatiate the floating IP in the data plane
return super(OVNL3RouterPlugin, self).create_floatingip(
context, floatingip, initial_status)
When a user asks for a floating IP, this one method answers...
...and its default answer is DOWN: reserved, not live.
The comment owns up: the generic database layer would mark it ACTIVE straight away...
...but OVN refuses to bluff, so the record is written switched off...
...because nothing has been plumbed yet...
...and making it real in the data plane is a separate job, done later.
(Yes, instatiate. The typo is faithfully preserved from the real file.)
Then hand the paperwork to the standard database layer...
...one ledger entry, zero packets moved.
Create: address reserved, status DOWN
Hold it as long as you like: it costs OVN nothing
Associate to a port: the plumbing happens, status ACTIVE
The plumbing function itself opens with a guard that proves the point. No router to hang the letterbox on, no letterbox.
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 889-891
router_id = floatingip.get('router_id')
if not router_id:
return
First question: which router does this floating IP hang off?
No router means no association yet...
...so write nothing to the northbound database and walk away. An unassociated floating IP is pure bookkeeping, which is why it is instant to attach.
The letterbox is one database row
Associate the floating IP with a VM port and here is everything OVN learns: one NAT row on the router, five keys.
203.0.113.7
external_ip
dnat_and_snat
one NAT row on the router
192.168.10.5
logical_ip · logical_port
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 921-925
columns = {'type': 'dnat_and_snat',
'logical_ip': floatingip['fixed_ip_address'],
'external_ip': floatingip['floating_ip_address'],
'logical_port': floatingip['port_id'],
'external_ids': ext_ids}
The row type dnat_and_snat means both directions: incoming mail gets its destination rewritten, outgoing mail gets its source re-stamped, overriding module 4 shared stamp for this one VM.
Which house the mail is walked to: the VM fixed IP.
The address painted on the letterbox: the public floating IP the world sees.
Exactly one nameplate: the specific Neutron port this letterbox forwards to.
And the usual sticky labels tying the OVN row back to the Neutron floating IP.
This is the entire public identity of your VM, and it lives on the router, not on the machine. The guest OS must never be configured with 203.0.113.7, because port security only lets the port speak as its fixed IP.
The same function has a fixed_ip_address is None case that hands off to a different handler entirely: port forwarding. One floating IP can be a bundle of per-TCP-port forwards instead of a full letterbox.
One column, two very different paths
By default, all floating IP traffic detours through the gateway chassis. One config option, enable_distributed_floating_ip, adds a single extra column that changes the route entirely.
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 935-937
if ovn_conf.is_ovn_distributed_floating_ip():
if self._nb_idl.lsp_get_up(floatingip['port_id']).execute():
columns['external_mac'] = port_db['mac_address']
Only if the cloud enabled distributed floating IPs...
...and only if the VM port is actually UP right now...
...add a sixth key: the port MAC address. That single value tells the VM own chassis to answer for the floating IP itself.
With external_mac set, the hypervisor hosting the VM answers ARP for the floating IP and rewrites addresses locally. Engineers call this distributed floating IP. Watch both runs below and keep an eye on the gateway chassis.
Run 1 uses the Geneve tunnel between chassis; run 2 never needs it.
Unscrew the letterbox
web-1 is misbehaving, so you point the floating IP at web-2 instead. One API call, and here is everything that changes in OVN.
Before · screwed to web-1
type
dnat_and_snat
external_ip
203.0.113.7
logical_ip
192.168.10.5
logical_port
port of web-1
After · screwed to web-2
type
dnat_and_snat · unchanged
external_ip
203.0.113.7 · unchanged, the world notices nothing
logical_ip
192.168.10.9 · swapped
logical_port
port of web-2 · swapped
Two fields swap, the public address stays put. Neither guest OS was touched, because neither ever knew the letterbox existed.
row exists?
Your new floating IP checklist, item 1: is there a dnat_and_snat row for this address at all?
logical_port right?
Item 2: does it point at the port you think it does?
external_mac?
Item 3: is it set, and should it be? That column alone decides centralised versus distributed.
$ openstack floating ip show 203.0.113.7 -c status -c fixed_ip_address -c port_id
| DOWN | None | None | # and no NAT row exists anywhere yet
# associated to web-1: the ledger flips ACTIVE once the plumbing lands
$ openstack floating ip show 203.0.113.7 -c status -c fixed_ip_address -c port_id
| ACTIVE | 192.168.10.5 | $PORT_ID |
# the letterbox itself: one dnat_and_snat row on the router
$ ovn-nbctl lr-nat-list neutron-$ROUTER_ID
| dnat_and_snat | 203.0.113.7 | 192.168.10.5 | fa:16:3e:… | $PORT_ID |
# the same row, key by key: your three-item checklist as one command
$ ovn-nbctl --columns=logical_ip,logical_port,external_mac find NAT type=dnat_and_snat external_ip=203.0.113.7
# external_mac present: distributed, the VM's own chassis answers ARP.
# external_mac empty: centralised, everything detours via the gateway chassis.
# it should match enable_distributed_floating_ip in the Neutron config
Check your letterbox instincts
A colleague wants to configure 203.0.113.7 inside the guest, so the VM knows its public address. Why is that a mistake?
Right after openstack floating ip create, the new address shows status DOWN. Is this a bug?
Debugging: a floating IP is unreachable. The dnat_and_snat row exists, external_mac is present, but the VM port is DOWN. What is the fix?
Floating IP traffic is saturating one gateway node even though the deployment intends distributed handling. Which column do you check first?
You now own every row in the story: islands, causeways, gateways and letterboxes. Module 6, Follow the Packet, walks one packet end to end and turns the whole course into a debugging playbook.
Follow the Packet
Same parcel, three trips. Replay the courier log line by line, then keep the lost-parcel checklist forever.
One parcel, three trips
The Way In screwed the last letterbox to the wall: every row this scenario writes now exists. This module makes packets use them.
Time to cash in five modules of rows: three packet walks, and a checklist that turns "no internet" tickets into five-minute diagnoses.
Couriers do not debate a lost parcel. They replay the delivery log and find the first checkpoint without a scan. Our courier makes exactly three kinds of trip:
Trip 1 · Across the island
East-west: web-1 (192.168.10.5) to db-1 (10.0.20.8). Two subnets, one router, and the parcel never leaves the hypervisors.
Trip 2 · Island to mainland
North-south out: web-1 to 8.8.8.8. Collected, stamped with the shared return address (SNAT), and shipped through one designated port town.
Trip 3 · Mainland to one house
North-south in: a stranger addresses 203.0.113.7 and the parcel lands at one specific door (DNAT).
Every reachability ticket in this scenario is one of these trips. Each trip has an ordered set of checkpoints, and at each one you ask: does the row exist, and is the responsible chassis healthy?
Trip 1: across the island, no ferry
The Bridge animated this trip, so here it is as the courier files it: four log lines. The router does its work right on compute-1, as rules in the local flow table.
web-1 hands the parcel to its default gateway, 192.168.10.1
compute-1 routes it in place, using the distributed router rules
Geneve tunnel straight to the chassis hosting db-1
db-1 (10.0.20.8) signs for it
Two checkpoints cover the whole trip:
LRP × 2
The router needs a foot on both subnets: two Logical_Router_Port rows, one per street.
ACTIVE × 2
Both VM ports up. A parcel delivered to a boarded-up door is still a lost parcel.
Trip 2: island to mainland
web-1 pings 8.8.8.8. The trip starts exactly like trip 1, then hits the one stop that cannot be distributed: someone specific must stamp the mail and face the mainland.
The routing decision leans on the default static route; everything after it leans on the election. Four checkpoints, in the order the parcel meets them:
Gateway_Chassis
Scheduled at all? This row ranks up to five chassis by priority; the top one owns the trip. Empty list, no internet.
no candidates are available
The module 4 log warning. A candidate needs enable-chassis-as-gw plus the external physnet.
type='snat'
Module 4 reminder: update_nat_rules writes one snat row per internal CIDR, all pointing at the router external IP. One missing row strands one subnet.
ovn-bridge-mappings
Is gw-1 alive, and does it map the physnet to a real network card? A won election with a dead uplink still delivers nothing.
And remember why the reply must come back through gw-1: the connection state lives there and nowhere else.
Checkpoint one, examined: who won the election?
Neutron picks the port town the moment you attach the gateway, using a scheduler you can read in full. Here is one entire policy, five lines long:
neutron/scheduler/l3_ovn_scheduler.py · lines 148-152
def _select_gateway_chassis(self, nb_idl, sb_idl, candidates,
priority_min, priority_max, target_lrouter):
candidates = copy.deepcopy(candidates)
random.shuffle(candidates)
return self._reorder_by_az(nb_idl, sb_idl, candidates)
The chance policy. Given every chassis eligible to serve, produce the ranked winners for one router...
...along with the range of priority numbers those winners will wear.
Work on a private copy of the candidate list, so the ballot box the caller holds is untouched.
The entire election: shuffle the candidates. Genuinely.
Then reorder so the top picks spread across availability zones, and hand back the ranking.
Two policies ship in that one small file, swappable with a single config option:
Shuffle the candidates, then spread across zones. The five lines above.
Count existing gateway ports per chassis and favour the least burdened, then spread across zones.
Scheduling policies are small, readable and swappable. The whole file is a ten-minute read, and now you know exactly why some chassis wears priority 5 for causeway-1.
Trip 3: mainland to one house
A stranger on the internet addresses 203.0.113.7, the floating IP. Who answers the door depends on one column you met in The Way In: external_mac.
Whoever answers that ARP receives the traffic, so trip 3 has four checkpoints:
'type': 'dnat_and_snat'
Module 5 reminder: the letterbox row itself. No row means the floating IP was never associated, and nothing on the mainland knows the address.
logical_port
Pointing at the right VM port? A letterbox screwed to the wrong house fails exactly like a missing one.
columns['external_mac'] = port_db['mac_address']
Module 5 reminder: the guard writes this only when the distributed option is on and the port is up. Its presence decides who answers the ARP.
ACTIVE
The VM port must be up. The rows can be perfect while the door stays boarded shut.
The lost-parcel checklist
Print this on your mental tea towel. Five symptoms, each mapped to its trip and its checkpoints, in the order you should check them.
Neighbours unreachable, same network
Not a routing trip at all. Sibling course territory: is the port ACTIVE, and do the security groups allow the traffic?
Cross-subnet unreachable
Trip 1. Is the router attached to both subnets, two LRPs present? Are both VM ports ACTIVE?
No internet from any VM behind causeway-1
Trip 2, in order: gateway port scheduled? Candidates warning in the neutron log? Elected chassis alive with its physnet uplink? snat rows present, one per CIDR?
Floating IP dead
Trip 3, in order: dnat_and_snat row exists? logical_port names the right port? external_mac matches intent? VM port ACTIVE?
FIP works, but slow or asymmetric
Centralised versus distributed mismatch. Compare external_mac on the row against enable_distributed_floating_ip in the config.
Across both courses, every mechanism had the same shape: Neutron writes intent rows, OVN daemons make them real, events flow back up. Master that loop and any Neutron plus OVN feature reads the same way: load balancers, port forwarding, QoS.
$ ovn-nbctl lrp-list neutron-$ROUTER_ID → one lrp- per attached subnet
$ openstack port show $PORT_ID -c status → ACTIVE, on both ends
# trip 2, island to mainland: door, route, stamps, winner, in that order
$ ovn-nbctl lr-route-list neutron-$ROUTER_ID | grep 0.0.0.0/0
$ ovn-nbctl lr-nat-list neutron-$ROUTER_ID | grep snat → one row per CIDR
$ ovn-nbctl lrp-get-gateway-chassis lrp-$GW_PORT_ID → the ranked winners
$ grep "was not scheduled on any chassis" neutron-server.log # empty roll?
# trip 3, mainland to one house: the letterbox row, all four checkpoints at once
$ ovn-nbctl --columns=logical_ip,logical_port,external_mac find NAT type=dnat_and_snat external_ip=203.0.113.7
# slow or asymmetric floating IP? compare external_mac above against
# enable_distributed_floating_ip in the Neutron config: they must agree
The on-call drill
Four tickets, straight off the queue. Walk the trip before you click.
Ticket: "VMs behind causeway-1 reach other subnets fine, but not the internet. Other routers are unaffected." Your first three checks, in order?
Ticket: "The FIP pings, but ssh drops under load, and all FIP traffic funnels through one node in a deployment that is supposed to be distributed." Where do you look?
You inherit a cloud and want to know which chassis carry internet traffic today. Where do you look first?
A teammate suggests: "Just reboot the network node, that always fixed routing before." Why does that rarely help with OVN?
Journey complete: the whole map, one last look
Seven steps in The Island Story. Now every step has a row you can name, inspect and hand to a teammate or an AI assistant without ambiguity.
openstack network create
The island itself: one switch row every chassis can materialise.
openstack subnet create
Street addresses: the CIDR plus the boot-time answers, stored as data, not as a daemon.
openstack router create
A causeway with no ends yet: one row, no process started anywhere.
openstack router add subnet
The causeway touches down: a Logical_Router_Port paired with its switch-side twin.
openstack router set --external-gateway
The way out: a default static route, one snat row per CIDR, and a ranked election result.
openstack floating ip create + server add floating ip
The letterbox: five keys on one NAT row, with external_mac as the optional sixth.
ovn-northd compiled every row above into flow rules. The three trips in this module rode them.
Where to explore next
Three files carried this whole course. Open them with any question, or point your AI assistant at them by name:
This course started at a network that already existed. For the boot-time story, how a VM port comes up, gets its address and is marked ACTIVE, take the sibling course on how a VM gets its network. Same loop, different tables.
Ticket closed. Next time someone says the cloud is down, you can name the trip, the row and the chassis. 🎉