First, a house handover

Booting a VM works like a new house handover. The builder has finished the house, but nobody gets the keys until the electrician issues the electrical safety certificate.

The boot, replayed as a group chat

Here is a real boot of a VM called web-1, replayed as a chat between the services. Step through it and notice who waits for whom.

The whole story in seven steps

That chat is the entire course in miniature. Here is the same story as a map, with the module that unpacks each step.

And Module 2? It introduces the full cast properly before the action starts.

Where the story starts in the code

Every boot begins with one web call to Neutron's REST API. The code that answers it lives in neutron/plugins/ml2/plugin.py, inside Neutron's ML2 core.

    def create_port(self, context, port):        self._before_create_port(context, port)        result, mech_context = self._create_port_db(context, port)        return self._after_create_port(context, result, mech_context)

neutron/plugins/ml2/plugin.py · lines 1584 to 1587 · unmodified

Three scenarios from the field

No definitions, no memory tests. Use the story you just watched.

One council, many construction sites

Think of your cloud as a city council and its construction system. The control plane is the council building, where plans are lodged and work orders are drawn up. The data plane is the construction sites, one on every compute host, each with its own permanently stationed crew.

Click every box below. Notice how OVN intent only ever flows through databases, never direct phone calls.

The full cast list

Seven characters, two homes: the controller nodes, and every compute host. When something breaks, the first question is always "whose job was that?".

They do not phone each other. They share two databases.

Classic Neutron backends shouted commands over an RPC message queue and hoped an agent heard them. With OVN, Neutron mostly writes desired state into the NB database, and a pipeline of daemons makes reality match.

Watch one intent travel the pipeline. Neutron wants a new ACL to exist:

The council checks the crew's pulse

Neutron treats each host's ovn-controller as an agent, backed by its SB heartbeat. Before it will bind a port to a host, the OVN driver asks two questions: is a crew registered there, and is it alive?

neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py

        agents = n_agent.AgentCache().get_agents(
            {'host': bind_host,
             'agent_type': ovn_const.OVN_CONTROLLER_TYPES})
        if not agents:
            LOG.warning('Refusing to bind port %(port_id)s due to '
                        'no OVN chassis for host: %(host)s',
                        {'port_id': port['id'], 'host': bind_host})
            return
        agent = agents[0]
        if not agent.alive:
            LOG.warning("Refusing to bind port %(pid)s to dead agent:  "
                        "%(agent)s", {'pid': context.current['id'],
                                      'agent': agent})
            return

The full binding story, including what Nova does with a refusal, is module 4.

Your turn on the tools

Three situations. Use the cast, not your memory of the words.

Cast introduced. Next, module 3: A Port Is Born, where we follow one port-create call from the council counter all the way into the Northbound registry.

The Registry Office

You met the cast in Meet the Cast; now watch them handle their first piece of business together. Your VM's IP address was decided before the VM existed, and this is the moment it happened.

When Nova requests a network card for the new VM, Neutron acts like a civil registry issuing a birth certificate. The identity papers go into Neutron's own ledger first; the physical wire comes much later.

One Request, Two Ledgers

Follow one POST /v2.0/ports request from Nova all the way in and back out. Count the writes: there are two, into two different databases, and everything in this module hangs on the gap between them.

Notice the ending: Nova already holds the MAC and IP, yet the port's status is DOWN. Identity first, wiring later.

Phase One: Inside the Ledger

Phase one is the precommit hook, which the ML2 plugin runs inside the very SQL transaction that records the port. Everything here is atomic: if any line fails, the whole certificate is torn up. Straight from neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:

        port = context.current        if ovn_utils.is_lsp_ignored(port):            return        ovn_utils.validate_and_get_data_from_binding_profile(port)        self._validate_port_extra_dhcp_opts(port)        if self._is_port_provisioning_required(port, context.host):            self._insert_port_provisioning_block(context.plugin_context,                                                 port['id'])         ovn_revision_numbers_db.create_initial_revision(            context.plugin_context, port['id'], ovn_const.TYPE_PORTS,            std_attr_id=context.current['standard_attr_id'])

Phase Two: Publishing the Blueprint

The instant the ledger commits, ML2 calls the postcommit hook. There, the OVN mechanism driver photocopies the record and hands it to the OVN client, its clerk for the blueprint registry:

        port = copy.deepcopy(context.current)        port['network'] = context.network.current        self._ovn_client.create_port(context.plugin_context, port)        self._notify_dhcp_updated(port['id'])

The clerk now creates the port's twin, a Logical_Switch_Port, in the NB database. From neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py, original comment and all:

        with self._nb_idl.transaction(check_error=True) as txn:            dhcpv4_options, dhcpv6_options = self.update_port_dhcp_options(                port_info, txn=txn)            # The lport_name *must* be neutron port['id'].  It must match the            # iface-id set in the Interfaces table of the Open_vSwitch            # database which nova sets to be the port ID.             kwargs = {                'lport_name': port['id'],                'lswitch_name': lswitch_name,                'addresses': port_info.addresses,                'external_ids': external_ids,                'parent_name': port_info.parent_name,                'tag': port_info.tag,                'enabled': port.get('admin_state_up'),                'options': port_info.options,                'type': port_info.type,                'port_security': port_info.port_security,                'dhcpv4_options': dhcpv4_options,                'dhcpv6_options': dhcpv6_options            }

The Gatehouse: Security Groups Become Port Groups

There is no iptables firewall on the hypervisor here. A security group becomes an OVN port group, and its rules become ACLs attached to that group in the NB database.

Think of a gated estate: a new resident starts on the no-visitors list, and each security group is a guest list that opens the gate to named callers.

Still inside the very same NB transaction, in ovn_client.py:

            sg_ids = utils.get_lsp_security_groups(port)            # If this is not a trusted port and port security is enabled,            # add it to the default drop Port Group so that all traffic            # is dropped by default.            if not utils.is_lsp_trusted(port) and port_info.port_security:                self._add_port_to_drop_port_group(port_cmd, txn)            # Just add the port to its Port Group.            for sg in sg_ids:                txn.add(self._nb_idl.pg_add_ports(                    utils.ovn_port_group_name(sg), port_cmd))

Prove It in the Field

Four situations straight from real operations. Apply what you have just seen; each answer comes with the why.

Next: the port has papers but no home; in Finding a Home, Neutron picks the compute host it will live on and binding moves it in.

A Ship Without a Berth

A Port Is Born left you holding fresh papers: a port with an ID, a MAC and an IP, status DOWN, and a provisioning block IOU promising a real wire. Now the port needs somewhere to live.

A port with no host is a passport with no address. Binding is Neutron approving the address Nova picked, and it can say no.

Picture a harbourmaster assigning a ship to a berth. Six characters run this port town:

The Negotiation, End to End

Watch one PUT request turn a homeless port into a bound one. Every stop on this route is a place where binding can die.

Notice who does the very last step. Neutron only stamps paperwork; the physical plugging is done by Nova through os-vif, which pushes a TAP device into br-int.

A Negotiation with Retries

The world keeps moving while the harbourmaster thinks: hosts die, live migrations land, other API calls touch the same port. So ML2 treats binding as optimistic and wraps it in a retry loop rather than trusting one attempt.

neutron/plugins/ml2/plugin.py

        for count in range(1, MAX_BIND_TRIES + 1):            if count > 1:                # yield for binding retries so that we give other threads a                # chance to do their work                greenthread.sleep(0)                 # multiple attempts shouldn't happen very often so we log each                # attempt after the 1st.                LOG.info("Attempt %(count)s to bind port %(port)s",                         {'count': count, 'port': context.current['id']})

Three Ways to Say No

Before stamping anything, the OVN driver runs the berth checklist: right cargo type, crew on site, berth equipped. Three questions, three refusals, and each one leaves a plain log line you will grep for on a bad night.

Check 1, the cargo type: neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py

        port = context.current        vnic_type = port.get(portbindings.VNIC_TYPE, portbindings.VNIC_NORMAL)        if vnic_type not in self.supported_vnic_types:            LOG.debug('Refusing to bind port %(port_id)s due to unsupported '                      'vnic_type: %(vnic_type)s',                      {'port_id': port['id'], 'vnic_type': vnic_type})            return

Check 2 is the crew check you watched in Meet the Cast: is a chassis registered for the host, and is its heartbeat alive? A registered but silent crew counts as dead, and each refusal has its own log line:

Check 3 is the equipment. Networks arrive as segments, and flat and VLAN segments ride a named physnet. The host's bridge mappings must include that physnet, or this berth simply lacks the right crane:

Check 3, the equipment: neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py

            if ((network_type in [const.TYPE_FLAT, const.TYPE_VLAN]) and                    (physical_network not in chassis_physnets)):                LOG.info('Refusing to bind port %(port_id)s on '                         'host %(host)s due to the OVN chassis '                         'bridge mapping physical networks '                         '%(chassis_physnets)s not supporting '                         'physical network: %(physical_network)s',                         {'port_id': port['id'],                          'host': bind_host,                          'chassis_physnets': chassis_physnets,                          'physical_network': physical_network})

The Stamp and the Mooring

Every check passed, so the harbourmaster writes the mooring plan and stamps it. This is the hero moment of the whole negotiation:

neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py

                else:                    vif_type = portbindings.VIF_TYPE_OVS                    vif_details = copy.deepcopy(self.vif_details[vif_type])                 ovn_bridge = ovn_utils.get_ovn_bridge_from_chassis_private(                    agent.chassis_private)                dp_type = ovn_utils.get_datapath_type(bind_host, self.sb_ovn)                vif_details.update({                    portbindings.VIF_DETAILS_BRIDGE_NAME: ovn_bridge,                    portbindings.OVS_DATAPATH_TYPE: dp_type,                })                context.set_binding(segment_to_bind[api.ID], vif_type,                                    vif_details)                break

Nova reads three things off the stamped plan, then does its side of the bargain on the host:

Prove It on the Docks

binding_failed is one of the most common boot failures you will ever meet. Match each symptom to its most likely cause; the chips are the causes.

Now three situations from the field. Think like the harbourmaster.

Next: the cable is plugged and name-tagged, yet the port still says DOWN; in The Wake-Up Call, ovn-controller on compute-7 spots that tag, wires up the flows, and everyone finds out the ship has arrived.

Mission Control Runs a Go/No-Go Poll

Finding a Home ended with the TAP plugged into br-int and name-tagged with the port ID, but nobody has said power on. Nova is still holding the boot of web-1, phone in hand.

Neutron runs this moment the way mission control runs a rocket launch: a go/no-go poll. Every station must call GO, and until the checklist is empty, launch status stays red and the port stays out of ACTIVE.

The station was added to the checklist back in module 3, at the moment of port creation. From neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:

    def _insert_port_provisioning_block(self, context, port_id):        # Insert a provisioning block to prevent the port from        # transitioning to active until OVN reports back that        # the port is up.        provisioning_blocks.add_provisioning_component(            context, port_id, resources.PORT,            provisioning_blocks.L2_AGENT_ENTITY        )

Five Links, Zero Polling

Nothing in this chain polls. Every link is event-driven: each hop wakes the next one the instant something real changes.

Step through the five links. Each one has a symptom when it breaks, which makes this animation your map for bisecting a vif plugging timeout.

The Subscription: Wake Me Only for the Flip

neutron-server never asks the database "is it up yet?". It lodges a subscription through OVSDB: wake me when a row in the Logical_Switch_Port table flips from DOWN to UP.

The handler class is called LogicalSwitchPortUpdateUpEvent, and its own documentation says it plainly: "This happens when the VM goes up". From neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:

    def match_fn(self, event, row, old):        if not (utils.is_lsp_up(row) and utils.is_lsp_enabled(row)):            return False         if hasattr(old, 'up') and not utils.is_lsp_up(old):            # The port has transitioned from DOWN to UP, and the admin state            # is UP (lsp.enabled=True)            return True        if hasattr(old, 'enabled') and not utils.is_lsp_enabled(old):            # The user has set the admin state to UP and the port is UP too.            return True        return False     def run(self, event, row, old):        self.driver.set_port_status_up(row.name)

GO, and the Checklist Empties

The GO call lands in the driver, back in mech_driver.py. Two jobs: log the moment for posterity, then tell the checklist the L2 station has reported.

    def set_port_status_up(self, port_id):        # Port provisioning is complete now that OVN has reported that the        # port is up. Any provisioning block (possibly added during port        # creation or when OVN reports that the port is down) must be removed.        LOG.info("OVN reports status up for port: %s", port_id)         self._update_dnat_entry_if_needed(port_id)         admin_context = n_context.get_admin_context()        provisioning_blocks.provisioning_complete(            admin_context,            port_id,            resources.PORT,            provisioning_blocks.L2_AGENT_ENTITY)

The checklist desk itself lives in neutron/db/provisioning_blocks.py, and it only ever announces a launch when the list is empty:

        # now with that committed, check if any records are left. if None, emit        # an event that provisioning is complete.        if pb_obj.ProvisioningBlock.objects_exist(                context, standard_attr_id=standard_attr_id):            return     LOG.debug("Provisioning complete for %(otype)s %(oid)s triggered by "              "entity %(entity)s.", log_dict)    registry.publish(object_type, PROVISIONING_COMPLETE, entity,                     payload=events.DBEventPayload(                         context, resource_id=object_id))

The Call Nova Has Been Waiting For

The status flip is what dials the phone. Neutron's Nova notifier watches port status changes and, in neutron/notifiers/nova.py, keeps a strict rule about which flip means "plugged":

        # We only notify nova when a vif is plugged which only occurs        # when the status goes from:        # NO_VALUE/DOWN/BUILD -> ACTIVE/ERROR.        elif (previous_port_status in [sql_attr.NO_VALUE,                                       constants.PORT_STATUS_DOWN,                                       constants.PORT_STATUS_BUILD] and              current_port_status in [constants.PORT_STATUS_ACTIVE,                                      constants.PORT_STATUS_ERROR]):            event_name = VIF_PLUGGED

The notifier holds queued events for a moment (batching), then sends one POST to Nova's external events API. Module 1 opened with Nova waiting by the phone; here is the call being made, hop by hop.

That is the whole course in one phone call. The cliffhanger from module 1 is closed: the certificate arrived, and Nova stops pacing by the phone.

Prove You Can Bisect It

Four timeouts and design questions from the field. Each one is a link in the chain; find it.

Next: web-1 is powering on, and its first words are two questions: what is my address, and what is my config? First Breath follows DHCP and metadata, both answered the OVN way.

Checked In, Phone in Hand

The Wake-Up Call ended with the certificate: network-vif-plugged landed, Nova resumed the boot, and inside the VM an operating system is now starting for the very first time.

Think of a hotel guest who has just stepped into their room. Their first two acts: ask the building who they are on this network, then ring guest services to find out who they are supposed to become.

Neither act involves a DHCP server process, and the VM holds no passwords or credentials at all. Both work anyway, and this module shows you how.

The Room Number Is in the Wiring

Older Neutron backends ran a dnsmasq process per network to answer these questions. Under OVN that job is abolished: there is no daemon, only data compiled into switch rules.

Where did the answers come from? Neutron computed them when the port was born, and they landed in the NB database's DHCP_Options table. Here is the computation, from ovn_client.py:

        default_lease_time = str(ovn_conf.get_ovn_dhcp_default_lease_time())        mtu = network['mtu']        options = {            'server_id': service_id,            'lease_time': default_lease_time,            'mtu': str(mtu),        }

neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 2332 to 2338 · unmodified

And the Enable DHCP tick-box in the dashboard? In the same file, it is one if statement deciding whether the options box gets filled:

        dhcp_options = {'cidr': subnet['cidr'], 'options': {},                        'external_ids': external_ids}         if subnet['enable_dhcp']:            if subnet['ip_version'] == const.IP_VERSION_4:                dhcp_options['options'] = self._get_ovn_dhcpv4_opts(                    subnet, network, server_mac=server_mac)            else:                dhcp_options['options'] = self._get_ovn_dhcpv6_opts(                    subnet, server_id=server_mac)

neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py · lines 2279 to 2288 · unmodified

Dialling 9: The Metadata Call

Room number sorted, cloud-init picks up the room phone: a plain HTTP request to 169.254.169.254, the same magic number on every cloud in the world.

The call carries no password and no token, yet it must be answered with secrets like your ssh keys. Step through how the switchboard pulls that off; the machinery behind each actor comes next.

The welcome pack is what makes the VM yours: hostname, your ssh public key, and your user-data script. When ssh just works a minute after launch, this call is why.

The Switchboard Room

Who built that room? Every compute host runs an OVN metadata agent, and for each network with a VM on the host it fits out one switchboard room, made of three parts:

Here is the room being fitted out, from agent.py, original comments and all:

        LOG.info("Provisioning metadata for network %s", net_name)        # Create the VETH pair if it's not created. Also the add_veth function        # will create the namespace for us.        namespace = self._get_namespace_name(net_name)        veth_name = self._get_veth_name(net_name)         ip1 = ip_lib.IPDevice(veth_name[0])        if ip_lib.device_exists(veth_name[1], namespace):            ip2 = ip_lib.IPDevice(veth_name[1], namespace)        else:            LOG.debug("Creating VETH %s in %s namespace", veth_name[1],                      namespace)            # Might happen that the end in the root namespace exists even            # though the other end doesn't. Make sure we delete it first if            # that's the case.            if ip1.exists():                ip1.link.delete()            ip1, ip2 = ip_lib.IPWrapper().add_veth(                veth_name[0], veth_name[1], namespace)

neutron/agent/ovn/metadata/agent.py · lines 770 to 788 · unmodified

The Signature That Opens Doors

One puzzle remains: the request that reaches the agent is anonymous. The switchboard identifies the caller by provenance: the call arrived in network X's room from a particular source IP, and Neutron's records map that pair to exactly one port and one instance.

Then the agent vouches for the caller in writing. From server.py, the note it staples to the request before forwarding it:

        headers = {            'X-Forwarded-For': req.headers.get('X-Forwarded-For'),            'X-Instance-ID': instance_id,            'X-Tenant-ID': tenant_id,            'X-Instance-ID-Signature': common_utils.sign_instance_id(                self.conf, instance_id)        }

neutron/agent/ovn/metadata/server.py · lines 141 to 147 · unmodified

When First Breath Fails

Two of the most common day-two complaints in any OpenStack cloud are born right here: my instance got no IP, and my ssh key never arrived. You now know the machinery, so you know the first place to look.

Four field scenarios. Answer with the machinery, not with folklore.

The Whole Minute, Machinery Named

In module 1 this was seven mysterious steps. Here it is again, with every piece of machinery named. This is the map you keep.

Where to Explore Next in the Code

Everything in this course came from real files in the Neutron tree. These four places are where the story lives; each one rewards a slow read.

That was the invisible half of your minute. Next time an instance sticks in BUILD, boots with no IP, or arrives without your key, you will not be guessing: you can name the link that broke and point a teammate, or an AI assistant, straight at it.

end of course · Plugged In: how Neutron wires a VM with OVN