Problem description
Description
The subnet-create-full-v6 E2E test flaked on the Flamingo environment in PR #835, job 85394176988. The tests passed on Gazpacho and Epoxy. This is not an ORC bug — it's an upstream Neutron/OVN race condition.
Root Cause
Neutron returned HTTP 201 (Created) for the router creation request, but internally the router was immediately deleted due to an ovsdbapp crash in the OVN post_commit hook. The entire create-and-delete happened within a single API request (req-6cbc813e).
ovsdbapp traceback:
ERROR ovsdbapp.backend.ovs_idl.transaction:
File "ovsdbapp/backend/ovs_idl/command.py", line 102, in post_commit
row = self.api.tables[self.table_name].rows[real_uuid]
KeyError: UUID('d971425a-2cfb-4e3a-985c-f4aa7c40a3dd')
ERROR neutron.services.ovn_l3.service_providers.ovn:
Unable to create lrouter 2f49960b-a54e-4f39-9e48-f612a73f8af3:
KeyError: UUID('d971425a-2cfb-4e3a-985c-f4aa7c40a3dd')
Timeline
| Time |
Event |
| 14:38:42 |
ORC sends POST /networking/v2.0/routers to create router subnet-create-full-v6 |
| 14:38:42 |
Neutron DB commits the router (ID: 2f49960b...) |
| 14:38:44 |
OVN post_commit fails with KeyError — Neutron deletes the router internally |
| 14:38:45 |
Neutron returns HTTP 201 to ORC (response was already prepared before the error) |
| 14:38:45 |
ORC stores the router ID, then does GET /routers/2f49960b... → 404 |
| 14:38:45 |
ORC marks the Router as terminal error: "resource has been deleted from OpenStack" |
| 14:38:45–14:42:43 |
Router is not Available → RouterInterface controller skips it → RouterInterface has no status |
| 14:42:43 |
KUTTL times out after ~4 minutes waiting for RouterInterface to become Available |
Evidence from Neutron logs: No add_router_interface API call was ever made. The only subsequent requests for router 2f49960b... were GETs, all returning 404.
Impact
- 1 test failure out of 73 tests
- Only affects Flamingo; Gazpacho and Epoxy passed
- ORC handled the situation correctly (terminal error for a vanished resource)
- The test has no workaround on the ORC side — it depends on Neutron behaving correctly
Upstream
If this recurs, consider filing against neutron or ovsdbapp. The race condition appears to be in the post_commit phase of LrAddCommand where the OVN NB row for a logical router port no longer exists by the time the result is read back.
ORC version
main
Additional information
No response
Relevant log output
Problem description
Description
The
subnet-create-full-v6E2E test flaked on the Flamingo environment in PR #835, job 85394176988. The tests passed on Gazpacho and Epoxy. This is not an ORC bug — it's an upstream Neutron/OVN race condition.Root Cause
Neutron returned HTTP 201 (Created) for the router creation request, but internally the router was immediately deleted due to an
ovsdbappcrash in the OVNpost_commithook. The entire create-and-delete happened within a single API request (req-6cbc813e).ovsdbapp traceback:
Timeline
POST /networking/v2.0/routersto create routersubnet-create-full-v62f49960b...)post_commitfails withKeyError— Neutron deletes the router internallyGET /routers/2f49960b...→ 404"resource has been deleted from OpenStack"Evidence from Neutron logs: No
add_router_interfaceAPI call was ever made. The only subsequent requests for router2f49960b...were GETs, all returning 404.Impact
Upstream
If this recurs, consider filing against neutron or ovsdbapp. The race condition appears to be in the
post_commitphase ofLrAddCommandwhere the OVN NB row for a logical router port no longer exists by the time the result is read back.ORC version
main
Additional information
No response
Relevant log output