Speed up entity_orientations by hoisting per-entity work out of the loop - #5343
Speed up entity_orientations by hoisting per-entity work out of the loop#5343pbrubeck wants to merge 1 commit into
Conversation
`_compute_orientation_simplex()` allocated and freed two scratch arrays with `PetscMalloc1`/`PetscFree` on every call. On a tet mesh it is called for the 11 non-vertex entities of each cell's closure, so `BoxMesh(79, 79, 79)` made about 130M allocator round-trips to hold cones of at most four points. It now takes the two arrays from the caller, which is what the sibling `_compute_orientation_interval_tensor_product()` already did, and `entity_orientations()` allocates them once at `maxConeSize` beside the work arrays it already had. Every cell has the same FIAT cell, so entity `e` has the same polytope type on every cell. `entity_orientations()` now looks the type up once per entity rather than once per (cell, entity), which drops a `DMPlexGetCellType()` call per closure entry, and skips vertices without a call at all. `2**dim` on a `PetscInt` compiles to a call to Cython's integer power helper. The two occurrences in the tensor-product path are shifts instead. Output is bit-identical on both dispatch paths. `entity_orientations` on `BoxMesh(79, 79, 79)` goes from 3.66 s to 1.11 s, and on a hexahedral `BoxMesh(40, 40, 40)` from 0.282 s to 0.164 s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| @@ -1125,7 +1140,7 @@ cdef inline PetscInt _compute_orientation_interval_tensor_product(PetscInt *fiat | |||
| else: | |||
| raise RuntimeError("Found inconsistent fiat_cone and plex_cone") | |||
| assert dim1 == 0 | |||
| return <PetscInt> (2**dim) * eo + io | |||
| return (1 << dim) * eo + io | |||
There was a problem hiding this comment.
Does bitshifting here actually make a noticeable difference in the performance?
Unless it makes a very significant difference I don't think it's worth the loss in readability.
There was a problem hiding this comment.
I have made similar questions in the PR that proceeded this one #5231 and the answer for this one is yes
There was a problem hiding this comment.
This is an entity loop, ** calls a much heavier math routine
There was a problem hiding this comment.
the answer for this one is yes
Do you have numbers?
There was a problem hiding this comment.
I lost the benchmark but I can regenerate it if you really insist. 2**N is ~150 times slower than 1<<N for small N <= 4. If you call it O(10^6) times the 2**N approach would take around 0.5 seconds, so it does dominate for this particular benchmark.
Description
AI-assisted (Claude Code, Opus 5)
Goal
#5231 took
entity_orientationsfrom 19.7 s to 2.1 s onBoxMesh(79, 79, 79)and leftnogil/prangeout of scope. This picks up the serial work that was still on the table: three pieces of per-entity work that do not depend on the entity, hoisted out of the loop over the cell closure. No OpenMP, no GIL change, no new build dependency.entity_orientationsruns once per mesh topology, before any solve, so it sits on the startup path of every script that builds a function space.What's in it
_compute_orientation_simplex()no longer allocates. It calledPetscMalloc1/PetscFreetwice per invocation, to hold cones of at most four points. On a tet mesh it runs for the 11 non-vertex entities of each cell's closure, soBoxMesh(79, 79, 79)made roughly 33M calls and 130M allocator round-trips. It now takes the two scratch arrays from the caller, which is exactly what its sibling_compute_orientation_interval_tensor_product()already did, andentity_orientations()allocates them once atmaxConeSizealongside the three work arrays it already had.ehas the same type on every cell.entity_orientations()fills a small array indexed byeand passes the type in, dropping aDMPlexGetCellType()call per closure entry — about 45M of them on the same mesh. Vertices are then skipped without a call at all, since their orientation is always 0 and the output array starts zeroed.2**dimis a shift. On aPetscInt, Cython compiles**to a call to its integer power helper. The two occurrences in the tensor-product path become1 << dim, which is exact for thedimvalues that reach them.The loop order is unchanged.
cell_closureis C-contiguous ine, so cell-outer/entity-inner is already the cache-friendly order; swapping it to hoist the type lookup differently would stride bynumEntities.Measurements
entity_orientationstimed in isolation, minimum of 5 repeats, on one machine:BoxMesh(79, 79, 79)(2,958,234 tets)BoxMesh(40, 40, 40, hexahedral=True)(64,000 hexes)Taken in order on the tet mesh: 3.66 s → 1.68 s for the allocation removal, → 1.11 s for the type cache. The absolute numbers are higher than #5231's because the machine is slower; the baseline above was measured on the same machine as the result.
Correctness
Output is bit-identical, not merely equivalent.
np.array_equalagainst the valuesmainproduces, on eight meshes chosen to reach every branch of the dispatch:UnitIntervalMesh,UnitSquareMesh(triangle and quadrilateral),BoxMesh(tetrahedral and hexahedral),UnitIcosahedralSphereMesh,UnitCubedSphereMesh, and the unstructured quadrilateral mesh fromtests/firedrake/meshes. The last two carry the non-trivial quadrilateral orientations that a structured mesh does not produce.Tests
tests/firedrake/regression/test_facet_orientation.pyis the existing coverage for this code: it checks that cells sharing a node agree on it across a CG space, which is what these orientations feed throughget_cell_nodes. It passes at 1 and 2 processes, over both simplex and tensor-product meshes including the unstructured quadrilateral one.No test is added. The change alters no behaviour — the bit-identical comparison above is the claim, and a new test asserting orientation values would restate what
test_facet_orientation.pyalready exercises end to end.Notes for review
The one judgement call is the polytope-type cache: it assumes a single cell type across the mesh.
entity_orientations()already rejects anything that is not exactly aMeshTopology, and it builds its entity-cone map from a singleas_fiat_cell(mesh.ufl_cell()), so a per-cell type would already have broken the map. The cache does not weaken an assumption the function was not already making.