Merge pull request #278 from bgpkit/dev/memory-optimizations

Memory usage optimizations - 61% reduction

Author: digizeph

CHANGELOG.md

@@ -54,6 +54,13 @@ All notable changes to this project will be documented in this file.
 
 ### Performance improvements
 
+* **Memory usage reduction (61% improvement)**: Optimized memory allocation patterns resulting in significant peak memory reduction:
+  - **NLRI Add-Path clone-free parsing**: Replaced input buffer cloning with speculative byte-slice parsing. The new `try_parse_prefix` function parses from a byte slice without consuming, returning `(prefix, bytes_consumed)`. The `read_nlri_prefix` method is now a thin wrapper that calls `try_parse_prefix` and advances the cursor. This eliminates the expensive `input.clone()` operation while maintaining correct Add-Path heuristic retry behavior.
+  - **Attribute vector pre-allocation**: Estimate capacity from data size (`remaining / 3` bytes per attribute) instead of fixed 20, reducing reallocations for BGP messages with many attributes.
+  - **RIS Live vector pre-allocation**: Calculate capacity from announcements + withdrawals counts before allocation, preventing growth reallocations.
+  - **Measured improvement**: Peak memory reduced from 2,037 MB to 789 MB (61.3% reduction) when parsing full BGP table dump (RouteViews LINX RIB, 151MB compressed).
+  - **Code quality**: Single implementation of prefix parsing logic in `try_parse_prefix`, eliminating duplication with `read_nlri_prefix`.
+
 * Use zerocopy for MRT header parsing
   - Replaced manual byte parsing with zerocopy's `FromBytes` trait for `RawMrtCommonHeader` and `RawMrtEtCommonHeader`
   - Reduces bounds checking overhead by using compile-time verified struct layouts

src/parser/bgp/attributes/mod.rs

@@ -180,7 +180,10 @@ pub fn parse_attributes(
     safi: Option<Safi>,
     prefixes: Option<&[NetworkPrefix]>,
 ) -> Result<Attributes, ParserError> {
-    let mut attributes: Vec<Attribute> = Vec::with_capacity(20);
+    // Estimate capacity from data size: each attribute is at least 3 bytes
+    // (flag + type + length). Cap at 256 to avoid over-allocation for corrupted data.
+    let estimated_attrs = (data.remaining() / 3).min(256);
+    let mut attributes: Vec<Attribute> = Vec::with_capacity(estimated_attrs.max(8));
     let mut validation_warnings: Vec<BgpValidationWarning> = Vec::new();
     // boolean flags for seen attributes - small dataset in hot loop.
     let mut seen_attributes: [bool; 256] = [false; 256];

src/parser/rislive/mod.rs

@@ -103,7 +103,14 @@ pub fn parse_ris_live_message(msg_str: &str) -> Result<Vec<BgpElem>, ParserRisli
                     announcements,
                     withdrawals,
                 } => {
-                    let mut elems: Vec<BgpElem> = vec![];
+                    // Pre-allocate capacity based on announcements + withdrawals
+                    let announce_count: usize = announcements
+                        .as_ref()
+                        .map(|a| a.iter().map(|ann| ann.prefixes.len()).sum())
+                        .unwrap_or(0);
+                    let withdraw_count: usize = withdrawals.as_ref().map(|w| w.len()).unwrap_or(0);
+                    let mut elems: Vec<BgpElem> =
+                        Vec::with_capacity(announce_count + withdraw_count);
 
                     // parse community
                     let communities = community.map(|values| {