Hardening integer overflow guards in escape helper functions for atta…

[SABITHSAHEB]

server/util: Harden escape helpers against allocation size overflows

Summary

This patch adds integer overflow validation to several escape helper functions in server/util.c before calculating allocation sizes.

These functions derive buffer sizes from input lengths using expressions such as 3 * len + 1 and 10 * len. Without validation, extremely large values could cause arithmetic wraparound, resulting in an undersized allocation and potential memory corruption during subsequent writes.

Changes

• Added overflow checks before allocation size calculations.
• Prevented allocations when the computed size would overflow size_t.
• Preserved existing behavior for valid inputs.
• Added regression tests covering boundary and overflow cases.

Security Impact

This change hardens memory allocation logic against integer-overflow conditions that could otherwise lead to heap buffer overflows when processing unusually large inputs.

Testing

• Built successfully with GCC and MSVC.
• Added regression tests for safe boundary values and overflow conditions.
• Verified normal behavior remains unchanged for valid inputs.

Notes

• No API changes.
• No new dependencies.
• Backward compatible.

[notroj]

Using ap_assert in these cases is correct, don't remove that, if these paths are reachable you are just introducing some new error path which would crash differently anyway. Don't add new such paths anywhere, just use ap_assert throughout.

[SABITHSAHEB]

Makes sense, reverted to ap_assert across the board: restored the ones I'd removed and used it for the new checks too instead of the error-return paths. I also pulled the unrelated mod_core atoi changes and the cookie reverse-map sizing churn so this stays focused on the escape helpers. Pushed.

[SABITHSAHEB]

any update?

[notroj]

mod_rewrite.c: In function 'escape_backref':
mod_rewrite.c:727:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
  727 |     char *copy = apr_palloc(p, 3 * plen + 1);
      |     ^~~~

this doesn't compile in C90 in various places. Build with ./configure --enable-maintainer-mode

[SABITHSAHEB]

Fixed and pushed. The offenders were escape_backref in mod_rewrite.c and ftp_escape_globbingchars in mod_proxy_ftp.c, where the new declarations ended up after statements; moved them up into the declaration blocks. Rebuilt with --enable-maintainer-mode and the touched files compile clean now under -Wdeclaration-after-statement. The only failure left in my tree was the pre-existing SSL_get_handshake_rtt apr_uint64_t vs uint64_t mismatch on macOS with homebrew apr, unrelated to this patch.