Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
b30bdae
Fix log_retention test: use correct test run variable (#12884)
brbzull0 Feb 13, 2026
24b3558
Check Curl support for TLS 1.0 and 1.1 (#12887)
bryancall Feb 13, 2026
d51d01c
Add AuTest and Doc for DELETE method (#12883)
masaori335 Feb 13, 2026
13f5721
First cut at a CoPilot review agent (#12889)
zwoop Feb 17, 2026
3a7048f
autest: add missing newline before worker output in parallel runner (…
brbzull0 Feb 19, 2026
cc24790
tests: stabilize log-filenames diags check (#12901)
bneradt Feb 20, 2026
55f2201
Fix: set UA_BEGIN_WRITE milestone unconditionally (#12898)
bryancall Feb 21, 2026
0c13f60
Fix: difference_msec epoch leak when start milestone is unset (#12897)
bryancall Feb 21, 2026
ffa27be
Fix: Build with OpenSSL 3.5+ which removed engine API (#12902)
bryancall Feb 21, 2026
88a2593
Fix: msdms log fields emit dash instead of -1 for unset milestones (#…
bryancall Feb 21, 2026
308df56
Add support for PP2_SUBTYPE_SSL_GROUP (#12921)
maskit Mar 3, 2026
0d88f65
ArgParser: apply default values after dependency validation (#12934)
brbzull0 Mar 5, 2026
36b80b2
Docs: Add cache inspector removal to upgrading guide (#12945)
bryancall Mar 6, 2026
a692ed1
Docs: Fix misleading compress plugin cache option description (#12946)
bryancall Mar 9, 2026
cc42ce9
Fix nullptr crash in RecConfigOverrideFromEnvironment with runroot (#…
brbzull0 Mar 10, 2026
7394dd0
Fix header_rewrite MaxMind geo lookups for GeoIP2/GeoLite2 mmdb datab…
bryancall Mar 16, 2026
763b126
Fix cache_fill Content-Length greater-than config parsing (Coverity C…
bryancall Mar 16, 2026
aa1c4a0
Add AuTest for large chunked contents (#12962)
masaori335 Mar 17, 2026
ec68fbf
Fix uninitialized variables in txn_box plugin (#12979)
bryancall Mar 18, 2026
4a5ff32
Fix uninitialized fields in header_rewrite and ja4_fingerprint plugin…
bryancall Mar 18, 2026
cbf00b1
Fix uninitialized pointer/field members in core classes (#12976)
bryancall Mar 18, 2026
2df54e9
Initialize uninitialized local variables in core production code (#12…
bryancall Mar 18, 2026
84629b9
Add CAP_CHOWN to permitted capability set (#12908)
bryancall Mar 18, 2026
a418679
Fix uninitialized variables in test and benchmark code (#12981)
bryancall Mar 18, 2026
1f769e8
Clarify hostdb host_file.path reload timing and tip usage (#12994)
bryancall Mar 19, 2026
cb92d76
Add a setting to adjust the maximum PP header size (#12961)
maskit Mar 21, 2026
65a417e
Fix remaining uninitialized variable and field Coverity defects (#13004)
bryancall Mar 23, 2026
1304cfc
Test: add msdms milestone logging field validation autest (#12899)
bryancall Mar 23, 2026
67c4300
autest: print worker diagnostics for exceptions (#13017)
bneradt Mar 24, 2026
8a3f9e7
Fix flaky autests for timeout, sigusr2, and thread_config (#13012)
bryancall Mar 25, 2026
f193c43
log-milestone-fields: stabilize across ms truncation (#13019)
bneradt Mar 25, 2026
c025c8d
Initialize uninitialized pointer and scalar members in QUIC and tscpp…
bryancall Mar 26, 2026
01e29cc
sigusr2 autest: simplify Process and Ready objects (#13021)
bneradt Mar 27, 2026
486e5b5
Fix three high-impact Coverity defects (#13030)
bryancall Mar 30, 2026
dada1bc
Fix memory leaks in plugins (#13028)
bryancall Mar 30, 2026
9288fe8
Fix memory leaks in core shutdown path (#13027)
bryancall Mar 30, 2026
403ccc8
Install rustup instead of cargo to use a newer version of rustc (#13057)
hnakamur Apr 7, 2026
bde540a
Clean up stale LSAN suppression entries (#13029)
bryancall Apr 13, 2026
6442606
Fix memory leaks in SSL subsystem (#13026)
bryancall Apr 13, 2026
42e2223
Add 204 and 308 to heuristically cacheable status codes (#13015)
bryancall Apr 13, 2026
549aa95
Remove unused member variables across multiple subsystems (#13005)
bryancall Apr 17, 2026
1bbdf46
Add doxygen comment guidance to AGENTS.md (#13112)
bneradt Apr 21, 2026
f55e3c3
Clarify HostDBInfo state (#13092)
masaori335 Apr 22, 2026
cbd0a58
Add support for TLS Certificate Compression (RFC 8879) (#13088)
maskit Apr 22, 2026
d040e5b
Cap uncompressed length in TLS Certificate Compression (#13117)
maskit Apr 24, 2026
fdceace
Support per-remap geo DB handles in header_rewrite (#13042)
cmcfarlen Apr 27, 2026
8e1c793
Update dependencies for h3 tools (#13136)
maskit May 5, 2026
943e7c1
Parallelize dir-sync on graceful shutdowns (#12996)
zwoop May 6, 2026
bcef33d
Update dockerfile (#13153)
cmcfarlen May 12, 2026
93cbbf4
Fix AIO callback from_api completion lifetime (#13151)
bneradt May 12, 2026
a499f42
Honor RECA_NO_ACCESS in record lookup RPC encoder (#13141)
brbzull0 May 13, 2026
5190a10
mgmt/rpc: refuse RECA_READ_ONLY/RECA_NO_ACCESS writes (#13142)
brbzull0 May 13, 2026
5879de4
Avoid confusing AIO callback lifetime test (#13159)
bneradt May 13, 2026
5f7d43b
nexthop health status don't overflow fail count (#13164)
traeak May 14, 2026
d59f326
Limit regex remap substitutions (#13139)
bneradt May 15, 2026
c316e13
Fix use-of-uninitialized-value problem from fuzzing (#13140)
shukitchan May 17, 2026
9211982
Zero hdrtoken heap to fix use-of-uninitialized-value (#13172)
shukitchan May 18, 2026
5d3a3a4
Add cache key hash logging field and public API (#13134)
zwoop May 18, 2026
fd4c857
Fix marshal_proxy_protocol_tls_group for LogData refactor
cmcfarlen Jun 2, 2026
70bb098
Stabilize post and slow_post autests under parallel runs (#12886)
bryancall Feb 19, 2026
7ff8b97
Fix uninitialized members and variables flagged by Coverity. (#13000)
bryancall Mar 19, 2026
25c6e2f
Relocate HostDB tests and benchmark to standard directories (#13115)
masaori335 Apr 22, 2026
bdf9ea3
add mstsms log field, slow log as a csv field (#12885)
traeak Feb 21, 2026
239d701
Use AddAwaitFileContainsTestRun more in autests (#13034)
bneradt Mar 30, 2026
a527004
tests/gold_tests/headers tests: use ATSReplayTest (#13033)
bneradt Apr 3, 2026
5d33962
Upgrade bundled RAT to 0.17 (#13071)
bneradt Apr 8, 2026
efb3777
Quiet ESI streaming gunzip zero-output logs (#13171)
bneradt Jun 1, 2026
4700bbc
Rename: PreTransactionLogData -> NonHttpSmLogData (#13154)
bneradt Jun 1, 2026
a005443
cripts: shrink Context from 3408 to 1920 bytes, expand data[] to 16 (…
zwoop Jun 2, 2026
8bc82d1
Update cert compression reporting (#13197)
bneradt Jun 1, 2026
84a0698
proxy.config.ssl.client.CA.cert.filename: overridable (#13174)
bneradt Jun 1, 2026
69426f9
Fix tls_cert_comp autest for 10.2.x ssl_multicert.config
cmcfarlen Jun 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
428 changes: 428 additions & 0 deletions .github/copilot-instructions.md

Large diffs are not rendered by default.

223 changes: 223 additions & 0 deletions .github/instructions/HRW.instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
---
applyTo:
- "plugins/header_rewrite/**/*"
- "tools/hrw4u/**/*"
---

# Header Rewrite Plugin and HRW4U Transpiler

## Overview

Two closely related components that must be kept in sync:

1. **header_rewrite plugin** (`plugins/header_rewrite/`) - ATS plugin for modifying HTTP headers
2. **hrw4u transpiler** (`tools/hrw4u/`) - DSL compiler for generating header_rewrite configurations

## Critical Requirement: Feature Synchronization

**Features added to either component may require corresponding changes in the other.**

### When to Update Both

- **New operator in header_rewrite** → Add syntax and code generation in hrw4u
- **New condition in header_rewrite** → Add parsing and symbols in hrw4u
- **New variable/resource in header_rewrite** → Update hrw4u symbol tables and types
- **New hook in header_rewrite** → Add hook syntax in hrw4u
- **New hrw4u syntax** → Ensure correct header_rewrite config generation

### Bidirectional Compilation

Both directions must work:
- **hrw4u** (forward): HRW4U source → header_rewrite config
- **u4wrh** (reverse): header_rewrite config → HRW4U source

Round-trip test: `hrw4u example.hrw4u | u4wrh` should produce equivalent output.

## Header Rewrite Plugin

### Architecture

**Core files:**
- `parser.cc/h` - Configuration syntax parser
- `factory.cc/h` - Factory for operators and conditions
- `operators.cc/h` - Header manipulation operations
- `conditions.cc/h` - Conditional logic
- `resources.cc/h` - Available variables (headers, IPs, etc.)
- `statement.cc/h` - Rule statement abstraction
- `ruleset.cc/h` - Rule collection and execution
- `matcher.cc/h` - Pattern matching
- `value.cc/h` - Value extraction and manipulation

### Adding Features

**New operator:**
1. Define class in `operators.h`, implement in `operators.cc`
2. Register in `factory.cc`
3. Update hrw4u: `tables.py` (forward mapping tables), `visitor.py` (forward compiler - HRW4UVisitor), and `generators.py` (reverse-resolution tables used by u4wrh)

**New condition:**
1. Define class in `conditions.h`, implement in `conditions.cc`
2. Register in `factory.cc`
3. Update hrw4u: `visitor.py` for parsing, `tables.py` for symbol maps

**New resource/variable:**
1. Define in `resources.h`, implement in `resources.cc`
2. Update hrw4u: `types.py` for type system, `tables.py` (OPERATOR_MAP/CONDITION_MAP/etc.) for symbol tables, `symbols.py` for resolver wiring, and `generators.py` for reverse mappings

## HRW4U Transpiler

### Purpose

Provides readable DSL syntax that compiles to header_rewrite configuration.

**Requirements:** Python 3.11+, ANTLR4

### Project Structure

```
tools/hrw4u/
├── src/ # Python source
│ ├── common.py # Shared utilities
│ ├── types.py # Type system
│ ├── symbols.py # Symbol resolution
│ ├── hrw_symbols.py # Header rewrite symbols
│ ├── tables.py # Symbol/type tables
│ ├── visitor.py # Forward compiler (HRW4UVisitor - hrw4u script)
│ ├── hrw_visitor.py # Reverse compiler (HRWInverseVisitor - u4wrh script)
│ ├── generators.py # Reverse-resolution table generation
│ ├── validation.py # Semantic validation
│ └── lsp/ # LSP server
├── scripts/ # CLI tools
│ ├── hrw4u # Forward compiler (hrw4u → HRW config)
│ ├── u4wrh # Reverse compiler (HRW config → hrw4u)
│ └── hrw4u-lsp # LSP server
├── grammar/ # ANTLR4 grammars
└── tests/ # Test suite
```

### Key Modules

**Type System (`types.py`):**
- HRW4U type hierarchy
- Variable types (string, int, bool, IP, etc.)
- Operator signatures
- Type checking and inference

**Symbol Resolution (`symbols.py`, `hrw_symbols.py`, `tables.py`):**
- Symbol tables for variables, operators, functions
- Scope management
- Built-in symbols for header_rewrite resources

**Reverse-Resolution Tables (`generators.py`):**
- Generates derived tables and reverse mappings from primary forward tables
- Used by u4wrh (reverse compiler) to map HRW config back to hrw4u syntax
- Eliminates duplication by maintaining single source of truth in forward tables

**Visitors:**
- `visitor.py` (HRW4UVisitor) - Forward compilation: hrw4u DSL → header_rewrite config
- `hrw_visitor.py` (HRWInverseVisitor) - Reverse compilation: header_rewrite config → hrw4u DSL
- `kg_visitor.py` (KnowledgeGraphVisitor) - Extracts structured graph data for analysis/visualization (used by `hrw4u-kg` script, rarely modified)

### Adding Features

**New operator:**
1. Update grammar if new syntax needed
2. Add symbol definition in `hrw_symbols.py`
3. Add type signature in `types.py`
4. Update forward compiler in `visitor.py` (HRW4UVisitor) to handle new operator
5. Update `generators.py` to generate reverse mappings for u4wrh
6. Update reverse compiler in `hrw_visitor.py` (HRWInverseVisitor) if special handling needed
7. Add tests in `tests/test_ops.py` and `tests/test_ops_reverse.py`
8. Update corresponding header_rewrite plugin code

**New condition:**
1. Update grammar if needed
2. Add symbol definition in `hrw_symbols.py` and type info in `types.py`
3. Update forward compiler in `visitor.py` (HRW4UVisitor)
4. Update `generators.py` for reverse mappings
5. Update reverse compiler in `hrw_visitor.py` (HRWInverseVisitor) if needed
6. Add tests
7. Update header_rewrite plugin

**New variable:**
1. Add to symbol tables (`tables.py`, `hrw_symbols.py`)
2. Add type definition (`types.py`)
3. Update forward compiler in `visitor.py` (HRW4UVisitor) for property access
4. Update `generators.py` for reverse mappings
5. Add tests
6. Ensure header_rewrite supports it

### Code Style

**Python (3.11+):**
- 4-space indentation (never tabs)
- Type hints on all functions
- Dataclasses for structured data
- Modern Python features (match/case, walrus operator)

**C++ (header_rewrite):**
- Follow ATS C++20 standards
- CamelCase classes, snake_case functions/variables
- 2-space indentation
- Empty line after declarations

## Feature Addition Example

**Hypothetical example to illustrate the workflow:**

Adding a `has-prefix` operator (this operator does not exist):

1. **header_rewrite plugin:**
```cpp
// operators.h
class OperatorHasPrefix : public Operator {
void exec(const Resources &res) override;
};

// operators.cc - implement exec()
// factory.cc - register operator
```

2. **hrw4u transpiler:**
```python
# hrw_symbols.py
OPERATORS = {
'has-prefix': OperatorSymbol(
name='has-prefix',
params=['target', 'prefix'],
return_type=BoolType()
),
}

# generators.py
def generate_has_prefix_op(target, prefix):
return f'has-prefix {target} {prefix}'

# tests/test_ops.py
def test_has_prefix():
# Test forward compilation

# tests/test_ops_reverse.py
def test_has_prefix_reverse():
# Test reverse compilation
```

3. **Verify round-trip:**
```bash
echo 'REMAP { if req.Host has-prefix "www." { } }' | hrw4u | u4wrh
```

## Common Pitfalls

1. **Forgetting to update both components** - Changes often need coordination
2. **Breaking round-trip** - Always test `hrw4u | u4wrh` round-trip
3. **Symbol table drift** - Keep hrw4u symbols synced with plugin capabilities
4. **Type mismatches** - Ensure type system matches plugin runtime behavior
5. **Missing tests** - Add tests for both forward and reverse compilation

## Documentation

- User docs: `doc/admin-guide/plugins/header_rewrite.en.html`
- Plugin README: `plugins/header_rewrite/README`
- HRW4U README: `tools/hrw4u/README.md`
- LSP README: `tools/hrw4u/LSP_README.md`
32 changes: 32 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,38 @@ SMDebug(dbg_ctl, "Processing request for URL: %s", url);
- UPPER_CASE for macros and constants: `HTTP_SM_SET_DEFAULT_HANDLER`
- Private member variables have the `m_` prefix.

**Doxygen Comments:**

When adding doxygen comments:

- `@brief` is assumed for the first sentence, so give a brief summary right
after `/** ` without using `@brief`.
- In the description of classes, functions, and member variables, convey the
responsibility of the item being described (its role and intent), not just
what the code obviously does.
- Use `@a <name>` to reference a function argument by name in prose
(e.g. "If @a size is zero..."). Use `@c <text>` for inline code or
constants (e.g. `@c true`, `@c NULL`, `@c MyEnum::VALUE`).
- Use `@ref`, `@see`, or `@sa` to cross-reference related types or functions
when that helps convey how items interact.
- Use `@param` with `[in]`, `[out]`, or `[in,out]` to indicate the
parameter's direction, followed by a description of its meaning.
- Use `@return` to describe the semantics of the returned value. Don't
restate the type; that is obvious from the signature and rendered docs.
- Use `@note` for non-obvious caveats and `@warning` for hazards (e.g. lock
ordering, lifetime, or threading constraints).
- Use `@code` ... `@endcode` for embedded usage examples.
- For templates, document type parameters with `@tparam`.

Conventions specific to this codebase:

- Every new header file should start with a `/** @file` block (see existing
headers in `include/` for the standard license/section layout).
- Prefer trailing briefs for data members and enumerators, e.g.
`int max_conns{0}; ///< Maximum concurrent connections.`
- Use single-line `///` briefs for short function or type docs where a full
`/** ... */` block would be overkill.

**Modern C++ Patterns (Preferred):**
```cpp
// GOOD - Modern C++20
Expand Down
11 changes: 7 additions & 4 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -541,13 +541,15 @@ check_symbol_exists(SSL_error_description "openssl/ssl.h" HAVE_SSL_ERROR_DESCRIP
check_symbol_exists(SSL_CTX_set_ciphersuites "openssl/ssl.h" TS_USE_TLS_SET_CIPHERSUITES)
check_symbol_exists(SSL_CTX_set_keylog_callback "openssl/ssl.h" TS_HAS_TLS_KEYLOGGING)
check_symbol_exists(SSL_CTX_set_tlsext_ticket_key_cb "openssl/ssl.h" HAVE_SSL_CTX_SET_TLSEXT_TICKET_KEY_CB)
check_symbol_exists(SSL_CTX_add_cert_compression_alg "openssl/ssl.h" HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG)
check_symbol_exists(SSL_CTX_set1_cert_comp_preference "openssl/ssl.h" HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE)
check_symbol_exists(SSL_get_all_async_fds openssl/ssl.h TS_USE_TLS_ASYNC)
check_symbol_exists(OSSL_PARAM_construct_end "openssl/params.h" HAVE_OSSL_PARAM_CONSTRUCT_END)
check_symbol_exists(TLS1_3_VERSION "openssl/ssl.h" TS_USE_TLS13)
check_symbol_exists(MD5_Init "openssl/md5.h" HAVE_MD5_INIT)
check_symbol_exists(ENGINE_load_dynamic "include/openssl/engine.h" HAVE_ENGINE_LOAD_DYNAMIC)
check_symbol_exists(ENGINE_get_default_RSA "include/openssl/engine.h" HAVE_ENGINE_GET_DEFAULT_RSA)
check_symbol_exists(ENGINE_load_private_key "include/openssl/engine.h" HAVE_ENGINE_LOAD_PRIVATE_KEY)
check_symbol_exists(ENGINE_load_dynamic "openssl/engine.h" HAVE_ENGINE_LOAD_DYNAMIC)
check_symbol_exists(ENGINE_get_default_RSA "openssl/engine.h" HAVE_ENGINE_GET_DEFAULT_RSA)
check_symbol_exists(ENGINE_load_private_key "openssl/engine.h" HAVE_ENGINE_LOAD_PRIVATE_KEY)
check_symbol_exists(sysctlbyname "sys/sysctl.h" HAVE_SYSCTLBYNAME)

if(SSLLIB_IS_OPENSSL3)
Expand Down Expand Up @@ -917,7 +919,8 @@ endif()
add_custom_target(
rat
COMMENT "Running Apache RAT"
COMMAND java -jar ${CMAKE_SOURCE_DIR}/ci/apache-rat-0.13-SNAPSHOT.jar -E ${CMAKE_SOURCE_DIR}/ci/rat-regex.txt -d
COMMAND java -jar ${CMAKE_SOURCE_DIR}/ci/apache-rat-0.17.jar --input-exclude-file
${CMAKE_SOURCE_DIR}/ci/rat-exclude.txt --input-include-file ${CMAKE_SOURCE_DIR}/ci/rat-include.txt --
${CMAKE_SOURCE_DIR}
)

Expand Down
Binary file removed ci/apache-rat-0.13-SNAPSHOT.jar
Binary file not shown.
Binary file added ci/apache-rat-0.17.jar
Binary file not shown.
3 changes: 0 additions & 3 deletions ci/asan_leak_suppression/regression.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ leak:RegressionTest_PARENTSELECTION
leak:ParentConfig::reconfigure
leak:RegressionTest_SDK_API_TSHttpConnectIntercept
leak:RegressionTest_SDK_API_TSHttpConnectServerIntercept
leak:make_log_host
leak:ReRegressionSM::clone
leak:RegressionTest_ram_cache
leak:RegressionTest_HttpTransact_is_request_valid
Expand All @@ -11,9 +10,7 @@ leak:MakeTextLogFormat
leak:RegressionTest_HttpTransact_handle_trace_and_options_requests
leak:CRYPTO_malloc
leak:RegressionTest_SDK_API_TSMgmtGet
leak:RegressionTest_Cache_vol
leak:RegressionTest_SDK_API_TSCache
leak:RegressionTest_Hdrs
leak:RegressionTest_SDK_API_TSPortDescriptor
leak:RegressionTest_HostDBProcessor
leak:RegressionTest_DNS
Expand Down
12 changes: 3 additions & 9 deletions ci/asan_leak_suppression/unit_tests.txt
Original file line number Diff line number Diff line change
@@ -1,10 +1,4 @@
# leaks in test_X509HostnameValidator
leak:libcrypto.so.1.1
# for OpenSSL 1.0.2:
leak:CRYPTO_malloc
leak:CRYPTO_realloc
leak:ConsCell
# PR#10295
leak:pcre_jit_stack_alloc
# PR#10541
# marshal_hdr in test_http_hdr_print_and_copy_aux is intentionally
# not destroyed because it holds a reference to a stack-allocated
# TestRefCountObj whose free() override calls exit(1).
leak:test_http_hdr_print_and_copy_aux
Loading