diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 00000000000..72ed6387e1a --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,428 @@ +# Apache Traffic Server - GitHub Copilot Instructions + +## Repository Overview + +Apache Traffic Server (ATS) is a high-performance HTTP/HTTPS caching proxy server written in C++20. It processes large-scale web traffic using an event-driven, multi-threaded architecture with a sophisticated plugin system. + +**Key Facts:** +- ~500K lines of C++20 code (strict - no C++23) +- Event-driven architecture using Continuation callbacks +- Handles HTTP/1.1, HTTP/2, HTTP/3 (QUIC) +- Supports TLS termination and caching +- Extensible via C/C++ plugins + +## Code Style Requirements + +### C++ Style Guidelines + +Some of these rules are enforced automatically by CI (via clang-format and clang-tidy); others are recommended conventions that will not themselves cause CI failures but should still be followed in code reviews. +**Formatting Rules:** +- **Line length:** 132 characters maximum +- **Indentation:** 2 spaces (never tabs) +- **Braces:** Linux kernel style (opening brace on same line) +```cpp +if (condition) { + // code here +} +``` +- **Pointer/reference alignment:** Right side +```cpp +Type *ptr; // Correct +Type &ref; // Correct +Type* ptr; // Wrong +``` +- **Empty line after declarations:** +```cpp +void function() { + int x = 5; + std::string name = "test"; + + // Empty line before first code statement + process_data(x, name); +} +``` +- **Always use braces** for if/while/for (no naked conditions): +```cpp +if (x > 0) { // Correct + foo(); +} + +if (x > 0) // Wrong - missing braces + foo(); +``` +- **Keep variable declarations together:** +```cpp +// Correct - declarations grouped together +void function() { + int count = 0; + std::string name = "test"; + auto *buffer = new_buffer(); + + // Empty line before first code statement + process_data(count, name, buffer); +} + +// Wrong - declarations scattered +void function() { + int count = 0; + process_count(count); + std::string name = "test"; // Don't scatter declarations +} +``` + +**Naming Conventions:** +- Classes: `CamelCase` → `HttpSM`, `NetVConnection`, `CacheProcessor` +- Functions/variables: `snake_case` → `handle_request()`, `server_port`, `cache_key` +- Constants/macros: `UPPER_CASE` → `HTTP_STATUS_OK`, `MAX_BUFFER_SIZE` +- Member variables: `snake_case` with no prefix → `connection_count`, `buffer_size` + +**C++20 Patterns (Use These):** +```cpp +// GOOD - Modern C++20 +auto buffer = std::make_unique(size); +for (const auto &entry : container) { + if (auto *ptr = entry.get(); ptr != nullptr) { + process(ptr); + } +} + +// AVOID - Legacy C-style +MIOBuffer *buffer = (MIOBuffer*)malloc(sizeof(MIOBuffer)); +for (int i = 0; i < container.size(); i++) { + process(container[i]); +} +``` + +**Memory Management:** +- Use RAII and smart pointers (`std::unique_ptr`, `std::shared_ptr`) +- Prefer smart pointers over raw `new`/`delete` when possible +- Use `ats_malloc()`/`ats_free()` for large allocations (not `malloc`) +- Use `IOBuffer` for network data (zero-copy design) +- Note: Some subsystems legitimately use explicit deletes / `delete this` (e.g., continuation-based code) + +**What NOT to Use:** +- ❌ C++23 features (code must compile with C++20) +- ❌ `malloc`/`free` for large allocations (use `ats_malloc`), or prefer heaps or stack allocations +- ❌ Blocking operations in event threads +- ❌ Creating threads manually (use async event system) + +### Comments and Documentation + +**Minimal comments philosophy:** +- Only add comments where code isn't self-explanatory +- **Don't** describe what the code does (the code already shows that) +- **Do** explain *why* something is done if not obvious +- Avoid stating the obvious + +```cpp +// BAD - stating the obvious +// Increment the counter +counter++; + +// GOOD - explaining why +// Skip the first element since it's always the sentinel value +counter++; + +// BAD - describing what +// Loop through all connections and close them +for (auto &conn : connections) { + conn.close(); +} + +// GOOD - explaining why (if not obvious) +// Must close connections before destroying the acceptor to avoid use-after-free +for (auto &conn : connections) { + conn.close(); +} +``` + +**When to add comments:** +- Non-obvious algorithms or math +- Workarounds for bugs in dependencies +- Performance optimizations that reduce clarity +- Security-critical sections +- Complex state machine transitions + +**When NOT to add comments:** +- Self-documenting code +- Obvious operations +- Function/variable names that explain themselves + +### Python Style + +- Python 3.11+ with type hints +- 4-space indentation (never tabs) +- Type annotations on all function signatures + +### License Headers + +**New source and test files** must start with Apache License 2.0 header (`.cc`, `.h`, `.py`, and other code files): +```cpp +/** @file + * + * Brief description of file + * + * @section license License + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +``` + +## Architecture Patterns + +### Event-Driven Model (CRITICAL) + +ATS uses **Continuation-based** asynchronous programming: + +```cpp +// Continuation is the core callback pattern +class MyContinuation : public Continuation { + int handle_event(int event, void *data) { + switch (event) { + case EVENT_SUCCESS: + // Handle success + return EVENT_DONE; + case EVENT_ERROR: + // Handle error + return EVENT_ERROR; + } + return EVENT_CONT; + } +}; +``` + +**Key Rules:** +- ⚠️ **NEVER block in event threads** - use async I/O or thread pools +- All async operations use `Continuation` callbacks +- Return `EVENT_DONE`, `EVENT_CONT`, or `EVENT_ERROR` from handlers +- Use `EThread::schedule()` for deferred work + +### HTTP State Machine + +The `HttpSM` class orchestrates HTTP request processing: + +```cpp +// HttpSM is the central state machine +class HttpSM : public Continuation { + // Processes requests through various states + // Hook into appropriate stage via plugin hooks + // Access transaction state via HttpTxn +}; +``` + +**Common hooks:** +- `TS_HTTP_READ_REQUEST_HDR_HOOK` - After reading client request +- `TS_HTTP_SEND_REQUEST_HDR_HOOK` - Before sending to origin +- `TS_HTTP_READ_RESPONSE_HDR_HOOK` - After reading origin response +- `TS_HTTP_SEND_RESPONSE_HDR_HOOK` - Before sending to client + +### Threading Model + +- **Event threads:** Handle most async work (never block here) +- **DNS threads:** Dedicated DNS resolution pool +- **Disk I/O threads:** Cache disk operations +- **Network threads:** Actually event threads handling network I/O + +**Rule:** Don't create threads. Use the event system or existing thread pools. + +### Debug Logging Pattern + +```cpp +// At file scope +static DbgCtl dbg_ctl{"http_sm"}; + +// In code (preferred) +Dbg(dbg_ctl, "Processing request for URL: %s", url); + +// Alternative (less common) +DbgPrint(dbg_ctl, "Processing request for URL: %s", url); +``` + +**Note:** Use `Dbg()` for new code. `DbgPrint()` exists but is rarely used (~60 vs ~3400 uses). + +## Project Structure (Key Paths) + +``` +trafficserver/ +├── src/ +│ ├── iocore/ # I/O subsystem +│ │ ├── eventsystem/ # Event engine (Continuation.h is core) +│ │ ├── cache/ # Cache implementation +│ │ ├── net/ # Network I/O, TLS, QUIC +│ │ └── dns/ # DNS resolution +│ ├── proxy/ # HTTP proxy logic +│ │ ├── http/ # HTTP/1.1 (HttpSM.cc is central) +│ │ ├── http2/ # HTTP/2 +│ │ ├── http3/ # HTTP/3 +│ │ ├── hdrs/ # Header parsing +│ │ └── logging/ # Logging +│ ├── tscore/ # Core utilities +│ ├── tsutil/ # Utilities (metrics, debugging) +│ └── api/ # Plugin API implementation +│ +├── include/ +│ ├── ts/ # Public plugin API (ts.h) +│ ├── tscpp/ # C++ plugin API +│ └── iocore/ # Internal headers +│ +├── plugins/ # Stable plugins +│ ├── header_rewrite/ # Header manipulation (see HRW.instructions.md) +│ └── experimental/ # Experimental plugins +│ +└── tools/ + └── hrw4u/ # Header Rewrite DSL compiler + +``` + +### Key Files to Understand + +- `include/iocore/eventsystem/Continuation.h` - Core async pattern +- `src/proxy/http/HttpSM.cc` - HTTP state machine (most important) +- `src/iocore/cache/Cache.cc` - Cache implementation +- `include/ts/ts.h` - Plugin API (most stable interface) +- `include/tscore/ink_memory.h` - Memory allocation functions + +## Common Patterns + +### Finding Examples + +**Before writing new code, look for similar existing code:** +- Plugin examples: `example/plugins/` for simple patterns +- Stable plugins: `plugins/` for production patterns +- Experimental plugins: `plugins/experimental/` for newer approaches + +**Pattern discovery:** +- Search for similar functionality in existing code +- Check `include/ts/ts.h` for plugin API patterns +- Look at tests in `tests/gold_tests/` for usage examples + +### Code Organization + +**Typical file structure for a plugin:** +``` +plugins/my_plugin/ +├── my_plugin.cc # Main plugin logic +├── handler.cc # Request/response handlers +├── handler.h # Handler interface +├── config.cc # Configuration parsing +└── CMakeLists.txt # Build configuration +``` + +**Typical class structure:** +- Inherit from `Continuation` for async operations +- Implement `handle_event()` for event processing +- Store state in class members, not globals +- Clean up resources in destructor (RAII) + +### Async Operation Pattern + +**General structure for async operations:** +1. Create continuation with callback +2. Initiate async operation (returns `Action*`) +3. Handle callback events in `handle_event()` +4. Return `EVENT_DONE` when complete + +**Always async, never blocking:** +- Network I/O → Use VConnection +- Cache operations → Use CacheProcessor +- DNS lookups → Use DNSProcessor +- Delayed work → Use `schedule_in()` or `schedule_at()` + +### Error Handling + +**Recoverable errors:** +- Return error codes +- Log with appropriate severity +- Clean up resources (RAII helps) + +**Unrecoverable errors:** +- Use `ink_release_assert()` for conditions that should never happen +- Log detailed context before asserting + +### Testing Approach + +**When adding new functionality:** +1. Check if unit tests exist in same directory (Catch2) +2. Add integration tests in `tests/gold_tests/` (autest) +3. Prefer `Test.ATSReplayTest()` with `replay.yaml` format (Proxy Verifier) +4. Test both success and error paths + +## Configuration + +### Adding New Configuration Records + +1. Define in `src/records/RecordsConfig.cc`: +```cpp +{RECT_CONFIG, "proxy.config.my_feature.enabled", RECD_INT, "0", RECU_RESTART_TS, RR_NULL, RECC_INT, nullptr, RECA_NULL} +``` + +2. Read in code: +```cpp +int enabled = 0; +REC_ReadConfigInteger(enabled, "proxy.config.my_feature.enabled"); +``` + +## What to Avoid + +### Common Mistakes + +❌ **Blocking in event threads:** +```cpp +// WRONG - blocks event thread +sleep(5); +blocking_network_call(); +``` + +✅ **Use async operations:** +```cpp +// CORRECT - schedules continuation +eventProcessor.schedule_in(this, HRTIME_SECONDS(5)); +``` + +❌ **Manual memory management:** +```cpp +// WRONG +auto *obj = new MyObject(); +// ... might leak if exception thrown +delete obj; +``` + +✅ **Use RAII:** +```cpp +// CORRECT +auto obj = std::make_unique(); +// Automatically cleaned up +``` + +❌ **Creating threads:** +```cpp +// WRONG +std::thread t([](){ do_work(); }); +``` + +✅ **Use event system:** +```cpp +// CORRECT +eventProcessor.schedule_imm(continuation, ET_CALL); +``` + +## Additional Resources + +- Plugin API: `include/ts/ts.h` +- Event system: `include/iocore/eventsystem/` +- HTTP state machine: `src/proxy/http/HttpSM.cc` +- Documentation: `doc/developer-guide/` diff --git a/.github/instructions/HRW.instructions.md b/.github/instructions/HRW.instructions.md new file mode 100644 index 00000000000..11f0036400e --- /dev/null +++ b/.github/instructions/HRW.instructions.md @@ -0,0 +1,223 @@ +--- +applyTo: + - "plugins/header_rewrite/**/*" + - "tools/hrw4u/**/*" +--- + +# Header Rewrite Plugin and HRW4U Transpiler + +## Overview + +Two closely related components that must be kept in sync: + +1. **header_rewrite plugin** (`plugins/header_rewrite/`) - ATS plugin for modifying HTTP headers +2. **hrw4u transpiler** (`tools/hrw4u/`) - DSL compiler for generating header_rewrite configurations + +## Critical Requirement: Feature Synchronization + +**Features added to either component may require corresponding changes in the other.** + +### When to Update Both + +- **New operator in header_rewrite** → Add syntax and code generation in hrw4u +- **New condition in header_rewrite** → Add parsing and symbols in hrw4u +- **New variable/resource in header_rewrite** → Update hrw4u symbol tables and types +- **New hook in header_rewrite** → Add hook syntax in hrw4u +- **New hrw4u syntax** → Ensure correct header_rewrite config generation + +### Bidirectional Compilation + +Both directions must work: +- **hrw4u** (forward): HRW4U source → header_rewrite config +- **u4wrh** (reverse): header_rewrite config → HRW4U source + +Round-trip test: `hrw4u example.hrw4u | u4wrh` should produce equivalent output. + +## Header Rewrite Plugin + +### Architecture + +**Core files:** +- `parser.cc/h` - Configuration syntax parser +- `factory.cc/h` - Factory for operators and conditions +- `operators.cc/h` - Header manipulation operations +- `conditions.cc/h` - Conditional logic +- `resources.cc/h` - Available variables (headers, IPs, etc.) +- `statement.cc/h` - Rule statement abstraction +- `ruleset.cc/h` - Rule collection and execution +- `matcher.cc/h` - Pattern matching +- `value.cc/h` - Value extraction and manipulation + +### Adding Features + +**New operator:** +1. Define class in `operators.h`, implement in `operators.cc` +2. Register in `factory.cc` +3. Update hrw4u: `tables.py` (forward mapping tables), `visitor.py` (forward compiler - HRW4UVisitor), and `generators.py` (reverse-resolution tables used by u4wrh) + +**New condition:** +1. Define class in `conditions.h`, implement in `conditions.cc` +2. Register in `factory.cc` +3. Update hrw4u: `visitor.py` for parsing, `tables.py` for symbol maps + +**New resource/variable:** +1. Define in `resources.h`, implement in `resources.cc` +2. Update hrw4u: `types.py` for type system, `tables.py` (OPERATOR_MAP/CONDITION_MAP/etc.) for symbol tables, `symbols.py` for resolver wiring, and `generators.py` for reverse mappings + +## HRW4U Transpiler + +### Purpose + +Provides readable DSL syntax that compiles to header_rewrite configuration. + +**Requirements:** Python 3.11+, ANTLR4 + +### Project Structure + +``` +tools/hrw4u/ +├── src/ # Python source +│ ├── common.py # Shared utilities +│ ├── types.py # Type system +│ ├── symbols.py # Symbol resolution +│ ├── hrw_symbols.py # Header rewrite symbols +│ ├── tables.py # Symbol/type tables +│ ├── visitor.py # Forward compiler (HRW4UVisitor - hrw4u script) +│ ├── hrw_visitor.py # Reverse compiler (HRWInverseVisitor - u4wrh script) +│ ├── generators.py # Reverse-resolution table generation +│ ├── validation.py # Semantic validation +│ └── lsp/ # LSP server +├── scripts/ # CLI tools +│ ├── hrw4u # Forward compiler (hrw4u → HRW config) +│ ├── u4wrh # Reverse compiler (HRW config → hrw4u) +│ └── hrw4u-lsp # LSP server +├── grammar/ # ANTLR4 grammars +└── tests/ # Test suite +``` + +### Key Modules + +**Type System (`types.py`):** +- HRW4U type hierarchy +- Variable types (string, int, bool, IP, etc.) +- Operator signatures +- Type checking and inference + +**Symbol Resolution (`symbols.py`, `hrw_symbols.py`, `tables.py`):** +- Symbol tables for variables, operators, functions +- Scope management +- Built-in symbols for header_rewrite resources + +**Reverse-Resolution Tables (`generators.py`):** +- Generates derived tables and reverse mappings from primary forward tables +- Used by u4wrh (reverse compiler) to map HRW config back to hrw4u syntax +- Eliminates duplication by maintaining single source of truth in forward tables + +**Visitors:** +- `visitor.py` (HRW4UVisitor) - Forward compilation: hrw4u DSL → header_rewrite config +- `hrw_visitor.py` (HRWInverseVisitor) - Reverse compilation: header_rewrite config → hrw4u DSL +- `kg_visitor.py` (KnowledgeGraphVisitor) - Extracts structured graph data for analysis/visualization (used by `hrw4u-kg` script, rarely modified) + +### Adding Features + +**New operator:** +1. Update grammar if new syntax needed +2. Add symbol definition in `hrw_symbols.py` +3. Add type signature in `types.py` +4. Update forward compiler in `visitor.py` (HRW4UVisitor) to handle new operator +5. Update `generators.py` to generate reverse mappings for u4wrh +6. Update reverse compiler in `hrw_visitor.py` (HRWInverseVisitor) if special handling needed +7. Add tests in `tests/test_ops.py` and `tests/test_ops_reverse.py` +8. Update corresponding header_rewrite plugin code + +**New condition:** +1. Update grammar if needed +2. Add symbol definition in `hrw_symbols.py` and type info in `types.py` +3. Update forward compiler in `visitor.py` (HRW4UVisitor) +4. Update `generators.py` for reverse mappings +5. Update reverse compiler in `hrw_visitor.py` (HRWInverseVisitor) if needed +6. Add tests +7. Update header_rewrite plugin + +**New variable:** +1. Add to symbol tables (`tables.py`, `hrw_symbols.py`) +2. Add type definition (`types.py`) +3. Update forward compiler in `visitor.py` (HRW4UVisitor) for property access +4. Update `generators.py` for reverse mappings +5. Add tests +6. Ensure header_rewrite supports it + +### Code Style + +**Python (3.11+):** +- 4-space indentation (never tabs) +- Type hints on all functions +- Dataclasses for structured data +- Modern Python features (match/case, walrus operator) + +**C++ (header_rewrite):** +- Follow ATS C++20 standards +- CamelCase classes, snake_case functions/variables +- 2-space indentation +- Empty line after declarations + +## Feature Addition Example + +**Hypothetical example to illustrate the workflow:** + +Adding a `has-prefix` operator (this operator does not exist): + +1. **header_rewrite plugin:** + ```cpp + // operators.h + class OperatorHasPrefix : public Operator { + void exec(const Resources &res) override; + }; + + // operators.cc - implement exec() + // factory.cc - register operator + ``` + +2. **hrw4u transpiler:** + ```python + # hrw_symbols.py + OPERATORS = { + 'has-prefix': OperatorSymbol( + name='has-prefix', + params=['target', 'prefix'], + return_type=BoolType() + ), + } + + # generators.py + def generate_has_prefix_op(target, prefix): + return f'has-prefix {target} {prefix}' + + # tests/test_ops.py + def test_has_prefix(): + # Test forward compilation + + # tests/test_ops_reverse.py + def test_has_prefix_reverse(): + # Test reverse compilation + ``` + +3. **Verify round-trip:** + ```bash + echo 'REMAP { if req.Host has-prefix "www." { } }' | hrw4u | u4wrh + ``` + +## Common Pitfalls + +1. **Forgetting to update both components** - Changes often need coordination +2. **Breaking round-trip** - Always test `hrw4u | u4wrh` round-trip +3. **Symbol table drift** - Keep hrw4u symbols synced with plugin capabilities +4. **Type mismatches** - Ensure type system matches plugin runtime behavior +5. **Missing tests** - Add tests for both forward and reverse compilation + +## Documentation + +- User docs: `doc/admin-guide/plugins/header_rewrite.en.html` +- Plugin README: `plugins/header_rewrite/README` +- HRW4U README: `tools/hrw4u/README.md` +- LSP README: `tools/hrw4u/LSP_README.md` diff --git a/AGENTS.md b/AGENTS.md index 793910a3df3..75308cecdfb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -308,6 +308,38 @@ SMDebug(dbg_ctl, "Processing request for URL: %s", url); - UPPER_CASE for macros and constants: `HTTP_SM_SET_DEFAULT_HANDLER` - Private member variables have the `m_` prefix. +**Doxygen Comments:** + +When adding doxygen comments: + +- `@brief` is assumed for the first sentence, so give a brief summary right + after `/** ` without using `@brief`. +- In the description of classes, functions, and member variables, convey the + responsibility of the item being described (its role and intent), not just + what the code obviously does. +- Use `@a ` to reference a function argument by name in prose + (e.g. "If @a size is zero..."). Use `@c ` for inline code or + constants (e.g. `@c true`, `@c NULL`, `@c MyEnum::VALUE`). +- Use `@ref`, `@see`, or `@sa` to cross-reference related types or functions + when that helps convey how items interact. +- Use `@param` with `[in]`, `[out]`, or `[in,out]` to indicate the + parameter's direction, followed by a description of its meaning. +- Use `@return` to describe the semantics of the returned value. Don't + restate the type; that is obvious from the signature and rendered docs. +- Use `@note` for non-obvious caveats and `@warning` for hazards (e.g. lock + ordering, lifetime, or threading constraints). +- Use `@code` ... `@endcode` for embedded usage examples. +- For templates, document type parameters with `@tparam`. + +Conventions specific to this codebase: + +- Every new header file should start with a `/** @file` block (see existing + headers in `include/` for the standard license/section layout). +- Prefer trailing briefs for data members and enumerators, e.g. + `int max_conns{0}; ///< Maximum concurrent connections.` +- Use single-line `///` briefs for short function or type docs where a full + `/** ... */` block would be overkill. + **Modern C++ Patterns (Preferred):** ```cpp // GOOD - Modern C++20 diff --git a/CMakeLists.txt b/CMakeLists.txt index cfbdf96e0ab..2dcc13163ce 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -541,13 +541,15 @@ check_symbol_exists(SSL_error_description "openssl/ssl.h" HAVE_SSL_ERROR_DESCRIP check_symbol_exists(SSL_CTX_set_ciphersuites "openssl/ssl.h" TS_USE_TLS_SET_CIPHERSUITES) check_symbol_exists(SSL_CTX_set_keylog_callback "openssl/ssl.h" TS_HAS_TLS_KEYLOGGING) check_symbol_exists(SSL_CTX_set_tlsext_ticket_key_cb "openssl/ssl.h" HAVE_SSL_CTX_SET_TLSEXT_TICKET_KEY_CB) +check_symbol_exists(SSL_CTX_add_cert_compression_alg "openssl/ssl.h" HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG) +check_symbol_exists(SSL_CTX_set1_cert_comp_preference "openssl/ssl.h" HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE) check_symbol_exists(SSL_get_all_async_fds openssl/ssl.h TS_USE_TLS_ASYNC) check_symbol_exists(OSSL_PARAM_construct_end "openssl/params.h" HAVE_OSSL_PARAM_CONSTRUCT_END) check_symbol_exists(TLS1_3_VERSION "openssl/ssl.h" TS_USE_TLS13) check_symbol_exists(MD5_Init "openssl/md5.h" HAVE_MD5_INIT) -check_symbol_exists(ENGINE_load_dynamic "include/openssl/engine.h" HAVE_ENGINE_LOAD_DYNAMIC) -check_symbol_exists(ENGINE_get_default_RSA "include/openssl/engine.h" HAVE_ENGINE_GET_DEFAULT_RSA) -check_symbol_exists(ENGINE_load_private_key "include/openssl/engine.h" HAVE_ENGINE_LOAD_PRIVATE_KEY) +check_symbol_exists(ENGINE_load_dynamic "openssl/engine.h" HAVE_ENGINE_LOAD_DYNAMIC) +check_symbol_exists(ENGINE_get_default_RSA "openssl/engine.h" HAVE_ENGINE_GET_DEFAULT_RSA) +check_symbol_exists(ENGINE_load_private_key "openssl/engine.h" HAVE_ENGINE_LOAD_PRIVATE_KEY) check_symbol_exists(sysctlbyname "sys/sysctl.h" HAVE_SYSCTLBYNAME) if(SSLLIB_IS_OPENSSL3) @@ -917,7 +919,8 @@ endif() add_custom_target( rat COMMENT "Running Apache RAT" - COMMAND java -jar ${CMAKE_SOURCE_DIR}/ci/apache-rat-0.13-SNAPSHOT.jar -E ${CMAKE_SOURCE_DIR}/ci/rat-regex.txt -d + COMMAND java -jar ${CMAKE_SOURCE_DIR}/ci/apache-rat-0.17.jar --input-exclude-file + ${CMAKE_SOURCE_DIR}/ci/rat-exclude.txt --input-include-file ${CMAKE_SOURCE_DIR}/ci/rat-include.txt -- ${CMAKE_SOURCE_DIR} ) diff --git a/ci/apache-rat-0.13-SNAPSHOT.jar b/ci/apache-rat-0.13-SNAPSHOT.jar deleted file mode 100644 index 8b386a7cbed..00000000000 Binary files a/ci/apache-rat-0.13-SNAPSHOT.jar and /dev/null differ diff --git a/ci/apache-rat-0.17.jar b/ci/apache-rat-0.17.jar new file mode 100644 index 00000000000..f0f505aaa46 Binary files /dev/null and b/ci/apache-rat-0.17.jar differ diff --git a/ci/asan_leak_suppression/regression.txt b/ci/asan_leak_suppression/regression.txt index b3dfcf83631..3bcd60f553a 100644 --- a/ci/asan_leak_suppression/regression.txt +++ b/ci/asan_leak_suppression/regression.txt @@ -2,7 +2,6 @@ leak:RegressionTest_PARENTSELECTION leak:ParentConfig::reconfigure leak:RegressionTest_SDK_API_TSHttpConnectIntercept leak:RegressionTest_SDK_API_TSHttpConnectServerIntercept -leak:make_log_host leak:ReRegressionSM::clone leak:RegressionTest_ram_cache leak:RegressionTest_HttpTransact_is_request_valid @@ -11,9 +10,7 @@ leak:MakeTextLogFormat leak:RegressionTest_HttpTransact_handle_trace_and_options_requests leak:CRYPTO_malloc leak:RegressionTest_SDK_API_TSMgmtGet -leak:RegressionTest_Cache_vol leak:RegressionTest_SDK_API_TSCache -leak:RegressionTest_Hdrs leak:RegressionTest_SDK_API_TSPortDescriptor leak:RegressionTest_HostDBProcessor leak:RegressionTest_DNS diff --git a/ci/asan_leak_suppression/unit_tests.txt b/ci/asan_leak_suppression/unit_tests.txt index da34e83c615..a553604f1bf 100644 --- a/ci/asan_leak_suppression/unit_tests.txt +++ b/ci/asan_leak_suppression/unit_tests.txt @@ -1,10 +1,4 @@ -# leaks in test_X509HostnameValidator -leak:libcrypto.so.1.1 -# for OpenSSL 1.0.2: -leak:CRYPTO_malloc -leak:CRYPTO_realloc -leak:ConsCell -# PR#10295 -leak:pcre_jit_stack_alloc -# PR#10541 +# marshal_hdr in test_http_hdr_print_and_copy_aux is intentionally +# not destroyed because it holds a reference to a stack-allocated +# TestRefCountObj whose free() override calls exit(1). leak:test_http_hdr_print_and_copy_aux diff --git a/ci/rat-exclude.txt b/ci/rat-exclude.txt new file mode 100644 index 00000000000..39ad1b11ca8 --- /dev/null +++ b/ci/rat-exclude.txt @@ -0,0 +1,81 @@ +BUILDS/** +autom4te.cache/** +m4/** +build-aux/** +**/body_factory/** +**/config.log +**/config.nice +**/config.status +**/libtool +blib/** +**/stamp-h1 +**/*.txt +**/*uv.lock +**/*.cfg +**/*.in +**/*.dot +**/*.svg +**/*.sed +**/*.dtd +**/*.json +**/*.yaml +**/*.md +**/*.default +**/*.default.in +**/*.config +**/*.gold +**/*.hrw4u +**/.gitignore +**/.gitmodules +**/.perltidyrc +**/.indent.pro +**/.vimrc +**/.clang-* +**/.ripgreprc +**/Doxyfile +**/CHANGES +**/CHANGELOG* +**/LAYOUT +**/README* +**/TODO +**/REVIEWERS +**/INSTALL.SKIP +**/MANIFEST +**/configure +**/ink_rand.cc +**/ink_rand.h +**/ink_res_init.cc +**/ink_res_mkquery.cc +**/hashtable.cc +**/logstats.summary +**/test_AIO.sample +**/stats.memo +**/port.h +**/diags.log +**/pm_to_blib +**/*rubbish* +**/short +%regex[.*/\[.+\]$] +**/MurmurHash3.cc +**/MurmurHash3.h +**/HashFNV.cc +**/HashSip.cc +**/TsConfigGrammar.? +**/*.crt +**/*.key +**/*.pem +**/*.doc +**/protocol_binary.h +**/override.css +**/catch.hpp +**/configuru.hpp +**/Catch2/** +**/yamlcpp/** +**/systemtap/** +**/swoc/** +tests/gold_tests/autest-site/min_cfg +tests/gold_tests/h2/rules/huge_resp_hdrs.conf +tools/http_load/** +**/clang-tidy.conf +build*/** +cmake-build*/** diff --git a/ci/rat-include.txt b/ci/rat-include.txt new file mode 100644 index 00000000000..e41bf9c5415 --- /dev/null +++ b/ci/rat-include.txt @@ -0,0 +1 @@ +**/CMakeLists.txt diff --git a/ci/rat-regex.txt b/ci/rat-regex.txt deleted file mode 100644 index 58b3602f099..00000000000 --- a/ci/rat-regex.txt +++ /dev/null @@ -1,77 +0,0 @@ -^BUILDS$ -^autom4te\.cache$ -^m4$ -^build-aux$ -^body_factory$ -^config\.log$ -^config\.nice$ -^config\.status$ -^libtool$ -^blib$ -^stamp-h1$ -.*(? /arch; else echo "amd64" > /arch; fi \ - && wget -qO- https://go.dev/dl/go1.21.6.linux-$(cat /arch).tar.gz | tar -C ${BASE} -xzf - + && wget -qO- https://go.dev/dl/go${GO_VERSION}.linux-$(cat /arch).tar.gz | tar -C ${BASE} -xzf - ENV CC=clang-${LLVM_VERSION} ENV CXX=clang++-${LLVM_VERSION} @@ -80,7 +89,6 @@ RUN git clone https://boringssl.googlesource.com/boringssl \ -DCMAKE_INSTALL_PREFIX=${BASE}/boringssl \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_CXX_FLAGS='-Wno-error=ignored-attributes -UBORINGSSL_HAVE_LIBUNWIND' \ - -DCMAKE_C_FLAGS=${BSSL_C_FLAGS} \ -DBUILD_SHARED_LIBS=1 \ && cmake \ -B build-static \ @@ -100,7 +108,7 @@ RUN git clone https://boringssl.googlesource.com/boringssl \ ENV QUICHE_BASE="${BASE}/quiche" -RUN git clone -b 0.22.0 --depth 1 https://github.com/cloudflare/quiche.git \ +RUN git clone -b 0.28.0 --depth 1 https://github.com/cloudflare/quiche.git \ && cd quiche \ && QUICHE_BSSL_PATH=${BASE}/boringssl/lib QUICHE_BSSL_LINK_KIND=dylib \ cargo build -j$(nproc) --package quiche --release --features ffi,pkg-config-meta,qlog \ @@ -108,7 +116,7 @@ RUN git clone -b 0.22.0 --depth 1 https://github.com/cloudflare/quiche.git \ && mkdir -p ${QUICHE_BASE}/include \ && cp target/release/libquiche.a ${QUICHE_BASE}/lib/ \ && cp target/release/libquiche.so ${QUICHE_BASE}/lib/ \ - && ln -s ${QUICHE_BASE}/lib/libquiche.so ${QUICHE_BASE}/lib/libquiche.so.0 \ + && ln -sf ${QUICHE_BASE}/lib/libquiche.so ${QUICHE_BASE}/lib/libquiche.so.0 \ && cp quiche/include/quiche.h ${QUICHE_BASE}/include/ \ && cp target/release/quiche.pc ${QUICHE_BASE}/lib/pkgconfig \ && cd .. \ @@ -119,12 +127,13 @@ ENV CFLAGS="-O3" ENV CXXFLAGS="-O3" ENV PKG_CONFIG_PATH="${BASE}/lib/pkgconfig:${BASE}/boringssl/lib/pkgconfig:${BASE}/quiche/lib/pkgconfig" -RUN git clone --depth 1 -b v1.2.0 https://github.com/ngtcp2/nghttp3.git \ +RUN git clone --depth 1 -b v1.15.0 https://github.com/ngtcp2/nghttp3.git \ && cd nghttp3 \ && git submodule update --init \ && autoreconf -if \ && ./configure \ --prefix=${BASE} \ + PKG_CONFIG_PATH=${BASE}/lib/pkgconfig:${BASE}/boringssl/lib/pkgconfig \ CFLAGS="${CFLAGS}" \ CXXFLAGS="${CXXFLAGS}" \ LDFLAGS="${LDFLAGS}" \ @@ -135,7 +144,7 @@ RUN git clone --depth 1 -b v1.2.0 https://github.com/ngtcp2/nghttp3.git \ && rm -rf nghttp3 -RUN git clone --depth 1 -b v1.4.0 https://github.com/ngtcp2/ngtcp2.git \ +RUN git clone --depth 1 -b v1.22.1 https://github.com/ngtcp2/ngtcp2.git \ && cd ngtcp2 \ && autoreconf -if \ && ./configure \ @@ -152,7 +161,7 @@ RUN git clone --depth 1 -b v1.4.0 https://github.com/ngtcp2/ngtcp2.git \ && cd .. \ && rm -rf ngtcp2 -RUN git clone --depth 1 -b v1.60.0 https://github.com/tatsuhiro-t/nghttp2.git \ +RUN git clone --depth 1 -b v1.69.0 https://github.com/tatsuhiro-t/nghttp2.git \ && cd nghttp2 \ && git submodule update --init \ && autoreconf -if \ @@ -170,7 +179,7 @@ RUN git clone --depth 1 -b v1.60.0 https://github.com/tatsuhiro-t/nghttp2.git \ && cd .. \ && rm -rf nghttp2 -RUN git clone --depth 1 -b curl-8_7_1 https://github.com/curl/curl.git \ +RUN git clone --depth 1 -b curl-8_20_0 https://github.com/curl/curl.git \ && cd curl \ && autoreconf -fi \ && ./configure \ @@ -187,6 +196,10 @@ RUN git clone --depth 1 -b curl-8_7_1 https://github.com/curl/curl.git \ && cd .. \ && rm -rf curl +FROM build-setup as build + +ARG ATS_VERSION=10.1.0 + RUN git clone --depth 1 -b ${ATS_VERSION} https://github.com/apache/trafficserver.git \ && cmake \ -Strafficserver \ diff --git a/doc/admin-guide/files/records.yaml.en.rst b/doc/admin-guide/files/records.yaml.en.rst index 55e47e08d35..cfb11e82cb9 100644 --- a/doc/admin-guide/files/records.yaml.en.rst +++ b/doc/admin-guide/files/records.yaml.en.rst @@ -3314,14 +3314,19 @@ HostDB Set the file path for an external host file. If this is set (non-empty) then the file is presumed to be a hosts file in - the standard . - It is read and the entries there added to the HostDB. The file is - periodically checked for a more recent modification date in which case it is - reloaded. The interval is set with :ts:cv:`proxy.config.hostdb.host_file.interval`. + the standard format. It is read and the entries there are added to HostDB. - While not technically reloadable, the value is read every time the file is - to be checked so that if changed the new value will be used on the next - check and the file will be treated as modified. + This setting is not immediately reloadable. |TS| checks + :ts:cv:`proxy.config.hostdb.host_file.path` during the periodic host file + check controlled by :ts:cv:`proxy.config.hostdb.host_file.interval` + (default: ``86400`` seconds). If the path value has changed, |TS| uses the + new path on that next check and treats the file as modified. + + .. tip:: + + For faster pickup during testing, temporarily reduce + :ts:cv:`proxy.config.hostdb.host_file.interval`, then restore it after + verification. .. ts:cv:: CONFIG proxy.config.hostdb.host_file.interval INT 86400 :units: seconds @@ -4353,6 +4358,40 @@ SSL Termination ``1`` Enables the use of Kernel TLS.. ===== ====================================================================== +.. ts:cv:: CONFIG proxy.config.ssl.server.cert_compression.algorithms STRING + :reloadable: + + A comma-separated list of compression algorithms that |TS| is willing to + use for TLS Certificate Compression + (`RFC 8879 `_) when |TS| + acts as a TLS server (i.e. accepting connections from clients). When a + connecting client advertises support for one of these algorithms, |TS| will + send its certificate in compressed form, reducing handshake size. + + Supported values: ``zlib``, ``brotli``, ``zstd``. The order determines the + server's preference. An empty value (the default) disables certificate + compression. + + ``brotli`` and ``zstd`` are only available when |TS| is compiled with the + corresponding libraries. + + Example:: + + proxy.config.ssl.server.cert_compression.algorithms: zlib,brotli + +.. ts:cv:: CONFIG proxy.config.ssl.client.cert_compression.algorithms STRING + :reloadable: + + A comma-separated list of compression algorithms that |TS| advertises for + TLS Certificate Compression + (`RFC 8879 `_) when |TS| + acts as a TLS client (i.e. connecting to origin servers). When the origin + supports one of these algorithms, |TS| will accept and decompress the + certificate. + + Supported values: ``zlib``, ``brotli``, ``zstd``. An empty value (the + default) disables certificate compression. + Client-Related Configuration ---------------------------- @@ -4435,6 +4474,7 @@ Client-Related Configuration .. ts:cv:: CONFIG proxy.config.ssl.client.CA.cert.path STRING NULL :reloadable: + :overridable: Specifies the location of the certificate authority file against which the origin server will be verified. @@ -5311,6 +5351,19 @@ UDP Configuration Enables (``1``) or disables (``0``) UDP GRO. When enabled, |TS| will try to use it when reading the UDP socket. + +PROXY protocol Configuration +============================= + +.. ts:cv:: CONFIG proxy.config.proxy_protocol.max_header_size INT 109 + :reloadable: + + Sets the maximum size of PROXY protocol header to receive. + The default size is enough for PROXY protocol version 1. The size needs to be increased + if the version 2 is used with many TLV fields. Although you can set a number up to 65535, + setting a large number can affect performance. + + Plug-in Configuration ===================== diff --git a/doc/admin-guide/logging/formatting.en.rst b/doc/admin-guide/logging/formatting.en.rst index f504cab2d64..b121ad440b8 100644 --- a/doc/admin-guide/logging/formatting.en.rst +++ b/doc/admin-guide/logging/formatting.en.rst @@ -151,6 +151,7 @@ Cache Details .. _crc: .. _crsc: .. _chm: +.. _ckh: .. _cwr: .. _cwtr: .. _crra: @@ -166,6 +167,10 @@ Field Source Description cluc Client Request Cache Lookup URL, also known as the :term:`cache key`, which is the canonicalized version of the client request URL. +ckh Proxy Cache Cache Key Hash. The base64-encoded cryptographic hash of the + effective cache key used for cache lookup and storage. This + is the actual key used to index cache objects. Empty + (``-``) when no cache lookup was performed. crc Proxy Cache Cache Result Code. The result of |TS| attempting to obtain the object from cache; :ref:`admin-logging-cache-results`. crsc Proxy Cache Cache Result Sub-Code. More specific code to complement the @@ -582,6 +587,8 @@ pptc Proxy Protocol The TLS cipher from Proxy Protocol context from the LB TLS Cipher to the |TS| pptv Proxy Protocol The TLS version from Proxy Protocol context from the LB TLS version to the |TS| +pptg Proxy Protocol The TLS group from Proxy Protocol context from the LB + TLS group to the |TS| ===== ============== ========================================================== .. note:: @@ -802,6 +809,7 @@ Timestamps and Durations .. _crat: .. _ms: .. _msdms: +.. _mstsms: .. _stms: .. _stmsh: .. _stmsf: @@ -821,54 +829,56 @@ Other fields in this category provide variously formatted timestamps of particular events within the current transaction (e.g. the time at which a client request was received by |TS|). -===== ======================= ================================================= -Field Source Description -===== ======================= ================================================= -cqtd Client Request Client request timestamp. Specifies the date of - the client request in the format ``YYYY-MM-DD`` - (four digit year, two digit month, two digit day - - with leading zeros as necessary for the latter - two). -cqtn Client Request Client request timestamp in the Netscape - timestamp format. -cqtq Client Request The time at which the client request was received - expressed as fractional (floating point) seconds - since midnight January 1, 1970 UTC (epoch), with - millisecond resolution. -cqts Client Request Same as cqtq_, but as an integer without - sub-second resolution. -cqth Client Request Same as cqts_, but represented in hexadecimal. -cqtt Client Request Client request timestamp in the 24-hour format - ``hh:mm:ss`` (two digit hour, minutes, and - seconds - with leading zeros as necessary). -crat Origin Response Retry-After time in seconds if specified in the - origin server response. -ms Proxy Timestamp in milliseconds of a specific milestone - for this request. See note below about specifying - which milestone to use. -msdms Proxy Difference in milliseconds between the timestamps - of two milestones. See note below about - specifying which milestones to use. -stms Proxy-Origin Connection Time (in milliseconds) spent accessing the origin - server. Measured from the time the connection - between proxy and origin is established to the - time it was closed. -stmsh Proxy-Origin Connection Same as stms_, but represented in hexadecimal. -stmsf Proxy-Origin Connection Same as stms_, but in fractional (floating point) - seconds. -sts Proxy-Origin Connection Same as stms_, but in integer seconds (no - sub-second precision). -ttms Client-Proxy Connection Time in milliseconds spent by |TS| processing the - entire client request. Measured from the time the - connection between the client and |TS| proxy was - established until the last byte of the proxy - response was delivered to the client. -ttmsh Client-Proxy Connection Same as ttms_, but represented in hexadecimal. -ttmsf Client-Proxy Connection Same as ttms_, but in fraction (floating point) - seconds. -tts Client Request Same as ttms_, but in integer seconds (no - sub-second precision). -===== ======================= ================================================= +====== ======================= ================================================= +Field Source Description +====== ======================= ================================================= +cqtd Client Request Client request timestamp. Specifies the date of + the client request in the format ``YYYY-MM-DD`` + (four digit year, two digit month, two digit day + - with leading zeros as necessary for the latter + two). +cqtn Client Request Client request timestamp in the Netscape + timestamp format. +cqtq Client Request The time at which the client request was received + expressed as fractional (floating point) seconds + since midnight January 1, 1970 UTC (epoch), with + millisecond resolution. +cqts Client Request Same as cqtq_, but as an integer without + sub-second resolution. +cqth Client Request Same as cqts_, but represented in hexadecimal. +cqtt Client Request Client request timestamp in the 24-hour format + ``hh:mm:ss`` (two digit hour, minutes, and + seconds - with leading zeros as necessary). +crat Origin Response Retry-After time in seconds if specified in the + origin server response. +ms Proxy Timestamp in milliseconds of a specific milestone + for this request. See note below about specifying + which milestone to use. +msdms Proxy Difference in milliseconds between the timestamps + of two milestones. See note below about + specifying which milestones to use. +mstsms Proxy Slow log report in milliseconds as CSV. + See note below about what timestamps are used. +stms Proxy-Origin Connection Time (in milliseconds) spent accessing the origin + server. Measured from the time the connection + between proxy and origin is established to the + time it was closed. +stmsh Proxy-Origin Connection Same as stms_, but represented in hexadecimal. +stmsf Proxy-Origin Connection Same as stms_, but in fractional (floating point) + seconds. +sts Proxy-Origin Connection Same as stms_, but in integer seconds (no + sub-second precision). +ttms Client-Proxy Connection Time in milliseconds spent by |TS| processing the + entire client request. Measured from the time the + connection between the client and |TS| proxy was + established until the last byte of the proxy + response was delivered to the client. +ttmsh Client-Proxy Connection Same as ttms_, but represented in hexadecimal. +ttmsf Client-Proxy Connection Same as ttms_, but in fraction (floating point) + seconds. +tts Client Request Same as ttms_, but in integer seconds (no + sub-second precision). +====== ======================= ================================================= .. note:: @@ -883,6 +893,32 @@ tts Client Request Same as ttms_, but in integer seconds (no For more information on transaction milestones in |TS|, refer to the documentation on :func:`TSHttpTxnMilestoneGet`. +.. note:: + + A full milestone report can be generated as a CSV string that matches + the example slow log. Fields are: + + 1. tls_handshake + 2. ua_begin + 3. ua_first_read + 4. ua_read_header_done + 5. cache_open_read_begin + 6. cache_open_read_end + 7. cache_open_write_begin + 8. cache_open_write_end + 9. dns_lookup_begin + 10. dns_lookup_end + 11. server_connect + 12. server_connect_end + 13. server_first_read + 14. server_read_header_done + 15. server_close + 16. ua_write + 17. ua_close + 18. sm_finish + 19. plugin_active + 20. plugin_total + .. _admin-logging-fields-urls: URLs, Schemes, and Paths diff --git a/doc/admin-guide/monitoring/statistics/core/ssl.en.rst b/doc/admin-guide/monitoring/statistics/core/ssl.en.rst index efef309c222..c3dc1bb7994 100644 --- a/doc/admin-guide/monitoring/statistics/core/ssl.en.rst +++ b/doc/admin-guide/monitoring/statistics/core/ssl.en.rst @@ -389,3 +389,69 @@ Stats for Pre-warming TLS Tunnel is registered dynamically. The ``POOL`` in belo :type: counter Represents the total number of pre-warming retry. + +.. ts:stat:: global proxy.process.ssl.cert_compress.zlib integer + :type: counter + + The number of times a server certificate was compressed with zlib during a + TLS handshake. + +.. ts:stat:: global proxy.process.ssl.cert_compress.zlib_failure integer + :type: counter + + The number of times zlib compression of a server certificate failed. + +.. ts:stat:: global proxy.process.ssl.cert_decompress.zlib integer + :type: counter + + The number of times a certificate received from an origin server was + decompressed with zlib. + +.. ts:stat:: global proxy.process.ssl.cert_decompress.zlib_failure integer + :type: counter + + The number of times zlib decompression of a certificate failed. + +.. ts:stat:: global proxy.process.ssl.cert_compress.brotli integer + :type: counter + + The number of times a server certificate was compressed with Brotli during a + TLS handshake. + +.. ts:stat:: global proxy.process.ssl.cert_compress.brotli_failure integer + :type: counter + + The number of times Brotli compression of a server certificate failed. + +.. ts:stat:: global proxy.process.ssl.cert_decompress.brotli integer + :type: counter + + The number of times a certificate received from an origin server was + decompressed with Brotli. + +.. ts:stat:: global proxy.process.ssl.cert_decompress.brotli_failure integer + :type: counter + + The number of times Brotli decompression of a certificate failed. + +.. ts:stat:: global proxy.process.ssl.cert_compress.zstd integer + :type: counter + + The number of times a server certificate was compressed with zstd during a + TLS handshake. + +.. ts:stat:: global proxy.process.ssl.cert_compress.zstd_failure integer + :type: counter + + The number of times zstd compression of a server certificate failed. + +.. ts:stat:: global proxy.process.ssl.cert_decompress.zstd integer + :type: counter + + The number of times a certificate received from an origin server was + decompressed with zstd. + +.. ts:stat:: global proxy.process.ssl.cert_decompress.zstd_failure integer + :type: counter + + The number of times zstd decompression of a certificate failed. diff --git a/doc/admin-guide/plugins/compress.en.rst b/doc/admin-guide/plugins/compress.en.rst index 77d2bb82868..f8058a2d31d 100644 --- a/doc/admin-guide/plugins/compress.en.rst +++ b/doc/admin-guide/plugins/compress.en.rst @@ -64,8 +64,9 @@ It can be enabled globally for |TS| by adding the following to your With no further options, this will enable the following default behavior: -* Enable caching of both compressed and uncompressed versions of origin - responses as :term:`alternates `. +* Enable caching of compressed responses. Uncompressed and compressed versions + are maintained as separate :term:`alternates ` via + ``Vary: Accept-Encoding``. * Compress objects with `text/*` content types for every origin. @@ -98,10 +99,24 @@ Per site configuration for remap plugin should be ignored. cache ----- -When set to ``true``, causes |TS| to cache both the compressed and uncompressed -versions of the content as :term:`alternates `. When set to -``false``, |TS| will cache only the compressed or decompressed variant returned -by the origin. Enabled by default. +Controls which version of the response is stored in cache when the compress +transform runs (i.e., when the client sends ``Accept-Encoding``). + +When set to ``true``, the compressed (transformed) response is cached. When set +to ``false``, the uncompressed (untransformed) response is cached and +compression is performed on-the-fly for subsequent cache hits. + +.. note:: + + The plugin always adds ``Vary: Accept-Encoding`` to compressible responses. + This causes |TS| to use separate cache :term:`alternates ` (keys) + for requests with different ``Accept-Encoding`` values. Which body + representation is actually stored in cache still depends on the ``cache`` + option: with ``cache true`` the compressed response is cached, while with + ``cache false`` only the uncompressed response is cached and compression is + performed on-the-fly for clients that send ``Accept-Encoding``. + +Enabled by default. range-request ------------- diff --git a/doc/admin-guide/storage/index.en.rst b/doc/admin-guide/storage/index.en.rst index 9fbdc4437e9..d6f650e89c1 100644 --- a/doc/admin-guide/storage/index.en.rst +++ b/doc/admin-guide/storage/index.en.rst @@ -15,6 +15,8 @@ specific language governing permissions and limitations under the License. +.. include:: ../../common.defs + .. _admin-cache-storage: Cache Storage @@ -312,6 +314,27 @@ the object. The next request for that object will result in a fresh copy of the object fetched. Users may still see the old (removed) content if it was cached by intermediary caches or by the end-users' web browser. +Conditional Cache Invalidation on DELETE Requests +================================================= + +In compliance with :rfc:`9111` Section 4.4 "Invalidating Stored Responses", |TS| +automatically invalidates cached responses when it receives a **successful** +(non-error, i.e., 2xx or 3xx status code) response to a ``DELETE`` request for the same URL. +This behavior ensures cache consistency when resources are deleted at the origin server. + +This means: + +- If a ``DELETE`` request to the origin server returns a success response + (e.g., ``200 OK``, ``204 No Content``), the cached object for that URL is invalidated. +- If the ``DELETE`` request returns an error response (e.g., ``404 Not Found``, + ``405 Method Not Allowed``), the cached object remains unchanged. + +.. note:: + + This automatic invalidation differs from the ``PURGE`` method, which is a |TS| specific + mechanism for explicit cache removal. The ``DELETE`` method invalidation follows standard + HTTP semantics and requires the request to be forwarded to the origin server. + Pushing an Object into the Cache ================================ diff --git a/doc/developer-guide/api/functions/TSHttpOverridableConfig.en.rst b/doc/developer-guide/api/functions/TSHttpOverridableConfig.en.rst index 9a175f4e15f..623b40f3a13 100644 --- a/doc/developer-guide/api/functions/TSHttpOverridableConfig.en.rst +++ b/doc/developer-guide/api/functions/TSHttpOverridableConfig.en.rst @@ -190,6 +190,7 @@ TSOverridableConfigKey Value Config :enumerator:`TS_CONFIG_SSL_CLIENT_CERT_FILENAME` :ts:cv:`proxy.config.ssl.client.cert.filename` :enumerator:`TS_CONFIG_SSL_CLIENT_PRIVATE_KEY_FILENAME` :ts:cv:`proxy.config.ssl.client.private_key.filename` :enumerator:`TS_CONFIG_SSL_CLIENT_CA_CERT_FILENAME` :ts:cv:`proxy.config.ssl.client.CA.cert.filename` +:enumerator:`TS_CONFIG_SSL_CLIENT_CA_CERT_PATH` :ts:cv:`proxy.config.ssl.client.CA.cert.path` :enumerator:`TS_CONFIG_HTTP_HOST_RESOLUTION_PREFERENCE` :ts:cv:`proxy.config.hostdb.ip_resolve` :enumerator:`TS_CONFIG_PLUGIN_VC_DEFAULT_BUFFER_INDEX` :ts:cv:`proxy.config.plugin.vc.default_buffer_index` :enumerator:`TS_CONFIG_PLUGIN_VC_DEFAULT_BUFFER_WATER_MARK` :ts:cv:`proxy.config.plugin.vc.default_buffer_water_mark` diff --git a/doc/developer-guide/api/functions/TSHttpTxnCacheKeyDigestGet.en.rst b/doc/developer-guide/api/functions/TSHttpTxnCacheKeyDigestGet.en.rst new file mode 100644 index 00000000000..28d8ea3fc91 --- /dev/null +++ b/doc/developer-guide/api/functions/TSHttpTxnCacheKeyDigestGet.en.rst @@ -0,0 +1,64 @@ +.. Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed + with this work for additional information regarding copyright + ownership. The ASF licenses this file to you under the Apache + License, Version 2.0 (the "License"); you may not use this file + except in compliance with the License. You may obtain a copy of + the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied. See the License for the specific language governing + permissions and limitations under the License. + +.. include:: ../../../common.defs + +.. default-domain:: cpp + +TSHttpTxnCacheKeyDigestGet +************************** + +Synopsis +======== + +.. code-block:: cpp + + #include + +.. function:: TSReturnCode TSHttpTxnCacheKeyDigestGet(TSHttpTxn txnp, char *buffer, int *length) + +Description +=========== + +Get the effective cache key digest (cryptographic hash) that was used for +cache lookup or storage on this transaction. This is the raw hash bytes, +not a hex or base64 encoding. + +The digest size depends on the build configuration: 16 bytes for MD5 +(default) or 32 bytes for SHA-256 (FIPS mode). A 32-byte buffer is +sufficient for either mode: + +.. code-block:: c + + char digest[32]; + int digest_len = sizeof(digest); + if (TSHttpTxnCacheKeyDigestGet(txnp, digest, &digest_len) == TS_SUCCESS) { + // digest_len contains the actual number of bytes written + } + +Pass :code:`nullptr` for *buffer* to query the digest size without +copying. + +Returns :enumerator:`TS_SUCCESS` if a cache key was computed for the +transaction. Returns :enumerator:`TS_ERROR` if no cache lookup was +performed, or if *buffer* is non-null and *\*length* is smaller than the +digest size. In all cases *\*length* is set to the required digest size +on return. + +See Also +======== + +:func:`TSHttpTxnCacheLookupUrlGet` diff --git a/doc/developer-guide/api/types/TSOverridableConfigKey.en.rst b/doc/developer-guide/api/types/TSOverridableConfigKey.en.rst index 56d325e6192..9d200845c66 100644 --- a/doc/developer-guide/api/types/TSOverridableConfigKey.en.rst +++ b/doc/developer-guide/api/types/TSOverridableConfigKey.en.rst @@ -154,6 +154,7 @@ Enumeration Members .. enumerator:: TS_CONFIG_SSL_CLIENT_SNI_POLICY .. enumerator:: TS_CONFIG_SSL_CLIENT_PRIVATE_KEY_FILENAME .. enumerator:: TS_CONFIG_SSL_CLIENT_CA_CERT_FILENAME +.. enumerator:: TS_CONFIG_SSL_CLIENT_CA_CERT_PATH .. enumerator:: TS_CONFIG_HTTP_HOST_RESOLUTION_PREFERENCE .. enumerator:: TS_CONFIG_PLUGIN_VC_DEFAULT_BUFFER_INDEX .. enumerator:: TS_CONFIG_PLUGIN_VC_DEFAULT_BUFFER_WATER_MARK diff --git a/doc/developer-guide/core-architecture/hostdb.en.rst b/doc/developer-guide/core-architecture/hostdb.en.rst index 18a0e06e61d..0666c8bfb28 100644 --- a/doc/developer-guide/core-architecture/hostdb.en.rst +++ b/doc/developer-guide/core-architecture/hostdb.en.rst @@ -50,12 +50,10 @@ a flag, where a value of ``TS_TIME_ZERO`` indicates a live target and any other down info. If an info is marked down (has a non-zero last failure time) there is a "fail window" during which -no connections are permitted. After this time the info is considered to be a "zombie". If all infos +no connections are permitted. After this time the info is considered to be a "suspect". If all infos for a record are down then a specific error message is generated (body factory tag -"connect#all_down"). Otherwise if the selected info is a zombie, a request is permitted but the -zombie is immediately marked down again, preventing any additional requests until either the fail -window has passed or the single connection succeeds. A successful connection clears the last file -time and the info becomes alive. +"connect#all_down"). Otherwise if the selected info is a suspect, connections are permitted and the +info will transition back to up on success or down on failure. Runtime Structure ================= @@ -152,8 +150,8 @@ Future There is still some work to be done in future PRs. -* The fail window and the zombie window should be separate values. It is quite reasonable to want - to configure a very short fail window (possibly 0) with a moderately long zombie window so that +* The fail window and the suspect window should be separate values. It is quite reasonable to want + to configure a very short fail window (possibly 0) with a moderately long suspect window so that probing connections can immediately start going upstream at a low rate. * Failing an upstream should be more loosely connected to transactions. Currently there is a one @@ -189,7 +187,7 @@ This version has several major architectural changes from the previous version. * State information has been promoted to atomics and updates are immediate rather than scheduled. This also means the data in the state machine is a reference to a shared object, not a local copy. - The promotion was necessary to coordinate zombie connections to upstreams marked down across transactions. + The promotion was necessary to coordinate suspect connections to upstreams marked down across transactions. * The "resolve key" is now a separate data object from the HTTP request. This is a subtle but major change. The effect is requests can be routed to different upstreams without changing diff --git a/doc/release-notes/upgrading.en.rst b/doc/release-notes/upgrading.en.rst index 90df999654d..5fc96c58186 100644 --- a/doc/release-notes/upgrading.en.rst +++ b/doc/release-notes/upgrading.en.rst @@ -36,6 +36,7 @@ removed in the next major release of ATS. * Removed Features * HostDB no longer supports persistent storage for DNS resolution * Removed support for the MMH crypto hash function + * Removed the built-in stats and cache inspector pages that were previously accessible via the |TS| HTTP interface * Traffic Manager is no longer part of |TS|. Administrative tools now interact with |TS| directly by using the :ref:`jsonrpc-node`. diff --git a/example/plugins/c-api/client_context_dump/client_context_dump.cc b/example/plugins/c-api/client_context_dump/client_context_dump.cc index d416c72884b..f8572d775ce 100644 --- a/example/plugins/c-api/client_context_dump/client_context_dump.cc +++ b/example/plugins/c-api/client_context_dump/client_context_dump.cc @@ -163,6 +163,7 @@ CB_context_dump(TSCont, TSEvent, void *edata) for (int i = 0; i < count; i += 2) { dump_context(results[i], results[i + 1]); } + free(results); } } TSTextLogObjectFlush(context_dump_log); diff --git a/include/cripts/Configs.hpp b/include/cripts/Configs.hpp index 6826f64247f..c2f4597a6a7 100644 --- a/include/cripts/Configs.hpp +++ b/include/cripts/Configs.hpp @@ -357,6 +357,7 @@ class Proxy { public: cripts::StringConfig filename{"proxy.config.ssl.client.CA.cert.filename"}; + cripts::StringConfig path{"proxy.config.ssl.client.CA.cert.path"}; }; // End class Cert public: diff --git a/include/cripts/Connections.hpp b/include/cripts/Connections.hpp index a8e851d7ae1..88a76e896e2 100644 --- a/include/cripts/Connections.hpp +++ b/include/cripts/Connections.hpp @@ -459,10 +459,9 @@ class ConnBase void virtual _initialize() { _initialized = true; } - cripts::Transaction *_state = nullptr; - struct sockaddr const *_socket = nullptr; - TSVConn _vc = nullptr; - char _str[INET6_ADDRSTRLEN + 1]; + cripts::Transaction *_state = nullptr; + struct sockaddr const *_socket = nullptr; + TSVConn _vc = nullptr; bool _initialized = false; }; // End class ConnBase diff --git a/include/cripts/Context.hpp b/include/cripts/Context.hpp index 00bc36eafc6..19b92bc77a5 100644 --- a/include/cripts/Context.hpp +++ b/include/cripts/Context.hpp @@ -18,6 +18,7 @@ #pragma once #include +#include #include #include "ts/ts.h" #include "ts/remap.h" @@ -28,7 +29,7 @@ #include "cripts/Connections.hpp" // These are pretty arbitrary for now -constexpr int CONTEXT_DATA_SLOTS = 4; +constexpr int CONTEXT_DATA_SLOTS = 16; namespace cripts { @@ -132,23 +133,16 @@ class Context } _cache; struct _UrlBlock { - cripts::Client::URL &request; - cripts::Pristine::URL pristine; - cripts::Parent::URL parent; + cripts::Client::URL &request; + std::unique_ptr pristine; + std::unique_ptr parent; struct { - cripts::Remap::From::URL from; - cripts::Remap::To::URL to; + std::unique_ptr from; + std::unique_ptr to; } remap; - _UrlBlock(Context *ctx, cripts::Client::URL &alias) : request(alias) - { - request.set_context(ctx); - pristine.set_context(ctx); - parent.set_context(ctx); - remap.from.set_context(ctx); - remap.to.set_context(ctx); - } + _UrlBlock(Context *ctx, cripts::Client::URL &alias) : request(alias) { request.set_context(ctx); } } _urls; }; // End class Context diff --git a/include/cripts/Error.hpp b/include/cripts/Error.hpp index da674abe600..f0957c7bf68 100644 --- a/include/cripts/Error.hpp +++ b/include/cripts/Error.hpp @@ -17,6 +17,7 @@ */ #pragma once +#include #include #include "ts/ts.h" @@ -132,10 +133,10 @@ class Error void Execute(cripts::Context *context); private: - Reason _reason; - Status _status; - bool _failed = false; - bool _redirect = false; + std::unique_ptr _reason; + Status _status; + bool _failed = false; + bool _redirect = false; }; } // namespace cripts diff --git a/include/cripts/Urls.hpp b/include/cripts/Urls.hpp index c28f1d1dab8..73580aa068f 100644 --- a/include/cripts/Urls.hpp +++ b/include/cripts/Urls.hpp @@ -18,6 +18,7 @@ #pragma once #include +#include #include #include #include @@ -338,7 +339,7 @@ class Url cripts::string_view GetSV() override; cripts::string operator+=(cripts::string_view add); - self_type operator=(cripts::string_view path); + self_type &operator=(cripts::string_view path); String operator[](Segments::size_type ix); void @@ -346,8 +347,10 @@ class Url { auto p = operator[](ix); - _size -= p.size(); - p.operator=(""); + if (_state && ix < _state->segments.size()) { + _state->size -= p.size(); + p.operator=(""); + } } void @@ -368,7 +371,7 @@ class Url void Flush() { - if (_modified) { + if (_state && _state->modified) { operator=(GetSV()); } } @@ -376,10 +379,23 @@ class Url private: void _parser(); - bool _modified = false; - Segments _segments; // Lazy loading on this - cripts::string _storage; // Used when recombining the segments into a full path - cripts::string::size_type _size = 0; // Mostly a guestimate for managing _storage + struct State { + bool modified = false; + Segments segments; // Ordered list of path segments + cripts::string storage; // Used when recombining the segments into a full path + cripts::string::size_type size = 0; // Mostly a guestimate for managing storage + }; + + State & + _ensure_state() + { + if (!_state) { + _state = std::make_unique(); + } + return *_state; + } + + std::unique_ptr _state; // Lazily allocated when path is parsed or modified }; // End class Url::Path @@ -461,18 +477,18 @@ class Url using Component::Component; - Query(cripts::string_view load) + Query(cripts::string_view load) : _state(std::make_unique()) { - _data = load; - _size = load.size(); - _loaded = true; - _standalone = true; + _data = load; + _state->size = load.size(); + _loaded = true; + _state->standalone = true; } void Reset() override; cripts::string_view GetSV() override; - self_type operator=(cripts::string_view query); + self_type &operator=(cripts::string_view query); cripts::string operator+=(cripts::string_view add); Parameter operator[](cripts::string_view param); void Erase(cripts::string_view param); @@ -482,7 +498,9 @@ class Url Erase() { operator=(""); - _size = 0; + if (_state) { + _state->size = 0; + } } void @@ -503,14 +521,14 @@ class Url // Make sure the hash and vector are populated _parser(); - std::ranges::sort(_ordered); - _modified = true; + std::ranges::sort(_state->ordered); + _state->modified = true; } void Flush() { - if (_modified) { + if (_state && _state->modified) { operator=(GetSV()); } } @@ -518,19 +536,33 @@ class Url private: void _parser(); - bool _modified = false; - bool _standalone = false; // This component is used outside of a URL owner, not common - OrderedParams _ordered; // Ordered vector of all parameters, can be sorted etc. - HashParams _hashed; // Unordered map to go from "name" to the query parameter - cripts::string _storage; // Used when recombining the query params into a - // full query string - cripts::string::size_type _size = 0; // Mostly a guesttimate + struct State { + bool modified = false; + bool standalone = false; // This component is used outside of a URL owner, not common + OrderedParams ordered; // Ordered vector of all parameters, can be sorted etc. + HashParams hashed; // Unordered map to go from "name" to the query parameter + cripts::string storage; // Used when recombining the query params into a full query string + cripts::string::size_type size = 0; // Mostly a guesttimate + }; + + State & + _ensure_state() + { + if (!_state) { + _state = std::make_unique(); + } + return *_state; + } + + std::unique_ptr _state; // Lazily allocated when query is parsed or modified }; // End class Url::Query public: Url() : scheme(this), host(this), port(this), path(this), query(this) {} + virtual ~Url() = default; + // Clear anything "cached" in the Url, this is rather draconian, but it's safe... virtual void Reset() diff --git a/include/iocore/aio/AIO.h b/include/iocore/aio/AIO.h index dfd14fa925b..332e2ca43c4 100644 --- a/include/iocore/aio/AIO.h +++ b/include/iocore/aio/AIO.h @@ -58,7 +58,6 @@ struct ink_aiocb { off_t aio_offset = 0; /* file offset */ int aio_lio_opcode = 0; /* listio operation */ - int aio_state = 0; /* state flag for List I/O */ }; bool ink_aio_thread_num_set(int thread_num); @@ -80,9 +79,10 @@ struct AIOCallback : public Continuation { EThread *thread = AIO_CALLBACK_THREAD_ANY; AIOCallback *then = nullptr; // set on return from aio_read/aio_write - int64_t aio_result = 0; - AIO_Reqs *aio_req = nullptr; - ink_hrtime sleep_time = 0; + int64_t aio_result = 0; + AIO_Reqs *aio_req = nullptr; + ink_hrtime sleep_time = 0; + bool from_ts_api = false; SLINK(AIOCallback, alink); /* for AIO_Reqs::aio_temp_list */ #if TS_USE_LINUX_IO_URING iovec iov = {}; // this is to support older kernels that only support readv/writev diff --git a/include/iocore/eventsystem/Event.h b/include/iocore/eventsystem/Event.h index 804040979c2..33bff3e62d4 100644 --- a/include/iocore/eventsystem/Event.h +++ b/include/iocore/eventsystem/Event.h @@ -267,20 +267,6 @@ class Event : public Action | UNIX/non-NT Interface | \*-------------------------------------------------------*/ -#ifdef ONLY_USED_FOR_FIB_AND_BIN_HEAP - void *node_pointer; - void - set_node_pointer(void *x) - { - node_pointer = x; - } - void * - get_node_pointer() - { - return node_pointer; - } -#endif - #if defined(__GNUC__) ~Event() override {} #endif diff --git a/include/iocore/hostdb/HostDBProcessor.h b/include/iocore/hostdb/HostDBProcessor.h index fecb1abef7b..c369aace4fa 100644 --- a/include/iocore/hostdb/HostDBProcessor.h +++ b/include/iocore/hostdb/HostDBProcessor.h @@ -123,10 +123,53 @@ enum class HostDBType : uint8_t { }; /** Information about a single target. + * + * Each instance tracks the health state of one upstream address. The state is derived from @c _last_failure and the caller-supplied + * @a fail_window: + * + * | State | Description | + * |---------|-----------------------------------------------------------------------------------| + * | Up | No known failure; eligible for normal selection. | + * | Down | Blocked; no connections permitted until @c _last_failure + @a fail_window elapses | + * | Suspect | Fail window has elapsed; connections are permitted. | + * | | On success transitions to Up (@c mark_up); on failure returns to Down. | + * + * State transition diagram: + * + * @startuml + * hide empty description + * + * [*] --> Up + * Up --> Down : connect failure\n(mark_down) + * Down --> Suspect : fail_window elapses + * Suspect --> Up : connect success\n(mark_up) + * Suspect --> Down : connect failure\n(mark_down) + * @enduml + * + * State transition and `fail_window` time chart: + * + * @code + * |<-- fail_window -->| + * -+----------+--------------------+--------------------+----------+----> time + * | Up | Down | Suspect | Up | + * -+----------+--------------------+--------------------+----------+----> + * ^ ^ ^ + * \ \ \ + * (_last_failure) (_last_failure + fail_window) (connect success) + * @endcode */ -struct HostDBInfo { +class HostDBInfo +{ +public: using self_type = HostDBInfo; ///< Self reference type. + /// Health state of this target. + enum class State { + UP, + DOWN, + SUSPECT, + }; + /// Default constructor. HostDBInfo() = default; @@ -134,50 +177,23 @@ struct HostDBInfo { /// Absolute time of when this target failed. /// A value of zero (@c TS_TIME_ZERO ) indicates no failure. - ts_time last_fail_time() const; - - /// Target is alive - no known failure. - bool is_alive(); - - /// Target has failed and is still in the blocked time window. - bool is_down(ts_time now, ts_seconds fail_window); - - /** Select this target. - * - * @param now Current time. - * @param fail_window Failure window. - * @return Status of the selection. - * - * If a zombie is selected the failure time is updated to make it appear down to other threads in a thread safe - * manner. The caller should check @c last_fail_time to see if a zombie was selected. - */ - bool select(ts_time now, ts_seconds fail_window) const; + ts_time last_fail_time() const; + uint8_t fail_count() const; + char const *srvname() const; - /** Mark the entry as down. - * - * @param now Time of the failure. - * @return @c true if @a this was marked down, @c false if not. - * - * This can return @c false if the entry is already marked down, in which case the failure time is not updated. - */ - bool mark_down(ts_time now); + /// Return the current health state of this target. + State state(ts_time now, ts_seconds fail_window) const; - std::pair increment_fail_count(ts_time now, uint8_t max_retries); + // Sugars of checking state + bool is_up() const; + bool is_down(ts_time now, ts_seconds fail_window) const; + bool is_suspect(ts_time now, ts_seconds fail_window) const; - /** Mark the target as up / alive. - * - * @return Previous alive state of the target. - */ - bool mark_up(); - - char const *srvname() const; + // State controllers + bool mark_up(); + bool mark_down(ts_time now, ts_seconds fail_window); + std::pair increment_fail_count(ts_time now, uint8_t max_retries, ts_seconds fail_window); - /** Migrate data after a DNS update. - * - * @param that Source item. - * - * This moves only specific state information, it is not a generic copy. - */ void migrate_from(self_type const &that); /// A target is either an IP address or an SRV record. @@ -187,16 +203,8 @@ struct HostDBInfo { SRVInfo srv; ///< SRV record. } data{IpAddr{}}; - /// Data that migrates after updated DNS records are processed. - /// @see migrate_from - /// @{ - /// Last time a failure was recorded. - std::atomic last_failure{TS_TIME_ZERO}; - /// Count of connection failures - std::atomic fail_count{0}; /// Expected HTTP version of the target based on earlier transactions. HTTPVersion http_version = HTTP_INVALID; - /// @} self_type &assign(IpAddr const &addr); @@ -207,96 +215,11 @@ struct HostDBInfo { HostDBType type = HostDBType::UNSPEC; ///< Invalid data. friend HostDBContinuation; -}; -inline HostDBInfo & -HostDBInfo::operator=(HostDBInfo const &that) -{ - if (this != &that) { - memcpy(static_cast(this), static_cast(&that), sizeof(*this)); - } - return *this; -} - -inline ts_time -HostDBInfo::last_fail_time() const -{ - return last_failure; -} - -inline bool -HostDBInfo::is_alive() -{ - return this->last_fail_time() == TS_TIME_ZERO; -} - -/** - Check if this HostDBInfo is currently marked DOWN (true) or UP (false). Returns true while within the `fail_window` period after - `last_failure`. Once `fail_window` expires, the host is treated as UP and this function returns false. - - |<-- fail_window -->| - ----------------+-------------------+-----------------> time - UP | DOWN | UP - (is_down=false) | (is_down=true) | (is_down=false) - | | - ^ ^ - \ \ - last_failure last_failure + fail_window - */ -inline bool -HostDBInfo::is_down(ts_time now, ts_seconds fail_window) -{ - auto last_fail = this->last_fail_time(); - return (last_fail != TS_TIME_ZERO) && (now <= last_fail + fail_window); -} - -inline bool -HostDBInfo::mark_up() -{ - auto t = last_failure.exchange(TS_TIME_ZERO); - bool was_down = t != TS_TIME_ZERO; - if (was_down) { - fail_count.store(0); - } - return was_down; -} - -inline bool -HostDBInfo::mark_down(ts_time now) -{ - auto t0{TS_TIME_ZERO}; - return last_failure.compare_exchange_strong(t0, now); -} - -inline std::pair -HostDBInfo::increment_fail_count(ts_time now, uint8_t max_retries) -{ - auto fcount = ++fail_count; - bool marked_down = false; - if (fcount >= max_retries) { - marked_down = mark_down(now); - } - return std::make_pair(marked_down, fcount); -} - -inline bool -HostDBInfo::select(ts_time now, ts_seconds fail_window) const -{ - auto t0 = this->last_fail_time(); - if (t0 == TS_TIME_ZERO) { - return true; // it's alive and so is valid for selection. - } - // Return true and give it a try if enough time is elapsed since the last failure - return (t0 + fail_window < now); -} - -inline void -HostDBInfo::migrate_from(HostDBInfo::self_type const &that) -{ - this->last_failure = that.last_failure.load(); - this->fail_count = that.fail_count.load(); - this->http_version = that.http_version; -} +private: + std::atomic _last_failure{TS_TIME_ZERO}; ///< Last time a failure was recorded + std::atomic _fail_count{0}; ///< Count of connection failures +}; // ---- /** Root item for HostDB. @@ -371,15 +294,12 @@ class HostDBRecord : public RefCountObj /** Pick the next round robin and update the record atomically. * - * @note This may select a zombie server and reserve it for the caller, therefore the caller must - * attempt to connect to the selected target if possible. - * - * @param now Current time to use for aliveness calculations. - * @param fail_window Blackout time for down servers. - * @return Status of the updated target. + * @note This may select a suspect server. The caller must attempt to connect to the selected + * target if possible. * - * If the return value is @c HostDBInfo::Status::DOWN this means all targets are down and there is - * no valid upstream. + * @param[in] now Current time to use for HostDBInfo state calculations. + * @param[in] fail_window Blackout time for down servers. + * @return The selected target, or @c nullptr if all targets are down. * * @note Concurrency - this is not done under lock and depends on the caller for correct use. * For strict round robin, it is a feature that every call will get a distinct index. For @@ -434,9 +354,9 @@ class HostDBRecord : public RefCountObj * This accounts for the round robin setting. The default is to use "client affinity" in * which case @a hash_addr is as a hash seed to select the target. * - * This may select a zombie target, which can be detected by checking the target's last - * failure time. If it is not @c TS_TIME_ZERO the target is a zombie. Other transactions will - * be blocked from selecting that target until @a fail_window time has passed. + * This may select a suspect target (fail window elapsed, connections permitted again), which can + * be detected by checking the target's last failure time. If it is not @c TS_TIME_ZERO the target + * is a suspect. Multiple threads may concurrently select the same suspect target. * * In cases other than strict round robin, a base target is selected. If valid, that is returned, * but if not then the targets in this record are searched until a valid one is found. The result @@ -588,7 +508,7 @@ struct ResolveInfo { /// Keep a reference to the base HostDB object, so it doesn't get GC'd. Ptr record; - HostDBInfo *active = nullptr; ///< Active host record. + HostDBInfo *active = nullptr; ///< Active HostDBInfo /// Working address. The meaning / source of the value depends on other elements. /// This is the "resolved" address if @a resolved_p is @c true. @@ -646,19 +566,20 @@ struct ResolveInfo { */ bool resolve_immediate(); - /** Mark the active target as down. + /** Mark the active target as DOWN. * - * @param now Time of failure. + * @param[in] now Time of failure. + * @param[in] fail_window The fail window duration (proxy.config.http.down_server.cache_time). * @return @c true if the server was marked as down, @c false if not. * */ - bool mark_active_server_down(ts_time now); + bool mark_active_server_down(ts_time now, ts_seconds fail_window); - /** Mark the active target as alive. + /** Mark the active target as UP. * * @return @c true if the target changed state. */ - bool mark_active_server_alive(); + bool mark_active_server_up(); /// Select / resolve to the next RR entry for the record. bool select_next_rr(); @@ -863,15 +784,15 @@ ResolveInfo::set_active(sockaddr const *s) } inline bool -ResolveInfo::mark_active_server_alive() +ResolveInfo::mark_active_server_up() { return active->mark_up(); } inline bool -ResolveInfo::mark_active_server_down(ts_time now) +ResolveInfo::mark_active_server_down(ts_time now, ts_seconds fail_window) { - return active != nullptr && active->mark_down(now); + return active != nullptr && active->mark_down(now, fail_window); } inline bool diff --git a/include/iocore/net/NetVConnection.h b/include/iocore/net/NetVConnection.h index fedb7328ad4..4da60535390 100644 --- a/include/iocore/net/NetVConnection.h +++ b/include/iocore/net/NetVConnection.h @@ -505,7 +505,7 @@ class NetVConnection : public VConnection, public PluginUserArgs S *get_service() const; diff --git a/include/iocore/net/ProxyProtocol.h b/include/iocore/net/ProxyProtocol.h index 5c73019144d..d3a7c73671f 100644 --- a/include/iocore/net/ProxyProtocol.h +++ b/include/iocore/net/ProxyProtocol.h @@ -54,6 +54,7 @@ constexpr uint8_t PP2_SUBTYPE_SSL_CN = 0x22; constexpr uint8_t PP2_SUBTYPE_SSL_CIPHER = 0x23; constexpr uint8_t PP2_SUBTYPE_SSL_SIG_ALG = 0x24; constexpr uint8_t PP2_SUBTYPE_SSL_KEY_ALG = 0x25; +constexpr uint8_t PP2_SUBTYPE_SSL_GROUP = 0x26; constexpr uint8_t PP2_TYPE_NETNS = 0x30; class ProxyProtocol @@ -88,6 +89,7 @@ class ProxyProtocol std::optional get_tlv(const uint8_t tlvCode) const; std::optional get_tlv_ssl_version() const; std::optional get_tlv_ssl_cipher() const; + std::optional get_tlv_ssl_group() const; ProxyProtocolVersion version = ProxyProtocolVersion::UNDEFINED; uint16_t ip_family = AF_UNSPEC; @@ -143,6 +145,7 @@ class ProxyProtocol const size_t PPv1_CONNECTION_HEADER_LEN_MAX = 108; const size_t PPv2_CONNECTION_HEADER_LEN = 16; +extern bool proxy_protocol_detect(swoc::TextView tv); extern size_t proxy_protocol_parse(ProxyProtocol *pp_info, swoc::TextView tv); extern size_t proxy_protocol_build(uint8_t *buf, size_t max_buf_len, const ProxyProtocol &pp_info, ProxyProtocolVersion force_version = ProxyProtocolVersion::UNDEFINED); diff --git a/include/iocore/net/SSLMultiCertConfigLoader.h b/include/iocore/net/SSLMultiCertConfigLoader.h index 1d53afd2190..3891e18c5c8 100644 --- a/include/iocore/net/SSLMultiCertConfigLoader.h +++ b/include/iocore/net/SSLMultiCertConfigLoader.h @@ -111,6 +111,7 @@ class SSLMultiCertConfigLoader virtual bool _set_npn_callback(SSL_CTX *ctx); virtual bool _set_alpn_callback(SSL_CTX *ctx); virtual bool _set_keylog_callback(SSL_CTX *ctx); + virtual bool _enable_cert_compression(SSL_CTX *ctx); virtual bool _enable_ktls(SSL_CTX *ctx); virtual bool _enable_early_data(SSL_CTX *ctx); }; diff --git a/include/iocore/net/TLSEventSupport.h b/include/iocore/net/TLSEventSupport.h index e46220d784a..e8904d26f8a 100644 --- a/include/iocore/net/TLSEventSupport.h +++ b/include/iocore/net/TLSEventSupport.h @@ -78,7 +78,7 @@ class TLSEventSupport private: static int _ex_data_index; - SSL *_ssl; + SSL *_ssl{nullptr}; bool _first_handshake_hooks_pre = true; bool _first_handshake_hooks_outbound_pre = true; diff --git a/include/iocore/net/TLSSNISupport.h b/include/iocore/net/TLSSNISupport.h index 346c2c1c0af..2ce9556f431 100644 --- a/include/iocore/net/TLSSNISupport.h +++ b/include/iocore/net/TLSSNISupport.h @@ -84,7 +84,7 @@ class TLSSNISupport ClientHelloContainer _chc; #if HAVE_SSL_CTX_SET_CLIENT_HELLO_CB int *_ext_ids = nullptr; - size_t _ext_len; + size_t _ext_len{0}; #endif }; diff --git a/include/iocore/net/quic/QUICTransferProgressProvider.h b/include/iocore/net/quic/QUICTransferProgressProvider.h index 77f93b97c5a..2b6929fa5fe 100644 --- a/include/iocore/net/quic/QUICTransferProgressProvider.h +++ b/include/iocore/net/quic/QUICTransferProgressProvider.h @@ -54,7 +54,7 @@ class QUICTransferProgressProviderSA : public QUICTransferProgressProvider bool is_cancelled() const override; private: - QUICStreamAdapter *_adapter; + QUICStreamAdapter *_adapter{nullptr}; }; class QUICTransferProgressProviderVIO : public QUICTransferProgressProvider diff --git a/include/iocore/net/quic/QUICTypes.h b/include/iocore/net/quic/QUICTypes.h index 7517db2f33b..706e807cadb 100644 --- a/include/iocore/net/quic/QUICTypes.h +++ b/include/iocore/net/quic/QUICTypes.h @@ -214,7 +214,7 @@ class QUICStreamError : public QUICError QUICStreamError(const QUICStream *s, const QUICAppErrorCode error_code, const char *error_msg = nullptr) : QUICError(QUICErrorClass::APPLICATION, static_cast(error_code), error_msg), stream(s){}; - const QUICStream *stream; + const QUICStream *stream{nullptr}; }; using QUICErrorUPtr = std::unique_ptr; @@ -575,20 +575,20 @@ struct QUICSentPacketInfo { private: QUICFrameId _id = 0; - QUICFrameGenerator *_generator; + QUICFrameGenerator *_generator{nullptr}; }; // Recovery A.1.1. Sent Packet Fields - QUICPacketNumber packet_number; - bool ack_eliciting; - bool in_flight; - size_t sent_bytes; - ink_hrtime time_sent; + QUICPacketNumber packet_number{0}; + bool ack_eliciting{false}; + bool in_flight{false}; + size_t sent_bytes{0}; + ink_hrtime time_sent{0}; // Additional fields - QUICPacketType type; + QUICPacketType type{QUICPacketType::UNINITIALIZED}; std::vector frames; - QUICPacketNumberSpace pn_space; + QUICPacketNumberSpace pn_space{QUICPacketNumberSpace::INITIAL}; // End of additional fields }; diff --git a/include/proxy/FetchSM.h b/include/proxy/FetchSM.h index 2cf4b3bdef6..2ea205fe4bd 100644 --- a/include/proxy/FetchSM.h +++ b/include/proxy/FetchSM.h @@ -165,7 +165,7 @@ class FetchSM : public Continuation HTTPParser http_parser; HTTPHdr client_response_hdr; ChunkedHandler chunked_handler; - TSFetchEvent callback_events; + TSFetchEvent callback_events{}; TSFetchWakeUpOptions callback_options = NO_CALLBACK; bool req_finished = false; bool header_done = false; diff --git a/include/proxy/HostStatus.h b/include/proxy/HostStatus.h index 7e81f116461..fdf2de2d3bb 100644 --- a/include/proxy/HostStatus.h +++ b/include/proxy/HostStatus.h @@ -99,17 +99,17 @@ struct HostStatuses { // host status POD struct HostStatRec { - TSHostStatus status; - unsigned int reasons; + TSHostStatus status{TSHostStatus::TS_HOST_STATUS_INIT}; + unsigned int reasons{0}; // time the host was marked down for a given reason. - time_t active_marked_down; - time_t local_marked_down; - time_t manual_marked_down; - time_t self_detect_marked_down; + time_t active_marked_down{0}; + time_t local_marked_down{0}; + time_t manual_marked_down{0}; + time_t self_detect_marked_down{0}; // number of seconds that the host should be marked down for a given reason. - unsigned int active_down_time; - unsigned int local_down_time; - unsigned int manual_down_time; + unsigned int active_down_time{0}; + unsigned int local_down_time{0}; + unsigned int manual_down_time{0}; HostStatRec(); HostStatRec(std::string str); diff --git a/include/proxy/Milestones.h b/include/proxy/Milestones.h index 622b5738ae8..8f681357397 100644 --- a/include/proxy/Milestones.h +++ b/include/proxy/Milestones.h @@ -64,7 +64,7 @@ template class Milestones int64_t difference_msec(T ms_start, T ms_end, int64_t missing = -1) const // Return "missing" when Milestone is not set { - if (this->_milestones[static_cast(ms_end)] == 0) { + if (this->_milestones[static_cast(ms_start)] == 0 || this->_milestones[static_cast(ms_end)] == 0) { return missing; } return ink_hrtime_to_msec(this->_milestones[static_cast(ms_end)] - this->_milestones[static_cast(ms_start)]); diff --git a/include/proxy/PreTransactionLogData.h b/include/proxy/NonHttpSmLogData.h similarity index 61% rename from include/proxy/PreTransactionLogData.h rename to include/proxy/NonHttpSmLogData.h index 0ec03c646bb..552e09622ea 100644 --- a/include/proxy/PreTransactionLogData.h +++ b/include/proxy/NonHttpSmLogData.h @@ -1,7 +1,7 @@ /** @file - PreTransactionLogData populates LogData for requests that fail before - HttpSM creation. + NonHttpSmLogData populates LogData for access-log entries that cannot be + backed by an HttpSM. @section license License @@ -30,23 +30,32 @@ #include -/** Populate LogData for requests that never created an HttpSM. +/** Owns access-log data for entries that cannot be backed by an @c HttpSM. * - * Malformed HTTP/2 or HTTP/3 request headers can be rejected while the - * connection is still decoding and validating the stream, before the request - * progresses far enough to create an HttpSM. This class carries the - * copied request and session metadata needed to emit a best-effort - * transaction log entry for those failures. + * Normal transaction access logging is expected to use data extracted from a + * live @c HttpSM. This type is for exceptional client-facing failures that need + * transaction-log visibility but occur before an @c HttpSM exists, such as + * malformed HTTP/2 or HTTP/3 request headers rejected during protocol + * validation. It may also be used for connection-level failures, such as TLS + * handshake errors, when operators need those events in the access log. * - * This class owns its milestones, addresses, and strings because the - * originating stream is about to be destroyed. + * Because the protocol stream or connection state may be destroyed immediately + * after the failure is handled, this object owns the copied headers, + * addresses, milestones, protocol strings, and outcome fields needed by + * @c LogAccess. Fields that require an @c HttpSM, origin transaction, cache + * lookup, or server response are intentionally left unset and marshal through + * the normal default values. + * + * This path should remain narrow. If an @c HttpSM exists, prefer the standard + * @c HttpSM-backed logging path so normal transactions do not pay for extra + * copying or exceptional state. */ -class PreTransactionLogData +class NonHttpSmLogData { public: - PreTransactionLogData() = default; + NonHttpSmLogData() = default; - ~PreTransactionLogData() + ~NonHttpSmLogData() { if (owned_client_request.valid()) { owned_client_request.destroy(); diff --git a/include/proxy/ParentConsistentHash.h b/include/proxy/ParentConsistentHash.h index 4a38c16a615..1b0fee2e9b0 100644 --- a/include/proxy/ParentConsistentHash.h +++ b/include/proxy/ParentConsistentHash.h @@ -41,7 +41,6 @@ class ParentConsistentHash : public ParentSelectionStrategy std::unique_ptr hash[2]; std::unique_ptr chash[2]; pRecord *parents[2]; - bool foundParents[2][MAX_PARENTS]; bool ignore_query; int secondary_mode; ParentHashAlgorithm selected_algorithm; diff --git a/include/proxy/ProxySession.h b/include/proxy/ProxySession.h index a94bdf4dd45..398130bdee4 100644 --- a/include/proxy/ProxySession.h +++ b/include/proxy/ProxySession.h @@ -184,7 +184,7 @@ class ProxySession : public VConnection, public PluginUserArgs IpAllow::ACL acl; ///< IpAllow based method ACL. - HttpSessionAccept::Options const *accept_options; ///< connection info // L7R TODO: set in constructor + HttpSessionAccept::Options const *accept_options{nullptr}; ///< connection info protected: // Hook dispatching state diff --git a/include/proxy/ProxyTransaction.h b/include/proxy/ProxyTransaction.h index 7316be293a2..32b24c6b5bc 100644 --- a/include/proxy/ProxyTransaction.h +++ b/include/proxy/ProxyTransaction.h @@ -146,18 +146,19 @@ class ProxyTransaction : public VConnection void mark_as_tunnel_endpoint() override; - /** Emit a best-effort access log entry for a request that failed before - * HttpSM creation. + /** Emit a best-effort access log entry for a request without an HttpSM. * * Call this when a malformed request is rejected at the protocol layer * (e.g. during HTTP/2 or HTTP/3 header decoding) and no HttpSM was - * created. The method populates a PreTransactionLogData from the + * created. The method populates a NonHttpSmLogData from the * session and the partially decoded request, then invokes Log::access. + * If an HttpSM exists, callers should use the normal transaction logging + * path instead. * * @param[in] request The decoded (possibly partial) request header. * @param[in] protocol_str Protocol string for the log entry (e.g. "http/2"). */ - void log_pre_transaction_access(HTTPHdr const *request, const char *protocol_str); + void log_non_http_sm_access(HTTPHdr const *request, const char *protocol_str); /// Variables // diff --git a/include/proxy/hdrs/HdrHeap.h b/include/proxy/hdrs/HdrHeap.h index 0966ff40e50..75a3620a070 100644 --- a/include/proxy/hdrs/HdrHeap.h +++ b/include/proxy/hdrs/HdrHeap.h @@ -148,7 +148,7 @@ class HdrStrHeap : public RefCountObj HdrStrHeap(uint32_t total_size) : _total_size{total_size} {} uint32_t const _total_size; - uint32_t _avail_size; + uint32_t _avail_size{0}; }; inline bool diff --git a/include/proxy/http/ConnectingEntry.h b/include/proxy/http/ConnectingEntry.h index 3b14fbb59b4..2b92d4a600f 100644 --- a/include/proxy/http/ConnectingEntry.h +++ b/include/proxy/http/ConnectingEntry.h @@ -54,7 +54,6 @@ class ConnectingEntry : public Continuation MIOBuffer *_netvc_read_buffer = nullptr; IOBufferReader *_netvc_reader = nullptr; Action *_pending_action = nullptr; - NetVCOptions opt; }; struct IpHelper { diff --git a/include/proxy/http/Http1ClientTransaction.h b/include/proxy/http/Http1ClientTransaction.h index 3fd9db4410e..daa210ddb06 100644 --- a/include/proxy/http/Http1ClientTransaction.h +++ b/include/proxy/http/Http1ClientTransaction.h @@ -41,10 +41,4 @@ class Http1ClientTransaction : public Http1Transaction void transaction_done() override; void increment_transactions_stat() override; void decrement_transactions_stat() override; - - //////////////////// - // Variables - -protected: - bool outbound_transparent{false}; }; diff --git a/include/proxy/http/Http1ServerTransaction.h b/include/proxy/http/Http1ServerTransaction.h index 7d3226d1243..5ad321cdae2 100644 --- a/include/proxy/http/Http1ServerTransaction.h +++ b/include/proxy/http/Http1ServerTransaction.h @@ -44,10 +44,4 @@ class Http1ServerTransaction : public Http1Transaction void transaction_done() override; void force_close(); - - //////////////////// - // Variables - -protected: - bool outbound_transparent{false}; }; diff --git a/include/proxy/http/HttpCacheSM.h b/include/proxy/http/HttpCacheSM.h index 41623103d65..b1379e0555e 100644 --- a/include/proxy/http/HttpCacheSM.h +++ b/include/proxy/http/HttpCacheSM.h @@ -129,6 +129,12 @@ class HttpCacheSM : public Continuation return cache_read_vc ? (cache_read_vc->is_compressed_in_ram()) : false; } + const HttpCacheKey & + get_cache_key() const + { + return cache_key; + } + void set_open_read_tries(int value) { diff --git a/include/proxy/http/HttpConfig.h b/include/proxy/http/HttpConfig.h index a1d613465ed..c5989c3620a 100644 --- a/include/proxy/http/HttpConfig.h +++ b/include/proxy/http/HttpConfig.h @@ -773,6 +773,7 @@ struct OverridableHttpConfigParams { char *ssl_client_cert_filename = nullptr; char *ssl_client_private_key_filename = nullptr; char *ssl_client_ca_cert_filename = nullptr; + char *ssl_client_ca_cert_path = nullptr; char *ssl_client_alpn_protocols = nullptr; // Host Resolution order @@ -888,6 +889,7 @@ struct HttpConfigParams : public ConfigInfo { MgmtInt http_request_line_max_size = 65535; MgmtInt http_hdr_field_max_size = 131070; + MgmtInt pp_hdr_max_size = 109; MgmtByte http_host_sni_policy = 0; MgmtByte scheme_proto_mismatch_policy = 2; @@ -1030,6 +1032,7 @@ inline HttpConfigParams::~HttpConfigParams() ats_free(oride.ssl_client_cert_filename); ats_free(oride.ssl_client_private_key_filename); ats_free(oride.ssl_client_ca_cert_filename); + ats_free(oride.ssl_client_ca_cert_path); ats_free(connect_ports_string); ats_free(reverse_proxy_no_host_redirect); ats_free(redirect_actions_string); diff --git a/include/proxy/http/HttpSM.h b/include/proxy/http/HttpSM.h index 3bbbd802a14..fc3e1252452 100644 --- a/include/proxy/http/HttpSM.h +++ b/include/proxy/http/HttpSM.h @@ -483,10 +483,9 @@ class HttpSM : public Continuation, public PluginUserArgs */ void setup_client_request_plugin_agents(HttpTunnelProducer *p, int num_header_bytes = 0); - HttpTransact::StateMachineAction_t last_action = HttpTransact::StateMachineAction_t::UNDEFINED; - int (HttpSM::*m_last_state)(int event, void *data) = nullptr; - virtual void set_next_state(); - void call_transact_and_set_next_state(TransactEntryFunc_t f); + HttpTransact::StateMachineAction_t last_action = HttpTransact::StateMachineAction_t::UNDEFINED; + virtual void set_next_state(); + void call_transact_and_set_next_state(TransactEntryFunc_t f); bool is_http_server_eos_truncation(HttpTunnelProducer *); bool is_bg_fill_necessary(HttpTunnelConsumer *c); @@ -564,11 +563,7 @@ class HttpSM : public Continuation, public PluginUserArgs APIHook const *cur_hook = nullptr; HttpHookState hook_state; - // Continuation time keeper - int64_t prev_hook_start_time = 0; - int reentrancy_count = 0; - int cur_hooks = 0; HttpApiState_t callout_state = HttpApiState_t::NO_CALLOUT; // api_hooks must not be changed directly diff --git a/include/proxy/http/HttpTransact.h b/include/proxy/http/HttpTransact.h index be5f45ab346..a6d48a12f7f 100644 --- a/include/proxy/http/HttpTransact.h +++ b/include/proxy/http/HttpTransact.h @@ -657,10 +657,8 @@ class HttpTransact }; using ResponseAction = struct _ResponseAction { - bool handled = false; - TSResponseAction action; - - _ResponseAction() {} + bool handled{false}; + TSResponseAction action{}; }; struct State { diff --git a/include/proxy/http/HttpTunnel.h b/include/proxy/http/HttpTunnel.h index 961c3f6cdfa..f245a8452f8 100644 --- a/include/proxy/http/HttpTunnel.h +++ b/include/proxy/http/HttpTunnel.h @@ -132,7 +132,7 @@ struct ChunkedHandler { //@{ /// The maximum chunk size. /// This is the preferred size as well, used whenever possible. - int64_t max_chunk_size; + int64_t max_chunk_size{DEFAULT_MAX_CHUNK_SIZE}; /// Caching members to avoid using printf on every chunk. /// It holds the header for a maximal sized chunk which will cover /// almost all output chunks. diff --git a/include/proxy/http/HttpVCTable.h b/include/proxy/http/HttpVCTable.h index e1157e4075d..2521f3cf439 100644 --- a/include/proxy/http/HttpVCTable.h +++ b/include/proxy/http/HttpVCTable.h @@ -41,17 +41,17 @@ enum class HttpVC_t { }; struct HttpVCTableEntry { - VConnection *vc; - MIOBuffer *read_buffer; - MIOBuffer *write_buffer; - VIO *read_vio; - VIO *write_vio; - HttpSMHandler vc_read_handler; - HttpSMHandler vc_write_handler; - HttpVC_t vc_type; - HttpSM *sm; - bool eos; - bool in_tunnel; + VConnection *vc{nullptr}; + MIOBuffer *read_buffer{nullptr}; + MIOBuffer *write_buffer{nullptr}; + VIO *read_vio{nullptr}; + VIO *write_vio{nullptr}; + HttpSMHandler vc_read_handler{}; + HttpSMHandler vc_write_handler{}; + HttpVC_t vc_type{HttpVC_t::UNKNOWN}; + HttpSM *sm{nullptr}; + bool eos{false}; + bool in_tunnel{false}; }; struct HttpVCTable { diff --git a/include/proxy/http/OverridableConfigDefs.h b/include/proxy/http/OverridableConfigDefs.h index d70c4c54caa..a21e57d70aa 100644 --- a/include/proxy/http/OverridableConfigDefs.h +++ b/include/proxy/http/OverridableConfigDefs.h @@ -250,6 +250,7 @@ X(HTTP_CONNECT_ATTEMPTS_RETRY_BACKOFF_BASE, connect_attempts_retry_backoff_base, "proxy.config.http.connect_attempts_retry_backoff_base", INT, GENERIC) \ X(HTTP_NEGATIVE_REVALIDATING_LIST, negative_revalidating_list, "proxy.config.http.negative_revalidating_list", STRING, HttpStatusCodeList_Conv) \ X(HTTP_CACHE_POST_METHOD, cache_post_method, "proxy.config.http.cache.post_method", INT, GENERIC) \ - X(HTTP_CACHE_TARGETED_CACHE_CONTROL_HEADERS, targeted_cache_control_headers, "proxy.config.http.cache.targeted_cache_control_headers", STRING, TargetedCacheControlHeaders_Conv) + X(HTTP_CACHE_TARGETED_CACHE_CONTROL_HEADERS, targeted_cache_control_headers, "proxy.config.http.cache.targeted_cache_control_headers", STRING, TargetedCacheControlHeaders_Conv) \ + X(SSL_CLIENT_CA_CERT_PATH, ssl_client_ca_cert_path, "proxy.config.ssl.client.CA.cert.path", STRING, NONE) // clang-format on diff --git a/include/proxy/logging/Log.h b/include/proxy/logging/Log.h index 0064c9dd8e0..debde4d2dee 100644 --- a/include/proxy/logging/Log.h +++ b/include/proxy/logging/Log.h @@ -43,7 +43,7 @@ @section example Example usage of the API @code - // Populate a TransactionLogData, then: + // Populate a TransactionLogData source, then: LogAccess entry(data); int ret = Log::access(&entry); @endcode diff --git a/include/proxy/logging/LogAccess.h b/include/proxy/logging/LogAccess.h index facfd5f836c..dccb620ba37 100644 --- a/include/proxy/logging/LogAccess.h +++ b/include/proxy/logging/LogAccess.h @@ -123,8 +123,8 @@ class LogAccess * The caller retains ownership of @a data, which must outlive the * synchronous Log::access() call that marshals this entry. * - * @param[in] data Populated TransactionLogData for a completed or - * pre-transaction entry. + * @param[in] data Populated TransactionLogData for an HttpSM-backed or + * non-HttpSM entry. */ explicit LogAccess(TransactionLogData &data); @@ -255,6 +255,7 @@ class LogAccess // int marshal_cache_write_code(char *); // INT int marshal_cache_write_transform_code(char *); // INT + int marshal_cache_key_hash(char *); // STR // other fields // @@ -284,6 +285,7 @@ class LogAccess int marshal_proxy_protocol_authority(char *); // STR int marshal_proxy_protocol_tls_cipher(char *); // STR int marshal_proxy_protocol_tls_version(char *); // STR + int marshal_proxy_protocol_tls_group(char *); // STR // named fields from within a http header // @@ -316,6 +318,7 @@ class LogAccess int marshal_milestone_fmt_time(TSMilestonesType ms, char *buf); int marshal_milestone_fmt_ms(TSMilestonesType ms, char *buf); int marshal_milestone_diff(TSMilestonesType ms1, TSMilestonesType ms2, char *buf); + int marshal_milestones_csv(char *buf); bool has_http_header_field(LogField::Container container, const char *field) const; void set_http_header_field(LogField::Container container, char *field, char *buf, int len); @@ -330,6 +333,7 @@ class LogAccess static int unmarshal_itox(int64_t val, char *dest, int field_width = 0, char leading_char = ' '); static int unmarshal_int_to_str(char **buf, char *dest, int len); static int unmarshal_int_to_str_hex(char **buf, char *dest, int len); + static int unmarshal_milestone_diff(char **buf, char *dest, int len); static int unmarshal_str(char **buf, char *dest, int len, LogSlice *slice, LogEscapeType escape_type); static int unmarshal_ttmsf(char **buf, char *dest, int len); static int unmarshal_int_to_date_str(char **buf, char *dest, int len); diff --git a/include/proxy/logging/LogBuffer.h b/include/proxy/logging/LogBuffer.h index 5e85057db08..4f8295ae444 100644 --- a/include/proxy/logging/LogBuffer.h +++ b/include/proxy/logging/LogBuffer.h @@ -284,7 +284,7 @@ class LogBufferList class LogBufferIterator { public: - LogBufferIterator(LogBufferHeader *header, bool in_network_order = false); + LogBufferIterator(LogBufferHeader *header); ~LogBufferIterator(); LogEntryHeader *next(); @@ -295,7 +295,6 @@ class LogBufferIterator LogBufferIterator &operator=(const LogBufferIterator &) = delete; private: - bool m_in_network_order; char *m_next; unsigned m_iter_entry_count; unsigned m_buffer_entry_count; @@ -311,8 +310,8 @@ class LogBufferIterator within a given LogBuffer. -------------------------------------------------------------------------*/ -inline LogBufferIterator::LogBufferIterator(LogBufferHeader *header, bool in_network_order) - : m_in_network_order(in_network_order), m_next(nullptr), m_iter_entry_count(0), m_buffer_entry_count(0) +inline LogBufferIterator::LogBufferIterator(LogBufferHeader *header) + : m_next(nullptr), m_iter_entry_count(0), m_buffer_entry_count(0) { ink_assert(header); diff --git a/include/proxy/logging/TransactionLogData.h b/include/proxy/logging/TransactionLogData.h index b2942024ca3..d617458bd6d 100644 --- a/include/proxy/logging/TransactionLogData.h +++ b/include/proxy/logging/TransactionLogData.h @@ -31,20 +31,20 @@ #include class HttpSM; -class PreTransactionLogData; +class NonHttpSmLogData; -/** Provide access-log data from either a completed HttpSM or pre-transaction storage. +/** Provide access-log data from either a completed HttpSM or non-HttpSM storage. * * The common completed-transaction path reads directly from @c HttpSM. The - * rare pre-transaction path reads from @c PreTransactionLogData, which owns - * copied request/session state for malformed requests rejected before HttpSM - * creation. + * rare non-HttpSM path reads from @c NonHttpSmLogData, which owns copied + * request/session state for exceptional access-log entries that cannot be + * backed by an @c HttpSM. */ class TransactionLogData { public: explicit TransactionLogData(HttpSM *sm); - explicit TransactionLogData(PreTransactionLogData const &pre_data); + explicit TransactionLogData(NonHttpSmLogData const &non_http_sm_data); void *http_sm_for_plugins() const; @@ -75,8 +75,9 @@ class TransactionLogData int get_unmapped_url_len() const; // ===== Cache lookup URL ===== - char *get_cache_lookup_url_str() const; - int get_cache_lookup_url_len() const; + char *get_cache_lookup_url_str() const; + int get_cache_lookup_url_len() const; + const ts::CryptoHash *get_cache_lookup_hash() const; // ===== Client addressing ===== sockaddr const *get_client_addr() const; @@ -186,7 +187,7 @@ class TransactionLogData // ===== Server response Transfer-Encoding ===== std::string_view get_server_response_transfer_encoding() const; - // ===== Fallback fields for pre-transaction logging ===== + // ===== Fallback fields for non-HttpSM logging ===== std::string_view get_method() const; std::string_view get_scheme() const; std::string_view get_client_protocol_str() const; @@ -195,8 +196,8 @@ class TransactionLogData TransactionLogData &operator=(const TransactionLogData &) = delete; private: - HttpSM *m_http_sm = nullptr; - PreTransactionLogData const *m_pre_data = nullptr; + HttpSM *m_http_sm = nullptr; + NonHttpSmLogData const *m_non_http_sm_data = nullptr; // Cached values for fields that require computation or string formatting. mutable char m_client_rx_error_code[10] = {'-', '\0'}; diff --git a/include/records/RecCore.h b/include/records/RecCore.h index d0a6ed5b4fb..a355bd39ccb 100644 --- a/include/records/RecCore.h +++ b/include/records/RecCore.h @@ -25,6 +25,8 @@ #include #include +#include +#include #include "tscore/Diags.h" @@ -69,9 +71,34 @@ std::string RecConfigReadConfigPath(const char *file_variable, const char *defau // Return a copy of the persistent stats file. This is $RUNTIMEDIR/records.snap. std::string RecConfigReadPersistentStatsPath(); -// Test whether the named configuration value is overridden by an environment variable. Return either -// the overridden value, or the original value. Caller MUST NOT free the result. -const char *RecConfigOverrideFromEnvironment(const char *name, const char *value); +/// Indicates why RecConfigOverrideFromEnvironment() chose its returned value. +enum class RecConfigOverrideSource { + NONE, ///< No override — the original value was kept. + ENV, ///< Overridden by a PROXY_CONFIG_* environment variable. + RUNROOT, ///< Overridden with the resolved Layout path because runroot manages this record. +}; + +/// Label for the override source (for logging). +constexpr const char * +RecConfigOverrideSourceName(RecConfigOverrideSource src) +{ + switch (src) { + case RecConfigOverrideSource::ENV: + return "environment variable"; + case RecConfigOverrideSource::RUNROOT: + return "runroot"; + case RecConfigOverrideSource::NONE: + return "none"; + default: + return "unknown"; + } +} + +// Test whether the named configuration value is overridden by the execution +// environment — either a PROXY_CONFIG_* environment variable or the runroot +// mechanism. Returns the resolved value together with the source that +// produced it. +std::pair RecConfigOverrideFromEnvironment(const char *name, const char *value); //------------------------------------------------------------------------- // Stat Registration diff --git a/include/ts/apidefs.h.in b/include/ts/apidefs.h.in index cc491e47060..8f1e8929b20 100644 --- a/include/ts/apidefs.h.in +++ b/include/ts/apidefs.h.in @@ -913,6 +913,7 @@ enum TSOverridableConfigKey { TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_LIST, TS_CONFIG_HTTP_CACHE_POST_METHOD, TS_CONFIG_HTTP_CACHE_TARGETED_CACHE_CONTROL_HEADERS, + TS_CONFIG_SSL_CLIENT_CA_CERT_PATH, TS_CONFIG_LAST_ENTRY, }; diff --git a/include/ts/ts.h b/include/ts/ts.h index cb698c7e00a..49ab23eb9df 100644 --- a/include/ts/ts.h +++ b/include/ts/ts.h @@ -2768,6 +2768,25 @@ TSReturnCode TSHttpTxnCachedRespModifiableGet(TSHttpTxn txnp, TSMBuffer *bufp, T TSReturnCode TSHttpTxnCacheLookupStatusSet(TSHttpTxn txnp, int cachelookup); TSReturnCode TSHttpTxnCacheLookupUrlGet(TSHttpTxn txnp, TSMBuffer bufp, TSMLoc obj); TSReturnCode TSHttpTxnCacheLookupUrlSet(TSHttpTxn txnp, TSMBuffer bufp, TSMLoc obj); + +/** + Gets the effective cache key digest (cryptographic hash) that was + used for cache lookup or storage on this transaction. The digest + is returned as raw bytes — 16 bytes for MD5 (default) or 32 bytes + for SHA-256 (FIPS mode). A buffer of at least 32 bytes is + recommended to accommodate either configuration. + + @param[in] txnp the transaction. + @param[out] buffer caller-provided buffer to receive the raw hash + bytes. If @c nullptr, only @a length is set (size query). + @param[in,out] length capacity of @a buffer in bytes on input; actual + digest size in bytes on output. + + @return @c TS_SUCCESS if a cache key was computed for this + transaction, @c TS_ERROR if no cache lookup was performed or if + @a buffer is non-null and too small. + */ +TSReturnCode TSHttpTxnCacheKeyDigestGet(TSHttpTxn txnp, char *buffer, int *length); TSReturnCode TSHttpTxnPrivateSessionSet(TSHttpTxn txnp, int private_session); const char *TSHttpTxnCacheDiskPathGet(TSHttpTxn txnp, int *length); int TSHttpTxnBackgroundFillStarted(TSHttpTxn txnp); diff --git a/include/tscore/ArgParser.h b/include/tscore/ArgParser.h index a4bd5916feb..fdb6086eba0 100644 --- a/include/tscore/ArgParser.h +++ b/include/tscore/ArgParser.h @@ -226,6 +226,8 @@ class ArgParser void validate_mutex_groups(Arguments &ret) const; // Helper method to validate option dependencies void validate_dependencies(Arguments &ret) const; + // Helper method to apply default values for options not explicitly set + void apply_option_defaults(Arguments &ret) const; // The command name and help message std::string _name; std::string _description; diff --git a/include/tscore/BaseLogFile.h b/include/tscore/BaseLogFile.h index cde3581c41e..d7d5e027dd8 100644 --- a/include/tscore/BaseLogFile.h +++ b/include/tscore/BaseLogFile.h @@ -91,18 +91,18 @@ class BaseMetaInfo }; private: - char *_filename; // the name of the meta file - time_t _creation_time; // file creation time - uint64_t _log_object_signature; // log object signature - int _flags; // metainfo status flags - char _buffer[BUF_SIZE]; // read/write buffer + char *_filename{nullptr}; // the name of the meta file + time_t _creation_time{0}; // file creation time + uint64_t _log_object_signature{0}; // log object signature + int _flags{0}; // metainfo status flags + char _buffer[BUF_SIZE]; // read/write buffer void _read_from_file(); void _write_to_file(); void _build_name(const char *filename); public: - BaseMetaInfo(const char *filename) : _flags(0) + BaseMetaInfo(const char *filename) { _build_name(filename); _read_from_file(); diff --git a/include/tscore/HashFNV.h b/include/tscore/HashFNV.h index 0ea09bbe08a..57a7c4b0bb1 100644 --- a/include/tscore/HashFNV.h +++ b/include/tscore/HashFNV.h @@ -45,7 +45,8 @@ struct ATSHash32FNV1a : ATSHash32 { void clear() override; private: - uint32_t hval; + static constexpr uint32_t fnv_init = 0x811c9dc5u; + uint32_t hval{fnv_init}; }; template @@ -76,7 +77,8 @@ struct ATSHash64FNV1a : ATSHash64 { void clear() override; private: - uint64_t hval; + static constexpr uint64_t fnv_init = 0xcbf29ce484222325ull; + uint64_t hval{fnv_init}; }; template diff --git a/include/tscore/ink_aiocb.h b/include/tscore/ink_aiocb.h index ac587b0598b..3b5740a9230 100644 --- a/include/tscore/ink_aiocb.h +++ b/include/tscore/ink_aiocb.h @@ -49,6 +49,4 @@ struct ink_aiocb { off_t aio_offset; /* file offset */ int aio_lio_opcode; /* listio operation */ - int aio_state; /* state flag for List I/O */ - int aio__pad[1]; /* extension padding */ }; diff --git a/include/tscore/ink_cap.h b/include/tscore/ink_cap.h index 86a8f31f4d5..a18c3449ac1 100644 --- a/include/tscore/ink_cap.h +++ b/include/tscore/ink_cap.h @@ -81,8 +81,8 @@ class ElevateAccess FILE_PRIVILEGE = 0x1u, ///< Access filesystem objects with privilege TRACE_PRIVILEGE = 0x2u, ///< Trace other processes with privilege LOW_PORT_PRIVILEGE = 0x4u, ///< Bind to privilege ports. - OWNER_PRIVILEGE = 0x8u ///< Bypass permission checks on operations that normally require - /// filesystem UID & process UID to match + OWNER_PRIVILEGE = 0x8u, ///< Owner-only operations on unowned files (CAP_FOWNER) + CHOWN_PRIVILEGE = 0x10u ///< Change file ownership }; ElevateAccess(unsigned level = FILE_PRIVILEGE); diff --git a/include/tscore/ink_config.h.cmake.in b/include/tscore/ink_config.h.cmake.in index bf012cefecd..73c8b860fb9 100644 --- a/include/tscore/ink_config.h.cmake.in +++ b/include/tscore/ink_config.h.cmake.in @@ -186,6 +186,8 @@ const int DEFAULT_STACKSIZE = @DEFAULT_STACK_SIZE@; #cmakedefine01 HAVE_SSL_ERROR_DESCRIPTION #cmakedefine01 HAVE_OSSL_PARAM_CONSTRUCT_END #cmakedefine01 TS_USE_TLS_SET_CIPHERSUITES +#cmakedefine01 HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG +#cmakedefine01 HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE #define TS_BUILD_CANONICAL_HOST "@CMAKE_HOST@" diff --git a/include/tscpp/api/AsyncTimer.h b/include/tscpp/api/AsyncTimer.h index 9b511c28f46..0dc100cd6fb 100644 --- a/include/tscpp/api/AsyncTimer.h +++ b/include/tscpp/api/AsyncTimer.h @@ -86,7 +86,7 @@ class AsyncTimer : public AsyncProvider void cancel() override; private: - AsyncTimerState *state_; + AsyncTimerState *state_{nullptr}; }; } // namespace atscppapi diff --git a/include/tscpp/api/InterceptPlugin.h b/include/tscpp/api/InterceptPlugin.h index c1d69b9b123..37f5415362b 100644 --- a/include/tscpp/api/InterceptPlugin.h +++ b/include/tscpp/api/InterceptPlugin.h @@ -96,7 +96,7 @@ class InterceptPlugin : public TransactionPlugin bool setOutputComplete(); private: - State *state_; + State *state_{nullptr}; bool doRead(); void handleEvent(int, void *); diff --git a/plugins/compress/sample.compress.config b/plugins/compress/sample.compress.config index 451f8958349..8cfaf2d9c87 100644 --- a/plugins/compress/sample.compress.config +++ b/plugins/compress/sample.compress.config @@ -25,7 +25,8 @@ # - for when the proxy parses responses, and the resulting compression/decompression # is wasteful # -# cache: when set, the plugin stores the uncompressed and compressed response as alternates +# cache: when true, caches the compressed response; when false, caches only the uncompressed +# response and compresses on-the-fly from cache. Vary: Accept-Encoding handles alternates. # # compressible-content-type: wildcard pattern for matching compressible content types # diff --git a/plugins/esi/lib/EsiGunzip.cc b/plugins/esi/lib/EsiGunzip.cc index 47afa954af4..19bfc6c2057 100644 --- a/plugins/esi/lib/EsiGunzip.cc +++ b/plugins/esi/lib/EsiGunzip.cc @@ -30,15 +30,6 @@ using std::string; using namespace EsiLib; -EsiGunzip::EsiGunzip() : _downstream_length(0), _total_data_length(0) -{ - _init = false; - _success = true; - // zlib _zstrm variables are initialized when they are required in stream_decode - // coverity[uninit_member] - // coverity[uninit_ctor] -} - bool EsiGunzip::stream_finish() { @@ -81,29 +72,32 @@ EsiGunzip::stream_decode(const char *data, int data_len, std::string &udata) int32_t curr_buf_size; do { - _zstrm.next_out = reinterpret_cast(raw_buf); - _zstrm.avail_out = BUF_SIZE; - inflate_result = inflate(&_zstrm, Z_SYNC_FLUSH); - curr_buf_size = -1; - if ((inflate_result == Z_OK) || (inflate_result == Z_BUF_ERROR)) { - curr_buf_size = BUF_SIZE - _zstrm.avail_out; - } else if (inflate_result == Z_STREAM_END) { + _zstrm.next_out = reinterpret_cast(raw_buf); + _zstrm.avail_out = BUF_SIZE; + auto const avail_in_before = _zstrm.avail_in; + inflate_result = inflate(&_zstrm, Z_SYNC_FLUSH); + if ((inflate_result == Z_OK) || (inflate_result == Z_BUF_ERROR) || (inflate_result == Z_STREAM_END)) { curr_buf_size = BUF_SIZE - _zstrm.avail_out; + } else { + TSError("[%s] Failure while inflating; error code %d", __FUNCTION__, inflate_result); + _success = false; + return false; } if (curr_buf_size > BUF_SIZE) { TSError("[%s] buf too large", __FUNCTION__); break; } - if (curr_buf_size < 1) { - TSError("[%s] buf below zero", __FUNCTION__); + if (curr_buf_size < 1 && avail_in_before == _zstrm.avail_in) { break; } // push empty object onto list and add data to in-list object to // avoid data copy for temporary - buf_list.push_back(string()); - string &curr_buf = buf_list.back(); - curr_buf.assign(raw_buf, curr_buf_size); + if (curr_buf_size > 0) { + buf_list.push_back(string()); + string &curr_buf = buf_list.back(); + curr_buf.assign(raw_buf, curr_buf_size); + } if (inflate_result == Z_STREAM_END) { break; diff --git a/plugins/esi/lib/EsiGunzip.h b/plugins/esi/lib/EsiGunzip.h index b8467403f94..96245484090 100644 --- a/plugins/esi/lib/EsiGunzip.h +++ b/plugins/esi/lib/EsiGunzip.h @@ -30,7 +30,7 @@ class EsiGunzip { public: - EsiGunzip(); + EsiGunzip() = default; ~EsiGunzip(); @@ -45,10 +45,10 @@ class EsiGunzip bool stream_finish(); private: - int64_t _downstream_length; - int64_t _total_data_length; - z_stream _zstrm; + int64_t _downstream_length{0}; + int64_t _total_data_length{0}; + z_stream _zstrm{}; - bool _init; - bool _success; + bool _init{false}; + bool _success{true}; }; diff --git a/plugins/esi/lib/EsiGzip.cc b/plugins/esi/lib/EsiGzip.cc index 789b79d84e0..aeec64c99fe 100644 --- a/plugins/esi/lib/EsiGzip.cc +++ b/plugins/esi/lib/EsiGzip.cc @@ -30,13 +30,6 @@ using std::string; using namespace EsiLib; -EsiGzip::EsiGzip() : _downstream_length(0), _total_data_length(0), _crc(0) -{ - // Zlib _zstrm variables are initialized when they are required in runDeflateLoop - // coverity[uninit_member] - // coverity[uninit_ctor] -} - template inline void append(string &out, T data) diff --git a/plugins/esi/lib/EsiGzip.h b/plugins/esi/lib/EsiGzip.h index fc8204ae105..e15cdd24fe3 100644 --- a/plugins/esi/lib/EsiGzip.h +++ b/plugins/esi/lib/EsiGzip.h @@ -31,7 +31,7 @@ class EsiGzip { public: - EsiGzip(); + EsiGzip() = default; ~EsiGzip(); @@ -65,12 +65,12 @@ class EsiGzip private: /** The cumulative total number of bytes for the compressed stream. */ - int64_t _downstream_length; + int64_t _downstream_length{0}; /** The cumulative total number of uncompressed bytes that have been * compressed. */ - int64_t _total_data_length; - z_stream _zstrm; - uLong _crc; + int64_t _total_data_length{0}; + z_stream _zstrm{}; + uLong _crc{0}; }; diff --git a/plugins/esi/test/gzip_test.cc b/plugins/esi/test/gzip_test.cc index a6bd50fa593..5d19366a7ca 100644 --- a/plugins/esi/test/gzip_test.cc +++ b/plugins/esi/test/gzip_test.cc @@ -26,12 +26,16 @@ #include +#include "EsiGunzip.h" #include "Utils.h" #include "gzip.h" using std::string; using namespace EsiLib; +extern void enableFakeErrorLog(); +extern string gFakeErrorLog; + TEST_CASE("test esi plugin - gzip") { SECTION("===================== Test 1") @@ -102,4 +106,26 @@ TEST_CASE("test esi plugin - gzip") // check output of gunzip CHECK(gunzip(expected_cdata, 32, buf_list) == false); } + + SECTION("streaming gunzip does not log an error for chunks with no output") + { + const char expected_data[] = "Hello World!"; + + string cdata; + REQUIRE(gzip(expected_data, 12, cdata)); + + EsiGunzip gunzip; + string data; + + enableFakeErrorLog(); + REQUIRE(gunzip.stream_decode(cdata.data(), GZIP_HEADER_SIZE, data)); + CHECK(data.empty()); + CHECK(gFakeErrorLog.empty()); + + REQUIRE(gunzip.stream_decode(cdata.data() + GZIP_HEADER_SIZE, cdata.size() - GZIP_HEADER_SIZE, data)); + REQUIRE(gunzip.stream_finish()); + + CHECK(data == expected_data); + CHECK(gFakeErrorLog.empty()); + } } diff --git a/plugins/esi/test/print_funcs.cc b/plugins/esi/test/print_funcs.cc index 2596a462054..d73b973d654 100644 --- a/plugins/esi/test/print_funcs.cc +++ b/plugins/esi/test/print_funcs.cc @@ -34,9 +34,11 @@ static const int LINE_SIZE = 1024 * 1024; namespace { bool fakeDebugLogEnabled; -} +bool fakeErrorLogEnabled; +} // namespace std::string gFakeDebugLog; +std::string gFakeErrorLog; void enableFakeDebugLog() @@ -45,6 +47,13 @@ enableFakeDebugLog() gFakeDebugLog.assign(""); } +void +enableFakeErrorLog() +{ + fakeErrorLogEnabled = true; + gFakeErrorLog.assign(""); +} + void DbgCtl::print(const char *tag, const char * /* file */, const char * /* function */, int /* line */, const char *fmt, ...) { @@ -101,4 +110,7 @@ TSError(const char *fmt, ...) vsnprintf(buf, LINE_SIZE, fmt, ap); printf("Error: %s\n", buf); va_end(ap); + if (fakeErrorLogEnabled) { + gFakeErrorLog.append(buf); + } } diff --git a/plugins/experimental/cache_fill/configs.cc b/plugins/experimental/cache_fill/configs.cc index 26b2e1967ce..c9d1b7b3ff9 100644 --- a/plugins/experimental/cache_fill/configs.cc +++ b/plugins/experimental/cache_fill/configs.cc @@ -157,11 +157,11 @@ BgFetchConfig::readConfig(const char *config_file) swoc::bwprint(ts::bw_dbg, "adding background_fetch address range rule {} for {}: {}", exclude, cfg_name, cfg_value); Dbg(dbg_ctl, "%s", ts::bw_dbg.c_str()); } else if ("Content-Length"_tv == cfg_name) { - BgFetchRule::size_cmp_type::OP op; + BgFetchRule::size_cmp_type::OP op{BgFetchRule::size_cmp_type::LESS_THAN_OR_EQUAL}; if (cfg_value[0] == '<') { op = BgFetchRule::size_cmp_type::LESS_THAN_OR_EQUAL; } else if (cfg_value[0] == '>') { - op = BgFetchRule::size_cmp_type::LESS_THAN_OR_EQUAL; + op = BgFetchRule::size_cmp_type::GREATER_THAN_OR_EQUAL; } else { TSError("[%s] invalid Content-Length condition %.*s, skipping config value", PLUGIN_NAME, int(cfg_value.size()), cfg_value.data()); diff --git a/plugins/experimental/http_stats/http_stats.cc b/plugins/experimental/http_stats/http_stats.cc index 2e67570470d..1a79c01fcb7 100644 --- a/plugins/experimental/http_stats/http_stats.cc +++ b/plugins/experimental/http_stats/http_stats.cc @@ -81,7 +81,12 @@ struct HTTPStatsFormatter { struct HTTPStatsConfig { explicit HTTPStatsConfig() {} - ~HTTPStatsConfig() { TSContDestroy(cont); } + ~HTTPStatsConfig() + { + if (cont) { + TSContDestroy(cont); + } + } std::string mimeType; int maxAge = 0; @@ -89,7 +94,7 @@ struct HTTPStatsConfig { bool integer_counters = false; bool wrap_counters = false; - TSCont cont; + TSCont cont{nullptr}; }; struct HTTPStatsRequest; diff --git a/plugins/experimental/ja4_fingerprint/ja4.h b/plugins/experimental/ja4_fingerprint/ja4.h index 31b151dc0c0..d277f271059 100644 --- a/plugins/experimental/ja4_fingerprint/ja4.h +++ b/plugins/experimental/ja4_fingerprint/ja4.h @@ -53,8 +53,8 @@ class TLSClientHelloSummary public: using difference_type = std::iterator_traits::iterator>::difference_type; - Protocol protocol; - std::uint16_t TLS_version{0}; // 0 is not the default, this is only to not have it un-initialized. + Protocol protocol{Protocol::TLS}; // always overwritten by caller + std::uint16_t TLS_version{0}; // 0 is not the default, this is only to not have it un-initialized. std::string ALPN; std::vector const &get_ciphers() const; diff --git a/plugins/experimental/stale_response/stale_response.h b/plugins/experimental/stale_response/stale_response.h index 6c0454a8aff..a0a9071ea58 100644 --- a/plugins/experimental/stale_response/stale_response.h +++ b/plugins/experimental/stale_response/stale_response.h @@ -23,6 +23,8 @@ #pragma once +#include + #include "ts/apidefs.h" #include "ts_wrap.h" #include "ts/ts.h" @@ -63,6 +65,9 @@ struct ConfigInfo { if (this->body_data_mutex) { TSMutexDestroy(this->body_data_mutex); } + if (this->log_info.filename != PLUGIN_TAG) { + free(const_cast(this->log_info.filename)); + } } UintBodyMap *body_data = nullptr; TSMutex body_data_mutex; diff --git a/plugins/experimental/txn_box/plugin/src/Context.cc b/plugins/experimental/txn_box/plugin/src/Context.cc index e634b2cf5be..0700f0bc533 100644 --- a/plugins/experimental/txn_box/plugin/src/Context.cc +++ b/plugins/experimental/txn_box/plugin/src/Context.cc @@ -215,7 +215,7 @@ Context::extract(Expr const &expr) FeatureView Context::extract_view(const Expr &expr, std::initializer_list opts) { - FeatureView zret; + FeatureView zret{}; bool commit_p = false; bool cstr_p = false; diff --git a/plugins/experimental/txn_box/plugin/src/Ex_Base.cc b/plugins/experimental/txn_box/plugin/src/Ex_Base.cc index bf3ac5166fa..b9bb01d76da 100644 --- a/plugins/experimental/txn_box/plugin/src/Ex_Base.cc +++ b/plugins/experimental/txn_box/plugin/src/Ex_Base.cc @@ -299,7 +299,7 @@ Ex_txn_conf::validate(Config &cfg, Spec &spec, const TextView &arg) Feature Ex_txn_conf::extract(Context &ctx, const Extractor::Spec &spec) { - Feature zret{}; + Feature zret{NIL_FEATURE}; auto var = spec._data.span.rebind()[0]; auto &&[value, errata] = ctx._txn.override_fetch(*var); if (errata.is_ok()) { diff --git a/plugins/experimental/txn_box/plugin/src/Ex_HTTP.cc b/plugins/experimental/txn_box/plugin/src/Ex_HTTP.cc index b4fec29918f..367553e9bd8 100644 --- a/plugins/experimental/txn_box/plugin/src/Ex_HTTP.cc +++ b/plugins/experimental/txn_box/plugin/src/Ex_HTTP.cc @@ -354,7 +354,7 @@ class Ex_proxy_req_scheme : public StringExtractor Feature Ex_proxy_req_scheme::extract(Context &ctx, Spec const &) { - FeatureView zret; + FeatureView zret{}; zret._direct_p = true; if (auto hdr{ctx.proxy_req_hdr()}; hdr.is_valid()) { if (ts::URL url{hdr.url()}; url.is_valid()) { @@ -557,7 +557,7 @@ Ex_ua_req_port::validate(Config &, Spec &, const swoc::TextView &) Feature Ex_ua_req_port::extract(Context &ctx, Spec const &) { - Feature zret{}; + Feature zret{NIL_FEATURE}; if (auto hdr{ctx.ua_req_hdr()}; hdr.is_valid()) { if (ts::URL url{hdr.url()}; url.is_valid()) { zret = static_cast>(url.port()); @@ -585,7 +585,7 @@ Ex_proxy_req_port::validate(Config &, Spec &, const swoc::TextView &) Feature Ex_proxy_req_port::extract(Context &ctx, Spec const &) { - Feature zret{}; + Feature zret{NIL_FEATURE}; if (auto hdr{ctx.proxy_req_hdr()}; hdr.is_valid()) { if (ts::URL url{hdr.url()}; url.is_valid()) { zret = static_cast>(url.port()); @@ -964,7 +964,7 @@ Ex_ua_req_url_port::validate(Config &, Spec &, const swoc::TextView &) Feature Ex_ua_req_url_port::extract(Context &ctx, Spec const &) { - FeatureView zret; + FeatureView zret{}; zret._direct_p = true; if (auto hdr{ctx.ua_req_hdr()}; hdr.is_valid()) { if (ts::URL url{hdr.url()}; url.is_valid()) { @@ -993,7 +993,7 @@ Ex_proxy_req_url_port::validate(Config &, Spec &, const swoc::TextView &) Feature Ex_proxy_req_url_port::extract(Context &ctx, Spec const &) { - FeatureView zret; + FeatureView zret{}; zret._direct_p = true; if (auto hdr{ctx.proxy_req_hdr()}; hdr.is_valid()) { if (ts::URL url{hdr.url()}; url.is_valid()) { @@ -1022,7 +1022,7 @@ Ex_pre_remap_port::validate(Config &, Spec &, const swoc::TextView &) Feature Ex_pre_remap_port::extract(Context &ctx, Spec const &) { - Feature zret{}; + Feature zret{NIL_FEATURE}; if (auto url{ctx._txn.pristine_url_get()}; url.is_valid()) { zret = static_cast>(url.port()); } @@ -1048,7 +1048,7 @@ Ex_remap_target_port::validate(Config &, Spec &, const swoc::TextView &) Feature Ex_remap_target_port::extract(Context &ctx, Spec const &) { - Feature zret{}; + Feature zret{NIL_FEATURE}; if (ctx._remap_info) { if (ts::URL url{ctx._remap_info->requestBufp, ctx._remap_info->mapFromUrl}; url.is_valid()) { zret = static_cast>(url.port()); @@ -1077,7 +1077,7 @@ Ex_remap_replacement_port::validate(Config &, Spec &, const swoc::TextView &) Feature Ex_remap_replacement_port::extract(Context &ctx, Spec const &) { - Feature zret{}; + Feature zret{NIL_FEATURE}; if (ctx._remap_info) { if (ts::URL url{ctx._remap_info->requestBufp, ctx._remap_info->mapToUrl}; url.is_valid()) { zret = static_cast>(url.port()); diff --git a/plugins/experimental/txn_box/plugin/src/text_block.cc b/plugins/experimental/txn_box/plugin/src/text_block.cc index dc6e3413959..01644c27907 100644 --- a/plugins/experimental/txn_box/plugin/src/text_block.cc +++ b/plugins/experimental/txn_box/plugin/src/text_block.cc @@ -121,7 +121,7 @@ class Do_text_block_define : public Directive std::optional _text; ///< Default literal text (optional) feature_type_for _duration; ///< Time between update checks. std::atomic _last_check = Clock::now().time_since_epoch(); ///< Absolute time of the last alert. - Clock::time_point _last_modified; ///< Last modified time of the file. + Clock::time_point _last_modified{}; ///< Last modified time of the file. std::shared_ptr _content; ///< Content of the file. int _line_no = 0; ///< For debugging name conflicts. std::shared_mutex _content_mutex; ///< Lock for access @a content. @@ -480,7 +480,7 @@ Mod_as_text_block::load(Config &cfg, YAML::Node, TextView, TextView, YAML::Node Rv Mod_as_text_block::operator()(Context &ctx, Feature &feature) { - Feature zret{}; + Feature zret{NIL_FEATURE}; if (IndexFor(STRING) == feature.index()) { auto const &tag = std::get(feature); // get the name. zret = Ex_text_block::extract_block(ctx, tag); diff --git a/plugins/header_rewrite/CMakeLists.txt b/plugins/header_rewrite/CMakeLists.txt index 6eb3f5f32ab..619c0a1191e 100644 --- a/plugins/header_rewrite/CMakeLists.txt +++ b/plugins/header_rewrite/CMakeLists.txt @@ -60,7 +60,43 @@ if(BUILD_TESTING) target_link_libraries(test_header_rewrite PRIVATE header_rewrite_parser ts::inkevent ts::tscore) if(maxminddb_FOUND) + target_compile_definitions(test_header_rewrite PRIVATE TS_USE_HRW_MAXMINDDB=1) target_link_libraries(test_header_rewrite PRIVATE maxminddb::maxminddb) + + find_package( + Python3 + COMPONENTS Interpreter + QUIET + ) + if(Python3_FOUND) + execute_process( + COMMAND "${Python3_EXECUTABLE}" -c "import mmdb_writer; import netaddr" + RESULT_VARIABLE _mmdb_python_result + OUTPUT_QUIET ERROR_QUIET + ) + if(_mmdb_python_result EQUAL 0) + set(_mmdb_test_dir "${CMAKE_CURRENT_BINARY_DIR}/test_mmdb") + add_custom_command( + OUTPUT "${_mmdb_test_dir}/test_flat_geo.mmdb" "${_mmdb_test_dir}/test_nested_geo.mmdb" + COMMAND ${CMAKE_COMMAND} -E make_directory "${_mmdb_test_dir}" + COMMAND ${Python3_EXECUTABLE} "${CMAKE_CURRENT_SOURCE_DIR}/generate_test_mmdb.py" "${_mmdb_test_dir}" + DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/generate_test_mmdb.py" + COMMENT "Generating test MMDB files for header_rewrite" + ) + add_custom_target( + generate_test_mmdb DEPENDS "${_mmdb_test_dir}/test_flat_geo.mmdb" "${_mmdb_test_dir}/test_nested_geo.mmdb" + ) + add_dependencies(test_header_rewrite generate_test_mmdb) + set_tests_properties( + test_header_rewrite + PROPERTIES + ENVIRONMENT + "MMDB_TEST_FLAT=${_mmdb_test_dir}/test_flat_geo.mmdb;MMDB_TEST_NESTED=${_mmdb_test_dir}/test_nested_geo.mmdb" + ) + else() + message(STATUS "Python modules 'mmdb-writer'/'netaddr' not found; skipping test MMDB generation") + endif() + endif() endif() # add_executable(test_matcher matcher_tests.cc matcher.cc lulu.cc regex_helper.cc) diff --git a/plugins/header_rewrite/conditions.cc b/plugins/header_rewrite/conditions.cc index faf4a19a52a..35497aaeeb9 100644 --- a/plugins/header_rewrite/conditions.cc +++ b/plugins/header_rewrite/conditions.cc @@ -831,14 +831,14 @@ ConditionNow::eval(const Resources &res) } std::string -ConditionGeo::get_geo_string(const sockaddr * /* addr ATS_UNUSED */) const +ConditionGeo::get_geo_string(const sockaddr * /* addr ATS_UNUSED */, void * /* geo_handle ATS_UNUSED */) const { TSError("[%s] No Geo library available!", PLUGIN_NAME); return ""; } int64_t -ConditionGeo::get_geo_int(const sockaddr * /* addr ATS_UNUSED */) const +ConditionGeo::get_geo_int(const sockaddr * /* addr ATS_UNUSED */, void * /* geo_handle ATS_UNUSED */) const { TSError("[%s] No Geo library available!", PLUGIN_NAME); return 0; @@ -891,9 +891,9 @@ void ConditionGeo::append_value(std::string &s, const Resources &res) { if (is_int_type()) { - s += std::to_string(get_geo_int(getClientAddr(res.state.txnp, _txn_private_slot))); + s += std::to_string(get_geo_int(getClientAddr(res.state.txnp, _txn_private_slot), res.geo_handle)); } else { - s += get_geo_string(getClientAddr(res.state.txnp, _txn_private_slot)); + s += get_geo_string(getClientAddr(res.state.txnp, _txn_private_slot), res.geo_handle); } Dbg(pi_dbg_ctl, "Appending GEO() to evaluation value -> %s", s.c_str()); } @@ -905,7 +905,7 @@ ConditionGeo::eval(const Resources &res) Dbg(pi_dbg_ctl, "Evaluating GEO()"); if (is_int_type()) { - int64_t geo = get_geo_int(getClientAddr(res.state.txnp, _txn_private_slot)); + int64_t geo = get_geo_int(getClientAddr(res.state.txnp, _txn_private_slot), res.geo_handle); ret = static_cast *>(_matcher.get())->test(geo, res); } else { diff --git a/plugins/header_rewrite/conditions.h b/plugins/header_rewrite/conditions.h index 4393ecbab09..c393e602663 100644 --- a/plugins/header_rewrite/conditions.h +++ b/plugins/header_rewrite/conditions.h @@ -477,8 +477,8 @@ class ConditionGeo : public Condition } private: - virtual int64_t get_geo_int(const sockaddr *addr) const; - virtual std::string get_geo_string(const sockaddr *addr) const; + virtual int64_t get_geo_int(const sockaddr *addr, void *geo_handle) const; + virtual std::string get_geo_string(const sockaddr *addr, void *geo_handle) const; protected: bool diff --git a/plugins/header_rewrite/conditions_geo.h b/plugins/header_rewrite/conditions_geo.h index 6894ca3a590..224cff554b6 100644 --- a/plugins/header_rewrite/conditions_geo.h +++ b/plugins/header_rewrite/conditions_geo.h @@ -27,10 +27,10 @@ class MMConditionGeo : public ConditionGeo MMConditionGeo() {} virtual ~MMConditionGeo() {} - static void initLibrary(const std::string &path); + static void *initLibrary(const std::string &path); - virtual int64_t get_geo_int(const sockaddr *addr) const override; - virtual std::string get_geo_string(const sockaddr *addr) const override; + int64_t get_geo_int(const sockaddr *addr, void *geo_handle) const override; + std::string get_geo_string(const sockaddr *addr, void *geo_handle) const override; }; class GeoIPConditionGeo : public ConditionGeo @@ -39,8 +39,8 @@ class GeoIPConditionGeo : public ConditionGeo GeoIPConditionGeo() {} virtual ~GeoIPConditionGeo() {} - static void initLibrary(const std::string &path); + static void *initLibrary(const std::string &path); - virtual int64_t get_geo_int(const sockaddr *addr) const override; - virtual std::string get_geo_string(const sockaddr *addr) const override; + int64_t get_geo_int(const sockaddr *addr, void *geo_handle) const override; + std::string get_geo_string(const sockaddr *addr, void *geo_handle) const override; }; diff --git a/plugins/header_rewrite/conditions_geo_geoip.cc b/plugins/header_rewrite/conditions_geo_geoip.cc index 1bde085cb41..5e2ffe0756b 100644 --- a/plugins/header_rewrite/conditions_geo_geoip.cc +++ b/plugins/header_rewrite/conditions_geo_geoip.cc @@ -24,6 +24,7 @@ #include #include #include +#include #include "ts/ts.h" @@ -31,49 +32,65 @@ #include -GeoIP *gGeoIP[NUM_DB_TYPES]; +struct GeoIPHandleSet { + GeoIP *dbs[NUM_DB_TYPES] = {}; +}; -void +static std::mutex gGeoIPCacheMutex; +static GeoIPHandleSet *gGeoIPHandleSet = nullptr; + +void * GeoIPConditionGeo::initLibrary(const std::string &) { + std::lock_guard lock(gGeoIPCacheMutex); + + if (gGeoIPHandleSet != nullptr) { + return gGeoIPHandleSet; + } + + gGeoIPHandleSet = new GeoIPHandleSet; + GeoIPDBTypes dbs[] = {GEOIP_COUNTRY_EDITION, GEOIP_COUNTRY_EDITION_V6, GEOIP_ASNUM_EDITION, GEOIP_ASNUM_EDITION_V6}; for (auto &db : dbs) { - if (!gGeoIP[db] && GeoIP_db_avail(db)) { - // GEOIP_STANDARD seems to break threaded apps... - gGeoIP[db] = GeoIP_open_type(db, GEOIP_MMAP_CACHE); + if (!gGeoIPHandleSet->dbs[db] && GeoIP_db_avail(db)) { + gGeoIPHandleSet->dbs[db] = GeoIP_open_type(db, GEOIP_MMAP_CACHE); - char *db_info = GeoIP_database_info(gGeoIP[db]); + char *db_info = GeoIP_database_info(gGeoIPHandleSet->dbs[db]); Dbg(pi_dbg_ctl, "initialized GeoIP-DB[%d] %s", db, db_info); free(db_info); } } + + return gGeoIPHandleSet; } std::string -GeoIPConditionGeo::get_geo_string(const sockaddr *addr) const +GeoIPConditionGeo::get_geo_string(const sockaddr *addr, void *geo_handle) const { std::string ret = "(unknown)"; int v = 4; - if (addr) { + auto *handle = static_cast(geo_handle); + + if (addr && handle) { switch (_geo_qual) { // Country database case GEO_QUAL_COUNTRY: switch (addr->sa_family) { case AF_INET: - if (gGeoIP[GEOIP_COUNTRY_EDITION]) { + if (handle->dbs[GEOIP_COUNTRY_EDITION]) { uint32_t ip = ntohl(reinterpret_cast(addr)->sin_addr.s_addr); - ret = GeoIP_country_code_by_ipnum(gGeoIP[GEOIP_COUNTRY_EDITION], ip); + ret = GeoIP_country_code_by_ipnum(handle->dbs[GEOIP_COUNTRY_EDITION], ip); } break; case AF_INET6: { - if (gGeoIP[GEOIP_COUNTRY_EDITION_V6]) { + if (handle->dbs[GEOIP_COUNTRY_EDITION_V6]) { geoipv6_t ip = reinterpret_cast(addr)->sin6_addr; v = 6; - ret = GeoIP_country_code_by_ipnum_v6(gGeoIP[GEOIP_COUNTRY_EDITION_V6], ip); + ret = GeoIP_country_code_by_ipnum_v6(handle->dbs[GEOIP_COUNTRY_EDITION_V6], ip); } } break; default: @@ -86,18 +103,18 @@ GeoIPConditionGeo::get_geo_string(const sockaddr *addr) const case GEO_QUAL_ASN_NAME: switch (addr->sa_family) { case AF_INET: - if (gGeoIP[GEOIP_ASNUM_EDITION]) { + if (handle->dbs[GEOIP_ASNUM_EDITION]) { uint32_t ip = ntohl(reinterpret_cast(addr)->sin_addr.s_addr); - ret = GeoIP_name_by_ipnum(gGeoIP[GEOIP_ASNUM_EDITION], ip); + ret = GeoIP_name_by_ipnum(handle->dbs[GEOIP_ASNUM_EDITION], ip); } break; case AF_INET6: { - if (gGeoIP[GEOIP_ASNUM_EDITION_V6]) { + if (handle->dbs[GEOIP_ASNUM_EDITION_V6]) { geoipv6_t ip = reinterpret_cast(addr)->sin6_addr; v = 6; - ret = GeoIP_name_by_ipnum_v6(gGeoIP[GEOIP_ASNUM_EDITION_V6], ip); + ret = GeoIP_name_by_ipnum_v6(handle->dbs[GEOIP_ASNUM_EDITION_V6], ip); } } break; default: @@ -114,12 +131,14 @@ GeoIPConditionGeo::get_geo_string(const sockaddr *addr) const } int64_t -GeoIPConditionGeo::get_geo_int(const sockaddr *addr) const +GeoIPConditionGeo::get_geo_int(const sockaddr *addr, void *geo_handle) const { int64_t ret = -1; int v = 4; - if (!addr) { + auto *handle = static_cast(geo_handle); + + if (!addr || !handle) { return 0; } @@ -128,18 +147,18 @@ GeoIPConditionGeo::get_geo_int(const sockaddr *addr) const case GEO_QUAL_COUNTRY_ISO: switch (addr->sa_family) { case AF_INET: - if (gGeoIP[GEOIP_COUNTRY_EDITION]) { + if (handle->dbs[GEOIP_COUNTRY_EDITION]) { uint32_t ip = ntohl(reinterpret_cast(addr)->sin_addr.s_addr); - ret = GeoIP_id_by_ipnum(gGeoIP[GEOIP_COUNTRY_EDITION], ip); + ret = GeoIP_id_by_ipnum(handle->dbs[GEOIP_COUNTRY_EDITION], ip); } break; case AF_INET6: { - if (gGeoIP[GEOIP_COUNTRY_EDITION_V6]) { + if (handle->dbs[GEOIP_COUNTRY_EDITION_V6]) { geoipv6_t ip = reinterpret_cast(addr)->sin6_addr; v = 6; - ret = GeoIP_id_by_ipnum_v6(gGeoIP[GEOIP_COUNTRY_EDITION_V6], ip); + ret = GeoIP_id_by_ipnum_v6(handle->dbs[GEOIP_COUNTRY_EDITION_V6], ip); } } break; default: @@ -153,18 +172,18 @@ GeoIPConditionGeo::get_geo_int(const sockaddr *addr) const switch (addr->sa_family) { case AF_INET: - if (gGeoIP[GEOIP_ASNUM_EDITION]) { + if (handle->dbs[GEOIP_ASNUM_EDITION]) { uint32_t ip = ntohl(reinterpret_cast(addr)->sin_addr.s_addr); - asn_name = GeoIP_name_by_ipnum(gGeoIP[GEOIP_ASNUM_EDITION], ip); + asn_name = GeoIP_name_by_ipnum(handle->dbs[GEOIP_ASNUM_EDITION], ip); } break; case AF_INET6: - if (gGeoIP[GEOIP_ASNUM_EDITION_V6]) { + if (handle->dbs[GEOIP_ASNUM_EDITION_V6]) { geoipv6_t ip = reinterpret_cast(addr)->sin6_addr; v = 6; - asn_name = GeoIP_name_by_ipnum_v6(gGeoIP[GEOIP_ASNUM_EDITION_V6], ip); + asn_name = GeoIP_name_by_ipnum_v6(handle->dbs[GEOIP_ASNUM_EDITION_V6], ip); } break; } diff --git a/plugins/header_rewrite/conditions_geo_maxmind.cc b/plugins/header_rewrite/conditions_geo_maxmind.cc index 5fd905deca1..81e34ed1d9f 100644 --- a/plugins/header_rewrite/conditions_geo_maxmind.cc +++ b/plugins/header_rewrite/conditions_geo_maxmind.cc @@ -23,6 +23,9 @@ #include #include +#include +#include +#include #include "ts/ts.h" @@ -30,154 +33,182 @@ #include -MMDB_s *gMaxMindDB = nullptr; +enum class MmdbSchema { NESTED, FLAT }; -void +struct MmdbHandle { + MMDB_s db; + MmdbSchema schema = MmdbSchema::NESTED; +}; + +static std::map gMmdbCache; +static std::mutex gMmdbCacheMutex; + +// Detect whether the MMDB uses nested (GeoLite2) or flat (vendor) field layout +// by probing for the nested country path on a lookup result. +static MmdbSchema +detect_schema(MMDB_entry_s *entry) +{ + MMDB_entry_data_s probe; + int status = MMDB_get_value(entry, &probe, "country", "iso_code", NULL); + + if (MMDB_SUCCESS == status && probe.has_data && probe.type == MMDB_DATA_TYPE_UTF8_STRING) { + return MmdbSchema::NESTED; + } + + status = MMDB_get_value(entry, &probe, "country_code", NULL); + if (MMDB_SUCCESS == status && probe.has_data && probe.type == MMDB_DATA_TYPE_UTF8_STRING) { + return MmdbSchema::FLAT; + } + + return MmdbSchema::NESTED; +} + +static const char *probe_ips[] = {"8.8.8.8", "1.1.1.1", "128.0.0.1"}; + +void * MMConditionGeo::initLibrary(const std::string &path) { if (path.empty()) { Dbg(pi_dbg_ctl, "Empty MaxMind db path specified. Not initializing!"); - return; + return nullptr; } - if (gMaxMindDB != nullptr) { - Dbg(pi_dbg_ctl, "Maxmind library already initialized"); - return; + std::lock_guard lock(gMmdbCacheMutex); + + auto it = gMmdbCache.find(path); + if (it != gMmdbCache.end()) { + Dbg(pi_dbg_ctl, "Maxmind library already initialized for %s", path.c_str()); + return it->second; } - gMaxMindDB = new MMDB_s; + auto *handle = new MmdbHandle; + int status = MMDB_open(path.c_str(), MMDB_MODE_MMAP, &handle->db); - int status = MMDB_open(path.c_str(), MMDB_MODE_MMAP, gMaxMindDB); if (MMDB_SUCCESS != status) { Dbg(pi_dbg_ctl, "Cannot open %s - %s", path.c_str(), MMDB_strerror(status)); - delete gMaxMindDB; - return; - } - Dbg(pi_dbg_ctl, "Loaded %s", path.c_str()); + delete handle; + return nullptr; + } + + // Probe the database schema at load time so we know which field paths to + // use for country lookups. Try a few well-known IPs until one hits. + for (auto *ip : probe_ips) { + int gai_error, mmdb_error; + MMDB_lookup_result_s result = MMDB_lookup_string(&handle->db, ip, &gai_error, &mmdb_error); + if (gai_error == 0 && MMDB_SUCCESS == mmdb_error && result.found_entry) { + handle->schema = detect_schema(&result.entry); + Dbg(pi_dbg_ctl, "Loaded %s (schema: %s)", path.c_str(), handle->schema == MmdbSchema::FLAT ? "flat" : "nested"); + gMmdbCache[path] = handle; + return handle; + } + } + + Dbg(pi_dbg_ctl, "Loaded %s (schema: defaulting to nested, no probe IPs matched)", path.c_str()); + gMmdbCache[path] = handle; + return handle; } std::string -MMConditionGeo::get_geo_string(const sockaddr *addr) const +MMConditionGeo::get_geo_string(const sockaddr *addr, void *geo_handle) const { std::string ret = "(unknown)"; int mmdb_error; - if (gMaxMindDB == nullptr) { + auto *handle = static_cast(geo_handle); + + if (handle == nullptr) { Dbg(pi_dbg_ctl, "MaxMind not initialized; using default value"); return ret; } - MMDB_lookup_result_s result = MMDB_lookup_sockaddr(gMaxMindDB, addr, &mmdb_error); + MMDB_lookup_result_s result = MMDB_lookup_sockaddr(&handle->db, addr, &mmdb_error); if (MMDB_SUCCESS != mmdb_error) { Dbg(pi_dbg_ctl, "Error during sockaddr lookup: %s", MMDB_strerror(mmdb_error)); return ret; } - MMDB_entry_data_list_s *entry_data_list = nullptr; if (!result.found_entry) { Dbg(pi_dbg_ctl, "No entry for this IP was found"); return ret; } - int status = MMDB_get_entry_data_list(&result.entry, &entry_data_list); - if (MMDB_SUCCESS != status) { - Dbg(pi_dbg_ctl, "Error looking up entry data: %s", MMDB_strerror(status)); - return ret; - } - - if (entry_data_list == nullptr) { - Dbg(pi_dbg_ctl, "No data found"); - return ret; - } + MMDB_entry_data_s entry_data; + int status; - const char *field_name; switch (_geo_qual) { case GEO_QUAL_COUNTRY: - field_name = "country_code"; + if (handle->schema == MmdbSchema::FLAT) { + status = MMDB_get_value(&result.entry, &entry_data, "country_code", NULL); + } else { + status = MMDB_get_value(&result.entry, &entry_data, "country", "iso_code", NULL); + } break; case GEO_QUAL_ASN_NAME: - field_name = "autonomous_system_organization"; + status = MMDB_get_value(&result.entry, &entry_data, "autonomous_system_organization", NULL); break; default: Dbg(pi_dbg_ctl, "Unsupported field %d", _geo_qual); return ret; - break; } - MMDB_entry_data_s entry_data; - - status = MMDB_get_value(&result.entry, &entry_data, field_name, NULL); if (MMDB_SUCCESS != status) { - Dbg(pi_dbg_ctl, "ERROR on get value asn value: %s", MMDB_strerror(status)); + Dbg(pi_dbg_ctl, "Error looking up geo string field: %s", MMDB_strerror(status)); return ret; } - ret = std::string(entry_data.utf8_string, entry_data.data_size); - if (nullptr != entry_data_list) { - MMDB_free_entry_data_list(entry_data_list); + if (entry_data.has_data && entry_data.type == MMDB_DATA_TYPE_UTF8_STRING) { + ret = std::string(entry_data.utf8_string, entry_data.data_size); } return ret; } int64_t -MMConditionGeo::get_geo_int(const sockaddr *addr) const +MMConditionGeo::get_geo_int(const sockaddr *addr, void *geo_handle) const { int64_t ret = -1; int mmdb_error; - if (gMaxMindDB == nullptr) { + auto *handle = static_cast(geo_handle); + + if (handle == nullptr) { Dbg(pi_dbg_ctl, "MaxMind not initialized; using default value"); return ret; } - MMDB_lookup_result_s result = MMDB_lookup_sockaddr(gMaxMindDB, addr, &mmdb_error); + MMDB_lookup_result_s result = MMDB_lookup_sockaddr(&handle->db, addr, &mmdb_error); if (MMDB_SUCCESS != mmdb_error) { Dbg(pi_dbg_ctl, "Error during sockaddr lookup: %s", MMDB_strerror(mmdb_error)); return ret; } - MMDB_entry_data_list_s *entry_data_list = nullptr; if (!result.found_entry) { Dbg(pi_dbg_ctl, "No entry for this IP was found"); return ret; } - int status = MMDB_get_entry_data_list(&result.entry, &entry_data_list); - if (MMDB_SUCCESS != status) { - Dbg(pi_dbg_ctl, "Error looking up entry data: %s", MMDB_strerror(status)); - return ret; - } - - if (entry_data_list == nullptr) { - Dbg(pi_dbg_ctl, "No data found"); - return ret; - } + MMDB_entry_data_s entry_data; + int status; - const char *field_name; switch (_geo_qual) { case GEO_QUAL_ASN: - field_name = "autonomous_system_number"; + // GeoLite2-ASN / DBIP-ASN store this as a top-level uint32 field + status = MMDB_get_value(&result.entry, &entry_data, "autonomous_system_number", NULL); break; default: Dbg(pi_dbg_ctl, "Unsupported field %d", _geo_qual); return ret; - break; } - MMDB_entry_data_s entry_data; - - status = MMDB_get_value(&result.entry, &entry_data, field_name, NULL); if (MMDB_SUCCESS != status) { - Dbg(pi_dbg_ctl, "ERROR on get value asn value: %s", MMDB_strerror(status)); + Dbg(pi_dbg_ctl, "Error looking up geo int field: %s", MMDB_strerror(status)); return ret; } - ret = entry_data.uint32; - if (nullptr != entry_data_list) { - MMDB_free_entry_data_list(entry_data_list); + if (entry_data.has_data && entry_data.type == MMDB_DATA_TYPE_UINT32) { + ret = entry_data.uint32; } return ret; diff --git a/plugins/header_rewrite/generate_test_mmdb.py b/plugins/header_rewrite/generate_test_mmdb.py new file mode 100644 index 00000000000..8cd34ac8a41 --- /dev/null +++ b/plugins/header_rewrite/generate_test_mmdb.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Generate test MMDB files for header_rewrite geo lookup unit tests. + +Two schemas exist in the wild: + + Nested (GeoLite2/GeoIP2/DBIP): country -> iso_code + Flat (vendor-specific): country_code (top-level) + +This script generates one MMDB file for each schema so the C++ test +can verify that auto-detection works for both. + +Requires: pip install mmdb-writer netaddr +""" + +import os +import sys + +try: + from mmdb_writer import MMDBWriter, MmdbU32 + import netaddr +except ImportError: + print("SKIP: mmdb-writer or netaddr not installed (pip install mmdb-writer netaddr)", file=sys.stderr) + sys.exit(1) + + +def net(cidr): + return netaddr.IPSet([netaddr.IPNetwork(cidr)]) + + +def generate_flat(path): + """Flat schema: country_code at top level (vendor databases).""" + w = MMDBWriter(ip_version=4, database_type="Test-Flat-GeoIP") + w.insert_network( + net("8.8.8.0/24"), { + "country_code": "US", + "autonomous_system_number": MmdbU32(15169), + "autonomous_system_organization": "GOOGLE", + }) + w.insert_network( + net("1.2.3.0/24"), { + "country_code": "KR", + "autonomous_system_number": MmdbU32(9286), + "autonomous_system_organization": "KINX", + }) + w.to_db_file(path) + + +def generate_nested(path): + """Nested schema: country/iso_code (GeoLite2, GeoIP2, DBIP).""" + w = MMDBWriter(ip_version=4, database_type="Test-Nested-GeoIP2") + w.insert_network( + net("8.8.8.0/24"), { + "country": { + "iso_code": "US", + "names": { + "en": "United States" + } + }, + "autonomous_system_number": MmdbU32(15169), + "autonomous_system_organization": "GOOGLE", + }) + w.insert_network( + net("1.2.3.0/24"), { + "country": { + "iso_code": "KR", + "names": { + "en": "South Korea" + } + }, + "autonomous_system_number": MmdbU32(9286), + "autonomous_system_organization": "KINX", + }) + w.to_db_file(path) + + +if __name__ == "__main__": + outdir = sys.argv[1] if len(sys.argv) > 1 else "." + + flat_path = os.path.join(outdir, "test_flat_geo.mmdb") + nested_path = os.path.join(outdir, "test_nested_geo.mmdb") + + generate_flat(flat_path) + generate_nested(nested_path) + + print(f"Generated {flat_path} ({os.path.getsize(flat_path)} bytes)") + print(f"Generated {nested_path} ({os.path.getsize(nested_path)} bytes)") diff --git a/plugins/header_rewrite/header_rewrite.cc b/plugins/header_rewrite/header_rewrite.cc index a2601be4c59..16f8b0c6170 100644 --- a/plugins/header_rewrite/header_rewrite.cc +++ b/plugins/header_rewrite/header_rewrite.cc @@ -41,7 +41,6 @@ // Debugs namespace header_rewrite_ns { -std::once_flag initGeoLibs; std::once_flag initPlugin; PluginFactory plugin_factory; } // namespace header_rewrite_ns @@ -52,19 +51,21 @@ initPluginFactory() header_rewrite_ns::plugin_factory.setRuntimeDir(RecConfigReadRuntimeDir()).addSearchDir(RecConfigReadPluginDir()); } -static void +static void * initGeoLibraries(const std::string &dbPath) { if (dbPath.empty()) { - return; + return nullptr; } Dbg(pi_dbg_ctl, "Loading geo db %s", dbPath.c_str()); #if TS_USE_HRW_GEOIP - GeoIPConditionGeo::initLibrary(dbPath); + return GeoIPConditionGeo::initLibrary(dbPath); #elif TS_USE_HRW_MAXMINDDB - MMConditionGeo::initLibrary(dbPath); + return MMConditionGeo::initLibrary(dbPath); +#else + return nullptr; #endif } @@ -76,7 +77,8 @@ static int cont_rewrite_headers(TSCont, TSEvent, void *); class RulesConfig { public: - RulesConfig(int timezone, int inboundIpSource) : _timezone(timezone), _inboundIpSource(inboundIpSource) + RulesConfig(int timezone, int inboundIpSource, void *geo_handle = nullptr) + : _timezone(timezone), _inboundIpSource(inboundIpSource), _geo_handle(geo_handle) { Dbg(dbg_ctl, "RulesConfig CTOR"); _cont = TSContCreate(cont_rewrite_headers, nullptr); @@ -119,6 +121,12 @@ class RulesConfig return _inboundIpSource; } + [[nodiscard]] void * + geo_handle() const + { + return _geo_handle; + } + bool parse_config(const std::string &fname, TSHttpHookID default_hook, char *from_url = nullptr, char *to_url = nullptr); private: @@ -130,6 +138,8 @@ class RulesConfig int _timezone = 0; int _inboundIpSource = 0; + + void *_geo_handle = nullptr; }; void @@ -509,6 +519,8 @@ cont_rewrite_headers(TSCont contp, TSEvent event, void *edata) RuleSet *rule = conf->rule(hook); Resources res(txnp, contp); + res.geo_handle = conf->geo_handle(); + // Get the resources necessary to process this event res.gather(conf->resid(hook), hook); @@ -602,12 +614,12 @@ TSPluginInit(int argc, const char *argv[]) Dbg(pi_dbg_ctl, "Global geo db %s", geoDBpath.c_str()); - std::call_once(initGeoLibs, [&geoDBpath]() { initGeoLibraries(geoDBpath); }); + void *geo_handle = initGeoLibraries(geoDBpath); std::call_once(initPlugin, initPluginFactory); // Parse the global config file(s). All rules are just appended // to the "global" Rules configuration. - auto *conf = new RulesConfig(timezone, inboundIpSource); + auto *conf = new RulesConfig(timezone, inboundIpSource, geo_handle); bool got_config = false; for (int i = optind; i < argc; ++i) { @@ -711,21 +723,19 @@ TSRemapNewInstance(int argc, char *argv[], void **ih, char * /* errbuf ATS_UNUSE } } + void *geo_handle = nullptr; + if (!geoDBpath.empty()) { if (!geoDBpath.starts_with('/')) { geoDBpath = std::string(TSConfigDirGet()) + '/' + geoDBpath; } Dbg(pi_dbg_ctl, "Remap geo db %s", geoDBpath.c_str()); - - // This MUST be called only if the geoDBpath is set. If called without a geoDBPath (i.e. outside of this if) then - // NO hrw remap rule can load a mmdb file. - // The call_once applies to every remap instance as its a plugin global - std::call_once(initGeoLibs, [&geoDBpath]() { initGeoLibraries(geoDBpath); }); + geo_handle = initGeoLibraries(geoDBpath); } std::call_once(initPlugin, initPluginFactory); - auto *conf = new RulesConfig(timezone, inboundIpSource); + auto *conf = new RulesConfig(timezone, inboundIpSource, geo_handle); for (int i = optind; i < argc; ++i) { Dbg(pi_dbg_ctl, "Loading remap configuration file %s", argv[i]); @@ -781,6 +791,8 @@ TSRemapDoRemap(void *ih, TSHttpTxn rh, TSRemapRequestInfo *rri) RuleSet *rule = conf->rule(TS_REMAP_PSEUDO_HOOK); Resources res(rh, rri); + res.geo_handle = conf->geo_handle(); + if (rule) { res.gather(conf->resid(TS_REMAP_PSEUDO_HOOK), TS_REMAP_PSEUDO_HOOK); diff --git a/plugins/header_rewrite/header_rewrite_test.cc b/plugins/header_rewrite/header_rewrite_test.cc index 8e92f7ae5f1..6f6026e3df3 100644 --- a/plugins/header_rewrite/header_rewrite_test.cc +++ b/plugins/header_rewrite/header_rewrite_test.cc @@ -22,11 +22,17 @@ #include #include +#include #include #include +#include #include "parser.h" +#if TS_USE_HRW_MAXMINDDB +#include +#endif + namespace header_rewrite_ns { const char PLUGIN_NAME[] = "TEST_header_rewrite"; @@ -538,6 +544,190 @@ test_tokenizer() return errors; } +#if TS_USE_HRW_MAXMINDDB + +static bool +file_exists(const char *path) +{ + struct stat st; + return stat(path, &st) == 0; +} + +static int +open_test_mmdb(MMDB_s &mmdb, const char *path) +{ + int status = MMDB_open(path, MMDB_MODE_MMAP, &mmdb); + if (MMDB_SUCCESS != status) { + std::cerr << "Cannot open " << path << ": " << MMDB_strerror(status) << std::endl; + return 1; + } + return 0; +} + +static std::string +lookup_country(MMDB_s &mmdb, const char *ip) +{ + int gai_error, mmdb_error; + MMDB_lookup_result_s result = MMDB_lookup_string(&mmdb, ip, &gai_error, &mmdb_error); + + if (gai_error != 0 || MMDB_SUCCESS != mmdb_error || !result.found_entry) { + return "(lookup_failed)"; + } + + MMDB_entry_data_s entry_data; + + // Try nested path first (GeoLite2 style) + int status = MMDB_get_value(&result.entry, &entry_data, "country", "iso_code", NULL); + if (MMDB_SUCCESS == status && entry_data.has_data && entry_data.type == MMDB_DATA_TYPE_UTF8_STRING) { + return std::string(entry_data.utf8_string, entry_data.data_size); + } + + // Try flat path (vendor style) + status = MMDB_get_value(&result.entry, &entry_data, "country_code", NULL); + if (MMDB_SUCCESS == status && entry_data.has_data && entry_data.type == MMDB_DATA_TYPE_UTF8_STRING) { + return std::string(entry_data.utf8_string, entry_data.data_size); + } + + return "(not_found)"; +} + +static uint32_t +lookup_asn(MMDB_s &mmdb, const char *ip) +{ + int gai_error, mmdb_error; + MMDB_lookup_result_s result = MMDB_lookup_string(&mmdb, ip, &gai_error, &mmdb_error); + + if (gai_error != 0 || MMDB_SUCCESS != mmdb_error || !result.found_entry) { + return 0; + } + + MMDB_entry_data_s entry_data; + int status = MMDB_get_value(&result.entry, &entry_data, "autonomous_system_number", NULL); + if (MMDB_SUCCESS == status && entry_data.has_data && entry_data.type == MMDB_DATA_TYPE_UINT32) { + return entry_data.uint32; + } + return 0; +} + +// Test that we can read country codes from a flat-schema MMDB (vendor databases +// where country_code is a top-level string field). +int +test_flat_schema() +{ + const char *path = getenv("MMDB_TEST_FLAT"); + if (path == nullptr || !file_exists(path)) { + std::cout << "SKIP: flat-schema test mmdb not found (set MMDB_TEST_FLAT)" << std::endl; + return 0; + } + + std::cout << "Testing flat-schema MMDB: " << path << std::endl; + int errors = 0; + MMDB_s mmdb; + if (open_test_mmdb(mmdb, path) != 0) { + return 1; + } + + std::string country = lookup_country(mmdb, "8.8.8.8"); + if (country != "US") { + std::cerr << "FAIL: flat schema 8.8.8.8 expected 'US', got '" << country << "'" << std::endl; + ++errors; + } else { + std::cout << " PASS: flat 8.8.8.8 country = " << country << std::endl; + } + + country = lookup_country(mmdb, "1.2.3.4"); + if (country != "KR") { + std::cerr << "FAIL: flat schema 1.2.3.4 expected 'KR', got '" << country << "'" << std::endl; + ++errors; + } else { + std::cout << " PASS: flat 1.2.3.4 country = " << country << std::endl; + } + + uint32_t asn = lookup_asn(mmdb, "8.8.8.8"); + if (asn != 15169) { + std::cerr << "FAIL: flat schema 8.8.8.8 expected ASN 15169, got " << asn << std::endl; + ++errors; + } else { + std::cout << " PASS: flat 8.8.8.8 ASN = " << asn << std::endl; + } + + // Loopback should not be found + country = lookup_country(mmdb, "127.0.0.1"); + if (country != "(lookup_failed)") { + std::cerr << "FAIL: flat schema 127.0.0.1 expected no entry, got '" << country << "'" << std::endl; + ++errors; + } else { + std::cout << " PASS: flat 127.0.0.1 correctly not found" << std::endl; + } + + MMDB_close(&mmdb); + return errors; +} + +// Test that we can read country codes from a nested-schema MMDB (GeoLite2/GeoIP2/DBIP +// where country is country -> iso_code). +int +test_nested_schema() +{ + const char *path = getenv("MMDB_TEST_NESTED"); + if (path == nullptr || !file_exists(path)) { + std::cout << "SKIP: nested-schema test mmdb not found (set MMDB_TEST_NESTED)" << std::endl; + return 0; + } + + std::cout << "Testing nested-schema MMDB: " << path << std::endl; + int errors = 0; + MMDB_s mmdb; + if (open_test_mmdb(mmdb, path) != 0) { + return 1; + } + + std::string country = lookup_country(mmdb, "8.8.8.8"); + if (country != "US") { + std::cerr << "FAIL: nested schema 8.8.8.8 expected 'US', got '" << country << "'" << std::endl; + ++errors; + } else { + std::cout << " PASS: nested 8.8.8.8 country = " << country << std::endl; + } + + country = lookup_country(mmdb, "1.2.3.4"); + if (country != "KR") { + std::cerr << "FAIL: nested schema 1.2.3.4 expected 'KR', got '" << country << "'" << std::endl; + ++errors; + } else { + std::cout << " PASS: nested 1.2.3.4 country = " << country << std::endl; + } + + uint32_t asn = lookup_asn(mmdb, "8.8.8.8"); + if (asn != 15169) { + std::cerr << "FAIL: nested schema 8.8.8.8 expected ASN 15169, got " << asn << std::endl; + ++errors; + } else { + std::cout << " PASS: nested 8.8.8.8 ASN = " << asn << std::endl; + } + + country = lookup_country(mmdb, "127.0.0.1"); + if (country != "(lookup_failed)") { + std::cerr << "FAIL: nested schema 127.0.0.1 expected no entry, got '" << country << "'" << std::endl; + ++errors; + } else { + std::cout << " PASS: nested 127.0.0.1 correctly not found" << std::endl; + } + + MMDB_close(&mmdb); + return errors; +} + +int +test_maxmind_geo() +{ + int errors = 0; + errors += test_flat_schema(); + errors += test_nested_schema(); + return errors; +} +#endif + int main() { @@ -545,5 +735,11 @@ main() return 1; } +#if TS_USE_HRW_MAXMINDDB + if (test_maxmind_geo()) { + return 1; + } +#endif + return 0; } diff --git a/plugins/header_rewrite/operators.cc b/plugins/header_rewrite/operators.cc index 9a3238d3af7..76c4f6197b6 100644 --- a/plugins/header_rewrite/operators.cc +++ b/plugins/header_rewrite/operators.cc @@ -1232,6 +1232,8 @@ OperatorSetPluginCntl::initialize(Parser &p) } else { TSError("[%s] Unknown value for INBOUND_IP_SOURCE control: %s", PLUGIN_NAME, value.c_str()); } + } else { + TSError("[%s] Unknown plugin control name: %s", PLUGIN_NAME, name.c_str()); } } diff --git a/plugins/header_rewrite/operators.h b/plugins/header_rewrite/operators.h index eee90c35810..2b48f813db8 100644 --- a/plugins/header_rewrite/operators.h +++ b/plugins/header_rewrite/operators.h @@ -456,8 +456,8 @@ class OperatorSetHttpCntl : public Operator bool exec(const Resources &res) const override; private: - bool _flag = false; - TSHttpCntlType _cntl_qual; + bool _flag{false}; + TSHttpCntlType _cntl_qual{TS_HTTP_CNTL_LOGGING_MODE}; // always overwritten by initialize() }; class OperatorSetPluginCntl : public Operator @@ -487,8 +487,8 @@ class OperatorSetPluginCntl : public Operator } private: - PluginCtrl _name; - int _value; + PluginCtrl _name{PluginCtrl::TIMEZONE}; // always overwritten by initialize() + int _value{0}; }; class RemapPluginInst; // Opaque to the HRW operator, but needed in the implementation. diff --git a/plugins/header_rewrite/resources.h b/plugins/header_rewrite/resources.h index 2fb45b08c44..80d1268106c 100644 --- a/plugins/header_rewrite/resources.h +++ b/plugins/header_rewrite/resources.h @@ -127,6 +127,7 @@ class Resources TransactionState state; // Without cripts, txnp / ssnp goes here #endif TSHttpStatus resp_status = TS_HTTP_STATUS_NONE; + void *geo_handle = nullptr; struct LifetimeExtension { std::string subject_storage; diff --git a/plugins/origin_server_auth/origin_server_auth.cc b/plugins/origin_server_auth/origin_server_auth.cc index 583ca0f785a..e2eaaa9a1c9 100644 --- a/plugins/origin_server_auth/origin_server_auth.cc +++ b/plugins/origin_server_auth/origin_server_auth.cc @@ -154,6 +154,8 @@ class S3Config; class ConfigCache { public: + ~ConfigCache(); + S3Config *get(const char *fname); private: @@ -601,6 +603,13 @@ class S3Config int _invalid_file_count = 0; }; +ConfigCache::~ConfigCache() +{ + for (auto &[key, data] : _cache) { + delete data.config.load(); + } +} + bool S3Config::parse_config(const std::string &config_fname) { diff --git a/plugins/slice/server.cc b/plugins/slice/server.cc index b304976abbc..3f2ca4f7c9f 100644 --- a/plugins/slice/server.cc +++ b/plugins/slice/server.cc @@ -26,6 +26,7 @@ #include "ts/apidefs.h" #include "util.h" +#include #include namespace @@ -460,14 +461,19 @@ handleNextServerHeader(Data *const data) data->m_blockstate = BlockState::PendingRef; // interior headers for new identifier reference + etaglen = std::min(etaglen, static_cast(sizeof(data->m_etag) - 1)); data->m_etaglen = etaglen; if (0 < etaglen) { - strncpy(data->m_etag, etag, etaglen); + memcpy(data->m_etag, etag, etaglen); } + data->m_etag[etaglen] = '\0'; + + lastmodifiedlen = std::min(lastmodifiedlen, static_cast(sizeof(data->m_lastmodified) - 1)); data->m_lastmodifiedlen = lastmodifiedlen; if (0 < lastmodifiedlen) { - strncpy(data->m_lastmodified, lastmodified, lastmodifiedlen); + memcpy(data->m_lastmodified, lastmodified, lastmodifiedlen); } + data->m_lastmodified[lastmodifiedlen] = '\0'; // potentially new content length data->m_contentlen = blockcr.m_length; diff --git a/plugins/slice/slice.cc b/plugins/slice/slice.cc index de694147a05..ba087fb2685 100644 --- a/plugins/slice/slice.cc +++ b/plugins/slice/slice.cc @@ -28,6 +28,7 @@ #include "ts/ts.h" #include +#include #include #include #include @@ -50,14 +51,13 @@ TSCont global_read_resp_hdr_contp; static bool should_skip_this_obj(TSHttpTxn txnp, Config *const config) { - int len = 0; - char *const urlstr = TSHttpTxnEffectiveUrlStringGet(txnp, &len); + int len = 0; + std::unique_ptr urlstr(TSHttpTxnEffectiveUrlStringGet(txnp, &len), TSfree); - bool const known_large = config->isKnownLargeObj({urlstr, static_cast(len)}); - TSfree(urlstr); + bool const known_large = config->isKnownLargeObj({urlstr.get(), static_cast(len)}); if (!known_large) { - DEBUG_LOG("Not a known large object, not slicing: %.*s", len, urlstr); + DEBUG_LOG("Not a known large object, not slicing: %.*s", len, urlstr.get()); return true; } diff --git a/src/api/InkAPI.cc b/src/api/InkAPI.cc index 1edd2f1bc2c..625358a40b0 100644 --- a/src/api/InkAPI.cc +++ b/src/api/InkAPI.cc @@ -4478,6 +4478,35 @@ TSHttpTxnCacheLookupUrlSet(TSHttpTxn txnp, TSMBuffer bufp, TSMLoc obj) return TS_SUCCESS; } +TSReturnCode +TSHttpTxnCacheKeyDigestGet(TSHttpTxn txnp, char *buffer, int *length) +{ + sdk_assert(sdk_sanity_check_txn(txnp) == TS_SUCCESS); + sdk_assert(length != nullptr); + + HttpSM *sm = reinterpret_cast(txnp); + const CryptoHash &hash = sm->get_cache_sm().get_cache_key().hash; + constexpr int size = CRYPTO_HASH_SIZE; + int provided_length = *length; + + *length = size; + + if (hash.is_zero()) { + return TS_ERROR; + } + + if (buffer == nullptr) { + return TS_SUCCESS; + } + + if (provided_length < size) { + return TS_ERROR; + } + + memcpy(buffer, hash.u8, size); + return TS_SUCCESS; +} + /** * timeout is in msec * overrides as proxy.config.http.transaction_active_timeout_out @@ -7068,11 +7097,13 @@ TSAIORead(int fd, off_t offset, char *buf, size_t buffSize, TSCont contp) pAIO->aiocb.aio_buf = buf; pAIO->action = pCont; pAIO->thread = pCont->mutex->thread_holding; + pAIO->from_ts_api = true; if (ink_aio_read(pAIO, 1) == 1) { return TS_SUCCESS; } + delete pAIO; return TS_ERROR; } @@ -7107,11 +7138,13 @@ TSAIOWrite(int fd, off_t offset, char *buf, const size_t bufSize, TSCont contp) pAIO->aiocb.aio_nbytes = bufSize; pAIO->action = pCont; pAIO->thread = pCont->mutex->thread_holding; + pAIO->from_ts_api = true; if (ink_aio_write(pAIO, 1) == 1) { return TS_SUCCESS; } + delete pAIO; return TS_ERROR; } @@ -7542,6 +7575,11 @@ TSHttpTxnConfigStringSet(TSHttpTxn txnp, TSOverridableConfigKey conf, const char s->t_state.my_txn_conf().ssl_client_ca_cert_filename = const_cast(value); } break; + case TS_CONFIG_SSL_CLIENT_CA_CERT_PATH: + if (value && length > 0) { + s->t_state.my_txn_conf().ssl_client_ca_cert_path = const_cast(value); + } + break; case TS_CONFIG_SSL_CLIENT_ALPN_PROTOCOLS: if (value && length > 0) { s->t_state.my_txn_conf().ssl_client_alpn_protocols = const_cast(value); @@ -7621,6 +7659,10 @@ TSHttpTxnConfigStringGet(TSHttpTxn txnp, TSOverridableConfigKey conf, const char *value = sm->t_state.txn_conf->server_session_sharing_match_str; *length = *value ? strlen(*value) : 0; break; + case TS_CONFIG_SSL_CLIENT_CA_CERT_PATH: + *value = sm->t_state.txn_conf->ssl_client_ca_cert_path; + *length = *value ? strlen(*value) : 0; + break; default: { MgmtConverter const *conv; const void *src = _conf_to_memberp(conf, sm->t_state.txn_conf, conv); diff --git a/src/cripts/Context.cc b/src/cripts/Context.cc index 76bcc822b2d..fe1a2ce389a 100644 --- a/src/cripts/Context.cc +++ b/src/cripts/Context.cc @@ -48,13 +48,12 @@ Context::reset() _server.request.Reset(); } - // Clear the initialized URLs before calling next hook - if (_urls.pristine.Initialized()) { - _urls.pristine.Reset(); - } - if (_urls.parent.Initialized()) { - _urls.parent.Reset(); - } + // Release lazy URLs entirely — they get recreated on demand for the next txn. + _urls.pristine.reset(); + _urls.parent.reset(); + _urls.remap.from.reset(); + _urls.remap.to.reset(); + if (_cache.url.Initialized()) { _cache.url.Reset(); } diff --git a/src/cripts/Error.cc b/src/cripts/Error.cc index 63935a52063..f93ea90b929 100644 --- a/src/cripts/Error.cc +++ b/src/cripts/Error.cc @@ -41,7 +41,10 @@ void Error::Reason::_set(cripts::Context *context, const cripts::string_view msg) { context->state.error.Fail(); - context->state.error._reason._setter(msg); + if (!context->state.error._reason) { + context->state.error._reason = std::make_unique(); + } + context->state.error._reason->_setter(msg); } // For convenience, an optional Reason message can also be specified with the status @@ -52,7 +55,10 @@ Error::Status::_set(cripts::Context *context, TSHttpStatus status, const cripts: context->state.error._status._setter(status); if (msg.size() > 0) { - context->state.error._reason._setter(msg); + if (!context->state.error._reason) { + context->state.error._reason = std::make_unique(); + } + context->state.error._reason->_setter(msg); } if (context->state.error.Redirected() || status == TS_HTTP_STATUS_MOVED_PERMANENTLY || diff --git a/src/cripts/Urls.cc b/src/cripts/Urls.cc index f47e49f371b..044b0c551a0 100644 --- a/src/cripts/Urls.cc +++ b/src/cripts/Urls.cc @@ -112,25 +112,27 @@ Url::Port::operator=(int port) cripts::string_view Url::Path::GetSV() { - if (_segments.size() > 0) { + if (_state && _state->segments.size() > 0) { std::ostringstream path; - std::ranges::copy(_segments, std::ostream_iterator(path, "/")); - _storage.reserve(_size); - _storage = std::string_view(path.str()); - if (_storage.size() > 0) { - _storage.pop_back(); // Removes the trailing / + std::ranges::copy(_state->segments, std::ostream_iterator(path, "/")); + _state->storage.reserve(_state->size); + _state->storage = std::string_view(path.str()); + if (_state->storage.size() > 0) { + _state->storage.pop_back(); // Removes the trailing / } - return {_storage}; + return {_state->storage}; } else if (_owner && _data.empty()) { const char *value = nullptr; int len = 0; _ensure_initialized(_owner); - value = TSUrlPathGet(_owner->_bufp, _owner->_urlp, &len); - _data = cripts::string_view(value, len); - _size = len; + value = TSUrlPathGet(_owner->_bufp, _owner->_urlp, &len); + _data = cripts::string_view(value, len); + if (_state) { + _state->size = len; + } _loaded = true; } @@ -144,14 +146,14 @@ Url::Path::operator[](Segments::size_type ix) _ensure_initialized(_owner); _parser(); // Make sure the segments are loaded - if (ix < _segments.size()) { - ret._initialize(_segments[ix], this, ix); + if (_state && ix < _state->segments.size()) { + ret._initialize(_state->segments[ix], this, ix); } return ret; // RVO } -Url::Path +Url::Path & Url::Path::operator=(cripts::string_view path) { _ensure_initialized(_owner); @@ -183,10 +185,11 @@ Url::Path::String::operator=(const cripts::string_view str) { _ensure_initialized(_owner->_owner); CAssert(!_owner->_owner->ReadOnly()); // This can not be a read-only URL - _owner->_size -= _owner->_segments[_ix].size(); - _owner->_segments[_ix] = str; - _owner->_size += str.size(); - _owner->_modified = true; + CAssert(_owner->_state); // Should have been allocated by operator[]/_parser() + _owner->_state->size -= _owner->_state->segments[_ix].size(); + _owner->_state->segments[_ix] = str; + _owner->_state->size += str.size(); + _owner->_state->modified = true; return *this; } @@ -196,51 +199,53 @@ Url::Path::Reset() { Component::Reset(); - _segments.clear(); - _storage.clear(); - _size = 0; - _modified = false; + _state.reset(); } void Url::Path::Push(cripts::string_view val) { _parser(); - _modified = true; - _segments.push_back(val); + auto &s = _ensure_state(); + s.modified = true; + s.segments.push_back(val); } void Url::Path::Insert(Segments::size_type ix, cripts::string_view val) { _parser(); - _modified = true; - _segments.insert(_segments.begin() + ix, val); + auto &s = _ensure_state(); + s.modified = true; + s.segments.insert(s.segments.begin() + ix, val); } void Url::Path::_parser() { - if (_segments.size() == 0) { - _segments = Split('/'); + auto &s = _ensure_state(); + + if (s.segments.size() == 0) { + s.segments = Split('/'); } } Url::Query::Parameter & Url::Query::Parameter::operator=(const cripts::string_view str) { - CAssert(!_owner->_standalone); + CAssert(!_owner->_state || !_owner->_state->standalone); _ensure_initialized(_owner->_owner); CAssert(!_owner->_owner->ReadOnly()); // This can not be a read-only URL - auto iter = _owner->_hashed.find(_name); + auto &s = _owner->_ensure_state(); + auto iter = s.hashed.find(_name); - if (iter != _owner->_hashed.end()) { + if (iter != s.hashed.end()) { iter->second = str; // Can be an empty string here! } else { - _owner->_ordered.push_back(_name); - _owner->_hashed[_name] = str; + s.ordered.push_back(_name); + s.hashed[_name] = str; } - _owner->_modified = true; + s.modified = true; return *this; } @@ -248,31 +253,31 @@ Url::Query::Parameter::operator=(const cripts::string_view str) cripts::string_view Url::Query::GetSV() { - if (!_standalone) { + if (!_state || !_state->standalone) { _ensure_initialized(_owner); } - if (_ordered.size() > 0) { - _storage.clear(); - _storage.reserve(_size); + if (_state && _state->ordered.size() > 0) { + _state->storage.clear(); + _state->storage.reserve(_state->size); // ToDo: This is wonky, has to be a better std:: iteration to do here - for (const auto key : _ordered) { - auto iter = _hashed.find(key); + for (const auto key : _state->ordered) { + auto iter = _state->hashed.find(key); - if (_storage.size() > 0) { - _storage += "&"; + if (_state->storage.size() > 0) { + _state->storage += "&"; } - if (iter != _hashed.end()) { - _storage += iter->first; + if (iter != _state->hashed.end()) { + _state->storage += iter->first; if (iter->second.size() > 0) { - _storage += '='; - _storage += iter->second; + _state->storage += '='; + _state->storage += iter->second; } } } - return {_storage}; + return {_state->storage}; } // This gets weird when we modify the query parameter components, and can possibly empty @@ -282,19 +287,21 @@ Url::Query::GetSV() const char *value = nullptr; int len = 0; - value = TSUrlHttpQueryGet(_owner->_bufp, _owner->_urlp, &len); - _data = cripts::string_view(value, len); - _size = len; + value = TSUrlHttpQueryGet(_owner->_bufp, _owner->_urlp, &len); + _data = cripts::string_view(value, len); + if (_state) { + _state->size = len; + } _loaded = true; } return _data; } -Url::Query +Url::Query & Url::Query::operator=(cripts::string_view query) { - CAssert(!_standalone); + CAssert(!_state || !_state->standalone); _ensure_initialized(_owner); CAssert(!_owner->ReadOnly()); // This can not be a read-only URL TSUrlHttpQuerySet(_owner->_bufp, _owner->_urlp, query.data(), query.size()); @@ -323,15 +330,15 @@ Url::Query::Parameter Url::Query::operator[](cripts::string_view param) { // Make sure the hash and vector are populated, but only if we have an owner - if (!_standalone) { + if (!_state || !_state->standalone) { _ensure_initialized(_owner); } _parser(); Parameter ret; - auto iter = _hashed.find(param); + auto iter = _state->hashed.find(param); - if (iter != _hashed.end()) { + if (iter != _state->hashed.end()) { ret._initialize(iter->first, iter->second, this); } else { ret._initialize(param, "", this); @@ -346,21 +353,25 @@ Url::Query::Erase(cripts::string_view param) // Make sure the hash and vector are populated _parser(); - auto iter = _hashed.find(param); - auto viter = std::ranges::find(_ordered, param); + auto &s = *_state; + auto iter = s.hashed.find(param); + auto viter = std::ranges::find(s.ordered, param); - if (iter != _hashed.end()) { - _size -= iter->second.size(); // Size of the erased value - _hashed.erase(iter); + if (iter != s.hashed.end()) { + s.size -= iter->second.size(); // Size of the erased value + s.hashed.erase(iter); - CAssert(viter != _ordered.end()); - _size -= viter->size(); // Length of the erased key - _ordered.erase(viter); + CAssert(viter != s.ordered.end()); + s.size -= viter->size(); // Length of the erased key + s.ordered.erase(viter); - if (_ordered.size() == 0) { + if (s.ordered.size() == 0) { Reset(); + // After Reset() the state is gone; re-create it just so _modified can be tracked. + _ensure_state().modified = true; + } else { + s.modified = true; } - _modified = true; // Make sure to set this after we reset above ... } } @@ -371,21 +382,23 @@ Url::Query::Erase(std::initializer_list list, bool keep) // Make sure the hash and vector are populated _parser(); - for (auto viter = _ordered.begin(); viter != _ordered.end();) { + auto &s = *_state; + + for (auto viter = s.ordered.begin(); viter != s.ordered.end();) { if (list.end() == std::ranges::find(list, *viter)) { - auto iter = _hashed.find(*viter); - - CAssert(iter != _hashed.end()); - _size -= iter->second.size(); // Size of the erased value - _size -= viter->size(); // Length of the erased key - _hashed.erase(iter); - viter = _ordered.erase(viter); - _modified = true; + auto iter = s.hashed.find(*viter); + + CAssert(iter != s.hashed.end()); + s.size -= iter->second.size(); // Size of the erased value + s.size -= viter->size(); // Length of the erased key + s.hashed.erase(iter); + viter = s.ordered.erase(viter); + s.modified = true; } else { ++viter; } } - if (_ordered.size() == 0) { + if (s.ordered.size() == 0) { Reset(); } } else { @@ -400,17 +413,15 @@ Url::Query::Reset() { Component::Reset(); - _ordered.clear(); - _hashed.clear(); - _storage.clear(); - _size = 0; - _modified = false; + _state.reset(); } void Url::Query::_parser() { - if (_ordered.size() == 0) { + auto &s = _ensure_state(); + + if (s.ordered.size() == 0) { for (const auto sv : Split('&')) { const auto eq = sv.find_first_of('='); cripts::string_view key = sv.substr(0, eq); @@ -420,8 +431,8 @@ Url::Query::_parser() val = sv.substr(eq + 1); } - _ordered.push_back(key); // Keep the order - _hashed[key] = val; + s.ordered.push_back(key); // Keep the order + s.hashed[key] = val; } } } @@ -447,17 +458,21 @@ Url::String() Pristine::URL & Pristine::URL::_get(cripts::Context *context) { - _ensure_initialized(&context->_urls.pristine); - return context->_urls.pristine; + auto &slot = context->_urls.pristine; + + if (!slot) { + slot = std::make_unique(); + slot->set_context(context); + } + _ensure_initialized(slot.get()); + return *slot; } void Pristine::URL::_initialize() { - Pristine::URL *url = &_context->_urls.pristine; - TSAssert(_context->state.txnp); - if (TSHttpTxnPristineUrlGet(_context->state.txnp, &url->_bufp, &url->_urlp) != TS_SUCCESS) { + if (TSHttpTxnPristineUrlGet(_context->state.txnp, &_bufp, &_urlp) != TS_SUCCESS) { _context->state.error.Fail(); } else { super_type::_initialize(); // Only if successful @@ -519,8 +534,14 @@ Remap::From::URL::_initialize() Remap::From::URL & Remap::From::URL::_get(cripts::Context *context) { - _ensure_initialized(&context->_urls.remap.from); - return context->_urls.remap.from; + auto &slot = context->_urls.remap.from; + + if (!slot) { + slot = std::make_unique(); + slot->set_context(context); + } + _ensure_initialized(slot.get()); + return *slot; } void @@ -539,8 +560,14 @@ Remap::To::URL::_initialize() Remap::To::URL & Remap::To::URL::_get(cripts::Context *context) { - _ensure_initialized(&context->_urls.remap.to); - return context->_urls.remap.to; + auto &slot = context->_urls.remap.to; + + if (!slot) { + slot = std::make_unique(); + slot->set_context(context); + } + _ensure_initialized(slot.get()); + return *slot; } Cache::URL & @@ -623,19 +650,24 @@ Cache::URL::_update() Parent::URL & Parent::URL::_get(cripts::Context *context) { - _ensure_initialized(&context->_urls.parent); - return context->_urls.parent; + auto &slot = context->_urls.parent; + + if (!slot) { + slot = std::make_unique(); + slot->set_context(context); + } + _ensure_initialized(slot.get()); + return *slot; } void Parent::URL::_initialize() { - Parent::URL *url = &_context->_urls.parent; Client::Request &req = Client::Request::_get(_context); // Repurpose / create the shared request object - if (TSUrlCreate(req.BufP(), &url->_urlp) == TS_SUCCESS) { + if (TSUrlCreate(req.BufP(), &_urlp) == TS_SUCCESS) { TSAssert(_context->state.txnp); - if (TSHttpTxnParentSelectionUrlGet(_context->state.txnp, req.BufP(), url->_urlp) != TS_SUCCESS) { + if (TSHttpTxnParentSelectionUrlGet(_context->state.txnp, req.BufP(), _urlp) != TS_SUCCESS) { _context->state.error.Fail(); return; } diff --git a/src/iocore/aio/AIO.cc b/src/iocore/aio/AIO.cc index 0f69f51d3ed..1a4fb4fe9bf 100644 --- a/src/iocore/aio/AIO.cc +++ b/src/iocore/aio/AIO.cc @@ -85,6 +85,11 @@ AIOCallback::io_complete(int event, void *data) { (void)event; (void)data; + // Store from_api's value ahead of time because handling the event in the + // midst of this function may delete `this`, making using from_api itself a + // use-after-free later. + bool const should_delete_self = from_ts_api; + if (aio_err_callback && !ok()) { AIOCallback *err_op = new AIOCallback(); err_op->aiocb.aio_fildes = this->aiocb.aio_fildes; @@ -99,6 +104,9 @@ AIOCallback::io_complete(int event, void *data) if (!action.cancelled && action.continuation) { action.continuation->handleEvent(AIO_EVENT_DONE, this); } + if (should_delete_self) { + delete this; + } return EVENT_DONE; } @@ -109,7 +117,6 @@ struct AIO_Reqs { ASLL(AIOCallback, alink) aio_temp_list; ink_mutex aio_mutex; ink_cond aio_cond; - int index = 0; /* position of this struct in the aio_reqs array */ int pending = 0; /* number of outstanding requests on the disk */ int queued = 0; /* total number of aio_todo requests */ int filedes = -1; /* the file descriptor for the requests or status IO_NOT_IN_PROGRESS */ @@ -298,13 +305,11 @@ aio_init_fildes(int fildes, int fromAPI = 0) RecInt thread_num; if (fromAPI) { - request->index = 0; request->filedes = -1; aio_reqs[0] = request; thread_is_created = 1; thread_num = api_config_threads_per_disk; } else { - request->index = num_filedes; request->filedes = fildes; aio_reqs[num_filedes] = request; thread_num = cache_config_threads_per_disk; diff --git a/src/iocore/aio/CMakeLists.txt b/src/iocore/aio/CMakeLists.txt index 4f6f4bc0b8b..e6f342b1430 100644 --- a/src/iocore/aio/CMakeLists.txt +++ b/src/iocore/aio/CMakeLists.txt @@ -33,6 +33,10 @@ if(BUILD_TESTING) COMMAND $ WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/src/iocore/aio ) + + add_executable(test_AIOCallback unit_tests/test_AIOCallback.cc) + target_link_libraries(test_AIOCallback PRIVATE ts::aio Catch2::Catch2WithMain) + add_catch2_test(NAME test_AIOCallback COMMAND test_AIOCallback) endif() clang_tidy_check(aio) diff --git a/src/iocore/aio/test_AIO.cc b/src/iocore/aio/test_AIO.cc index 0cff8b35c45..3ae808bbd17 100644 --- a/src/iocore/aio/test_AIO.cc +++ b/src/iocore/aio/test_AIO.cc @@ -93,23 +93,19 @@ int seq_write_size = 0; int rand_read_size = 0; struct AIO_Device : public Continuation { - char *path; - int fd; - int id; - char *buf; - ink_hrtime time_start, time_end; - int seq_reads; - int seq_writes; - int rand_reads; - int hotset_idx; - int mode; + char *path{nullptr}; + int fd{-1}; + int id{0}; + char *buf{nullptr}; + ink_hrtime time_start{0}; + ink_hrtime time_end{0}; + int seq_reads{0}; + int seq_writes{0}; + int rand_reads{0}; + int hotset_idx{0}; + int mode{0}; std::unique_ptr io; - AIO_Device(ProxyMutex *m) : Continuation(m), io{new_AIOCallback()} - { - hotset_idx = 0; - time_start = 0; - SET_HANDLER(&AIO_Device::do_hotset); - } + AIO_Device(ProxyMutex *m) : Continuation(m), io{new_AIOCallback()} { SET_HANDLER(&AIO_Device::do_hotset); } int select_mode(double p) { diff --git a/src/iocore/aio/unit_tests/test_AIOCallback.cc b/src/iocore/aio/unit_tests/test_AIOCallback.cc new file mode 100644 index 00000000000..39fc1dede17 --- /dev/null +++ b/src/iocore/aio/unit_tests/test_AIOCallback.cc @@ -0,0 +1,118 @@ +/** @file + + Catch based unit tests for AIOCallback. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include + +#include + +#include "iocore/aio/AIO.h" +#include "iocore/eventsystem/Event.h" + +namespace +{ + +struct AIOCompletionOwner : Continuation { + std::unique_ptr callback; + bool *completed = nullptr; + + explicit AIOCompletionOwner(bool &completion_flag) + : Continuation(nullptr), callback(std::make_unique()), completed(&completion_flag) + { + SET_HANDLER(&AIOCompletionOwner::handle_aio_complete); + + callback->action = this; + callback->aiocb.aio_nbytes = 0; + callback->aio_result = 0; + } + + int + handle_aio_complete(int event, void *data) + { + CHECK(event == AIO_EVENT_DONE); + CHECK(data == callback.get()); + + *completed = true; + callback.reset(); + return EVENT_DONE; + } +}; + +struct AIOCompletionHandler : Continuation { + AIOCallback *expected = nullptr; + bool *completed = nullptr; + + explicit AIOCompletionHandler(bool &completion_flag) : Continuation(nullptr), completed(&completion_flag) + { + SET_HANDLER(&AIOCompletionHandler::handle_aio_complete); + } + + int + handle_aio_complete(int event, void *data) + { + CHECK(event == AIO_EVENT_DONE); + CHECK(data == expected); + + *completed = true; + return EVENT_DONE; + } +}; + +struct DeletionTrackedAIOCallback : AIOCallback { + bool *destroyed = nullptr; + + explicit DeletionTrackedAIOCallback(bool &destroyed_flag) : destroyed(&destroyed_flag) {} + + ~DeletionTrackedAIOCallback() override { *destroyed = true; } +}; + +} // namespace + +TEST_CASE("AIOCallback completion tolerates callback deletion", "[iocore][aio]") +{ + bool completed = false; + AIOCompletionOwner owner(completed); + auto *callback = owner.callback.get(); + + // Without ASan, a broken implementation can still pass because the stale value of from_ts_api is typically false. + CHECK(callback->io_complete(EVENT_NONE, nullptr) == EVENT_DONE); + CHECK(completed); +} + +TEST_CASE("API AIOCallback completion deletes the callback after dispatch", "[iocore][aio]") +{ + bool completed = false; + bool destroyed = false; + + AIOCompletionHandler handler(completed); + auto *callback = new DeletionTrackedAIOCallback(destroyed); + + handler.expected = callback; + callback->action = &handler; + callback->from_ts_api = true; + callback->aiocb.aio_nbytes = 0; + callback->aio_result = 0; + + CHECK(callback->io_complete(EVENT_NONE, nullptr) == EVENT_DONE); + CHECK(completed); + CHECK(destroyed); +} diff --git a/src/iocore/cache/CacheDir.cc b/src/iocore/cache/CacheDir.cc index 413d11c6219..ee14d521ff3 100644 --- a/src/iocore/cache/CacheDir.cc +++ b/src/iocore/cache/CacheDir.cc @@ -33,6 +33,7 @@ #include "ts/ats_probe.h" #include "iocore/eventsystem/Tasks.h" +#include #include #ifdef LOOP_CHECK_MODE @@ -123,7 +124,6 @@ OpenDir::signal_readers(int /* event ATS_UNUSED */, Event * /* e ATS_UNUSED */) while ((c = delayed_readers.dequeue())) { CACHE_TRY_LOCK(lock, c->mutex, t); if (lock.is_locked()) { - c->f.open_read_timeout = 0; c->handleEvent(EVENT_IMMEDIATE, nullptr); continue; } @@ -958,20 +958,45 @@ Directory::entries_used() } /* - * this function flushes the cache meta data to disk when + * This function flushes the cache meta data to disk when * the cache is shutdown. Must *NOT* be used during regular - * operation. + * operation. Stripes are synced in parallel, one thread per + * physical disk. */ void sync_cache_dir_on_shutdown() { - Dbg(dbg_ctl_cache_dir_sync, "sync started"); - EThread *t = reinterpret_cast(0xdeadbeef); + Dbg(dbg_ctl_cache_dir_sync, "shutdown sync started"); + + std::unordered_map> drive_stripe_map; + for (int i = 0; i < gnstripes; i++) { - gstripes[i]->shutdown(t); + drive_stripe_map[gstripes[i]->disk].push_back(i); + } + + std::vector threads; + + threads.reserve(drive_stripe_map.size()); + for (auto &[disk, indices] : drive_stripe_map) { + Dbg(dbg_ctl_cache_dir_sync, "Disk %s: syncing %zu stripe(s)", disk->path, indices.size()); + auto stripe_indices = indices; + threads.emplace_back([stripe_indices]() { + // Use a thread_local variable to give each OS thread a unique EThread* sentinel instead of 0xdeadbeef. + thread_local char thread_sentinel; + EThread *t = reinterpret_cast(&thread_sentinel); + + for (int idx : stripe_indices) { + gstripes[idx]->shutdown(t); + } + }); } - Dbg(dbg_ctl_cache_dir_sync, "sync done"); + + for (auto &thr : threads) { + thr.join(); + } + + Dbg(dbg_ctl_cache_dir_sync, "shutdown sync done"); } int diff --git a/src/iocore/cache/CacheVC.h b/src/iocore/cache/CacheVC.h index 3ca74ce650b..bfd2d39b225 100644 --- a/src/iocore/cache/CacheVC.h +++ b/src/iocore/cache/CacheVC.h @@ -289,26 +289,26 @@ struct CacheVC : public CacheVConnection { union { uint32_t flags; struct { - unsigned int use_first_key : 1; - unsigned int overwrite : 1; // overwrite first_key Dir if it exists - unsigned int close_complete : 1; // WRITE_COMPLETE is final - unsigned int sync : 1; // write to be committed to durable storage before WRITE_COMPLETE - unsigned int evacuator : 1; - unsigned int single_fragment : 1; - unsigned int evac_vector : 1; - unsigned int lookup : 1; - unsigned int update : 1; - unsigned int remove : 1; - unsigned int remove_aborted_writers : 1; - unsigned int open_read_timeout : 1; // UNUSED - unsigned int data_done : 1; - unsigned int read_from_writer_called : 1; - unsigned int rewrite_resident_alt : 1; - unsigned int readers : 1; - unsigned int doc_from_ram_cache : 1; - unsigned int hit_evacuate : 1; - unsigned int compressed_in_ram : 1; // compressed state in ram cache - unsigned int allow_empty_doc : 1; // used for cache empty http document + unsigned int use_first_key : 1; + unsigned int overwrite : 1; // overwrite first_key Dir if it exists + unsigned int close_complete : 1; // WRITE_COMPLETE is final + unsigned int sync : 1; // write to be committed to durable storage before WRITE_COMPLETE + unsigned int evacuator : 1; + unsigned int single_fragment : 1; + unsigned int evac_vector : 1; + unsigned int lookup : 1; + unsigned int update : 1; + unsigned int remove : 1; + unsigned int remove_aborted_writers : 1; + unsigned int unused_open_read_timeout : 1; // UNUSED, reserved for bitfield layout + unsigned int data_done : 1; + unsigned int read_from_writer_called : 1; + unsigned int rewrite_resident_alt : 1; + unsigned int readers : 1; + unsigned int doc_from_ram_cache : 1; + unsigned int hit_evacuate : 1; + unsigned int compressed_in_ram : 1; // compressed state in ram cache + unsigned int allow_empty_doc : 1; // used for cache empty http document } f; }; // BTF optimization used to skip reading stuff in cache partition that doesn't contain any diff --git a/src/iocore/hostdb/CMakeLists.txt b/src/iocore/hostdb/CMakeLists.txt index 7cc9ada3248..3920522bd67 100644 --- a/src/iocore/hostdb/CMakeLists.txt +++ b/src/iocore/hostdb/CMakeLists.txt @@ -23,24 +23,5 @@ target_link_libraries(inkhostdb PUBLIC ts::inkdns ts::inkevent ts::tscore) clang_tidy_check(inkhostdb) if(BUILD_TESTING) - add_executable(benchmark_HostDB benchmark_HostDB.cc) - target_link_libraries( - benchmark_HostDB - PRIVATE ts::tscore - ts::tsutil - ts::inkevent - ts::http - ts::http_remap - ts::inkcache - ts::inkhostdb - ) - - add_executable(test_HostFile test_HostFile.cc HostFile.cc HostDBInfo.cc) - target_link_libraries(test_HostFile PRIVATE ts::tscore ts::tsutil ts::inkevent Catch2::Catch2WithMain) - add_catch2_test(NAME test_hostdb_HostFile COMMAND $) - - add_executable(test_RefCountCache test_RefCountCache.cc) - target_link_libraries(test_RefCountCache PRIVATE ts::tscore ts::tsutil ts::inkevent Catch2::Catch2WithMain) - add_catch2_test(NAME test_hostdb_RefCountCache COMMAND $) - + add_subdirectory(unit_tests) endif() diff --git a/src/iocore/hostdb/HostDB.cc b/src/iocore/hostdb/HostDB.cc index 1c0289f1497..f225aeae1cf 100644 --- a/src/iocore/hostdb/HostDB.cc +++ b/src/iocore/hostdb/HostDB.cc @@ -1317,14 +1317,14 @@ HostDBRecord::select_best_http(ts_time now, ts_seconds fail_window, sockaddr con // Starting at the current target, search for a valid one. for (unsigned short i = 0; i < rr_count; i++) { auto target = &info[this->rr_idx(i)]; - if (target->select(now, fail_window)) { + if (!target->is_down(now, fail_window)) { best_alive = target; break; } } } } else { - if (info[0].select(now, fail_window)) { + if (!info[0].is_down(now, fail_window)) { best_alive = &info[0]; } } @@ -1647,7 +1647,7 @@ HostDBRecord::select_next_rr(ts_time now, ts_seconds fail_window) auto rr_info = this->rr_info(); for (unsigned idx = 0, limit = rr_info.count(); idx < limit; ++idx) { auto &target = rr_info[this->next_rr()]; - if (target.select(now, fail_window)) { + if (!target.is_down(now, fail_window)) { return ⌖ } } @@ -1711,7 +1711,7 @@ ResolveInfo::select_next_rr() if (active) { if (auto rr_info{this->record->rr_info()}; rr_info.count() > 1) { unsigned limit = active - rr_info.data(), idx = (limit + 1) % rr_info.count(); - while ((idx = (idx + 1) % rr_info.count()) != limit && !rr_info[idx].is_alive()) {} + while ((idx = (idx + 1) % rr_info.count()) != limit && !rr_info[idx].is_up()) {} active = &rr_info[idx]; return idx != limit; // if the active record was actually changed. } diff --git a/src/iocore/hostdb/HostDBInfo.cc b/src/iocore/hostdb/HostDBInfo.cc index db76345a162..1bb4126359b 100644 --- a/src/iocore/hostdb/HostDBInfo.cc +++ b/src/iocore/hostdb/HostDBInfo.cc @@ -27,6 +27,8 @@ namespace { +DbgCtl dbg_ctl_hostdb_info{"hostdb_info"}; + /** Assign raw storage to an @c IpAddr * * @param ip Destination. @@ -95,3 +97,152 @@ HostDBInfo::srvname() const { return data.srv.srv_offset ? reinterpret_cast(this) + data.srv.srv_offset : nullptr; } + +HostDBInfo & +HostDBInfo::operator=(HostDBInfo const &that) +{ + if (this != &that) { + memcpy(static_cast(this), static_cast(&that), sizeof(*this)); + } + return *this; +} + +ts_time +HostDBInfo::last_fail_time() const +{ + return _last_failure; +} + +uint8_t +HostDBInfo::fail_count() const +{ + return _fail_count; +} + +HostDBInfo::State +HostDBInfo::state(ts_time now, ts_seconds fail_window) const +{ + auto last_fail = this->last_fail_time(); + if (last_fail == TS_TIME_ZERO) { + return State::UP; + } + + if (now <= last_fail + fail_window) { + return State::DOWN; + } else { + return State::SUSPECT; + } +} + +bool +HostDBInfo::is_up() const +{ + return this->last_fail_time() == TS_TIME_ZERO; +} + +bool +HostDBInfo::is_down(ts_time now, ts_seconds fail_window) const +{ + return this->state(now, fail_window) == State::DOWN; +} + +bool +HostDBInfo::is_suspect(ts_time now, ts_seconds fail_window) const +{ + return this->state(now, fail_window) == State::SUSPECT; +} + +/** Mark the target as UP + * + * @return @c true if the target was previously DOWN or SUSPECT (i.e., a state change occurred). + */ +bool +HostDBInfo::mark_up() +{ + auto t = _last_failure.exchange(TS_TIME_ZERO); + _fail_count.store(0); + + return t != TS_TIME_ZERO; +} + +/** Mark the entry as DOWN. + * + * @param[in] now Time of the failure. + * @param[in] fail_window The fail window duration (proxy.config.http.down_server.cache_time). + * @return @c true if @a this was marked down, @c false if not. + * + * Handles two transitions: + * - UP → DOWN: @c _last_failure is @c TS_TIME_ZERO; set via CAS. + * - SUSPECT → DOWN: @c fail_window has elapsed since the last failure; @c _last_failure is + * refreshed via CAS to restart the fail window. + * + * On a successful transition @c _fail_count is reset to zero so that the next SUSPECT window + * accumulates failures from a clean baseline. + * + * Returns @c false if the server is already DOWN (within the active fail window), so the + * fail window is not refreshed by concurrent failures. + */ +bool +HostDBInfo::mark_down(ts_time now, ts_seconds fail_window) +{ + // UP -> DOWN + auto t0{TS_TIME_ZERO}; + if (_last_failure.compare_exchange_strong(t0, now)) { + // Reset so the next SUSPECT window starts with a fresh failure count. + _fail_count.store(0); + return true; + } + + // After the failed CAS, t0 holds the current _last_failure value. + // SUSPECT -> DOWN: the fail window has elapsed; refresh _last_failure to restart it. + if (t0 + fail_window < now) { + if (_last_failure.compare_exchange_strong(t0, now)) { + // Reset so the next SUSPECT window starts with a fresh failure count. + _fail_count.store(0); + return true; + } + } + + // Already DOWN; don't refresh the fail window. + return false; +} + +/** Increment the connection failure counter and conditionally mark the target DOWN. + * + * @param[in] now Current time, used as the failure timestamp if the target is marked DOWN. + * @param[in] max_retries Number of failures that triggers a transition to DOWN. + * @param[in] fail_window The fail window duration (proxy.config.http.down_server.cache_time). + * @return A pair { @c marked_down, @c fail_count } where @c marked_down is @c true if this call + * caused the target to transition to DOWN (i.e. @c fail_count just reached @a max_retries + * and the @c mark_down CAS succeeded), and @c fail_count is the updated counter value. + * + * @note @c marked_down can be @c false even when @c fail_count >= @a max_retries if another + * thread concurrently marked the target DOWN first (the CAS on @c _last_failure will fail). + */ +std::pair +HostDBInfo::increment_fail_count(ts_time now, uint8_t max_retries, ts_seconds fail_window) +{ + auto fcount = ++_fail_count; + bool marked_down = false; + + Dbg(dbg_ctl_hostdb_info, "fail_count=%d max_retries=%d", fcount, max_retries); + + if (fcount >= max_retries) { + marked_down = mark_down(now, fail_window); + } + return std::make_pair(marked_down, fcount); +} + +/** Migrate data after a DNS update. + * + * @param[in] that Source item. + * + * This moves only specific state information, it is not a generic copy. + */ +void +HostDBInfo::migrate_from(HostDBInfo::self_type const &that) +{ + this->_last_failure = that._last_failure.load(); + this->_fail_count = that._fail_count.load(); + this->http_version = that.http_version; +} diff --git a/src/iocore/hostdb/unit_tests/CMakeLists.txt b/src/iocore/hostdb/unit_tests/CMakeLists.txt new file mode 100644 index 00000000000..bd38fc884ec --- /dev/null +++ b/src/iocore/hostdb/unit_tests/CMakeLists.txt @@ -0,0 +1,38 @@ +###################### +# +# Licensed to the Apache Software Foundation (ASF) under one or more contributor license +# agreements. See the NOTICE file distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed under the License +# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express +# or implied. See the License for the specific language governing permissions and limitations under +# the License. +# +###################### + +# test_hostdb +add_executable(test_hostdb test_HostDBInfo.cc "${CMAKE_CURRENT_SOURCE_DIR}/../HostDBInfo.cc") +target_link_libraries(test_hostdb PRIVATE Catch2::Catch2WithMain ts::tscore ts::tsutil ts::inkevent configmanager) +add_catch2_test(NAME test_hostdb COMMAND $) + +# test_HostFile +add_executable( + test_HostFile test_HostFile.cc "${CMAKE_CURRENT_SOURCE_DIR}/../HostFile.cc" + "${CMAKE_CURRENT_SOURCE_DIR}/../HostDBInfo.cc" +) +target_include_directories(test_HostFile PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/..") +target_link_libraries(test_HostFile PRIVATE ts::tscore ts::tsutil ts::inkevent configmanager Catch2::Catch2WithMain) +add_catch2_test(NAME test_hostdb_HostFile COMMAND $) + +# test_RefCountCache +add_executable(test_RefCountCache test_RefCountCache.cc) +target_include_directories(test_RefCountCache PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/..") +target_link_libraries( + test_RefCountCache PRIVATE ts::tscore ts::tsutil ts::inkevent configmanager Catch2::Catch2WithMain +) +add_catch2_test(NAME test_hostdb_RefCountCache COMMAND $) diff --git a/src/iocore/hostdb/unit_tests/test_HostDBInfo.cc b/src/iocore/hostdb/unit_tests/test_HostDBInfo.cc new file mode 100644 index 00000000000..431ef1fa1f0 --- /dev/null +++ b/src/iocore/hostdb/unit_tests/test_HostDBInfo.cc @@ -0,0 +1,214 @@ +/** @file + + Unit tests for HostDBInfo state transitions (UP / DOWN / SUSPECT). + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include + +#include "iocore/hostdb/HostDBProcessor.h" +#include "tscore/ink_time.h" + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +// fail_window used throughout the tests +static const ts_seconds FAIL_WINDOW{300}; + +static const ts_time T0 = ts_clock::now(); ///< A fixed anchor in time; + +static const ts_time T1 = T0 + ts_seconds(1); ///< A time within FAIL_WINDOW from T0 +static const ts_time T2 = T0 + FAIL_WINDOW + ts_seconds(1); ///< A time past FAIL_WINDOW from T0 + +static const ts_time T3 = T2 + ts_seconds(1); ///< A time within FAIL_WINDOW from T2 +static const ts_time T4 = T2 + FAIL_WINDOW + ts_seconds(1); ///< A time past FAIL_WINDOW from T2 +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +TEST_CASE("HostDBInfo state", "[hostdb]") +{ + SECTION("initial state is UP") + { + HostDBInfo info; + REQUIRE(info.state(T0, FAIL_WINDOW) == HostDBInfo::State::UP); + REQUIRE(info.is_up()); + REQUIRE(info.last_fail_time() == TS_TIME_ZERO); + REQUIRE(info.fail_count() == 0); + } + + SECTION("UP -> DOWN via mark_down; fail_count is reset") + { + HostDBInfo info; + REQUIRE(info.mark_down(T0, FAIL_WINDOW) == true); + REQUIRE(info.last_fail_time() == T0); + REQUIRE(info.fail_count() == 0); // reset so the next SUSPECT window starts fresh + REQUIRE(info.state(T1, FAIL_WINDOW) == HostDBInfo::State::DOWN); + } + + SECTION("mark_down on DOWN server returns false and does not refresh fail window") + { + HostDBInfo info; + info.mark_down(T0, FAIL_WINDOW); + REQUIRE(info.mark_down(T1, FAIL_WINDOW) == false); + REQUIRE(info.last_fail_time() == T0); // unchanged + } + + SECTION("DOWN -> SUSPECT when fail window elapses (time-based, no explicit call)") + { + HostDBInfo info; + info.mark_down(T0, FAIL_WINDOW); + REQUIRE(info.state(T2, FAIL_WINDOW) == HostDBInfo::State::SUSPECT); + REQUIRE(info.is_suspect(T2, FAIL_WINDOW)); + REQUIRE(info.last_fail_time() == T0); // _last_failure is unchanged + } + + SECTION("SUSPECT -> DOWN via mark_down; fail_count is reset; fail window restarts") + { + HostDBInfo info; + info.mark_down(T0, FAIL_WINDOW); + + REQUIRE(info.mark_down(T2, FAIL_WINDOW) == true); + REQUIRE(info.last_fail_time() == T2); // refreshed to the new failure time + REQUIRE(info.fail_count() == 0); // reset so the next SUSPECT window starts fresh + // server is DOWN again with a fresh window anchored at T2 + REQUIRE(info.state(T2 + FAIL_WINDOW / 2, FAIL_WINDOW) == HostDBInfo::State::DOWN); + } + + SECTION("SUSPECT -> UP via mark_up; returns true (was DOWN/SUSPECT)") + { + HostDBInfo info; + info.mark_down(T0, FAIL_WINDOW); + REQUIRE(info.mark_up() == true); + REQUIRE(info.state(T0, FAIL_WINDOW) == HostDBInfo::State::UP); + REQUIRE(info.last_fail_time() == TS_TIME_ZERO); + REQUIRE(info.fail_count() == 0); + } + + SECTION("mark_up on already-UP server returns false") + { + HostDBInfo info; + REQUIRE(info.mark_up() == false); + } + + SECTION("UP -> DOWN via increment_fail_count with max_retries=1") + { + HostDBInfo info; + auto [marked, count] = info.increment_fail_count(T0, 1, FAIL_WINDOW); + REQUIRE(marked == true); + REQUIRE(count == 1); + REQUIRE(info.state(T1, FAIL_WINDOW) == HostDBInfo::State::DOWN); + REQUIRE(info.fail_count() == 0); // reset by mark_down + } + + SECTION("UP -> DOWN via increment_fail_count with max_retries=3 requires 3 failures") + { + HostDBInfo info; + { + auto [marked, count] = info.increment_fail_count(T0, 3, FAIL_WINDOW); + REQUIRE(marked == false); + REQUIRE(count == 1); + } + { + auto [marked, count] = info.increment_fail_count(T0, 3, FAIL_WINDOW); + REQUIRE(marked == false); + REQUIRE(count == 2); + } + { + auto [marked, count] = info.increment_fail_count(T0, 3, FAIL_WINDOW); + REQUIRE(marked == true); + REQUIRE(count == 3); + } + REQUIRE(info.state(T1, FAIL_WINDOW) == HostDBInfo::State::DOWN); + REQUIRE(info.fail_count() == 0); // reset by mark_down + } + + SECTION("SUSPECT -> DOWN via increment_fail_count; fail_count starts from zero each SUSPECT window") + { + HostDBInfo info; + // Drive UP -> DOWN (R=1); fail_count is reset to 0 after mark_down. + info.increment_fail_count(T0, 1, FAIL_WINDOW); + REQUIRE(info.fail_count() == 0); + + // D=3: need exactly 3 fresh failures in the SUSPECT window. + { + auto [marked, count] = info.increment_fail_count(T2, 3, FAIL_WINDOW); + REQUIRE(marked == false); + REQUIRE(count == 1); + } + { + auto [marked, count] = info.increment_fail_count(T2, 3, FAIL_WINDOW); + REQUIRE(marked == false); + REQUIRE(count == 2); + } + { + auto [marked, count] = info.increment_fail_count(T2, 3, FAIL_WINDOW); + REQUIRE(marked == true); + REQUIRE(count == 3); + } + REQUIRE(info.last_fail_time() == T2); + REQUIRE(info.fail_count() == 0); // reset by mark_down SUSPECT->DOWN + } + + SECTION("full cycle: UP -> DOWN -> SUSPECT -> DOWN -> SUSPECT -> UP") + { + HostDBInfo info; + + // UP -> DOWN + info.increment_fail_count(T0, 1, FAIL_WINDOW); + REQUIRE(info.state(T1, FAIL_WINDOW) == HostDBInfo::State::DOWN); + REQUIRE(info.fail_count() == 0); + + // DOWN -> SUSPECT + REQUIRE(info.state(T2, FAIL_WINDOW) == HostDBInfo::State::SUSPECT); + + // SUSPECT -> DOWN (fail window restarts from T2) + info.increment_fail_count(T2, 1, FAIL_WINDOW); + REQUIRE(info.last_fail_time() == T2); + REQUIRE(info.state(T3, FAIL_WINDOW) == HostDBInfo::State::DOWN); + REQUIRE(info.fail_count() == 0); + + // DOWN -> SUSPECT again + REQUIRE(info.state(T4, FAIL_WINDOW) == HostDBInfo::State::SUSPECT); + + // SUSPECT -> UP + REQUIRE(info.mark_up() == true); + REQUIRE(info.state(T4, FAIL_WINDOW) == HostDBInfo::State::UP); + REQUIRE(info.fail_count() == 0); + } + + SECTION("partial failures in SUSPECT then success resets to UP cleanly") + { + HostDBInfo info; + info.increment_fail_count(T0, 1, FAIL_WINDOW); + + // Two failures below the D=3 threshold + info.increment_fail_count(T2, 3, FAIL_WINDOW); + info.increment_fail_count(T2, 3, FAIL_WINDOW); + REQUIRE(info.state(T2, FAIL_WINDOW) == HostDBInfo::State::SUSPECT); + + // Connection succeeds + info.mark_up(); + REQUIRE(info.state(T2, FAIL_WINDOW) == HostDBInfo::State::UP); + REQUIRE(info.last_fail_time() == TS_TIME_ZERO); + REQUIRE(info.fail_count() == 0); + } +} diff --git a/src/iocore/hostdb/test_HostFile.cc b/src/iocore/hostdb/unit_tests/test_HostFile.cc similarity index 100% rename from src/iocore/hostdb/test_HostFile.cc rename to src/iocore/hostdb/unit_tests/test_HostFile.cc diff --git a/src/iocore/hostdb/test_RefCountCache.cc b/src/iocore/hostdb/unit_tests/test_RefCountCache.cc similarity index 100% rename from src/iocore/hostdb/test_RefCountCache.cc rename to src/iocore/hostdb/unit_tests/test_RefCountCache.cc diff --git a/src/iocore/net/CMakeLists.txt b/src/iocore/net/CMakeLists.txt index a416ec18fe2..cb0d8e7957e 100644 --- a/src/iocore/net/CMakeLists.txt +++ b/src/iocore/net/CMakeLists.txt @@ -61,6 +61,7 @@ add_library( TLSSessionResumptionSupport.cc TLSSNISupport.cc TLSTunnelSupport.cc + TLSCertCompression.cc UDPEventIO.cc UDPIOEvent.cc UnixConnection.cc @@ -116,6 +117,21 @@ if(TS_USE_LINUX_IO_URING) target_link_libraries(inknet PUBLIC ts::inkuring) endif() +# Link cert compression libraries after OpenSSL so that OpenSSL include +# directories appear first in the search order, preventing broad system +# include paths (e.g. from Homebrew's zstd) from shadowing them. +if(HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG) + target_sources(inknet PRIVATE TLSCertCompression_zlib.cc) + if(HAVE_BROTLI_ENCODE_H) + target_sources(inknet PRIVATE TLSCertCompression_brotli.cc) + target_link_libraries(inknet PRIVATE brotli::brotlienc brotli::brotlidec) + endif() + if(HAVE_ZSTD_H) + target_sources(inknet PRIVATE TLSCertCompression_zstd.cc) + target_link_libraries(inknet PRIVATE zstd::zstd) + endif() +endif() + if(BUILD_TESTING) # libinknet_stub.cc is need because GNU ld is sensitive to the order of static libraries on the command line, and we have a cyclic dependency between inknet and proxy add_executable( diff --git a/src/iocore/net/NetVConnection.cc b/src/iocore/net/NetVConnection.cc index 94535843016..bacfc0e354d 100644 --- a/src/iocore/net/NetVConnection.cc +++ b/src/iocore/net/NetVConnection.cc @@ -51,11 +51,19 @@ DbgCtl dbg_ctl_ssl{"ssl"}; If the buffer has PROXY Protocol, it will be consumed by this function. */ bool -NetVConnection::has_proxy_protocol(IOBufferReader *reader) +NetVConnection::has_proxy_protocol(IOBufferReader *reader, int max_header_size) { - char buf[PPv1_CONNECTION_HEADER_LEN_MAX + 1]; swoc::TextView tv; - tv.assign(buf, reader->memcpy(buf, sizeof(buf), 0)); + + char preface[PPv2_CONNECTION_HEADER_LEN]; + tv.assign(preface, reader->memcpy(preface, sizeof(preface), 0)); + if (!proxy_protocol_detect(tv)) { + return false; + } + + int bufsize = max_header_size; + char buf[bufsize]; + tv.assign(buf, reader->memcpy(buf, bufsize, 0)); size_t len = proxy_protocol_parse(&this->pp_info, tv); diff --git a/src/iocore/net/P_SSLConfig.h b/src/iocore/net/P_SSLConfig.h index 1e157de0b11..15328a1aa2b 100644 --- a/src/iocore/net/P_SSLConfig.h +++ b/src/iocore/net/P_SSLConfig.h @@ -75,14 +75,14 @@ struct SSLConfigParams : public ConfigInfo { int configLoadConcurrency; int clientCertLevel; int verify_depth; - int ssl_origin_session_cache; - int ssl_origin_session_cache_size; - int ssl_session_cache; // SSL_SESSION_CACHE_MODE - int ssl_session_cache_size; - int ssl_session_cache_num_buckets; - int ssl_session_cache_skip_on_contention; - int ssl_session_cache_timeout; - int ssl_session_cache_auto_clear; + int ssl_origin_session_cache{0}; + int ssl_origin_session_cache_size{0}; + int ssl_session_cache{0}; // SSL_SESSION_CACHE_MODE + int ssl_session_cache_size{0}; + int ssl_session_cache_num_buckets{0}; + int ssl_session_cache_skip_on_contention{0}; + int ssl_session_cache_timeout{0}; + int ssl_session_cache_auto_clear{0}; char *clientCertPath; char *clientCertPathOnly; @@ -93,7 +93,6 @@ struct SSLConfigParams : public ConfigInfo { int clientCertExitOnLoadError; YamlSNIConfig::Policy verifyServerPolicy; YamlSNIConfig::Property verifyServerProperties; - bool tls_server_connection; int client_verify_depth; long ssl_ctx_options; long ssl_client_ctx_options; @@ -101,6 +100,9 @@ struct SSLConfigParams : public ConfigInfo { unsigned char alpn_protocols_array[MAX_ALPN_STRING]; int alpn_protocols_array_size = 0; + char *server_cert_compression_algorithms; + char *client_cert_compression_algorithms; + char *server_tls13_cipher_suites; char *client_tls13_cipher_suites; char *server_groups_list; diff --git a/src/iocore/net/P_SSLNetVConnection.h b/src/iocore/net/P_SSLNetVConnection.h index fdc52726e49..3ff284ff8cc 100644 --- a/src/iocore/net/P_SSLNetVConnection.h +++ b/src/iocore/net/P_SSLNetVConnection.h @@ -254,9 +254,6 @@ class SSLNetVConnection : public UnixNetVConnection, SSLNetVConnection(const SSLNetVConnection &) = delete; SSLNetVConnection &operator=(const SSLNetVConnection &) = delete; - bool protocol_mask_set = false; - unsigned long protocol_mask = 0; - bool peer_provided_cert() const override { diff --git a/src/iocore/net/ProxyProtocol.cc b/src/iocore/net/ProxyProtocol.cc index 96939412c8c..1b475c96c58 100644 --- a/src/iocore/net/ProxyProtocol.cc +++ b/src/iocore/net/ProxyProtocol.cc @@ -22,6 +22,7 @@ */ #include "iocore/net/ProxyProtocol.h" +#include "tscore/Diags.h" #include "tscore/ink_assert.h" #include "tscore/ink_string.h" #include "tscore/ink_inet.h" @@ -237,7 +238,7 @@ proxy_protocol_v2_parse(ProxyProtocol *pp_info, const swoc::TextView &msg) uint16_t tlv_len = 0; if (msg.size() < total_len) { - Dbg(dbg_ctl_proxyprotocol_v2, "The amount of available data is smaller than the expected size"); + Error("The size of PP header received (%zu) is smaller than the expected size (%zu)", msg.size(), total_len); return 0; } @@ -453,6 +454,18 @@ proxy_protocol_v2_build(uint8_t *buf, size_t max_buf_len, const ProxyProtocol &p } // namespace +bool +proxy_protocol_detect(swoc::TextView tv) +{ + if (tv.size() >= PPv1_CONNECTION_HEADER_LEN_MIN && tv.starts_with(PPv1_CONNECTION_PREFACE)) { + return true; + } else if (tv.size() >= PPv2_CONNECTION_HEADER_LEN && tv.starts_with(PPv2_CONNECTION_PREFACE)) { + return true; + } else { + return false; + } +} + /** PROXY Protocol Parser */ @@ -625,6 +638,12 @@ ProxyProtocol::get_tlv_ssl_cipher() const return this->_get_tlv_ssl_subtype(PP2_SUBTYPE_SSL_CIPHER); } +std::optional +ProxyProtocol::get_tlv_ssl_group() const +{ + return this->_get_tlv_ssl_subtype(PP2_SUBTYPE_SSL_GROUP); +} + int ProxyProtocol::set_additional_data(std::string_view data) { diff --git a/src/iocore/net/SSLClientUtils.cc b/src/iocore/net/SSLClientUtils.cc index fe08b43a4f5..453b971dd79 100644 --- a/src/iocore/net/SSLClientUtils.cc +++ b/src/iocore/net/SSLClientUtils.cc @@ -24,9 +24,11 @@ #include "P_SSLNetVConnection.h" #include "P_TLSKeyLogger.h" #include "SSLSessionCache.h" +#include "TLSCertCompression.h" #include "iocore/net/YamlSNIConfig.h" #include "iocore/net/SSLDiags.h" #include "tscore/ink_config.h" +#include "tscore/SimpleTokenizer.h" #include "tscore/Filenames.h" #include "tscore/X509HostnameValidator.h" @@ -247,6 +249,18 @@ SSLInitClientContext(const SSLConfigParams *params) } #endif + if (params->client_cert_compression_algorithms) { + std::vector algs; + SimpleTokenizer tok(params->client_cert_compression_algorithms, ','); + for (const char *token = tok.getNext(); token; token = tok.getNext()) { + algs.emplace_back(token); + } + if (register_certificate_compression_preference(client_ctx, algs) != 1) { + SSLError("invalid client certificate compression algorithm list in %s", ts::filename::RECORDS); + goto fail; + } + } + SSL_CTX_set_verify_depth(client_ctx, params->client_verify_depth); if (SSLConfigParams::init_ssl_ctx_cb) { SSLConfigParams::init_ssl_ctx_cb(client_ctx, false); diff --git a/src/iocore/net/SSLConfig.cc b/src/iocore/net/SSLConfig.cc index fd22831e051..5862a630758 100644 --- a/src/iocore/net/SSLConfig.cc +++ b/src/iocore/net/SSLConfig.cc @@ -199,6 +199,8 @@ SSLConfigParams::reset() clientKeyPath = clientCACertFilename = clientCACertPath = cipherSuite = client_cipherSuite = dhparamsFile = serverKeyPathOnly = clientKeyPathOnly = clientCertPathOnly = nullptr; ssl_ocsp_response_path_only = nullptr; + server_cert_compression_algorithms = nullptr; + client_cert_compression_algorithms = nullptr; server_tls13_cipher_suites = nullptr; client_tls13_cipher_suites = nullptr; server_groups_list = nullptr; @@ -242,11 +244,13 @@ SSLConfigParams::cleanup() ssl_ocsp_response_path_only = static_cast(ats_free_null(ssl_ocsp_response_path_only)); - server_tls13_cipher_suites = static_cast(ats_free_null(server_tls13_cipher_suites)); - client_tls13_cipher_suites = static_cast(ats_free_null(client_tls13_cipher_suites)); - server_groups_list = static_cast(ats_free_null(server_groups_list)); - client_groups_list = static_cast(ats_free_null(client_groups_list)); - keylog_file = static_cast(ats_free_null(keylog_file)); + server_cert_compression_algorithms = static_cast(ats_free_null(server_cert_compression_algorithms)); + client_cert_compression_algorithms = static_cast(ats_free_null(client_cert_compression_algorithms)); + server_tls13_cipher_suites = static_cast(ats_free_null(server_tls13_cipher_suites)); + client_tls13_cipher_suites = static_cast(ats_free_null(client_tls13_cipher_suites)); + server_groups_list = static_cast(ats_free_null(server_groups_list)); + client_groups_list = static_cast(ats_free_null(client_groups_list)); + keylog_file = static_cast(ats_free_null(keylog_file)); cleanupCTXTable(); reset(); @@ -567,7 +571,7 @@ SSLConfigParams::initialize() session_cache = new SSLSessionCache(); } - if (ssl_origin_session_cache == 1 && ssl_origin_session_cache_size > 0) { + if (ssl_origin_session_cache == 1 && ssl_origin_session_cache_size > 0 && origin_sess_cache == nullptr) { origin_sess_cache = new SSLOriginSessionCache(); } @@ -586,6 +590,7 @@ SSLConfigParams::initialize() set_paths_helper(ssl_ocsp_response_path, nullptr, &ssl_ocsp_response_path_only, nullptr); } if (auto rec_str{RecGetRecordStringAlloc("proxy.config.http.request_via_str")}; rec_str) { + ats_free(ssl_ocsp_user_agent); ssl_ocsp_user_agent = ats_stringdup(rec_str); } @@ -600,6 +605,13 @@ SSLConfigParams::initialize() server_groups_list = ats_stringdup(rec_str); } + if (auto rec_str{RecGetRecordStringAlloc("proxy.config.ssl.server.cert_compression.algorithms")}; rec_str) { + server_cert_compression_algorithms = ats_stringdup(rec_str); + } + if (auto rec_str{RecGetRecordStringAlloc("proxy.config.ssl.client.cert_compression.algorithms")}; rec_str) { + client_cert_compression_algorithms = ats_stringdup(rec_str); + } + // ++++++++++++++++++++++++ Client part ++++++++++++++++++++ client_verify_depth = 7; @@ -963,6 +975,11 @@ SSLTicketParams::cleanup() void cleanup_bio(BIO *&biop) { + // BIO_new_mem_buf sets BIO_FLAGS_MEM_RDONLY which prevents BIO_free from + // cleaning up internal BUF_MEM structures. Clear this flag so BIO_free + // properly releases them. BIO_NOCLOSE ensures the external data buffer + // (owned by the caller's std::string) is not freed. + BIO_clear_flags(biop, BIO_FLAGS_MEM_RDONLY); #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wunused-value" BIO_set_close(biop, BIO_NOCLOSE); diff --git a/src/iocore/net/SSLNetVConnection.cc b/src/iocore/net/SSLNetVConnection.cc index ab5eb9d32bd..43f945176e8 100644 --- a/src/iocore/net/SSLNetVConnection.cc +++ b/src/iocore/net/SSLNetVConnection.cc @@ -92,6 +92,17 @@ DbgCtl dbg_ctl_ssl_alpn{"ssl_alpn"}; DbgCtl dbg_ctl_ssl_origin_session_cache{"ssl.origin_session_cache"}; DbgCtl dbg_ctl_proxyprotocol{"proxyprotocol"}; +const char * +resolve_client_ca_cert_path(const SSLConfigParams *params, const char *path, std::string &storage) +{ + if (path == nullptr) { + return params->clientCACertPath; + } + + storage = Layout::get()->relative_to(Layout::get()->prefix, path); + return storage.c_str(); +} + } // namespace // @@ -1129,6 +1140,8 @@ SSLNetVConnection::_sslStartHandShake(int event, int &err) auto nps = sniParam->get_property_config(serverKey); shared_SSL_CTX sharedCTX = nullptr; SSL_CTX *clientCTX = nullptr; + std::string caCertPathStorage; + const char *caCertPath = resolve_client_ca_cert_path(params, options.ssl_client_ca_cert_path, caCertPathStorage); // First Look to see if there are override parameters Dbg(dbg_ctl_ssl, "Checking for outbound client cert override [%p]", options.ssl_client_cert_name.get()); @@ -1144,18 +1157,21 @@ SSLNetVConnection::_sslStartHandShake(int event, int &err) keyFilePath = Layout::get()->relative_to(params->clientKeyPathOnly, options.ssl_client_private_key_name); } if (options.ssl_client_ca_cert_name) { - caCertFilePath = Layout::get()->relative_to(params->clientCACertPath, options.ssl_client_ca_cert_name); + caCertFilePath = Layout::get()->relative_to(caCertPath, options.ssl_client_ca_cert_name); } Dbg(dbg_ctl_ssl, "Using outbound client cert `%s'", options.ssl_client_cert_name.get()); } else { Dbg(dbg_ctl_ssl, "Clearing outbound client cert"); } - sharedCTX = - params->getCTX(certFilePath, keyFilePath, caCertFilePath.empty() ? params->clientCACertFilename : caCertFilePath.c_str(), - params->clientCACertPath); - } else if (options.ssl_client_ca_cert_name) { - std::string caCertFilePath = Layout::get()->relative_to(params->clientCACertPath, options.ssl_client_ca_cert_name); - sharedCTX = params->getCTX(params->clientCertPath, params->clientKeyPath, caCertFilePath.c_str(), params->clientCACertPath); + sharedCTX = params->getCTX(certFilePath, keyFilePath, + caCertFilePath.empty() ? params->clientCACertFilename : caCertFilePath.c_str(), caCertPath); + } else if (options.ssl_client_ca_cert_name || options.ssl_client_ca_cert_path) { + std::string caCertFilePath; + if (options.ssl_client_ca_cert_name) { + caCertFilePath = Layout::get()->relative_to(caCertPath, options.ssl_client_ca_cert_name); + } + sharedCTX = params->getCTX(params->clientCertPath, params->clientKeyPath, + caCertFilePath.empty() ? params->clientCACertFilename : caCertFilePath.c_str(), caCertPath); } else if (nps && !nps->client_cert_file.empty()) { // If no overrides available, try the available nextHopProperty by reading from context mappings sharedCTX = diff --git a/src/iocore/net/SSLSessionCache.cc b/src/iocore/net/SSLSessionCache.cc index 9dcccd535ef..bf343593435 100644 --- a/src/iocore/net/SSLSessionCache.cc +++ b/src/iocore/net/SSLSessionCache.cc @@ -339,7 +339,13 @@ SSLSessionBucket::~SSLSessionBucket() {} SSLOriginSessionCache::SSLOriginSessionCache() {} -SSLOriginSessionCache::~SSLOriginSessionCache() {} +SSLOriginSessionCache::~SSLOriginSessionCache() +{ + while (auto *node = orig_sess_que.pop()) { + delete node; + } + orig_sess_map.clear(); +} void SSLOriginSessionCache::insert_session(const std::string &lookup_key, SSL_SESSION *sess, SSL *ssl) diff --git a/src/iocore/net/SSLStats.cc b/src/iocore/net/SSLStats.cc index fdec94b1e46..463e67599e3 100644 --- a/src/iocore/net/SSLStats.cc +++ b/src/iocore/net/SSLStats.cc @@ -213,6 +213,18 @@ SSLInitializeStatistics() // For now, register with the librecords global sync. RecRegNewSyncStatSync(SSLPeriodicMetricsUpdate); + ssl_rsb.cert_compress_zlib = Metrics::Counter::createPtr("proxy.process.ssl.cert_compress.zlib"); + ssl_rsb.cert_compress_zlib_failure = Metrics::Counter::createPtr("proxy.process.ssl.cert_compress.zlib_failure"); + ssl_rsb.cert_decompress_zlib = Metrics::Counter::createPtr("proxy.process.ssl.cert_decompress.zlib"); + ssl_rsb.cert_decompress_zlib_failure = Metrics::Counter::createPtr("proxy.process.ssl.cert_decompress.zlib_failure"); + ssl_rsb.cert_compress_brotli = Metrics::Counter::createPtr("proxy.process.ssl.cert_compress.brotli"); + ssl_rsb.cert_compress_brotli_failure = Metrics::Counter::createPtr("proxy.process.ssl.cert_compress.brotli_failure"); + ssl_rsb.cert_decompress_brotli = Metrics::Counter::createPtr("proxy.process.ssl.cert_decompress.brotli"); + ssl_rsb.cert_decompress_brotli_failure = Metrics::Counter::createPtr("proxy.process.ssl.cert_decompress.brotli_failure"); + ssl_rsb.cert_compress_zstd = Metrics::Counter::createPtr("proxy.process.ssl.cert_compress.zstd"); + ssl_rsb.cert_compress_zstd_failure = Metrics::Counter::createPtr("proxy.process.ssl.cert_compress.zstd_failure"); + ssl_rsb.cert_decompress_zstd = Metrics::Counter::createPtr("proxy.process.ssl.cert_decompress.zstd"); + ssl_rsb.cert_decompress_zstd_failure = Metrics::Counter::createPtr("proxy.process.ssl.cert_decompress.zstd_failure"); ssl_rsb.early_data_received_count = Metrics::Counter::createPtr("proxy.process.ssl.early_data_received"); ssl_rsb.error_async = Metrics::Counter::createPtr("proxy.process.ssl.ssl_error_async"); ssl_rsb.error_ssl = Metrics::Counter::createPtr("proxy.process.ssl.ssl_error_ssl"); diff --git a/src/iocore/net/SSLStats.h b/src/iocore/net/SSLStats.h index 82b84445302..0ecb5db3e19 100644 --- a/src/iocore/net/SSLStats.h +++ b/src/iocore/net/SSLStats.h @@ -37,6 +37,18 @@ using ts::Metrics; // for ssl_rsb.total_ticket_keys_renewed needs this initialization, but lets be // consistent at least. struct SSLStatsBlock { + Metrics::Counter::AtomicType *cert_compress_zlib = nullptr; + Metrics::Counter::AtomicType *cert_compress_zlib_failure = nullptr; + Metrics::Counter::AtomicType *cert_decompress_zlib = nullptr; + Metrics::Counter::AtomicType *cert_decompress_zlib_failure = nullptr; + Metrics::Counter::AtomicType *cert_compress_brotli = nullptr; + Metrics::Counter::AtomicType *cert_compress_brotli_failure = nullptr; + Metrics::Counter::AtomicType *cert_decompress_brotli = nullptr; + Metrics::Counter::AtomicType *cert_decompress_brotli_failure = nullptr; + Metrics::Counter::AtomicType *cert_compress_zstd = nullptr; + Metrics::Counter::AtomicType *cert_compress_zstd_failure = nullptr; + Metrics::Counter::AtomicType *cert_decompress_zstd = nullptr; + Metrics::Counter::AtomicType *cert_decompress_zstd_failure = nullptr; Metrics::Counter::AtomicType *early_data_received_count = nullptr; Metrics::Counter::AtomicType *error_async = nullptr; Metrics::Counter::AtomicType *error_ssl = nullptr; diff --git a/src/iocore/net/SSLUtils.cc b/src/iocore/net/SSLUtils.cc index 5235d9e10a9..c7b13f63e14 100644 --- a/src/iocore/net/SSLUtils.cc +++ b/src/iocore/net/SSLUtils.cc @@ -30,6 +30,7 @@ #include "SSLSessionCache.h" #include "SSLSessionTicket.h" #include "SSLDynlock.h" // IWYU pragma: keep - for ssl_dyn_* +#include "TLSCertCompression.h" #include "iocore/net/SSLMultiCertConfigLoader.h" #include "iocore/net/SSLAPIHooks.h" @@ -57,7 +58,9 @@ #include #include #include +#if HAVE_ENGINE_LOAD_DYNAMIC #include +#endif #include #include #include @@ -533,6 +536,26 @@ DH_get_2048_256() } #endif +bool +SSLMultiCertConfigLoader::_enable_cert_compression(SSL_CTX *ctx) +{ + std::vector algs; + + if (this->_params->server_cert_compression_algorithms) { + SimpleTokenizer tok(this->_params->server_cert_compression_algorithms, ','); + for (const char *token = tok.getNext(); token; token = tok.getNext()) { + algs.emplace_back(token); + } + } + + if (register_certificate_compression_preference(ctx, algs) == 1) { + return true; + } else { + SSLError("Failed to enable certificate compression"); + return false; + } +} + bool SSLMultiCertConfigLoader::_enable_ktls([[maybe_unused]] SSL_CTX *ctx) { @@ -1372,6 +1395,10 @@ SSLMultiCertConfigLoader::init_server_ssl_ctx(CertLoadData const &data, const SS goto fail; } + if (!this->_enable_cert_compression(ctx)) { + goto fail; + } + if (!this->_enable_ktls(ctx)) { goto fail; } diff --git a/src/iocore/net/TLSCertCompression.cc b/src/iocore/net/TLSCertCompression.cc new file mode 100644 index 00000000000..90192990f3a --- /dev/null +++ b/src/iocore/net/TLSCertCompression.cc @@ -0,0 +1,149 @@ +/** @file + + Functions for Certificate Compression + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include + +#include "tscore/Diags.h" +#include "TLSCertCompression.h" + +namespace +{ +DbgCtl dbg_ctl_ssl_cert_compress{"ssl_cert_compress"}; +} + +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG || HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE +constexpr unsigned int N_ALGORITHMS = 3; +#endif + +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG +#include "TLSCertCompression_zlib.h" + +#if HAVE_BROTLI_ENCODE_H +#include "TLSCertCompression_brotli.h" +#endif + +#if HAVE_ZSTD_H +#include "TLSCertCompression_zstd.h" +#endif +#endif + +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG || HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE +namespace +{ +struct alg_info { + const char *name; + int32_t number; +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG + ssl_cert_compression_func_t compress_func; + ssl_cert_decompression_func_t decompress_func; +#endif +} supported_algs[] = { +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG + {"zlib", 1, compression_func_zlib, decompression_func_zlib }, +#if HAVE_BROTLI_ENCODE_H + {"brotli", 2, compression_func_brotli, decompression_func_brotli}, +#endif +#if HAVE_ZSTD_H + {"zstd", 3, compression_func_zstd, decompression_func_zstd }, +#endif + {nullptr, 0, nullptr, nullptr }, +#elif HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE +#if !defined(OPENSSL_NO_ZLIB) + {"zlib", 1}, +#endif +#if !defined(OPENSSL_NO_BROTLI) + {"brotli", 2}, +#endif +#if !defined(OPENSSL_NO_ZSTD) + {"zstd", 3}, +#endif + {nullptr, 0}, +#else + {nullptr, 0}, +#endif +}; + +alg_info const * +find_algorithm(std::string const &name) +{ + for (auto const &alg : supported_algs) { + if (alg.name != nullptr && name == alg.name) { + return &alg; + } + } + return nullptr; +} + +} // end anonymous namespace +#endif + +int +register_certificate_compression_preference(SSL_CTX *ctx, const std::vector &specified_algs) +{ + ink_assert(ctx != nullptr); + if (specified_algs.empty()) { + return 1; + } + +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG + if (specified_algs.size() > N_ALGORITHMS) { + return 0; + } + + for (auto &&alg : specified_algs) { + auto const *info = find_algorithm(alg); + if (info == nullptr) { + Dbg(dbg_ctl_ssl_cert_compress, "Unsupported algorithm: %s", alg.c_str()); + return 0; + } + if (SSL_CTX_add_cert_compression_alg(ctx, info->number, info->compress_func, info->decompress_func) == 0) { + return 0; + } + Dbg(dbg_ctl_ssl_cert_compress, "Enabled %s", info->name); + } + return 1; +#elif HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE + if (specified_algs.size() > N_ALGORITHMS) { + return 0; + } + + int algs[N_ALGORITHMS]; + int n = 0; + + for (unsigned int i = 0; i < specified_algs.size(); ++i) { + auto const *info = find_algorithm(specified_algs[i]); + if (info == nullptr) { + Dbg(dbg_ctl_ssl_cert_compress, "Unsupported algorithm: %s", specified_algs[i].c_str()); + return 0; + } + algs[n++] = info->number; + Dbg(dbg_ctl_ssl_cert_compress, "Enabled %s", info->name); + } + return SSL_CTX_set1_cert_comp_preference(ctx, algs, n); +#else + // If Certificate Compression is unsupported there's nothing to do. + // No need to raise an error since handshake would be done successfully without compression. + Dbg(dbg_ctl_ssl_cert_compress, "Certificate Compression is unsupported"); + return 1; +#endif +} diff --git a/src/iocore/net/TLSCertCompression.h b/src/iocore/net/TLSCertCompression.h new file mode 100644 index 00000000000..fa7bbc6ee56 --- /dev/null +++ b/src/iocore/net/TLSCertCompression.h @@ -0,0 +1,43 @@ +/** @file + + Functions for Certificate Compression + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include +#include +#include + +// RFC 8879 uses uint24 for uncompressed_length, allowing up to ~16 MB. +// Real certificate chains rarely exceed 10-30 KB even with large RSA +// keys and multiple intermediates — 128 KB gives ample headroom while +// preventing excessive memory allocation from a malicious peer. +constexpr size_t MAX_CERT_UNCOMPRESSED_LEN = 128 * 1024; + +/** + * Common function to set certificate compression preference + * + * @param[in] ctx SSL_CTX + * @param[in] algs A vector that contains compression algorithm names ("zlib", "brotli", or "zstd") + * @return 1 on success + */ +int register_certificate_compression_preference(SSL_CTX *ctx, const std::vector &algs); diff --git a/src/iocore/net/TLSCertCompression_brotli.cc b/src/iocore/net/TLSCertCompression_brotli.cc new file mode 100644 index 00000000000..52cf22dada6 --- /dev/null +++ b/src/iocore/net/TLSCertCompression_brotli.cc @@ -0,0 +1,84 @@ +/** @file + + Functions for brotli compression/decompression + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "TLSCertCompression_brotli.h" +#include "TLSCertCompression.h" +#include "SSLStats.h" +#include +#include +#include + +int +compression_func_brotli(SSL * /* ssl */, CBB *out, const uint8_t *in, size_t in_len) +{ + // TODO Need a cache mechanism inside this function for better performance. + + uint8_t *buf; + unsigned long buf_len = BrotliEncoderMaxCompressedSize(in_len); + + if (CBB_reserve(out, &buf, buf_len) != 1) { + Metrics::Counter::increment(ssl_rsb.cert_compress_brotli_failure); + return 0; + } + + if (BrotliEncoderCompress(BROTLI_DEFAULT_QUALITY, BROTLI_DEFAULT_WINDOW, BROTLI_DEFAULT_MODE, in_len, in, &buf_len, buf) == + BROTLI_TRUE) { + CBB_did_write(out, buf_len); + Metrics::Counter::increment(ssl_rsb.cert_compress_brotli); + return 1; + } else { + CBB_did_write(out, 0); + Metrics::Counter::increment(ssl_rsb.cert_compress_brotli_failure); + return 0; + } +} + +int +decompression_func_brotli(SSL * /* ssl */, CRYPTO_BUFFER **out, size_t uncompressed_len, const uint8_t *in, size_t in_len) +{ + if (uncompressed_len > MAX_CERT_UNCOMPRESSED_LEN) { + *out = nullptr; + Metrics::Counter::increment(ssl_rsb.cert_decompress_brotli_failure); + return 0; + } + + uint8_t *buf; + + *out = CRYPTO_BUFFER_alloc(&buf, uncompressed_len); + if (*out == nullptr) { + Metrics::Counter::increment(ssl_rsb.cert_decompress_brotli_failure); + return 0; + } + + size_t dest_len = uncompressed_len; + + if (BrotliDecoderDecompress(in_len, in, &dest_len, buf) != BROTLI_DECODER_RESULT_SUCCESS || dest_len != uncompressed_len) { + CRYPTO_BUFFER_free(*out); + *out = nullptr; + Metrics::Counter::increment(ssl_rsb.cert_decompress_brotli_failure); + return 0; + } + + Metrics::Counter::increment(ssl_rsb.cert_decompress_brotli); + return 1; +} diff --git a/src/iocore/net/TLSCertCompression_brotli.h b/src/iocore/net/TLSCertCompression_brotli.h new file mode 100644 index 00000000000..7026f4ff70f --- /dev/null +++ b/src/iocore/net/TLSCertCompression_brotli.h @@ -0,0 +1,30 @@ +/** @file + + Functions for brotli compression/decompression + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include +#include + +int compression_func_brotli(SSL *ssl, CBB *out, const uint8_t *in, size_t in_len); +int decompression_func_brotli(SSL *ssl, CRYPTO_BUFFER **out, size_t uncompressed_len, const uint8_t *in, size_t in_len); diff --git a/src/iocore/net/TLSCertCompression_zlib.cc b/src/iocore/net/TLSCertCompression_zlib.cc new file mode 100644 index 00000000000..1d724ef541a --- /dev/null +++ b/src/iocore/net/TLSCertCompression_zlib.cc @@ -0,0 +1,82 @@ +/** @file + + Functions for zlib compression/decompression + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "TLSCertCompression_zlib.h" +#include "TLSCertCompression.h" +#include "SSLStats.h" +#include +#include + +int +compression_func_zlib(SSL * /* ssl */, CBB *out, const uint8_t *in, size_t in_len) +{ + // TODO Need a cache mechanism inside this function for better performance. + + uint8_t *buf; + unsigned long buf_len = compressBound(in_len); + + if (CBB_reserve(out, &buf, buf_len) != 1) { + Metrics::Counter::increment(ssl_rsb.cert_compress_zlib_failure); + return 0; + } + + if (compress(buf, &buf_len, in, in_len) == Z_OK) { + CBB_did_write(out, buf_len); + Metrics::Counter::increment(ssl_rsb.cert_compress_zlib); + return 1; + } else { + CBB_did_write(out, 0); + Metrics::Counter::increment(ssl_rsb.cert_compress_zlib_failure); + return 0; + } +} + +int +decompression_func_zlib(SSL * /* ssl */, CRYPTO_BUFFER **out, size_t uncompressed_len, const uint8_t *in, size_t in_len) +{ + if (uncompressed_len > MAX_CERT_UNCOMPRESSED_LEN) { + *out = nullptr; + Metrics::Counter::increment(ssl_rsb.cert_decompress_zlib_failure); + return 0; + } + + uint8_t *buf; + + *out = CRYPTO_BUFFER_alloc(&buf, uncompressed_len); + if (*out == nullptr) { + Metrics::Counter::increment(ssl_rsb.cert_decompress_zlib_failure); + return 0; + } + + unsigned long dest_len = uncompressed_len; + + if (uncompress(buf, &dest_len, in, in_len) != Z_OK || dest_len != uncompressed_len) { + CRYPTO_BUFFER_free(*out); + *out = nullptr; + Metrics::Counter::increment(ssl_rsb.cert_decompress_zlib_failure); + return 0; + } + + Metrics::Counter::increment(ssl_rsb.cert_decompress_zlib); + return 1; +} diff --git a/src/iocore/net/TLSCertCompression_zlib.h b/src/iocore/net/TLSCertCompression_zlib.h new file mode 100644 index 00000000000..622f8efb5a4 --- /dev/null +++ b/src/iocore/net/TLSCertCompression_zlib.h @@ -0,0 +1,30 @@ +/** @file + + Functions for zlib compression/decompression + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include +#include + +int compression_func_zlib(SSL *ssl, CBB *out, const uint8_t *in, size_t in_len); +int decompression_func_zlib(SSL *ssl, CRYPTO_BUFFER **out, size_t uncompressed_len, const uint8_t *in, size_t in_len); diff --git a/src/iocore/net/TLSCertCompression_zstd.cc b/src/iocore/net/TLSCertCompression_zstd.cc new file mode 100644 index 00000000000..85cf2a94f90 --- /dev/null +++ b/src/iocore/net/TLSCertCompression_zstd.cc @@ -0,0 +1,89 @@ +/** @file + + Functions for zstd compression/decompression + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "TLSCertCompression_zstd.h" +#include "TLSCertCompression.h" +#include "SSLStats.h" +#include +#include + +int +compression_func_zstd(SSL * /* ssl */, CBB *out, const uint8_t *in, size_t in_len) +{ + // TODO Need a cache mechanism inside this function for better performance. + + uint8_t *buf; + unsigned long buf_len = ZSTD_compressBound(in_len); + + if (ZSTD_isError(buf_len) == 1) { + Metrics::Counter::increment(ssl_rsb.cert_compress_zstd_failure); + return 0; + } + + if (CBB_reserve(out, &buf, buf_len) != 1) { + Metrics::Counter::increment(ssl_rsb.cert_compress_zstd_failure); + return 0; + } + + // For better performance ZSTD_compressCCtx, which reuses a context object, should be used. + // One context object need to be made for each thread. + size_t ret = ZSTD_compress(buf, buf_len, in, in_len, ZSTD_CLEVEL_DEFAULT); + if (ZSTD_isError(ret) == 1) { + Metrics::Counter::increment(ssl_rsb.cert_compress_zstd_failure); + return 0; + } else { + CBB_did_write(out, ret); + Metrics::Counter::increment(ssl_rsb.cert_compress_zstd); + return 1; + } +} + +int +decompression_func_zstd(SSL * /* ssl */, CRYPTO_BUFFER **out, size_t uncompressed_len, const uint8_t *in, size_t in_len) +{ + if (uncompressed_len > MAX_CERT_UNCOMPRESSED_LEN) { + *out = nullptr; + Metrics::Counter::increment(ssl_rsb.cert_decompress_zstd_failure); + return 0; + } + + uint8_t *buf; + + *out = CRYPTO_BUFFER_alloc(&buf, uncompressed_len); + if (*out == nullptr) { + Metrics::Counter::increment(ssl_rsb.cert_decompress_zstd_failure); + return 0; + } + + size_t ret = ZSTD_decompress(buf, uncompressed_len, in, in_len); + + if (ZSTD_isError(ret) || ret != uncompressed_len) { + CRYPTO_BUFFER_free(*out); + *out = nullptr; + Metrics::Counter::increment(ssl_rsb.cert_decompress_zstd_failure); + return 0; + } + + Metrics::Counter::increment(ssl_rsb.cert_decompress_zstd); + return 1; +} diff --git a/src/iocore/net/TLSCertCompression_zstd.h b/src/iocore/net/TLSCertCompression_zstd.h new file mode 100644 index 00000000000..bde6ef6d7b3 --- /dev/null +++ b/src/iocore/net/TLSCertCompression_zstd.h @@ -0,0 +1,30 @@ +/** @file + + Functions for zstd compression/decompression + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include +#include + +int compression_func_zstd(SSL *ssl, CBB *out, const uint8_t *in, size_t in_len); +int decompression_func_zstd(SSL *ssl, CRYPTO_BUFFER **out, size_t uncompressed_len, const uint8_t *in, size_t in_len); diff --git a/src/iocore/net/UnixNetProcessor.cc b/src/iocore/net/UnixNetProcessor.cc index 5c8d5111e1b..950ea5d2878 100644 --- a/src/iocore/net/UnixNetProcessor.cc +++ b/src/iocore/net/UnixNetProcessor.cc @@ -167,9 +167,11 @@ void UnixNetProcessor::stop_accept() { SCOPED_MUTEX_LOCK(lock, naVecMutex, this_ethread()); - for (auto &na : naVec) { + for (auto *na : naVec) { na->stop_accept(); + delete na; } + naVec.clear(); } Action * diff --git a/src/iocore/net/unit_tests/test_ProxyProtocol.cc b/src/iocore/net/unit_tests/test_ProxyProtocol.cc index 53958b30b27..ada8605f085 100644 --- a/src/iocore/net/unit_tests/test_ProxyProtocol.cc +++ b/src/iocore/net/unit_tests/test_ProxyProtocol.cc @@ -303,15 +303,16 @@ TEST_CASE("PROXY Protocol v2 Parser", "[ProxyProtocol][ProxyProtocolv2]") 0x55, 0x49, 0x54, 0x0A, ///< 0x21, ///< version & command 0x11, ///< protocol & family - 0x00, 0x2B, ///< len + 0x00, 0x32, ///< len 0xC0, 0x00, 0x02, 0x01, ///< src_addr 0xC6, 0x33, 0x64, 0x01, ///< dst_addr 0xC3, 0x50, ///< src_port 0x01, 0xBB, ///< dst_port 0x01, 0x00, 0x02, 0x68, 0x32, /// PP2_TYPE_ALPN (h2) 0x02, 0x00, 0x03, 0x61, 0x62, 0x63, /// PP2_TYPE_AUTHORITY (abc) - 0x20, 0x00, 0x11, 0x01, 0x00, 0x00, 0x00, 0x00, /// PP2_TYPE_SSL (client=0x01, verify=0) + 0x20, 0x00, 0x18, 0x01, 0x00, 0x00, 0x00, 0x00, /// PP2_TYPE_SSL (client=0x01, verify=0) 0x23, 0x00, 0x03, 0x58, 0x59, 0x5A, /// PP2_SUBTYPE_SSL_CIPHER (XYZ) + 0x26, 0x00, 0x04, 0x58, 0x31, 0x32, 0x33, /// PP2_SUBTYPE_SSL_GROUP(X123) 0x21, 0x00, 0x03, 0x54, 0x4C, 0x53, /// PP2_SUBTYPE_SSL_VERSION (TLS) }; @@ -332,6 +333,7 @@ TEST_CASE("PROXY Protocol v2 Parser", "[ProxyProtocol][ProxyProtocolv2]") CHECK(pp_info.tlv[PP2_TYPE_AUTHORITY] == "abc"); CHECK(pp_info.get_tlv_ssl_cipher() == "XYZ"); + CHECK(pp_info.get_tlv_ssl_group() == "X123"); CHECK(pp_info.get_tlv_ssl_version() == "TLS"); } diff --git a/src/mgmt/rpc/CMakeLists.txt b/src/mgmt/rpc/CMakeLists.txt index bee0f8929fd..6ab29e3dbdf 100644 --- a/src/mgmt/rpc/CMakeLists.txt +++ b/src/mgmt/rpc/CMakeLists.txt @@ -70,6 +70,12 @@ if(BUILD_TESTING) ) target_link_libraries(test_jsonrpcserver Catch2::Catch2WithMain ts::jsonrpc_server ts::inkevent libswoc::libswoc) add_catch2_test(NAME test_jsonrpcserver COMMAND test_jsonrpcserver) + + add_executable(test_record_yaml handlers/common/unit_tests/test_record_yaml.cc) + target_link_libraries( + test_record_yaml PRIVATE Catch2::Catch2WithMain ts::rpcpublichandlers ts::tscore yaml-cpp::yaml-cpp + ) + add_catch2_test(NAME test_record_yaml COMMAND test_record_yaml) endif() clang_tidy_check(jsonrpc_protocol) diff --git a/src/mgmt/rpc/handlers/common/RecordsUtils.cc b/src/mgmt/rpc/handlers/common/RecordsUtils.cc index 0d2291c0b4a..fb8811f861c 100644 --- a/src/mgmt/rpc/handlers/common/RecordsUtils.cc +++ b/src/mgmt/rpc/handlers/common/RecordsUtils.cc @@ -63,6 +63,10 @@ RPCRecordErrorCategory::message(int ev) const return {"Found record does not match the requested type"}; case rpc::handlers::errors::RecordError::INVALID_INCOMING_DATA: return {"Invalid request data provided"}; + case rpc::handlers::errors::RecordError::RECORD_READ_ONLY: + return {"Record is read-only and cannot be modified through the management interface."}; + case rpc::handlers::errors::RecordError::RECORD_NO_ACCESS: + return {"Record is not accessible through the management interface."}; default: return "Record error error " + std::to_string(ev); } diff --git a/src/mgmt/rpc/handlers/common/RecordsUtils.h b/src/mgmt/rpc/handlers/common/RecordsUtils.h index b94e66dc07d..5be131b8d48 100644 --- a/src/mgmt/rpc/handlers/common/RecordsUtils.h +++ b/src/mgmt/rpc/handlers/common/RecordsUtils.h @@ -42,7 +42,9 @@ enum class RecordError { GENERAL_ERROR, RECORD_WRITE_ERROR, REQUESTED_TYPE_MISMATCH, - INVALID_INCOMING_DATA + INVALID_INCOMING_DATA, + RECORD_READ_ONLY, + RECORD_NO_ACCESS }; std::error_code make_error_code(rpc::handlers::errors::RecordError e); } // namespace rpc::handlers::errors diff --git a/src/mgmt/rpc/handlers/common/convert.h b/src/mgmt/rpc/handlers/common/convert.h index 4105d495856..6d541398534 100644 --- a/src/mgmt/rpc/handlers/common/convert.h +++ b/src/mgmt/rpc/handlers/common/convert.h @@ -139,26 +139,41 @@ template <> struct convert { node[constants_rec::OVERRIDABLE] = (it == ts::Overridable_Txn_Vars.end()) ? "false" : "true"; } + // access_type lives in config_meta, which shares storage with stat_meta + // via a union; only inspect it when the record is actually a CONFIG + // record. Records registered with @c RECA_NO_ACCESS opt out of having + // their value exposed, so withhold the value fields while still emitting + // the type label so callers can tell which records exist. + const bool no_access = REC_TYPE_IS_CONFIG(record.rec_type) && record.config_meta.access_type == RECA_NO_ACCESS; + switch (record.data_type) { case RECD_INT: - node[constants_rec::DATA_TYPE] = "INT"; - node[constants_rec::CURRENT_VALUE] = record.data.rec_int; - node[constants_rec::DEFAULT_VALUE] = record.data_default.rec_int; + node[constants_rec::DATA_TYPE] = "INT"; + if (!no_access) { + node[constants_rec::CURRENT_VALUE] = record.data.rec_int; + node[constants_rec::DEFAULT_VALUE] = record.data_default.rec_int; + } break; case RECD_FLOAT: - node[constants_rec::DATA_TYPE] = "FLOAT"; - node[constants_rec::CURRENT_VALUE] = record.data.rec_float; - node[constants_rec::DEFAULT_VALUE] = record.data_default.rec_float; + node[constants_rec::DATA_TYPE] = "FLOAT"; + if (!no_access) { + node[constants_rec::CURRENT_VALUE] = record.data.rec_float; + node[constants_rec::DEFAULT_VALUE] = record.data_default.rec_float; + } break; case RECD_STRING: - node[constants_rec::DATA_TYPE] = "STRING"; - node[constants_rec::CURRENT_VALUE] = record.data.rec_string ? record.data.rec_string : "null"; - node[constants_rec::DEFAULT_VALUE] = record.data_default.rec_string ? record.data_default.rec_string : "null"; + node[constants_rec::DATA_TYPE] = "STRING"; + if (!no_access) { + node[constants_rec::CURRENT_VALUE] = record.data.rec_string ? record.data.rec_string : "null"; + node[constants_rec::DEFAULT_VALUE] = record.data_default.rec_string ? record.data_default.rec_string : "null"; + } break; case RECD_COUNTER: - node[constants_rec::DATA_TYPE] = "COUNTER"; - node[constants_rec::CURRENT_VALUE] = record.data.rec_counter; - node[constants_rec::DEFAULT_VALUE] = record.data_default.rec_counter; + node[constants_rec::DATA_TYPE] = "COUNTER"; + if (!no_access) { + node[constants_rec::CURRENT_VALUE] = record.data.rec_counter; + node[constants_rec::DEFAULT_VALUE] = record.data_default.rec_counter; + } break; default: // this is an error, internal we should flag it diff --git a/src/mgmt/rpc/handlers/common/unit_tests/test_record_yaml.cc b/src/mgmt/rpc/handlers/common/unit_tests/test_record_yaml.cc new file mode 100644 index 00000000000..2034a0bd3a4 --- /dev/null +++ b/src/mgmt/rpc/handlers/common/unit_tests/test_record_yaml.cc @@ -0,0 +1,146 @@ +/** + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +*/ + +#include + +#include +#include +#include + +#include "../convert.h" + +namespace +{ +// +// Build minimal RecRecord fixtures aimed at the YAML encoder under test in +// place. RecRecord embeds a RecMutex (with an atomic) and is therefore not +// copyable, so the helpers operate on an output reference and only populate +// the fields the encoder actually reads. The lock and the unused half of the +// stat_meta / config_meta union stay zero-initialized. +// +void +fill_string_config(RecRecord &r, const char *name, const char *current, const char *def, RecAccessT access) +{ + r.rec_type = RECT_CONFIG; + r.data_type = RECD_STRING; + r.name = name; + r.data.rec_string = const_cast(current); + r.data_default.rec_string = const_cast(def); + r.config_meta.access_type = access; +} + +void +fill_int_config(RecRecord &r, const char *name, RecInt current, RecInt def, RecAccessT access) +{ + r.rec_type = RECT_CONFIG; + r.data_type = RECD_INT; + r.name = name; + r.data.rec_int = current; + r.data_default.rec_int = def; + r.config_meta.access_type = access; +} + +void +fill_int_stat(RecRecord &r, const char *name, RecInt current) +{ + r.rec_type = RECT_PROCESS; // STAT category + r.data_type = RECD_INT; + r.name = name; + r.data.rec_int = current; + r.data_default.rec_int = 0; + + // stat_meta is the active union member after value-initialization of + // RecRecord; explicitly re-establish that to keep the contract clear + // and to give the encoder a well-defined object to read. + r.stat_meta = RecStatMeta{}; + + // Plant a RECA_NO_ACCESS bit pattern at the storage location that the + // overlaid RecConfigMeta would expose as access_type, without making + // config_meta the active union member. A hypothetical encoder that + // reads record.config_meta.access_type without a REC_TYPE_IS_CONFIG + // guard would observe RECA_NO_ACCESS and incorrectly suppress the + // value fields below; the well-defined encoder path only reads + // stat_meta for STAT records. + static_assert(sizeof(RecStatMeta) >= offsetof(RecConfigMeta, access_type) + sizeof(RecAccessT), + "RecStatMeta must fully overlap RecConfigMeta::access_type"); + const RecAccessT bad_access = RECA_NO_ACCESS; + std::memcpy(reinterpret_cast(&r.stat_meta) + offsetof(RecConfigMeta, access_type), &bad_access, sizeof(bad_access)); +} + +YAML::Node +encode_record_node(const RecRecord &record) +{ + // The encoder wraps the actual record fields in a {"record": ...} envelope. + return YAML::convert::encode(record)[constants_rec::REC]; +} +} // namespace + +TEST_CASE("Record YAML encoder exposes values for default-access config records", "[mgmt][rpc][record_yaml]") +{ + RecRecord rec{}; + fill_string_config(rec, "proxy.config.example.normal", "current", "default", RECA_NULL); + const YAML::Node node = encode_record_node(rec); + + REQUIRE(node[constants_rec::DATA_TYPE].as() == "STRING"); + REQUIRE(node[constants_rec::CURRENT_VALUE].as() == "current"); + REQUIRE(node[constants_rec::DEFAULT_VALUE].as() == "default"); +} + +TEST_CASE("Record YAML encoder withholds values for RECA_NO_ACCESS string records", "[mgmt][rpc][record_yaml][no_access]") +{ + RecRecord rec{}; + fill_string_config(rec, "proxy.config.example.secret", "supersecret", "secret-default", RECA_NO_ACCESS); + const YAML::Node node = encode_record_node(rec); + + // Type label and metadata are still expected so callers can enumerate the + // record's existence and tier. + REQUIRE(node[constants_rec::DATA_TYPE].as() == "STRING"); + REQUIRE(node[constants_rec::NAME].as() == "proxy.config.example.secret"); + REQUIRE(node[constants_rec::CONFIG_META][constants_rec::ACCESS_TYPE].as() == RECA_NO_ACCESS); + + // The protected value fields must not appear in the encoded output. + REQUIRE_FALSE(node[constants_rec::CURRENT_VALUE]); + REQUIRE_FALSE(node[constants_rec::DEFAULT_VALUE]); +} + +TEST_CASE("Record YAML encoder withholds values for RECA_NO_ACCESS int records", "[mgmt][rpc][record_yaml][no_access]") +{ + RecRecord rec{}; + fill_int_config(rec, "proxy.config.example.token", 42, 0, RECA_NO_ACCESS); + const YAML::Node node = encode_record_node(rec); + + REQUIRE(node[constants_rec::DATA_TYPE].as() == "INT"); + REQUIRE_FALSE(node[constants_rec::CURRENT_VALUE]); + REQUIRE_FALSE(node[constants_rec::DEFAULT_VALUE]); +} + +TEST_CASE("Record YAML encoder ignores access_type on STAT records", "[mgmt][rpc][record_yaml][union_safety]") +{ + // STAT records do not carry config_meta and must not be filtered by an + // access_type read out of the wrong half of the union. This guards the + // REC_TYPE_IS_CONFIG check that fences the new no-access logic. + RecRecord rec{}; + fill_int_stat(rec, "proxy.process.example.counter", 7); + const YAML::Node node = encode_record_node(rec); + + REQUIRE(node[constants_rec::DATA_TYPE].as() == "INT"); + REQUIRE(node[constants_rec::CURRENT_VALUE].as() == 7); + REQUIRE(node[constants_rec::DEFAULT_VALUE].as() == 0); +} diff --git a/src/mgmt/rpc/handlers/config/Configuration.cc b/src/mgmt/rpc/handlers/config/Configuration.cc index 0c57659d6fa..afe1c538913 100644 --- a/src/mgmt/rpc/handlers/config/Configuration.cc +++ b/src/mgmt/rpc/handlers/config/Configuration.cc @@ -110,8 +110,12 @@ set_config_records(std::string_view const & /* id ATS_UNUSED */, YAML::Node cons { swoc::Rv resp; - // we need the type and the update type for now. - using LookupContext = std::tuple; + // we need the type and the update type for now, plus the access tier so we + // can refuse writes to records the registrant marked as protected, and a + // flag tracking whether the lookup actually returned a CONFIG record (the + // RPC is for config writes; metric/process/plugin records must be + // refused even though RecLookupRecord finds them). + using LookupContext = std::tuple; for (auto const &kv : params) { SetRecordCmdInfo info; @@ -122,20 +126,25 @@ set_config_records(std::string_view const & /* id ATS_UNUSED */, YAML::Node cons continue; } - LookupContext recordCtx; + // Value-initialize the tuple so reads after the lookup are well defined + // even when the callback never assigns a field (non-CONFIG record, or + // an early failure inside the callback). + LookupContext recordCtx{}; // Get record info first. TODO: we may just want to get the full record and then send it back as a response. const auto ret = RecLookupRecord( info.name.c_str(), [](const RecRecord *record, void *data) { - auto &[dataType, checkType, pattern, updateType] = *static_cast(data); + auto &[dataType, checkType, pattern, updateType, accessType, is_config] = *static_cast(data); if (REC_TYPE_IS_CONFIG(record->rec_type)) { + is_config = true; dataType = record->data_type; checkType = record->config_meta.check_type; if (record->config_meta.check_expr) { pattern = record->config_meta.check_expr; } updateType = record->config_meta.update_type; + accessType = record->config_meta.access_type; } }, &recordCtx); @@ -147,7 +156,29 @@ set_config_records(std::string_view const & /* id ATS_UNUSED */, YAML::Node cons } // now set the value. - auto const &[dataType, checkType, pattern, updateType] = recordCtx; + auto const &[dataType, checkType, pattern, updateType, accessType, is_config] = recordCtx; + + // RecLookupRecord finds metrics, config records, etc. + // The set RPC only operates on config records; + // refuse anything else with a tier-specific error code so callers can + // distinguish "wrong record kind" from "record missing". + if (!is_config) { + auto const ec = std::error_code{err::RecordError::RECORD_NOT_CONFIG}; + resp.errata().assign(ec).note("{}", ec); + continue; + } + + // Honour the registrant's access tier. RECA_READ_ONLY records may only + // be changed by in-process code (config file load, environment override, + // etc.); RECA_NO_ACCESS records are not exposed to the management plane + // at all. Surface the per-tier error code in the JSONRPC error.data + // entry so callers can branch on the code instead of parsing messages. + if (accessType == RECA_READ_ONLY || accessType == RECA_NO_ACCESS) { + auto const ec = + std::error_code{accessType == RECA_READ_ONLY ? err::RecordError::RECORD_READ_ONLY : err::RecordError::RECORD_NO_ACCESS}; + resp.errata().assign(ec).note("{}", ec); + continue; + } // run the check only if we have something to check against it. if (pattern != nullptr && RecordValidityCheck(info.value.c_str(), checkType, pattern) == false) { diff --git a/src/mgmt/rpc/handlers/hostdb/HostDB.cc b/src/mgmt/rpc/handlers/hostdb/HostDB.cc index d90343bb1f9..9759a1cd19e 100644 --- a/src/mgmt/rpc/handlers/hostdb/HostDB.cc +++ b/src/mgmt/rpc/handlers/hostdb/HostDB.cc @@ -157,8 +157,8 @@ template <> struct convert { info_node["ip"] = std::string(buf); } - info_node["health"]["last_failure"] = info.last_failure.load().time_since_epoch().count(); - info_node["health"]["fail_count"] = static_cast(info.fail_count.load()); + info_node["health"]["last_failure"] = info.last_fail_time().time_since_epoch().count(); + info_node["health"]["fail_count"] = static_cast(info.fail_count()); node["info"].push_back(info_node); } diff --git a/src/proxy/IPAllow.cc b/src/proxy/IPAllow.cc index 56c52a621d8..d0e047ef98a 100644 --- a/src/proxy/IPAllow.cc +++ b/src/proxy/IPAllow.cc @@ -222,9 +222,10 @@ IpAllow::IpAllow(const char *ip_allow_config_var, const char *ip_categories_conf std::string_view::size_type s, e; for (s = 0, e = 0; s < subjects_sv.size() && e != subjects_sv.npos; s = e + 1) { e = subjects_sv.find(",", s); - std::string_view subject_sv = subjects_sv.substr(s, e); + std::string_view subject_sv = subjects_sv.substr(s, e == subjects_sv.npos ? subjects_sv.npos : e - s); if (i >= MAX_SUBJECTS) { Error("Too many ACL subjects were provided"); + break; } if (subject_sv == "PEER") { subjects[i] = Subject::PEER; diff --git a/src/proxy/ParentConsistentHash.cc b/src/proxy/ParentConsistentHash.cc index ce0cc46c216..3d2be5173a1 100644 --- a/src/proxy/ParentConsistentHash.cc +++ b/src/proxy/ParentConsistentHash.cc @@ -42,10 +42,8 @@ ParentConsistentHash::ParentConsistentHash(ParentRecord *parent_record) selected_algorithm = parent_record->consistent_hash_algorithm; hash_seed0 = parent_record->consistent_hash_seed0; hash_seed1 = parent_record->consistent_hash_seed1; - ink_zero(foundParents); - - hash[PRIMARY] = createHashInstance(selected_algorithm, hash_seed0, hash_seed1); - chash[PRIMARY] = std::make_unique(parent_record->consistent_hash_replicas); + hash[PRIMARY] = createHashInstance(selected_algorithm, hash_seed0, hash_seed1); + chash[PRIMARY] = std::make_unique(parent_record->consistent_hash_replicas); for (i = 0; i < parent_record->num_parents; i++) { chash[PRIMARY]->insert(&(parent_record->parents[i]), parent_record->parents[i].weight, hash[PRIMARY].get()); diff --git a/src/proxy/ProtocolProbeSessionAccept.cc b/src/proxy/ProtocolProbeSessionAccept.cc index 40ff286ea02..3ba94633f22 100644 --- a/src/proxy/ProtocolProbeSessionAccept.cc +++ b/src/proxy/ProtocolProbeSessionAccept.cc @@ -127,8 +127,10 @@ struct ProtocolProbeTrampoline : public Continuation, public ProtocolProbeSessio "ioCompletionEvent: proxy protocol DOES NOT have a configured allowlist of trusted IPs but proxy protocol is " "ernabled on this port - processing all connections"); } - - if (netvc->has_proxy_protocol(reader)) { + HttpConfigParams *param = HttpConfig::acquire(); + int max_header_size = param->pp_hdr_max_size; + HttpConfig::release(param); + if (netvc->has_proxy_protocol(reader, max_header_size)) { Dbg(dbg_ctl_proxyprotocol, "ioCompletionEvent: http has proxy protocol header"); } else { Dbg(dbg_ctl_proxyprotocol, "ioCompletionEvent: proxy protocol was enabled, but Proxy Protocol header was not present"); diff --git a/src/proxy/ProxyTransaction.cc b/src/proxy/ProxyTransaction.cc index c2a5a4584e7..a301ce20deb 100644 --- a/src/proxy/ProxyTransaction.cc +++ b/src/proxy/ProxyTransaction.cc @@ -23,7 +23,7 @@ #include "proxy/http/HttpSM.h" #include "proxy/Plugin.h" -#include "proxy/PreTransactionLogData.h" +#include "proxy/NonHttpSmLogData.h" #include "proxy/logging/LogAccess.h" #include "proxy/logging/TransactionLogData.h" #include "proxy/logging/Log.h" @@ -343,7 +343,7 @@ get_pseudo_header_value(HTTPHdr const &hdr, std::string_view name) } // end anonymous namespace void -ProxyTransaction::log_pre_transaction_access(HTTPHdr const *request, const char *protocol_str) +ProxyTransaction::log_non_http_sm_access(HTTPHdr const *request, const char *protocol_str) { if (get_sm() != nullptr) { return; @@ -358,7 +358,7 @@ ProxyTransaction::log_pre_transaction_access(HTTPHdr const *request, const char return; } - PreTransactionLogData data; + NonHttpSmLogData data; data.owned_client_request.create(HTTPType::REQUEST, request->version_get()); data.owned_client_request.copy(request); diff --git a/src/proxy/hdrs/HdrToken.cc b/src/proxy/hdrs/HdrToken.cc index 712a5ed7f73..6d2beabfec9 100644 --- a/src/proxy/hdrs/HdrToken.cc +++ b/src/proxy/hdrs/HdrToken.cc @@ -407,7 +407,7 @@ hdrtoken_init() heap_size += packed_prefix_str_len; } - _hdrtoken_strs_heap_f = static_cast(ats_malloc(heap_size)); + _hdrtoken_strs_heap_f = static_cast(ats_calloc(1, heap_size)); _hdrtoken_strs_heap_l = _hdrtoken_strs_heap_f + heap_size - 1; char *heap_ptr = const_cast(_hdrtoken_strs_heap_f); diff --git a/src/proxy/hdrs/unit_tests/test_Hdrs.cc b/src/proxy/hdrs/unit_tests/test_Hdrs.cc index f318723b25f..4e90846edb4 100644 --- a/src/proxy/hdrs/unit_tests/test_Hdrs.cc +++ b/src/proxy/hdrs/unit_tests/test_Hdrs.cc @@ -183,6 +183,13 @@ test_http_hdr_copy_over_aux(int testnum, const char *request, const char *respon HTTPHdr copy1; HTTPHdr copy2; + ts::PostScript cleanup([&]() -> void { + req_hdr.destroy(); + resp_hdr.destroy(); + copy1.destroy(); + copy2.destroy(); + }); + HTTPParser parser; const char *start; const char *end; @@ -236,15 +243,11 @@ test_http_hdr_copy_over_aux(int testnum, const char *request, const char *respon copy1.create(HTTPType::REQUEST); copy1.copy(&req_hdr); comp_str = comp_http_hdr(&req_hdr, ©1); - if (comp_str) { - goto done; - } - copy2.create(HTTPType::RESPONSE); - copy2.copy(&resp_hdr); - comp_str = comp_http_hdr(&resp_hdr, ©2); - if (comp_str) { - goto done; + if (!comp_str) { + copy2.create(HTTPType::RESPONSE); + copy2.copy(&resp_hdr); + comp_str = comp_http_hdr(&resp_hdr, ©2); } // The APIs for copying headers uses memcpy() which can be unsafe for @@ -252,32 +255,24 @@ test_http_hdr_copy_over_aux(int testnum, const char *request, const char *respon // created in the first place honestly, since nothing else does this. /*** (4) Gender bending copying ***/ - copy1.copy(&resp_hdr); - comp_str = comp_http_hdr(&resp_hdr, ©1); - if (comp_str) { - goto done; + if (!comp_str) { + copy1.copy(&resp_hdr); + comp_str = comp_http_hdr(&resp_hdr, ©1); } - copy2.copy(&req_hdr); - comp_str = comp_http_hdr(&req_hdr, ©2); - if (comp_str) { - goto done; + if (!comp_str) { + copy2.copy(&req_hdr); + comp_str = comp_http_hdr(&req_hdr, ©2); } -done: - req_hdr.destroy(); - resp_hdr.destroy(); - copy1.destroy(); - copy2.destroy(); - if (comp_str) { printf("FAILED: (test #%d) copy & compare: %s\n", testnum, comp_str); printf("REQ:\n[%.*s]\n", static_cast(strlen(request)), request); printf("RESP :\n[%.*s]\n", static_cast(strlen(response)), response); return (0); - } else { - return (1); } + + return (1); } int @@ -379,10 +374,16 @@ test_http_hdr_print_and_copy_aux(int testnum, const char *request, const char *r { ParseResult err; HTTPHdr hdr; + HTTPHdr new_hdr; HTTPParser parser; const char *start; const char *end; + ts::PostScript cleanup([&]() -> void { + hdr.destroy(); + new_hdr.destroy(); + }); + char prt_buf[2048]; int prt_bufsize = sizeof(prt_buf); int prt_bufindex, prt_dumpoffset, prt_ret; @@ -416,7 +417,7 @@ test_http_hdr_print_and_copy_aux(int testnum, const char *request, const char *r } /*** (2) copy the request header ***/ - HTTPHdr new_hdr, marshal_hdr; + HTTPHdr marshal_hdr; TestRefCountObj ref; // Pretend to pin this object with a refcount. @@ -524,9 +525,6 @@ test_http_hdr_print_and_copy_aux(int testnum, const char *request, const char *r return (0); } - hdr.destroy(); - new_hdr.destroy(); - if (test_http_hdr_copy_over_aux(testnum, request, response) == 0) { return 0; } diff --git a/src/proxy/http/HttpConfig.cc b/src/proxy/http/HttpConfig.cc index 9072c25ac24..aa9775eda13 100644 --- a/src/proxy/http/HttpConfig.cc +++ b/src/proxy/http/HttpConfig.cc @@ -127,8 +127,8 @@ static const ConfigEnumPair SessionSharingMatch bool HttpConfig::load_server_session_sharing_match(std::string_view key, MgmtByte &mask) { - MgmtByte value; - mask = 0; + MgmtByte value = 0; + mask = 0; // Parse through and build up mask size_t start = 0; size_t offset = 0; @@ -960,6 +960,7 @@ HttpConfig::startup() HttpEstablishStaticConfigLongLong(c.http_request_line_max_size, "proxy.config.http.request_line_max_size"); HttpEstablishStaticConfigLongLong(c.http_hdr_field_max_size, "proxy.config.http.header_field_max_size"); + HttpEstablishStaticConfigLongLong(c.pp_hdr_max_size, "proxy.config.proxy_protocol.max_header_size"); HttpEstablishStaticConfigByte(c.disable_ssl_parenting, "proxy.config.http.parent_proxy.disable_connect_tunneling"); HttpEstablishStaticConfigByte(c.oride.forward_connect_method, "proxy.config.http.forward_connect_method"); @@ -1272,6 +1273,7 @@ HttpConfig::reconfigure() params->http_request_line_max_size = m_master.http_request_line_max_size; params->http_hdr_field_max_size = m_master.http_hdr_field_max_size; + params->pp_hdr_max_size = m_master.pp_hdr_max_size; if (params->oride.connection_tracker_config.server_max > 0 && params->oride.connection_tracker_config.server_max < params->oride.connection_tracker_config.server_min) { diff --git a/src/proxy/http/HttpProxyServerMain.cc b/src/proxy/http/HttpProxyServerMain.cc index 3f9dc8e984f..ed4431adf07 100644 --- a/src/proxy/http/HttpProxyServerMain.cc +++ b/src/proxy/http/HttpProxyServerMain.cc @@ -376,4 +376,16 @@ stop_HttpProxyServer() { sslNetProcessor.stop_accept(); netProcessor.stop_accept(); + + for (auto &acceptor : HttpProxyAcceptors) { + delete acceptor._accept; + acceptor._accept = nullptr; + } + HttpProxyAcceptors.clear(); + + delete plugin_http_accept; + plugin_http_accept = nullptr; + + delete plugin_http_transparent_accept; + plugin_http_transparent_accept = nullptr; } diff --git a/src/proxy/http/HttpSM.cc b/src/proxy/http/HttpSM.cc index 4a862bd1770..1b9accd99f9 100644 --- a/src/proxy/http/HttpSM.cc +++ b/src/proxy/http/HttpSM.cc @@ -1694,6 +1694,7 @@ HttpSM::handle_api_return() break; } case HttpTransact::StateMachineAction_t::SERVER_READ: { + milestones.mark(TS_MILESTONE_UA_BEGIN_WRITE); if (unlikely(t_state.did_upgrade_succeed)) { // We've successfully handled the upgrade, let's now setup // a blind tunnel. @@ -1730,6 +1731,7 @@ HttpSM::handle_api_return() break; } case HttpTransact::StateMachineAction_t::SERVE_FROM_CACHE: { + milestones.mark(TS_MILESTONE_UA_BEGIN_WRITE); HttpTunnelProducer *p = setup_cache_read_transfer(); tunnel.tunnel_run(p); break; @@ -4773,10 +4775,10 @@ HttpSM::do_hostdb_update_if_necessary() if (track_connect_fail()) { this->mark_host_failure(&t_state.dns_info, ts_clock::from_time_t(t_state.client_request_time)); } else { - if (t_state.dns_info.mark_active_server_alive()) { + if (t_state.dns_info.mark_active_server_up()) { char addrbuf[INET6_ADDRPORTSTRLEN]; ats_ip_nptop(&t_state.current.server->dst_addr.sa, addrbuf, sizeof(addrbuf)); - ATS_PROBE2(mark_active_server_alive, sm_id, addrbuf); + ATS_PROBE2(mark_active_server_up, sm_id, addrbuf); if (t_state.dns_info.record->is_srv()) { SMDbg(dbg_ctl_http, "[%" PRId64 "] hostdb update marking SRV: %s(%s) as up", sm_id, t_state.dns_info.record->name(), addrbuf); @@ -5875,6 +5877,7 @@ HttpSM::do_http_server_open(bool raw, bool only_direct) opt.set_ssl_client_cert_name(t_state.txn_conf->ssl_client_cert_filename); opt.ssl_client_private_key_name = t_state.txn_conf->ssl_client_private_key_filename; opt.ssl_client_ca_cert_name = t_state.txn_conf->ssl_client_ca_cert_filename; + opt.ssl_client_ca_cert_path = t_state.txn_conf->ssl_client_ca_cert_path; if (is_private()) { // If the connection to origin is private, don't try to negotiate the higher overhead H2 opt.alpn_protocols_array_size = -1; @@ -5998,8 +6001,7 @@ HttpSM::do_api_callout_internal() } hook_state.init(cur_hook_id, http_global_hooks, _ua.get_txn() ? _ua.get_txn()->feature_hooks() : nullptr, &api_hooks); - cur_hook = nullptr; - cur_hooks = 0; + cur_hook = nullptr; return state_api_callout(0, nullptr); } @@ -6057,7 +6059,8 @@ HttpSM::mark_host_failure(ResolveInfo *info, ts_time time_down) if (time_down != TS_TIME_ZERO) { ats_ip_nptop(&t_state.current.server->dst_addr.sa, addrbuf, sizeof(addrbuf)); // Increment the fail_count - if (auto [down, fail_count] = info->active->increment_fail_count(time_down, t_state.txn_conf->connect_attempts_rr_retries); + if (auto [down, fail_count] = info->active->increment_fail_count(time_down, t_state.txn_conf->connect_attempts_rr_retries, + t_state.txn_conf->down_server_timeout); down) { char *url_str = t_state.hdr_info.client_request.url_string_get_ref(nullptr); std::string_view host_name{t_state.unmapped_url.host_get()}; diff --git a/src/proxy/http/HttpTransact.cc b/src/proxy/http/HttpTransact.cc index 224c728098d..fa2a9a71452 100644 --- a/src/proxy/http/HttpTransact.cc +++ b/src/proxy/http/HttpTransact.cc @@ -6574,7 +6574,8 @@ HttpTransact::is_response_cacheable(State *s, HTTPHdr *request, HTTPHdr *respons } if ((response_code == HTTPStatus::OK) || (response_code == HTTPStatus::NOT_MODIFIED) || - (response_code == HTTPStatus::NON_AUTHORITATIVE_INFORMATION) || (response_code == HTTPStatus::MOVED_PERMANENTLY) || + (response_code == HTTPStatus::NON_AUTHORITATIVE_INFORMATION) || (response_code == HTTPStatus::NO_CONTENT) || + (response_code == HTTPStatus::MOVED_PERMANENTLY) || (response_code == HTTPStatus::PERMANENT_REDIRECT) || (response_code == HTTPStatus::MULTIPLE_CHOICES) || (response_code == HTTPStatus::GONE)) { TxnDbg(dbg_ctl_http_trans, "YES response code seems fine"); return true; diff --git a/src/proxy/http/HttpVCTable.cc b/src/proxy/http/HttpVCTable.cc index 01690daee00..056e833c6de 100644 --- a/src/proxy/http/HttpVCTable.cc +++ b/src/proxy/http/HttpVCTable.cc @@ -30,11 +30,7 @@ class HttpSM; -HttpVCTable::HttpVCTable(HttpSM *mysm) -{ - memset(&vc_table, 0, sizeof(vc_table)); - sm = mysm; -} +HttpVCTable::HttpVCTable(HttpSM *mysm) : sm(mysm) {} HttpVCTableEntry * HttpVCTable::new_entry() diff --git a/src/proxy/http/PreWarmManager.cc b/src/proxy/http/PreWarmManager.cc index fafa7dd1402..c2172247644 100644 --- a/src/proxy/http/PreWarmManager.cc +++ b/src/proxy/http/PreWarmManager.cc @@ -576,6 +576,7 @@ PreWarmSM::_connect(const IpEndpoint &addr) opt.ssl_client_cert_name = http_conf_params->oride.ssl_client_cert_filename; opt.ssl_client_private_key_name = http_conf_params->oride.ssl_client_private_key_filename; opt.ssl_client_ca_cert_name = http_conf_params->oride.ssl_client_ca_cert_filename; + opt.ssl_client_ca_cert_path = http_conf_params->oride.ssl_client_ca_cert_path; SCOPED_MUTEX_LOCK(lock, mutex, this_ethread()); connect_action_handle = sslNetProcessor.connect_re(this, &addr.sa, opt); diff --git a/src/proxy/http/remap/NextHopHealthStatus.cc b/src/proxy/http/remap/NextHopHealthStatus.cc index 6c86c26bf30..35e42aa6115 100644 --- a/src/proxy/http/remap/NextHopHealthStatus.cc +++ b/src/proxy/http/remap/NextHopHealthStatus.cc @@ -120,8 +120,12 @@ NextHopHealthStatus::markNextHop(TSHttpTxn txn, const char *hostname, const int new_fail_count = h->failCount = 1; } } else if (result.retry == true) { - h->failedAt = _now; - new_fail_count = h->failCount += 1; + h->failedAt = _now; + if (h->failCount < UINT32_MAX) { + new_fail_count = ++h->failCount; + } else { + new_fail_count = UINT32_MAX; + } } } // end lock guard @@ -132,19 +136,25 @@ NextHopHealthStatus::markNextHop(TSHttpTxn txn, const char *hostname, const int { // lock guard std::lock_guard lock(h->_mutex); if ((h->failedAt.load() + retry_time) < _now) { + h->failedAt = _now; new_fail_count = h->failCount = 1; - h->failedAt = _now; } else { - new_fail_count = h->failCount += 1; + if (h->failCount < UINT32_MAX) { + new_fail_count = ++h->failCount; + } else { + new_fail_count = UINT32_MAX; + } } } // end of lock_guard - NH_Dbg(NH_DBG_CTL, "[%" PRId64 "] Parent fail count increased to %d for %s", sm_id, new_fail_count, h->hostname.c_str()); + NH_Dbg(NH_DBG_CTL, "[%" PRId64 "] Parent fail count increased to %" PRIu32 " for %s", sm_id, new_fail_count, + h->hostname.c_str()); } if (new_fail_count >= fail_threshold) { h->set_unavailable(); - NH_Note("[%" PRId64 "] Failure threshold met failcount:%d >= threshold:%" PRId64 ", http parent proxy %s marked down", sm_id, - new_fail_count, fail_threshold, h->hostname.c_str()); + NH_Note("[%" PRId64 "] Failure threshold met failcount:%" PRIu32 " >= threshold:%" PRId64 + ", http parent proxy %s marked down", + sm_id, new_fail_count, fail_threshold, h->hostname.c_str()); NH_Dbg(NH_DBG_CTL, "[%" PRId64 "] NextHop %s marked unavailable, h->available=%s", sm_id, h->hostname.c_str(), (h->available.load()) ? "true" : "false"); } diff --git a/src/proxy/http/remap/RemapConfig.cc b/src/proxy/http/remap/RemapConfig.cc index 95d51af39cf..23f859fb60d 100644 --- a/src/proxy/http/remap/RemapConfig.cc +++ b/src/proxy/http/remap/RemapConfig.cc @@ -1021,6 +1021,11 @@ process_regex_mapping_config(const char *from_host_lower, url_mapping *new_mappi Warning("Substitution id [%c] has no corresponding capture pattern in regex [%s]", to_host[i + 1], from_host_lower); goto lFail; } + if (reg_map->n_substitutions >= UrlRewrite::MAX_REGEX_SUBS) { + Warning("too many substitution markers in regex remap target [%.*s], saw %d markers, max %d", to_host_len, to_host.data(), + reg_map->n_substitutions + 1, UrlRewrite::MAX_REGEX_SUBS); + goto lFail; + } reg_map->substitution_markers[reg_map->n_substitutions] = i; reg_map->substitution_ids[reg_map->n_substitutions] = substitution_id; ++reg_map->n_substitutions; diff --git a/src/proxy/http/remap/unit-tests/plugin_testing_common.h b/src/proxy/http/remap/unit-tests/plugin_testing_common.h index fe01a4bcdfe..b4b163a1f51 100644 --- a/src/proxy/http/remap/unit-tests/plugin_testing_common.h +++ b/src/proxy/http/remap/unit-tests/plugin_testing_common.h @@ -65,8 +65,8 @@ class PluginDebugObject } /* Input fields used to set the test behavior of the plugin call-backs */ - bool fail = false; /* tell the plugin call-back to fail for testing purposuses */ - void *input_ih; /* the value to be returned by the plugin instance init function */ + bool fail{false}; /* tell the plugin call-back to fail for testing purposes */ + void *input_ih{nullptr}; /* the value to be returned by the plugin instance init function */ /* Output fields showing what happend during the test */ const PluginThreadContext *contextInit = nullptr; /* plugin initialization context */ diff --git a/src/proxy/http/remap/unit-tests/test_NextHopRoundRobin.cc b/src/proxy/http/remap/unit-tests/test_NextHopRoundRobin.cc index 908dbc35e5b..52ae3b511f0 100644 --- a/src/proxy/http/remap/unit-tests/test_NextHopRoundRobin.cc +++ b/src/proxy/http/remap/unit-tests/test_NextHopRoundRobin.cc @@ -381,3 +381,72 @@ SCENARIO("Testing NextHopRoundRobin class, using policy 'latched'", "[NextHopRou } } } + +SCENARIO("Testing NextHopHealthStatus failCount overflow saturation", "[NextHopHealthStatus]") +{ + // No thread setup, forbid use of thread local allocators. + cmd_disable_pfreelist = true; + http_init(); + + GIVEN("Loading the round-robin-tests.yaml config for overflow test.") + { + NextHopStrategyFactory nhf(TS_SRC_DIR "/round-robin-tests.yaml"); + NextHopSelectionStrategy *const strategy = nhf.strategyInstance("rr-strict-exhaust-ring"); + + REQUIRE(nhf.strategies_loaded == true); + REQUIRE(strategy != nullptr); + + WHEN("failCount is near UINT32_MAX and markNextHop is called") + { + HttpSM sm; + ParentResult *result = &sm.t_state.parent_result; + TSHttpTxn txnp = reinterpret_cast(&sm); + + // Select a host so result is populated. + build_request(20001, &sm, nullptr, "rabbit.net", nullptr); + strategy->findNextHop(txnp); + REQUIRE(result->result == ParentResultType::SPECIFIED); + REQUIRE(result->hostname != nullptr); + + // Get the HostRecord for the selected host. + std::shared_ptr host_rec; + for (auto &group : strategy->host_groups) { + for (auto &h : group) { + if (h->hostname == result->hostname) { + host_rec = h; + break; + } + } + if (host_rec) { + break; + } + } + REQUIRE(host_rec != nullptr); + + // Set failCount to UINT32_MAX - 1 to test saturation. + host_rec->failCount = UINT32_MAX - 1; + host_rec->failedAt = time(nullptr); + + // Set fail_threshold very high so set_unavailable doesn't interfere. + extern char _my_txn_conf[]; + auto *oride = reinterpret_cast(_my_txn_conf); + oride->parent_fail_threshold = static_cast(UINT32_MAX) + 1; + oride->parent_retry_time = 30; // large retry window so we stay in the increment path + + THEN("failCount saturates at UINT32_MAX and does not wrap to 0") + { + // First markNextHop: UINT32_MAX-1 -> UINT32_MAX + strategy->markNextHop(txnp, result->hostname, result->port, NHCmd::MARK_DOWN); + CHECK(host_rec->failCount.load() == UINT32_MAX); + + // Second markNextHop: should stay at UINT32_MAX (saturated) + strategy->markNextHop(txnp, result->hostname, result->port, NHCmd::MARK_DOWN); + CHECK(host_rec->failCount.load() == UINT32_MAX); + + // Verify host is still available (threshold not met since threshold > UINT32_MAX) + CHECK(host_rec->available.load() == true); + } + br_destroy(sm); + } + } +} diff --git a/src/proxy/http/remap/unit-tests/test_RemapRules.cc b/src/proxy/http/remap/unit-tests/test_RemapRules.cc index 012e8b55bd7..f51e39d0dd6 100644 --- a/src/proxy/http/remap/unit-tests/test_RemapRules.cc +++ b/src/proxy/http/remap/unit-tests/test_RemapRules.cc @@ -37,6 +37,7 @@ #include "tscore/BaseLogFile.h" #include "tsutil/PostScript.h" +#include #include #include @@ -122,6 +123,49 @@ SCENARIO("Parsing ACL named filters", "[proxy][remap]") } } +std::string +make_regex_remap_with_substitutions(int n_substitutions) +{ + std::string substitutions; + + substitutions.reserve(n_substitutions * 2); + for (int i = 0; i < n_substitutions; ++i) { + substitutions += "$0"; + } + + return "regex_map http://([^.]+)\\.example\\.com/ http://" + substitutions + ".origin.example.com/\n"; +} + +SCENARIO("Parsing regex remap substitutions", "[proxy][remap]") +{ + GIVEN("A regex remap target with the maximum number of substitution markers") + { + std::unique_ptr urlrw = std::make_unique(); + + auto cpath = write_test_remap(make_regex_remap_with_substitutions(UrlRewrite::MAX_REGEX_SUBS), "max-regex-substitutions"); + ts::PostScript file_cleanup([&]() -> void { std::filesystem::remove(cpath.c_str()); }); + + THEN("the remap parse succeeds") + { + REQUIRE(urlrw->BuildTable(cpath.c_str()) == TS_SUCCESS); + } + } + + GIVEN("A regex remap target with too many substitution markers") + { + std::unique_ptr urlrw = std::make_unique(); + + auto cpath = + write_test_remap(make_regex_remap_with_substitutions(UrlRewrite::MAX_REGEX_SUBS + 1), "too-many-regex-substitutions"); + ts::PostScript file_cleanup([&]() -> void { std::filesystem::remove(cpath.c_str()); }); + + THEN("the remap parse fails") + { + REQUIRE(urlrw->BuildTable(cpath.c_str()) != TS_SUCCESS); + } + } +} + struct EasyURL { URL url; HdrHeap *heap; diff --git a/src/proxy/http2/Http2ConnectionState.cc b/src/proxy/http2/Http2ConnectionState.cc index d7bf61b860f..5b8cea9be63 100644 --- a/src/proxy/http2/Http2ConnectionState.cc +++ b/src/proxy/http2/Http2ConnectionState.cc @@ -497,7 +497,7 @@ Http2ConnectionState::rcv_headers_frame(const Http2Frame &frame) "recv headers enhance your calm"); } else { if (!stream->trailing_header_is_possible() && !stream->is_outbound_connection()) { - stream->log_pre_transaction_access(stream->get_receive_header(), "http/2"); + stream->log_non_http_sm_access(stream->get_receive_header(), "http/2"); } return Http2Error(Http2ErrorClass::HTTP2_ERROR_CLASS_STREAM, Http2ErrorCode::HTTP2_ERROR_PROTOCOL_ERROR, "recv headers malformed request"); @@ -1112,7 +1112,7 @@ Http2ConnectionState::rcv_continuation_frame(const Http2Frame &frame) "continuation enhance your calm"); } else { if (!stream->trailing_header_is_possible() && !stream->is_outbound_connection()) { - stream->log_pre_transaction_access(stream->get_receive_header(), "http/2"); + stream->log_non_http_sm_access(stream->get_receive_header(), "http/2"); } return Http2Error(Http2ErrorClass::HTTP2_ERROR_CLASS_CONNECTION, Http2ErrorCode::HTTP2_ERROR_PROTOCOL_ERROR, "continuation malformed request"); diff --git a/src/proxy/http3/Http3Frame.cc b/src/proxy/http3/Http3Frame.cc index b3de9d2bffe..0060dad16d9 100644 --- a/src/proxy/http3/Http3Frame.cc +++ b/src/proxy/http3/Http3Frame.cc @@ -505,7 +505,7 @@ Http3FrameFactory::create(IOBufferReader &reader) ts::Http3Config::scoped_config params; Http3Frame *frame = nullptr; - uint8_t type_buf[FRAME_TYPE_MAX_BYTES]; + uint8_t type_buf[FRAME_TYPE_MAX_BYTES]{}; reader.memcpy(type_buf, sizeof(type_buf)); Http3FrameType type = Http3Frame::type(type_buf, sizeof(type_buf)); diff --git a/src/proxy/http3/Http3HeaderVIOAdaptor.cc b/src/proxy/http3/Http3HeaderVIOAdaptor.cc index 396733d6252..9d3e256b7f6 100644 --- a/src/proxy/http3/Http3HeaderVIOAdaptor.cc +++ b/src/proxy/http3/Http3HeaderVIOAdaptor.cc @@ -110,7 +110,7 @@ Http3HeaderVIOAdaptor::_on_qpack_decode_complete() NON_TRAILER)) { Dbg(dbg_ctl_http3, "Header is invalid"); if (this->_txn != nullptr) { - this->_txn->log_pre_transaction_access(&this->_header, "http/3"); + this->_txn->log_non_http_sm_access(&this->_header, "http/3"); } return -1; } @@ -118,7 +118,7 @@ Http3HeaderVIOAdaptor::_on_qpack_decode_complete() if (res != 0) { Dbg(dbg_ctl_http3, "ParseResult::ERROR"); if (this->_txn != nullptr) { - this->_txn->log_pre_transaction_access(&this->_header, "http/3"); + this->_txn->log_non_http_sm_access(&this->_header, "http/3"); } return -1; } diff --git a/src/proxy/logging/Log.cc b/src/proxy/logging/Log.cc index 3b7504177e6..f6aeed1599a 100644 --- a/src/proxy/logging/Log.cc +++ b/src/proxy/logging/Log.cc @@ -505,6 +505,10 @@ Log::init_fields() global_field_list.add(field, false); field_symbol_hash.emplace("cluc", field); + field = new LogField("cache_key_hash", "ckh", LogField::STRING, &LogAccess::marshal_cache_key_hash, &LogAccess::unmarshal_str); + global_field_list.add(field, false); + field_symbol_hash.emplace("ckh", field); + field = new LogField("client_sni_server_name", "cssn", LogField::STRING, &LogAccess::marshal_client_sni_server_name, &LogAccess::unmarshal_str); global_field_list.add(field, false); @@ -859,6 +863,10 @@ Log::init_fields() global_field_list.add(field, false); field_symbol_hash.emplace("sshv", field); + field = new LogField("milestones_csv", "mstsms", LogField::STRING, &LogAccess::marshal_milestones_csv, &LogAccess::unmarshal_str); + global_field_list.add(field, false); + field_symbol_hash.emplace("mstsms", field); + field = new LogField("server_resp_time", "stms", LogField::sINT, &LogAccess::marshal_server_resp_time_ms, &LogAccess::unmarshal_int_to_str); global_field_list.add(field, false); @@ -1045,6 +1053,11 @@ Log::init_fields() global_field_list.add(field, false); field_symbol_hash.emplace("pptv", field); + field = new LogField("proxy_protocol_tls_group", "pptg", LogField::STRING, &LogAccess::marshal_proxy_protocol_tls_group, + &LogAccess::unmarshal_str); + global_field_list.add(field, false); + field_symbol_hash.emplace("pptg", field); + field = new LogField("version_build_number", "vbn", LogField::STRING, &LogAccess::marshal_version_build_number, &LogAccess::unmarshal_str); global_field_list.add(field, false); diff --git a/src/proxy/logging/LogAccess.cc b/src/proxy/logging/LogAccess.cc index 282a04a2fc5..ec8eb803f34 100644 --- a/src/proxy/logging/LogAccess.cc +++ b/src/proxy/logging/LogAccess.cc @@ -35,6 +35,7 @@ #include "swoc/BufferWriter.h" #include "tscore/Encoding.h" #include "tscore/ink_inet.h" +#include "tscore/ink_base64.h" char INVALID_STR[] = "!INVALID_STR!"; @@ -663,6 +664,43 @@ LogAccess::unmarshal_int_to_str(char **buf, char *dest, int len) return -1; } +/*------------------------------------------------------------------------- + LogAccess::unmarshal_milestone_diff + + Unmarshal a milestone difference value. Returns "-" when the + marshalled value is -1 (the "missing" sentinel from difference_msec, + meaning one or both milestones were unset). Other negative values + (reversed milestone order) are preserved as numeric output for + debugging. + -------------------------------------------------------------------------*/ + +int +LogAccess::unmarshal_milestone_diff(char **buf, char *dest, int len) +{ + ink_assert(buf != nullptr); + ink_assert(*buf != nullptr); + ink_assert(dest != nullptr); + + int64_t val = unmarshal_int(buf); + if (val == -1) { + if (len >= 1) { + dest[0] = '-'; + return 1; + } + DBG_UNMARSHAL_DEST_OVERRUN + return -1; + } + + char val_buf[128]; + int val_len = unmarshal_itoa(val, val_buf + 127); + if (val_len < len) { + memcpy(dest, val_buf + 128 - val_len, val_len); + return val_len; + } + DBG_UNMARSHAL_DEST_OVERRUN + return -1; +} + /*------------------------------------------------------------------------- LogAccess::unmarshal_int_to_str_hex @@ -1688,6 +1726,25 @@ LogAccess::marshal_proxy_protocol_tls_version(char *buf) return len; } +int +LogAccess::marshal_proxy_protocol_tls_group(char *buf) +{ + int len = INK_MIN_ALIGN; + std::string_view group = m_data->get_pp_tls_group(); + + if (!group.empty()) { + len = padded_length(static_cast(group.size()) + 1); + if (buf) { + marshal_mem(buf, group.data(), static_cast(group.size()), len); + } + } else { + if (buf) { + marshal_mem(buf, nullptr, 0, len); + } + } + return len; +} + /*------------------------------------------------------------------------- -------------------------------------------------------------------------*/ int @@ -2976,6 +3033,35 @@ LogAccess::marshal_cache_write_transform_code(char *buf) return INK_MIN_ALIGN; } +/*------------------------------------------------------------------------- + -------------------------------------------------------------------------*/ + +int +LogAccess::marshal_cache_key_hash(char *buf) +{ + const ts::CryptoHash *hash = m_data->get_cache_lookup_hash(); + + if (!hash || hash->is_zero()) { + if (buf) { + marshal_str(buf, "-", padded_length(2)); + } + return padded_length(2); + } + + constexpr size_t b64_bufsize = ats_base64_encode_dstlen(CRYPTO_HASH_SIZE); + char b64_str[b64_bufsize]; + size_t b64_len = 0; + + ats_base64_encode(reinterpret_cast(hash->u8), CRYPTO_HASH_SIZE, b64_str, b64_bufsize, &b64_len); + + int len = padded_length(b64_len + 1); + + if (buf) { + marshal_str(buf, b64_str, len); + } + return len; +} + /*------------------------------------------------------------------------- -------------------------------------------------------------------------*/ @@ -3362,6 +3448,49 @@ LogAccess::marshal_milestone_diff(TSMilestonesType ms1, TSMilestonesType ms2, ch return INK_MIN_ALIGN; } +int +LogAccess::marshal_milestones_csv(char *buf) +{ + Dbg(dbg_ctl_log_unmarshal_data, "marshal_milestones_csv"); + + swoc::LocalBufferWriter<256> bw; + + TransactionMilestones const *milestones = m_data->get_milestones(); + if (milestones == nullptr) { + return 0; + } + + bw.print("{}", milestones->difference_msec(TS_MILESTONE_TLS_HANDSHAKE_START, TS_MILESTONE_TLS_HANDSHAKE_END)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_UA_BEGIN)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_UA_FIRST_READ)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_UA_READ_HEADER_DONE)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_CACHE_OPEN_READ_BEGIN)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_CACHE_OPEN_READ_END)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_CACHE_OPEN_WRITE_BEGIN)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_CACHE_OPEN_WRITE_END)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_DNS_LOOKUP_BEGIN)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_DNS_LOOKUP_END)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_SERVER_CONNECT)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_SERVER_CONNECT_END)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_SERVER_FIRST_READ)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_SERVER_READ_HEADER_DONE)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_SERVER_CLOSE)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_UA_BEGIN_WRITE)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_UA_CLOSE)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_SM_FINISH)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_PLUGIN_ACTIVE)); + bw.print(",{}", milestones->difference_msec(TS_MILESTONE_SM_START, TS_MILESTONE_PLUGIN_TOTAL)); + bw.print("\0"); + + auto const view = bw.view(); + int const len = LogAccess::padded_strlen(view.data()); + + if (nullptr != buf) { + marshal_str(buf, view.data(), len); + } + return len; +} + void LogAccess::set_http_header_field(LogField::Container container, char *field, char *buf, int len) { diff --git a/src/proxy/logging/LogField.cc b/src/proxy/logging/LogField.cc index ea3dbe84565..e3e03ea11ab 100644 --- a/src/proxy/logging/LogField.cc +++ b/src/proxy/logging/LogField.cc @@ -396,7 +396,7 @@ LogField::LogField(const char *field, Container container) if (0 != rv) { Note("Invalid milestone range in LogField ctor: %s", m_name); } - m_unmarshal_func = &(LogAccess::unmarshal_int_to_str); + m_unmarshal_func = &(LogAccess::unmarshal_milestone_diff); m_type = LogField::sINT; break; } diff --git a/src/proxy/logging/TransactionLogData.cc b/src/proxy/logging/TransactionLogData.cc index 1a6b0ff4de7..c1c613f03d9 100644 --- a/src/proxy/logging/TransactionLogData.cc +++ b/src/proxy/logging/TransactionLogData.cc @@ -22,7 +22,7 @@ */ #include "proxy/logging/TransactionLogData.h" -#include "proxy/PreTransactionLogData.h" +#include "proxy/NonHttpSmLogData.h" #include "proxy/http/HttpSM.h" #include "proxy/logging/LogAccess.h" #include "proxy/hdrs/MIME.h" @@ -92,7 +92,7 @@ TransactionLogData::TransactionLogData(HttpSM *sm) : m_http_sm(sm) ink_assert(sm != nullptr); } -TransactionLogData::TransactionLogData(PreTransactionLogData const &pre_data) : m_pre_data(&pre_data) {} +TransactionLogData::TransactionLogData(NonHttpSmLogData const &non_http_sm_data) : m_non_http_sm_data(&non_http_sm_data) {} void * TransactionLogData::http_sm_for_plugins() const @@ -111,7 +111,7 @@ TransactionLogData::get_milestones() const if (likely(m_http_sm != nullptr)) { return &m_http_sm->milestones; } - return &m_pre_data->owned_milestones; + return &m_non_http_sm_data->owned_milestones; } // ===== Headers ===== @@ -127,8 +127,8 @@ TransactionLogData::get_client_request() const return nullptr; } - if (m_pre_data->owned_client_request.valid()) { - return const_cast(&m_pre_data->owned_client_request); + if (m_non_http_sm_data->owned_client_request.valid()) { + return const_cast(&m_non_http_sm_data->owned_client_request); } return nullptr; } @@ -207,7 +207,7 @@ TransactionLogData::get_client_req_url_str() const cache_url_strings(); return m_client_req_url_str; } - return m_pre_data->owned_url.empty() ? nullptr : m_pre_data->owned_url.c_str(); + return m_non_http_sm_data->owned_url.empty() ? nullptr : m_non_http_sm_data->owned_url.c_str(); } int @@ -217,7 +217,7 @@ TransactionLogData::get_client_req_url_len() const cache_url_strings(); return m_client_req_url_len; } - return static_cast(m_pre_data->owned_url.size()); + return static_cast(m_non_http_sm_data->owned_url.size()); } const char * @@ -227,7 +227,7 @@ TransactionLogData::get_client_req_url_path_str() const cache_url_strings(); return m_client_req_url_path_str; } - return m_pre_data->owned_path.empty() ? nullptr : m_pre_data->owned_path.c_str(); + return m_non_http_sm_data->owned_path.empty() ? nullptr : m_non_http_sm_data->owned_path.c_str(); } int @@ -237,7 +237,7 @@ TransactionLogData::get_client_req_url_path_len() const cache_url_strings(); return m_client_req_url_path_len; } - return static_cast(m_pre_data->owned_path.size()); + return static_cast(m_non_http_sm_data->owned_path.size()); } // ===== Proxy response content-type / reason ===== @@ -366,6 +366,16 @@ TransactionLogData::get_cache_lookup_url_len() const return 0; } +const ts::CryptoHash * +TransactionLogData::get_cache_lookup_hash() const +{ + if (likely(m_http_sm != nullptr)) { + return &(m_http_sm->get_cache_sm().get_cache_key().hash); + } + + return nullptr; +} + // ===== Client addressing ===== sockaddr const * @@ -374,7 +384,7 @@ TransactionLogData::get_client_addr() const if (likely(m_http_sm != nullptr)) { return &m_http_sm->t_state.effective_client_addr.sa; } - return &m_pre_data->owned_client_addr.sa; + return &m_non_http_sm_data->owned_client_addr.sa; } sockaddr const * @@ -383,7 +393,7 @@ TransactionLogData::get_client_src_addr() const if (likely(m_http_sm != nullptr)) { return &m_http_sm->t_state.client_info.src_addr.sa; } - return &m_pre_data->owned_client_src_addr.sa; + return &m_non_http_sm_data->owned_client_src_addr.sa; } sockaddr const * @@ -392,7 +402,7 @@ TransactionLogData::get_client_dst_addr() const if (likely(m_http_sm != nullptr)) { return &m_http_sm->t_state.client_info.dst_addr.sa; } - return &m_pre_data->owned_client_dst_addr.sa; + return &m_non_http_sm_data->owned_client_dst_addr.sa; } sockaddr const * @@ -418,7 +428,7 @@ TransactionLogData::get_client_port() const } return 0; } - return m_pre_data->m_client_port; + return m_non_http_sm_data->m_client_port; } // ===== Server addressing ===== @@ -473,7 +483,7 @@ TransactionLogData::get_log_code() const if (likely(m_http_sm != nullptr)) { return m_http_sm->t_state.squid_codes.log_code; } - return m_pre_data->m_log_code; + return m_non_http_sm_data->m_log_code; } SquidSubcode @@ -491,7 +501,7 @@ TransactionLogData::get_hit_miss_code() const if (likely(m_http_sm != nullptr)) { return m_http_sm->t_state.squid_codes.hit_miss_code; } - return m_pre_data->m_hit_miss_code; + return m_non_http_sm_data->m_hit_miss_code; } SquidHierarchyCode @@ -500,7 +510,7 @@ TransactionLogData::get_hier_code() const if (likely(m_http_sm != nullptr)) { return m_http_sm->t_state.squid_codes.hier_code; } - return m_pre_data->m_hier_code; + return m_non_http_sm_data->m_hier_code; } // ===== Byte counters ===== @@ -585,7 +595,7 @@ TransactionLogData::get_connection_id() const if (likely(m_http_sm != nullptr)) { return m_http_sm->client_connection_id(); } - return m_pre_data->m_connection_id; + return m_non_http_sm_data->m_connection_id; } int @@ -594,7 +604,7 @@ TransactionLogData::get_transaction_id() const if (likely(m_http_sm != nullptr)) { return m_http_sm->client_transaction_id(); } - return m_pre_data->m_transaction_id; + return m_non_http_sm_data->m_transaction_id; } int @@ -643,7 +653,7 @@ TransactionLogData::get_client_protocol() const if (likely(m_http_sm != nullptr)) { return m_http_sm->get_user_agent().get_client_protocol(); } - return m_pre_data->owned_client_protocol_str.empty() ? nullptr : m_pre_data->owned_client_protocol_str.c_str(); + return m_non_http_sm_data->owned_client_protocol_str.empty() ? nullptr : m_non_http_sm_data->owned_client_protocol_str.c_str(); } const char * @@ -734,7 +744,7 @@ TransactionLogData::get_client_connection_is_ssl() const if (likely(m_http_sm != nullptr)) { return m_http_sm->get_user_agent().get_client_connection_is_ssl(); } - return m_pre_data->m_client_connection_is_ssl; + return m_non_http_sm_data->m_client_connection_is_ssl; } bool @@ -814,7 +824,7 @@ TransactionLogData::get_server_transact_count() const if (likely(m_http_sm != nullptr)) { return m_http_sm->server_transact_count; } - return m_pre_data->m_server_transact_count; + return m_non_http_sm_data->m_server_transact_count; } // ===== Finish status ===== @@ -1089,7 +1099,7 @@ TransactionLogData::get_server_response_transfer_encoding() const return {}; } -// ===== Fallback fields for pre-transaction logging ===== +// ===== Fallback fields for non-HttpSM logging ===== std::string_view TransactionLogData::get_method() const @@ -1097,7 +1107,7 @@ TransactionLogData::get_method() const if (likely(m_http_sm != nullptr)) { return {}; } - return m_pre_data->owned_method; + return m_non_http_sm_data->owned_method; } std::string_view @@ -1106,7 +1116,7 @@ TransactionLogData::get_scheme() const if (likely(m_http_sm != nullptr)) { return {}; } - return m_pre_data->owned_scheme; + return m_non_http_sm_data->owned_scheme; } std::string_view @@ -1115,5 +1125,5 @@ TransactionLogData::get_client_protocol_str() const if (likely(m_http_sm != nullptr)) { return {}; } - return m_pre_data->owned_client_protocol_str; + return m_non_http_sm_data->owned_client_protocol_str; } diff --git a/src/proxy/logging/unit-tests/test_LogAccess.cc b/src/proxy/logging/unit-tests/test_LogAccess.cc index aee2bd67287..94f6f8f91d6 100644 --- a/src/proxy/logging/unit-tests/test_LogAccess.cc +++ b/src/proxy/logging/unit-tests/test_LogAccess.cc @@ -23,7 +23,7 @@ #include -#include "proxy/PreTransactionLogData.h" +#include "proxy/NonHttpSmLogData.h" #include "proxy/logging/LogAccess.h" #include "proxy/logging/TransactionLogData.h" #include "tscore/ink_inet.h" @@ -101,8 +101,8 @@ set_socket_address(IpEndpoint &ep, std::string_view text) } void -populate_pre_transaction_data(PreTransactionLogData &data, std::string_view method, std::string_view scheme, - std::string_view authority, std::string_view path) +populate_non_http_sm_data(NonHttpSmLogData &data, std::string_view method, std::string_view scheme, std::string_view authority, + std::string_view path) { initialize_headers_once(); @@ -164,10 +164,10 @@ marshal_int_value(Marshal marshal) } } // namespace -TEST_CASE("LogAccess pre-transaction CONNECT fields", "[LogAccess]") +TEST_CASE("LogAccess non-HttpSM CONNECT fields", "[LogAccess]") { - PreTransactionLogData data; - populate_pre_transaction_data(data, "CONNECT", ""sv, "example.com:443", ""sv); + NonHttpSmLogData data; + populate_non_http_sm_data(data, "CONNECT", ""sv, "example.com:443", ""sv); TransactionLogData log_data(data); LogAccess access(log_data); @@ -187,8 +187,8 @@ TEST_CASE("LogAccess pre-transaction CONNECT fields", "[LogAccess]") TEST_CASE("LogAccess malformed CONNECT without authority falls back to path", "[LogAccess]") { - PreTransactionLogData data; - populate_pre_transaction_data(data, "CONNECT", "https"sv, ""sv, "/"sv); + NonHttpSmLogData data; + populate_non_http_sm_data(data, "CONNECT", "https"sv, ""sv, "/"sv); TransactionLogData log_data(data); LogAccess access(log_data); @@ -201,10 +201,10 @@ TEST_CASE("LogAccess malformed CONNECT without authority falls back to path", "[ CHECK(marshal_int_value([&](char *buf) { return access.marshal_transfer_time_ms(buf); }) == 5); } -TEST_CASE("LogAccess pre-transaction client host port is null-safe", "[LogAccess]") +TEST_CASE("LogAccess non-HttpSM client host port is null-safe", "[LogAccess]") { - PreTransactionLogData data; - populate_pre_transaction_data(data, "GET", "https"sv, "example.com", "/client-port"sv); + NonHttpSmLogData data; + populate_non_http_sm_data(data, "GET", "https"sv, "example.com", "/client-port"sv); TransactionLogData log_data(data); LogAccess access(log_data); diff --git a/src/proxy/shared/DiagsConfig.cc b/src/proxy/shared/DiagsConfig.cc index 9420f2df1f0..9b70c9fab06 100644 --- a/src/proxy/shared/DiagsConfig.cc +++ b/src/proxy/shared/DiagsConfig.cc @@ -41,9 +41,9 @@ void DiagsConfig::reconfigure_diags() { - int i; + int i = 0; DiagsConfigState c{}; - bool found, all_found; + bool found = false, all_found = false; static struct { const char *config_name; diff --git a/src/records/RecConfigParse.cc b/src/records/RecConfigParse.cc index 0819137f7d2..d80a1237af5 100644 --- a/src/records/RecConfigParse.cc +++ b/src/records/RecConfigParse.cc @@ -84,24 +84,20 @@ RecFileImport_Xmalloc(const char *file, char **file_buf, int *file_size) } //------------------------------------------------------------------------- -// RecConfigOverrideFromRunroot +// Records whose paths are managed by runroot. When runroot is active the +// value from records.yaml is replaced with the resolved Layout path. //------------------------------------------------------------------------- -bool -RecConfigOverrideFromRunroot(const char *name) -{ - if (!get_runroot().empty()) { - if (!strcmp(name, "proxy.config.bin_path") || !strcmp(name, "proxy.config.local_state_dir") || - !strcmp(name, "proxy.config.log.logfile_dir") || !strcmp(name, "proxy.config.plugin.plugin_dir")) { - return true; - } - } - return false; -} +static constexpr std::pair runroot_records[] = { + {"proxy.config.bin_path", &Layout::bindir }, + {"proxy.config.local_state_dir", &Layout::runtimedir}, + {"proxy.config.log.logfile_dir", &Layout::logdir }, + {"proxy.config.plugin.plugin_dir", &Layout::libexecdir}, +}; //------------------------------------------------------------------------- // RecConfigOverrideFromEnvironment //------------------------------------------------------------------------- -const char * +std::pair RecConfigOverrideFromEnvironment(const char *name, const char *value) { ats_scoped_str envname(ats_strdup(name)); @@ -121,12 +117,18 @@ RecConfigOverrideFromEnvironment(const char *name, const char *value) envval = getenv(envname.get()); if (envval) { - return envval; - } else if (RecConfigOverrideFromRunroot(name)) { - return nullptr; + return {envval, RecConfigOverrideSource::ENV}; + } + + if (!get_runroot().empty()) { + for (auto const &[rec_name, member] : runroot_records) { + if (rec_name == name) { + return {Layout::get()->*member, RecConfigOverrideSource::RUNROOT}; + } + } } - return value; + return {value ? value : "", RecConfigOverrideSource::NONE}; } //------------------------------------------------------------------------- @@ -141,10 +143,9 @@ RecConfigFileParse(const char *path, RecConfigEntryCallback handler) const char *line; int line_num; - char *rec_type_str, *name_str, *data_type_str, *data_str; - const char *value_str; - RecT rec_type; - RecDataT data_type; + char *rec_type_str, *name_str, *data_type_str, *data_str; + RecT rec_type; + RecDataT data_type; Tokenizer line_tok("\r\n"); tok_iter_state line_tok_state; @@ -245,8 +246,15 @@ RecConfigFileParse(const char *path, RecConfigEntryCallback handler) } // OK, we parsed the record, send it to the handler ... - value_str = RecConfigOverrideFromEnvironment(name_str, data_str); - handler(rec_type, data_type, name_str, value_str, value_str == data_str ? REC_SOURCE_EXPLICIT : REC_SOURCE_ENV); + { + auto [value_str, override_source] = RecConfigOverrideFromEnvironment(name_str, data_str); + if (override_source != RecConfigOverrideSource::NONE) { + RecDebug(DL_Debug, "'%s' overridden with '%s' by %s", name_str, value_str.c_str(), + RecConfigOverrideSourceName(override_source)); + } + handler(rec_type, data_type, name_str, value_str.c_str(), + override_source == RecConfigOverrideSource::NONE ? REC_SOURCE_EXPLICIT : REC_SOURCE_ENV); + } // update our g_rec_config_contents_xxx g_rec_config_contents_ht.emplace(name_str); diff --git a/src/records/RecYAMLDecoder.cc b/src/records/RecYAMLDecoder.cc index 29f2f12fa95..1bb7b9e45e9 100644 --- a/src/records/RecYAMLDecoder.cc +++ b/src/records/RecYAMLDecoder.cc @@ -156,11 +156,12 @@ SetRecordFromYAMLNode(CfgNode const &field, swoc::Errata &errata) std::string field_value = field.value_node.as(); // in case of a string, the library will give us the literal // 'null' which is exactly what we want. - std::string value_str = RecConfigOverrideFromEnvironment(record_name.c_str(), field_value.c_str()); - RecSourceT source = (field_value == value_str ? REC_SOURCE_EXPLICIT : REC_SOURCE_ENV); + auto [value_str, override_source] = RecConfigOverrideFromEnvironment(record_name.c_str(), field_value.c_str()); + RecSourceT source = (override_source == RecConfigOverrideSource::NONE) ? REC_SOURCE_EXPLICIT : REC_SOURCE_ENV; - if (source == REC_SOURCE_ENV) { - errata.note(ERRATA_DEBUG, "'{}' was override with '{}' using an env variable", record_name, value_str); + if (override_source != RecConfigOverrideSource::NONE) { + errata.note(ERRATA_DEBUG, "'{}' overridden with '{}' by {}", record_name, value_str, + RecConfigOverrideSourceName(override_source)); } if (!check_expr.empty() && RecordValidityCheck(value_str.c_str(), check_type, check_expr.c_str()) == false) { diff --git a/src/records/RecordsConfig.cc b/src/records/RecordsConfig.cc index c83d1bce8d0..bf80bb290dc 100644 --- a/src/records/RecordsConfig.cc +++ b/src/records/RecordsConfig.cc @@ -1255,6 +1255,10 @@ static constexpr RecordElement RecordsConfig[] = , {RECT_CONFIG, "proxy.config.ssl.ktls.enabled", RECD_INT, "0", RECU_RESTART_TS, RR_NULL, RECC_INT, "[0-1]", RECA_NULL} , + {RECT_CONFIG, "proxy.config.ssl.server.cert_compression.algorithms", RECD_STRING, nullptr, RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} + , + {RECT_CONFIG, "proxy.config.ssl.client.cert_compression.algorithms", RECD_STRING, nullptr, RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} + , //############################################################################## //# //# OCSP (Online Certificate Status Protocol) Stapling Configuration @@ -1554,7 +1558,14 @@ static constexpr RecordElement RecordsConfig[] = //# Thread watchdog //# //########### - {RECT_CONFIG, "proxy.config.exec_thread.watchdog.timeout_ms", RECD_INT, "0", RECU_RESTART_TS, RR_NULL, RECC_INT, "[0-10000]", RECA_NULL} + {RECT_CONFIG, "proxy.config.exec_thread.watchdog.timeout_ms", RECD_INT, "0", RECU_RESTART_TS, RR_NULL, RECC_INT, "[0-10000]", RECA_NULL}, + + //########### + //# + //# PROXY protocol + //# + //########### + {RECT_CONFIG, "proxy.config.proxy_protocol.max_header_size", RECD_INT, "109", RECU_DYNAMIC, RR_NULL, RECC_INT, "[109-65535]", RECA_NULL}, }; // clang-format on diff --git a/src/records/RecordsConfigUtils.cc b/src/records/RecordsConfigUtils.cc index f572a957495..9d244510616 100644 --- a/src/records/RecordsConfigUtils.cc +++ b/src/records/RecordsConfigUtils.cc @@ -49,9 +49,14 @@ initialize_record(const RecordElement *record, void *) access = record->access; if (REC_TYPE_IS_CONFIG(type)) { - const char *value = RecConfigOverrideFromEnvironment(record->name, record->value); - RecData data = {0}; - RecSourceT source = value == record->value ? REC_SOURCE_DEFAULT : REC_SOURCE_ENV; + auto [value, override_source] = RecConfigOverrideFromEnvironment(record->name, record->value); + RecData data = {0}; + RecSourceT source = (override_source == RecConfigOverrideSource::NONE) ? REC_SOURCE_DEFAULT : REC_SOURCE_ENV; + + if (override_source != RecConfigOverrideSource::NONE) { + RecDebug(DL_Debug, "'%s' overridden with '%s' by %s", record->name, value.c_str(), + RecConfigOverrideSourceName(override_source)); + } // If you specify a consistency check, you have to specify a regex expression. We abort here // so that this breaks QA completely. @@ -59,7 +64,11 @@ initialize_record(const RecordElement *record, void *) ink_fatal("%s has a consistency check but no regular expression", record->name); } - RecDataSetFromString(record->value_type, &data, value); + // When the built-in default is nullptr and no override was applied, preserve + // nullptr so optional records (e.g. keylog_file, groups_list) stay unset. + const char *value_ptr = + (override_source == RecConfigOverrideSource::NONE && record->value == nullptr) ? nullptr : value.c_str(); + RecDataSetFromString(record->value_type, &data, value_ptr); RecErrT reg_status{REC_ERR_FAIL}; switch (record->value_type) { case RECD_INT: diff --git a/src/traffic_layout/info.cc b/src/traffic_layout/info.cc index 0a4f44491c0..be431cb548e 100644 --- a/src/traffic_layout/info.cc +++ b/src/traffic_layout/info.cc @@ -23,6 +23,7 @@ #include #include +#include #include #include #include "tscore/Layout.h" @@ -53,6 +54,39 @@ #include #endif +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG +static constexpr int ts_has_cert_compression_callbacks = 1; +#else +static constexpr int ts_has_cert_compression_callbacks = 0; +#endif + +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG +static constexpr int ts_has_cert_compression_zlib = 1; +#elif HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE && !defined(OPENSSL_NO_ZLIB) +static constexpr int ts_has_cert_compression_zlib = 1; +#else +static constexpr int ts_has_cert_compression_zlib = 0; +#endif + +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG && HAVE_BROTLI_ENCODE_H +static constexpr int ts_has_cert_compression_brotli = 1; +#elif !HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG && HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE && !defined(OPENSSL_NO_BROTLI) +static constexpr int ts_has_cert_compression_brotli = 1; +#else +static constexpr int ts_has_cert_compression_brotli = 0; +#endif + +#if HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG && HAVE_ZSTD_H +static constexpr int ts_has_cert_compression_zstd = 1; +#elif !HAVE_SSL_CTX_ADD_CERT_COMPRESSION_ALG && HAVE_SSL_CTX_SET1_CERT_COMP_PREFERENCE && !defined(OPENSSL_NO_ZSTD) +static constexpr int ts_has_cert_compression_zstd = 1; +#else +static constexpr int ts_has_cert_compression_zstd = 0; +#endif + +static constexpr int ts_has_cert_compression = + ts_has_cert_compression_zlib | ts_has_cert_compression_brotli | ts_has_cert_compression_zstd; + // Produce output about compile time features, useful for checking how things were built static void print_feature(std::string_view name, int value, bool json, bool last = false) @@ -100,6 +134,11 @@ produce_features(bool json) #else print_feature("TS_HAS_ZSTD", 0, json); #endif + print_feature("TS_HAS_CERT_COMPRESSION", ts_has_cert_compression, json); + print_feature("TS_HAS_CERT_COMPRESSION_CALLBACKS", ts_has_cert_compression_callbacks, json); + print_feature("TS_HAS_CERT_COMPRESSION_ZLIB", ts_has_cert_compression_zlib, json); + print_feature("TS_HAS_CERT_COMPRESSION_BROTLI", ts_has_cert_compression_brotli, json); + print_feature("TS_HAS_CERT_COMPRESSION_ZSTD", ts_has_cert_compression_zstd, json); #ifdef F_GETPIPE_SZ print_feature("TS_HAS_PIPE_BUFFER_SIZE_CONFIG", 1, json); #else diff --git a/src/tscore/ArgParser.cc b/src/tscore/ArgParser.cc index 65298317a61..90c3c2f5e43 100644 --- a/src/tscore/ArgParser.cc +++ b/src/tscore/ArgParser.cc @@ -725,7 +725,14 @@ ArgParser::Command::append_option_data(Arguments &ret, AP_StrVec &args, int inde help_message(std::to_string(_option_list.at(it.first).arg_num) + " arguments expected by " + it.first); } } - // put in the default value of options +} + +// Apply default values for options not explicitly set by the user. +// This must be called AFTER validate_dependencies() so that default values +// (e.g. --timeout "0") don't falsely trigger dependency checks. +void +ArgParser::Command::apply_option_defaults(Arguments &ret) const +{ for (const auto &it : _option_list) { if (!it.second.default_value.empty() && ret.get(it.second.key).empty()) { std::istringstream ss(it.second.default_value); @@ -771,6 +778,11 @@ ArgParser::Command::parse(Arguments &ret, AP_StrVec &args) // Validate option dependencies validate_dependencies(ret); + + // Apply default values after validation so that defaults don't + // trigger dependency checks (e.g. --timeout with default "0" + // should not require --monitor when not explicitly used). + apply_option_defaults(ret); } if (command_called) { diff --git a/src/tscore/HashFNV.cc b/src/tscore/HashFNV.cc index b060ab4205a..b84d9a8f78a 100644 --- a/src/tscore/HashFNV.cc +++ b/src/tscore/HashFNV.cc @@ -9,14 +9,8 @@ #include "tscore/HashFNV.h" -static const uint32_t FNV_INIT_32 = 0x811c9dc5u; -static const uint64_t FNV_INIT_64 = 0xcbf29ce484222325ull; - -// FNV-1a 64bit -ATSHash32FNV1a::ATSHash32FNV1a() -{ - this->clear(); -} +// FNV-1a 32bit +ATSHash32FNV1a::ATSHash32FNV1a() = default; void ATSHash32FNV1a::final() @@ -32,14 +26,11 @@ ATSHash32FNV1a::get() const void ATSHash32FNV1a::clear() { - hval = FNV_INIT_32; + hval = fnv_init; } // FNV-1a 64bit -ATSHash64FNV1a::ATSHash64FNV1a() -{ - this->clear(); -} +ATSHash64FNV1a::ATSHash64FNV1a() = default; void ATSHash64FNV1a::final() { @@ -54,5 +45,5 @@ ATSHash64FNV1a::get() const void ATSHash64FNV1a::clear() { - hval = FNV_INIT_64; + hval = fnv_init; } diff --git a/src/tscore/ink_cap.cc b/src/tscore/ink_cap.cc index f464daad3b1..1e208f34cfb 100644 --- a/src/tscore/ink_cap.cc +++ b/src/tscore/ink_cap.cc @@ -273,7 +273,7 @@ RestrictCapabilities() cap_t caps_orig = cap_get_proc(); // Capabilities we need. - cap_value_t perm_list[] = {CAP_NET_ADMIN, CAP_NET_BIND_SERVICE, CAP_IPC_LOCK, CAP_DAC_OVERRIDE, CAP_FOWNER}; + cap_value_t perm_list[] = {CAP_NET_ADMIN, CAP_NET_BIND_SERVICE, CAP_IPC_LOCK, CAP_DAC_OVERRIDE, CAP_FOWNER, CAP_CHOWN}; static int const PERM_CAP_COUNT = sizeof(perm_list) / sizeof(*perm_list); cap_value_t eff_list[] = {CAP_NET_ADMIN, CAP_NET_BIND_SERVICE, CAP_IPC_LOCK}; static int const EFF_CAP_COUNT = sizeof(eff_list) / sizeof(*eff_list); @@ -436,7 +436,7 @@ void ElevateAccess::acquirePrivilege(unsigned priv_mask) { unsigned cap_count = 0; - cap_value_t cap_list[3]; + cap_value_t cap_list[4]; cap_t new_cap_state; Dbg(dbg_ctl_privileges, "[acquirePrivilege] level= %x", level); @@ -463,7 +463,12 @@ ElevateAccess::acquirePrivilege(unsigned priv_mask) ++cap_count; } - ink_release_assert(cap_count <= sizeof(cap_list)); + if (priv_mask & ElevateAccess::CHOWN_PRIVILEGE) { + cap_list[cap_count] = CAP_CHOWN; + ++cap_count; + } + + ink_release_assert(cap_count <= sizeof(cap_list) / sizeof(cap_list[0])); if (cap_count > 0) { this->cap_state = cap_get_proc(); // save current capabilities diff --git a/src/tscore/unit_tests/test_ArgParser.cc b/src/tscore/unit_tests/test_ArgParser.cc index 672a702a288..0a32502e964 100644 --- a/src/tscore/unit_tests/test_ArgParser.cc +++ b/src/tscore/unit_tests/test_ArgParser.cc @@ -148,3 +148,72 @@ TEST_CASE("Invoke test", "[invoke]") parsed_data.invoke(); REQUIRE(global == 2); } + +TEST_CASE("Case sensitive short options", "[parse]") +{ + ts::ArgParser cs_parser; + cs_parser.add_global_usage("test_prog [--SWITCH]"); + + // Add a command with two options that differ only in case: -t and -T + ts::ArgParser::Command &cmd = cs_parser.add_command("process", "process data"); + cmd.add_option("--tag", "-t", "a label", "", 1, ""); + cmd.add_option("--threshold", "-T", "a numeric value", "", 1, "100"); + + ts::Arguments parsed; + + // Use lowercase -t: should set "tag" only + const char *argv1[] = {"test_prog", "process", "-t", "my_tag", nullptr}; + parsed = cs_parser.parse(argv1); + REQUIRE(parsed.get("tag") == true); + REQUIRE(parsed.get("tag").value() == "my_tag"); + // threshold should still have its default + REQUIRE(parsed.get("threshold").value() == "100"); + + // Use uppercase -T: should set "threshold" only + const char *argv2[] = {"test_prog", "process", "-T", "200", nullptr}; + parsed = cs_parser.parse(argv2); + REQUIRE(parsed.get("threshold") == true); + REQUIRE(parsed.get("threshold").value() == "200"); + // tag should be empty (no default) + REQUIRE(parsed.get("tag").value() == ""); + + // Use both -t and -T together + const char *argv3[] = {"test_prog", "process", "-t", "foo", "-T", "500", nullptr}; + parsed = cs_parser.parse(argv3); + REQUIRE(parsed.get("tag") == true); + REQUIRE(parsed.get("tag").value() == "foo"); + REQUIRE(parsed.get("threshold") == true); + REQUIRE(parsed.get("threshold").value() == "500"); +} + +TEST_CASE("with_required does not trigger on default values", "[parse]") +{ + ts::ArgParser parser; + parser.add_global_usage("test_prog [OPTIONS]"); + + ts::ArgParser::Command &cmd = parser.add_command("scan", "scan targets"); + cmd.add_option("--tag", "-t", "a label", "", 1, ""); + cmd.add_option("--verbose", "-v", "enable verbose output"); + cmd.add_option("--threshold", "-T", "a numeric value", "", 1, "100").with_required("--verbose"); + + // -t alone should NOT trigger --threshold's dependency on --verbose. + // The default value "100" for --threshold must not count as "explicitly used". + const char *argv1[] = {"test_prog", "scan", "-t", "my_tag", nullptr}; + ts::Arguments parsed = parser.parse(argv1); + REQUIRE(parsed.get("tag").value() == "my_tag"); + // threshold default should still be applied after validation + REQUIRE(parsed.get("threshold").value() == "100"); + + // -T with -v should work fine + const char *argv2[] = {"test_prog", "scan", "-T", "200", "-v", nullptr}; + parsed = parser.parse(argv2); + REQUIRE(parsed.get("threshold").value() == "200"); + REQUIRE(parsed.get("verbose") == true); + + // -t and -T together with -v should work + const char *argv3[] = {"test_prog", "scan", "-t", "foo", "-T", "300", "-v", nullptr}; + parsed = parser.parse(argv3); + REQUIRE(parsed.get("tag").value() == "foo"); + REQUIRE(parsed.get("threshold").value() == "300"); + REQUIRE(parsed.get("verbose") == true); +} diff --git a/tests/autest-parallel.py.in b/tests/autest-parallel.py.in index be5f5d9129e..1f7f1eedea4 100755 --- a/tests/autest-parallel.py.in +++ b/tests/autest-parallel.py.in @@ -295,17 +295,17 @@ def parse_autest_output(output: str) -> dict: result['skipped'] = int(line.split(':')[-1].strip()) except ValueError: pass - elif 'Warning:' in line: + elif re.match(r'warnings?:', line, re.IGNORECASE): try: result['warnings'] = int(line.split(':')[-1].strip()) except ValueError: pass - elif 'Exception:' in line: + elif re.match(r'exceptions?:', line, re.IGNORECASE): try: result['exceptions'] = int(line.split(':')[-1].strip()) except ValueError: pass - elif 'Unknown:' in line: + elif re.match(r'unknowns?:', line, re.IGNORECASE): try: result['unknown'] = int(line.split(':')[-1].strip()) except ValueError: @@ -367,6 +367,60 @@ def extract_failure_output(output: str, failed_tests: List[str]) -> Dict[str, st return details +def extract_worker_diagnostics(output: str) -> str: + """ + Extract the most relevant worker diagnostics for exception/setup failures. + + When autest reports an exception without attributing it to a specific + failed test, the best fallback is the worker's captured output. This + helper prefers traceback/error sections but falls back to the full worker + output if no obvious diagnostic block is found. + """ + clean = strip_ansi(output).strip() + if not clean: + return "" + + lines = clean.split('\n') + ranges: List[Tuple[int, int]] = [] + + def is_marker(line: str) -> bool: + lower = line.lower() + return ( + 'traceback (most recent call last):' in lower or 'error:' in lower or 'timeout' in lower or + 'interrupted' in lower + ) + + for i, line in enumerate(lines): + if not is_marker(line): + continue + start = max(0, i - 5) + end = i + 1 + while end < len(lines): + candidate = lines[end].strip() + if not candidate and end > i + 1: + break + end += 1 + ranges.append((start, min(end, len(lines)))) + + if not ranges: + return clean + + merged_ranges: List[Tuple[int, int]] = [] + for start, end in ranges: + if merged_ranges and start <= merged_ranges[-1][1]: + merged_ranges[-1] = (merged_ranges[-1][0], max(merged_ranges[-1][1], end)) + else: + merged_ranges.append((start, end)) + + blocks = [] + for start, end in merged_ranges: + block = '\n'.join(lines[start:end]).rstrip() + if block: + blocks.append(block) + + return '\n\n...\n\n'.join(blocks) if blocks else clean + + def run_single_test(test: str, script_dir: Path, sandbox: Path, ats_bin: str, build_root: str, extra_args: List[str], env: dict) -> Tuple[str, float, str, str]: """ @@ -672,6 +726,20 @@ def print_summary(results: List[TestResult], total_duration: float, expected_tim print(details[test_name]) print("-" * 70) + worker_diagnostics = [ + r for r in results + if r.output and (r.exceptions > 0 or (r.return_code != 0 and not r.failed_tests)) + ] + if worker_diagnostics: + print("-" * 70) + print("WORKER DIAGNOSTICS:") + print("-" * 70) + for r in worker_diagnostics: + label = "Serial" if r.is_serial else f"Worker {r.worker_id}" + print(f"\n--- {label} (return code {r.return_code}) ---") + print(extract_worker_diagnostics(r.output)) + print("-" * 70) + # Check for timing discrepancies if expected_timings and actual_timings: timing_warnings = [] @@ -925,6 +993,7 @@ Examples: try: if partitions: print_progress() + print() # Newline so worker output starts on a fresh line with ProcessPoolExecutor(max_workers=len(partitions)) as executor: for worker_id, worker_tests in enumerate(partitions): future = executor.submit( diff --git a/tests/gold_tests/autest-site/ats_replay.test.ext b/tests/gold_tests/autest-site/ats_replay.test.ext index 976dc73b3d0..c19ff589674 100644 --- a/tests/gold_tests/autest-site/ats_replay.test.ext +++ b/tests/gold_tests/autest-site/ats_replay.test.ext @@ -35,9 +35,37 @@ def configure_ats(obj: 'TestRun', server: 'Process', ats_config: dict, dns: Opti name = ats_config.get('name', 'ts') process_config = ats_config.get('process_config', {}) ts = obj.MakeATSProcess(name, **process_config) + + # Configure records_config if specified. records_config = ats_config.get('records_config', {}) ts.Disk.records_config.update(records_config) + # TLS configs + enable_tls = process_config.get('enable_tls', False) + if enable_tls: + # Configure ssl_multicert.config if specified. + ssl_multicert_config = ats_config.get('ssl_multicert_config', []) + + # setup default cert and key ssl_multicert_config is empty + if ssl_multicert_config == []: + ts.addDefaultSSLFiles() + + ts.Disk.records_config.update( + { + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, + }) + + ssl_multicert_config = ["dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key"] + + for line in ssl_multicert_config: + ts.Disk.ssl_multicert_config.AddLine(line) + + # Configure sni.yaml if specified. + sni_yaml = ats_config.get('sni_yaml') + if sni_yaml != None: + ts.Disk.sni_yaml.AddLines(yaml.dump(sni_yaml).split('\n')) + # Configure plugin_config if specified. plugin_config = ats_config.get('plugin_config', []) for plugin_line in plugin_config: @@ -242,7 +270,8 @@ def ATSReplayTest(obj, replay_file: str): if not 'ats' in autest_config: raise ValueError(f"Replay file {replay_file} does not contain 'autest.ats' section") ats_config = autest_config['ats'] - enable_tls = ats_config.get('enable_tls', False) + process_config = ats_config.get('process_config', {}) + enable_tls = process_config.get('enable_tls', False) metric_checks = ats_config.get('metric_checks', []) log_validation = ats_config.get('log_validation', None) diff --git a/tests/gold_tests/autest-site/conditions.test.ext b/tests/gold_tests/autest-site/conditions.test.ext index e01fecdb85e..e228b19fae1 100644 --- a/tests/gold_tests/autest-site/conditions.test.ext +++ b/tests/gold_tests/autest-site/conditions.test.ext @@ -146,6 +146,67 @@ def HasCurlOption(self, option): return self.CheckOutput(['curl', '--help', 'all'], default, "Curl needs to support option: {option}".format(option=option)) +def HasCurlTLSVersionSupport(self, tls_version): + """Check whether curl can attempt a given TLS version. + + This probes curl directly because OpenSSL capability checks do not always + reflect curl runtime policy behavior on hardened systems. + """ + + def check_curl_tls_support(): + # Map semantic versions used by tests to curl flags. + version_map = { + "1.0": ("--tlsv1", "1.0"), + "1.1": ("--tlsv1.1", "1.1"), + "1.2": ("--tlsv1.2", "1.2"), + "1.3": ("--tlsv1.3", "1.3"), + } + if tls_version not in version_map: + return False + + tls_flag, tls_max = version_map[tls_version] + try: + # Connect to localhost closed port to avoid network dependencies. + # "connection refused" means curl accepted the TLS flags and tried. + result = subprocess.run( + [ + "curl", + "-svk", + "--connect-timeout", + "2", + "--max-time", + "3", + tls_flag, + "--tls-max", + tls_max, + "https://127.0.0.1:1", + ], + capture_output=True, + text=True, + timeout=5, + ) + output = (result.stdout + result.stderr).lower() + unsupported_markers = [ + "unsupported protocol", + "no protocols available", + "option --tlsv", + "unknown option", + "is unknown", + ] + if any(marker in output for marker in unsupported_markers): + return False + + # Any attempt to connect implies curl accepted the TLS setting. + return True + except subprocess.TimeoutExpired: + return False + except Exception: + return False + + return self.Condition( + check_curl_tls_support, "Curl does not support TLSv{version} in this environment".format(version=tls_version)) + + def HasATSFeature(self, feature): val = self.Variables.get(feature, None) @@ -175,5 +236,6 @@ ExtendCondition(HasATSFeature) ExtendCondition(HasCurlVersion) ExtendCondition(HasCurlFeature) ExtendCondition(HasCurlOption) +ExtendCondition(HasCurlTLSVersionSupport) ExtendCondition(PluginExists) ExtendCondition(CurlUsingUnixDomainSocket) diff --git a/tests/gold_tests/autest-site/when.test.ext b/tests/gold_tests/autest-site/when.test.ext index ad147e9b756..8975d27cb18 100644 --- a/tests/gold_tests/autest-site/when.test.ext +++ b/tests/gold_tests/autest-site/when.test.ext @@ -89,7 +89,7 @@ def AddAwaitFileContainsTestRun(test, name, file_path, needle, desired_count=1) ''' tr = test.AddTestRun(name) p = tr.Processes.Default - p.Command = f'echo waiting for {needle} in {file_path}' + p.Command = f'echo waiting for file content in {file_path}' await_process = tr.Processes.Process('await', 'sleep 60') await_process.Ready = When.FileContains(file_path, needle, desired_count) await_process.StartupTimeout = 30 diff --git a/tests/gold_tests/cache/cache-heuristic-status.test.py b/tests/gold_tests/cache/cache-heuristic-status.test.py new file mode 100644 index 00000000000..e191be26345 --- /dev/null +++ b/tests/gold_tests/cache/cache-heuristic-status.test.py @@ -0,0 +1,28 @@ +''' +Test heuristic caching of status codes per RFC 9110 Section 15.1. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Test heuristic caching of status codes per RFC 9110 Section 15.1. + +Verifies that responses with heuristically cacheable status codes (200, 203, +204, 300, 301, 308, 410) are cached when only Last-Modified is present, and +that non-cacheable codes (302, 307, 400, 403) are not. +''' + +Test.ATSReplayTest(replay_file="replay/cache-heuristic-status.replay.yaml") diff --git a/tests/gold_tests/cache/cache-request-method.test.py b/tests/gold_tests/cache/cache-request-method.test.py index 6e702a222e7..e08c0f471fd 100644 --- a/tests/gold_tests/cache/cache-request-method.test.py +++ b/tests/gold_tests/cache/cache-request-method.test.py @@ -32,3 +32,6 @@ # Verify correct HEAD response handling with cached GET response Test.ATSReplayTest(replay_file="replay/head_with_get_cached.replay.yaml") + +# Verify DELETE request handling - RFC 9111 4.4. Invalidating Stored Responses +Test.ATSReplayTest(replay_file="replay/delete_cached.replay.yaml") diff --git a/tests/gold_tests/cache/replay/cache-heuristic-status.replay.yaml b/tests/gold_tests/cache/replay/cache-heuristic-status.replay.yaml new file mode 100644 index 00000000000..fd6e9b204b9 --- /dev/null +++ b/tests/gold_tests/cache/replay/cache-heuristic-status.replay.yaml @@ -0,0 +1,507 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Verify heuristic caching behavior for various HTTP status codes per +# RFC 9110 Section 15.1. Responses contain only a Last-Modified header +# (no Cache-Control or Expires) so cacheability depends entirely on the +# status code allowlist in HttpTransact::is_response_cacheable(). +# +# Negative caching is disabled to isolate the heuristic path. +# + +meta: + version: "1.0" + +autest: + description: 'Verify heuristic caching by status code per RFC 9110 Section 15.1' + + dns: + name: 'dns-heuristic-status' + + server: + name: 'server-heuristic-status' + + client: + name: 'client-heuristic-status' + + ats: + name: 'ts-heuristic-status' + process_config: + enable_cache: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http' + proxy.config.http.insert_age_in_response: 0 + proxy.config.http.negative_caching_enabled: 0 + proxy.config.http.cache.required_headers: 0 + + remap_config: + - from: "http://example.com/" + to: "http://backend.example.com:{SERVER_HTTP_PORT}/" + + blocks: + - canary_response: &canary_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + +sessions: +- transactions: + + # ===================================================================== + # Heuristically cacheable status codes. + # + # Each test sends a request, gets a response with only Last-Modified + # (no CC/Expires), then repeats the request. The second request's + # server-response is a canary 200; if we get the original status back, + # the response was served from cache. + # ===================================================================== + + # --- 200 OK --- + - all: { headers: { fields: [[ uuid, 1 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/200 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + + proxy-response: + status: 200 + + - all: { headers: { fields: [[ uuid, 2 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/200 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 200 + + # --- 203 Non-Authoritative Information --- + - all: { headers: { fields: [[ uuid, 3 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/203 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 203 + reason: "Non-Authoritative Information" + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + + proxy-response: + status: 203 + + - all: { headers: { fields: [[ uuid, 4 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/203 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 203 + + # --- 204 No Content --- + - all: { headers: { fields: [[ uuid, 5 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/204 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 204 + reason: "No Content" + headers: + fields: + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + + proxy-response: + status: 204 + + - all: { headers: { fields: [[ uuid, 6 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/204 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 204 + + # --- 300 Multiple Choices --- + - all: { headers: { fields: [[ uuid, 7 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/300 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 300 + reason: "Multiple Choices" + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + - [ Location, "http://example.com/choice1" ] + + proxy-response: + status: 300 + + - all: { headers: { fields: [[ uuid, 8 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/300 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 300 + + # --- 301 Moved Permanently --- + - all: { headers: { fields: [[ uuid, 9 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/301 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 301 + reason: "Moved Permanently" + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + - [ Location, "http://example.com/new-location" ] + + proxy-response: + status: 301 + + - all: { headers: { fields: [[ uuid, 10 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/301 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 301 + + # --- 308 Permanent Redirect --- + - all: { headers: { fields: [[ uuid, 11 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/308 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 308 + reason: "Permanent Redirect" + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + - [ Location, "http://example.com/permanent" ] + + proxy-response: + status: 308 + + - all: { headers: { fields: [[ uuid, 12 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/308 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 308 + + # --- 410 Gone --- + - all: { headers: { fields: [[ uuid, 13 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/410 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 410 + reason: "Gone" + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + + proxy-response: + status: 410 + + - all: { headers: { fields: [[ uuid, 14 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /heuristic/410 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 410 + + # ===================================================================== + # Non-cacheable status codes without explicit cache directives. + # + # These are NOT in the heuristic allowlist and negative caching is + # disabled, so they should not be cached. The second request should + # go to the origin and return the canary 200. + # ===================================================================== + + # --- 302 Found (explicitly rejected) --- + - all: { headers: { fields: [[ uuid, 15 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /not-cacheable/302 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 302 + reason: "Found" + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + - [ Location, "http://example.com/temporary" ] + + proxy-response: + status: 302 + + - all: { headers: { fields: [[ uuid, 16 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /not-cacheable/302 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 200 + + # --- 307 Temporary Redirect (explicitly rejected) --- + - all: { headers: { fields: [[ uuid, 17 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /not-cacheable/307 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 307 + reason: "Temporary Redirect" + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + - [ Location, "http://example.com/temporary" ] + + proxy-response: + status: 307 + + - all: { headers: { fields: [[ uuid, 18 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /not-cacheable/307 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 200 + + # --- 400 Bad Request --- + - all: { headers: { fields: [[ uuid, 19 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /not-cacheable/400 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 400 + reason: "Bad Request" + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + + proxy-response: + status: 400 + + - all: { headers: { fields: [[ uuid, 20 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /not-cacheable/400 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 200 + + # --- 403 Forbidden --- + - all: { headers: { fields: [[ uuid, 21 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /not-cacheable/403 + headers: + fields: + - [ Host, example.com ] + + server-response: + status: 403 + reason: "Forbidden" + headers: + fields: + - [ Content-Length, 16 ] + - [ Last-Modified, "Mon, 16 Mar 2026 00:00:00 GMT" ] + + proxy-response: + status: 403 + + - all: { headers: { fields: [[ uuid, 22 ]]}} + client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /not-cacheable/403 + headers: + fields: + - [ Host, example.com ] + delay: 100ms + + <<: *canary_response + + proxy-response: + status: 200 diff --git a/tests/gold_tests/cache/replay/delete_cached.replay.yaml b/tests/gold_tests/cache/replay/delete_cached.replay.yaml new file mode 100644 index 00000000000..186be1b8f19 --- /dev/null +++ b/tests/gold_tests/cache/replay/delete_cached.replay.yaml @@ -0,0 +1,187 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +# Configuration section for autest integration +autest: + description: 'Verify DELETE method handling' + + dns: + name: 'dns-delete' + + server: + name: 'server-delete' + + client: + name: 'client-delete' + + ats: + name: 'ts-cache-delete' + process_config: + enable_cache: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http.*|cache.*' + proxy.config.http.insert_age_in_response: 0 + + remap_config: + - from: "http://example.com/" + to: "http://backend.example.com:{SERVER_HTTP_PORT}/" + +sessions: +- transactions: + + # Populate the cache with a response to a GET request. + - client-request: + method: "GET" + version: "1.1" + url: /some/path + headers: + fields: + - [ Host, example.com ] + - [ uuid, 1 ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ X-Response, first_get_response ] + + proxy-response: + status: 200 + + # Verify that we reply to the request out of the cache. + - client-request: + method: "GET" + version: "1.1" + url: /some/path + headers: + fields: + - [ Host, example.com ] + - [ uuid, 2 ] + + # Add a delay so ATS has time to finish any caching IO for the previous transaction. + delay: 100ms + + # The request should be served out of cache, so this 403 should not be + # received. + server-response: + status: 403 + reason: Forbidden + + # ATS should serve the cached 200 response. + proxy-response: + status: 200 + headers: + fields: + - [ X-Response, { value: first_get_response, as: equal} ] + + # Verify that we do NOT purge cache if origin returns error response for DELETE request + - client-request: + method: "DELETE" + version: "1.1" + url: /some/path + headers: + fields: + - [ Host, example.com ] + - [ uuid, 3 ] + + # DELETE request is not allowed + server-response: + status: 405 + reason: Method Not Allowed + + # ATS should forward the response from origin server + proxy-response: + status: 405 + reason: Method Not Allowed + + - client-request: + method: "GET" + version: "1.1" + url: /some/path + headers: + fields: + - [ Host, example.com ] + - [ uuid, 4 ] + + # The request should be served out of cache, so this 403 should not be + # received. + server-response: + status: 403 + reason: Forbidden + + # ATS should serve the cached 200 response. + proxy-response: + status: 200 + headers: + fields: + - [ X-Response, { value: first_get_response, as: equal} ] + + # Verify that we "purge" cache if origin returns non-error response for DELETE request + - client-request: + method: "DELETE" + version: "1.1" + url: /some/path + headers: + fields: + - [ Host, example.com ] + - [ uuid, 5 ] + + # DELETE request is accepted + server-response: + status: 204 + reason: No Content + + # ATS should forward the response from origin server + proxy-response: + status: 204 + reason: No Content + + # Verify that we cache is purged by DELETE request and returns cache miss + - client-request: + method: "GET" + version: "1.1" + url: /some/path + headers: + fields: + - [ Host, example.com ] + - [ uuid, 6 ] + + # Add a delay so ATS has time to finish any caching IO for the previous transaction. + delay: 100ms + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 32 ] + - [ Cache-Control, max-age=300 ] + - [ X-Response, second_get_response ] + + # ATS should serve the 200 response. + proxy-response: + status: 200 + headers: + fields: + - [ X-Response, { value: second_get_response, as: equal} ] diff --git a/tests/gold_tests/chunked_encoding/chunked_encoding.test.py b/tests/gold_tests/chunked_encoding/chunked_encoding.test.py index ab47b5e12e5..f4b0b129fc4 100644 --- a/tests/gold_tests/chunked_encoding/chunked_encoding.test.py +++ b/tests/gold_tests/chunked_encoding/chunked_encoding.test.py @@ -255,3 +255,6 @@ def _configure_client(self, tr: 'TestRun') -> 'Process': TestChunkedTrailers(configure_drop_trailers=True) TestChunkedTrailers(configure_drop_trailers=False) + +# Large chunked response from origin server +Test.ATSReplayTest(replay_file="replays/large_chunked.replay.yaml") diff --git a/tests/gold_tests/chunked_encoding/replays/large_chunked.replay.yaml b/tests/gold_tests/chunked_encoding/replays/large_chunked.replay.yaml new file mode 100644 index 00000000000..39c7fb59ce2 --- /dev/null +++ b/tests/gold_tests/chunked_encoding/replays/large_chunked.replay.yaml @@ -0,0 +1,170 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +# Configuration section for autest integration +autest: + description: 'Large chunked response from origin server' + + dns: + name: 'dns-chunked-origin' + + server: + name: 'server-chunked-origin' + + client: + name: 'client-chunked-origin' + + ats: + name: 'ts-chunked-large' + process_config: + enable_tls: true + enable_cache: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http' + proxy.config.http.response_via_str: 2 + + remap_config: + - from: "http://www.example.com/" + to: "http://backend.example.com:{SERVER_HTTP_PORT}/" + - from: "https://www.example.com/" + to: "http://backend.example.com:{SERVER_HTTP_PORT}/" + +sessions: + +# HTTP/1.0 - dechunk + +- transactions: + - client-request: + method: GET + url: /stream/1.0/ + version: '1.0' + headers: + fields: + - [ Host, www.example.com ] + - [ uuid, http-1.0-chunk ] + + server-response: + status: 200 + version: '1.1' + headers: + fields: + - [ Content-Type, text/plain ] + - [ Transfer-Encoding, chunked ] + - [ Cache-Control, "public, max-age=3600" ] + content: + size: 131072 + + proxy-response: + status: 200 + headers: + fields: + - [ Transfer-Encoding, { as: absent } ] + content: + size: 131072 + +# HTTP/1.1 - pass-through for user agent & dechunk for cache write + +- transactions: + - client-request: + method: GET + url: /stream/1.1/ + version: '1.1' + headers: + fields: + - [ Host, www.example.com ] + - [ uuid, http-1.1-chunk-0 ] + + server-response: + status: 200 + version: '1.1' + headers: + fields: + - [ Content-Type, text/plain ] + - [ Transfer-Encoding, chunked ] + - [ Cache-Control, "public, max-age=3600" ] + content: + size: 131072 + + proxy-response: + status: 200 + headers: + fields: + - [ Transfer-Encoding, { value: chunked, as: equal } ] + content: + size: {value: 131072, as: equal} + + # Expect: hit dechunked contents in the cache + - client-request: + method: GET + url: /stream/1.1/ + version: '1.1' + headers: + fields: + - [ Host, www.example.com ] + - [ uuid, http-1.1-chunk-1 ] + + server-response: + status: 503 + version: '1.1' + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: 131072, as: equal} ] + - [ Transfer-Encoding, { as: absent } ] + content: + size: {value: 131072, as: equal} + +# HTTP/2 - dechunk + +- protocol: + - name: http + version: 2 + transactions: + - client-request: + headers: + fields: + - [ ":method", "GET" ] + - [ ":scheme", https ] + - [ ":authority", www.example.com ] + - [ ":path", /stream/2/ ] + - [ uuid, http-2-chunk ] + + server-response: + status: 200 + version: 1.1 + headers: + fields: + - [ Content-Type, text/plain ] + - [ Transfer-Encoding, chunked ] + - [ Cache-Control, "public, max-age=3600" ] + content: + size: 131072 + + proxy-response: + headers: + fields: + - [ ":status", {value: '200', as: equal } ] + - [ transfer-encoding, { as: absent } ] + - [ content-length, { as: absent } ] + content: + size: {value: 131072, as: equal} diff --git a/tests/gold_tests/connect/connect.test.py b/tests/gold_tests/connect/connect.test.py index c1ff3a187df..a1674e4a34c 100644 --- a/tests/gold_tests/connect/connect.test.py +++ b/tests/gold_tests/connect/connect.test.py @@ -94,14 +94,14 @@ def __testCase0(self): self.__checkProcessAfter(tr) def __testAccessLog(self): - """Wait for log file to appear, then wait one extra second to make sure TS is done writing it.""" + """Wait for the access log entry to be written.""" Test.Disk.File(os.path.join(self.ts.Variables.LOGDIR, 'access.log'), exists=True, content='gold/connect_access.gold') - tr = Test.AddTestRun() - tr.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + - os.path.join(self.ts.Variables.LOGDIR, 'access.log')) - tr.Processes.Default.ReturnCode = 0 + Test.AddAwaitFileContainsTestRun( + 'Await CONNECT access log entry.', + os.path.join(self.ts.Variables.LOGDIR, 'access.log'), + 'CONNECT', + ) def run(self): self.__testCase0() diff --git a/tests/gold_tests/connect_down_policy/connect_down_policy.test.py b/tests/gold_tests/connect_down_policy/connect_down_policy.test.py index 4e935486822..0407dd25f9a 100644 --- a/tests/gold_tests/connect_down_policy/connect_down_policy.test.py +++ b/tests/gold_tests/connect_down_policy/connect_down_policy.test.py @@ -85,11 +85,11 @@ def _test_inactive_timeout(self): def _test_mark_down(self): if self._expect_mark_down: - # Wait for error.log to appear then verify it contains the mark-down entry. - tr = Test.AddTestRun(f"policy={self._policy}: check error.log for mark-down") - tr.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + - os.path.join(self._ts.Variables.LOGDIR, 'error.log')) + tr = Test.AddAwaitFileContainsTestRun( + f"policy={self._policy}: check error.log for mark-down", + os.path.join(self._ts.Variables.LOGDIR, 'error.log'), + "marking down", + ) self._ts.Disk.error_log.Content = Testers.ContainsExpression( "marking down", f"policy={self._policy}: origin should be marked down after inactive timeout") else: diff --git a/tests/gold_tests/dns/dns_host_down.test.py b/tests/gold_tests/dns/dns_host_down.test.py index 3d05805769a..80b59e0fb83 100644 --- a/tests/gold_tests/dns/dns_host_down.test.py +++ b/tests/gold_tests/dns/dns_host_down.test.py @@ -65,11 +65,11 @@ def _test_host_mark_down(self): # Verify error log marking host down exists def _test_error_log(self): - tr = Test.AddTestRun() - tr.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + - os.path.join(self._ts.Variables.LOGDIR, 'error.log')) - + tr = Test.AddAwaitFileContainsTestRun( + 'Await error.log mark-down entry.', + os.path.join(self._ts.Variables.LOGDIR, 'error.log'), + "/dns/mark/down' fail_count='1' marking down", + ) self._ts.Disk.error_log.Content = Testers.ContainsExpression( "/dns/mark/down' fail_count='1' marking down", "host should be marked down") diff --git a/tests/gold_tests/h2/httpbin.test.py b/tests/gold_tests/h2/httpbin.test.py index 342f175e435..14b5812aa65 100644 --- a/tests/gold_tests/h2/httpbin.test.py +++ b/tests/gold_tests/h2/httpbin.test.py @@ -122,8 +122,9 @@ test_run.Processes.Default.Streams.stderr = Testers.GoldFile("gold/httpbin_3_stderr.gold", case_insensitive=True) test_run.StillRunningAfter = httpbin -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. -test_run = Test.AddTestRun() -test_run.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + os.path.join(ts.Variables.LOGDIR, 'access.log')) -test_run.Processes.Default.ReturnCode = 0 +# Wait for the POST transaction to be logged. +Test.AddAwaitFileContainsTestRun( + 'Await POST access log entry.', + os.path.join(ts.Variables.LOGDIR, 'access.log'), + r'POST .*?/post', +) diff --git a/tests/gold_tests/headers/accept_webp.test.py b/tests/gold_tests/headers/accept_webp.test.py index 249534a1d12..1d0a4232c86 100644 --- a/tests/gold_tests/headers/accept_webp.test.py +++ b/tests/gold_tests/headers/accept_webp.test.py @@ -1,5 +1,5 @@ ''' -Test how we handle image/webp +Test how ATS handles image/webp alternates. ''' # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -18,71 +18,9 @@ # limitations under the License. Test.Summary = ''' -Checking that we don't serve image/webp to clients that do not support it +Check that ATS does not reuse a cached image/webp alternate for clients that do not advertise image/webp ''' -Test.SkipIf(Condition.CurlUsingUnixDomainSocket()) Test.ContinueOnFail = True -# Define default ATS -ts = Test.MakeATSProcess("ts") -server = Test.MakeOriginServer("server") - -testName = "accept_webp" -request_header = { - "headers": - "GET / HTTP/1.1\r\nHost: www.example.com\r\nAccept: image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5\r\n\r\n", - "timestamp": "1469733493.993", - "body": "" -} -response_header = { - "headers": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Type: image/webp\r\nCache-Control: max-age=300\r\n", - "timestamp": "1469733493.993", - "body": "xxx" -} -server.addResponse("sessionlog.json", request_header, response_header) - -# ATS Configuration -ts.Disk.records_config.update( - { - 'proxy.config.diags.debug.enabled': 1, - 'proxy.config.diags.debug.tags': 'http_match', - 'proxy.config.http.cache.ignore_accept_mismatch': 0, - 'proxy.config.http.insert_response_via_str': 3, - 'proxy.config.http.cache.http': 1, - 'proxy.config.http.wait_for_cache': 1, - }) - -ts.Disk.remap_config.AddLine('map http://www.example.com http://127.0.0.1:{0}'.format(server.Variables.Port)) - -# Test 1 - Request with image/webp support from the origin -tr = Test.AddTestRun() -tr.Processes.Default.StartBefore(server, ready=When.PortOpen(server.Variables.Port)) -tr.Processes.Default.StartBefore(Test.Processes.ts) -tr.MakeCurlCommand( - '-s -D - -v --ipv4 --http1.1 -H "Accept: image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5" -H "Host: www.example.com" http://localhost:{0}/' - .format(ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stderr = "gold/accept_webp.gold" -tr.StillRunningAfter = ts - -# Test 2 - Request with image/webp support from cache -tr = Test.AddTestRun() -tr.MakeCurlCommand( - '-s -D - -v --ipv4 --http1.1 -H "Accept: image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5" -H "Host: www.example.com" http://localhost:{0}/' - .format(ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stderr = "gold/accept_webp_cache.gold" -tr.StillRunningAfter = ts - -# Test 3 - Request without image/webp support going to the origin - NOTE: the origin can't change the content-type :( -tr = Test.AddTestRun() -tr.MakeCurlCommand( - '-s -D - -v --ipv4 --http1.1 -H "Accept: image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5" -H "Host: www.example.com" http://localhost:{0}/' - .format(ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stderr = "gold/accept_webp_jpeg.gold" -tr.StillRunningAfter = ts +Test.ATSReplayTest(replay_file="replays/accept_webp.replay.yaml") diff --git a/tests/gold_tests/headers/cachedDuplicateHeaders.test.py b/tests/gold_tests/headers/cachedDuplicateHeaders.test.py index c3c6be98d7f..eafe6cf81c1 100644 --- a/tests/gold_tests/headers/cachedDuplicateHeaders.test.py +++ b/tests/gold_tests/headers/cachedDuplicateHeaders.test.py @@ -1,5 +1,5 @@ ''' -Test cached responses and requests with bodies +Test cached duplicate headers during revalidation. ''' # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -18,43 +18,9 @@ # limitations under the License. Test.Summary = ''' -Test revalidating cached objects +Test cached duplicate headers during revalidation ''' -testName = "RevalidateCacheObject" -Test.ContinueOnFail = True - - -class CachedHeaderValidationTest: - replay_file = "replays/cache-test.replay.yaml" - - def __init__(self): - self.setupOriginServer() - self.setupTS() - - def setupOriginServer(self): - self.server = Test.MakeVerifierServerProcess("cached-header-verifier-server", self.replay_file) - - def setupTS(self): - self.ts = Test.MakeATSProcess("ts", enable_tls=True) - self.ts.Disk.plugin_config.AddLine('xdebug.so --enable=x-cache,x-cache-key,via') - self.ts.Disk.records_config.update( - { - 'proxy.config.diags.debug.enabled': 1, - 'proxy.config.diags.debug.tags': 'http', - 'proxy.config.http.response_via_str': 3, - }) - self.ts.Disk.remap_config.AddLine('map / http://127.0.0.1:{0}'.format(self.server.Variables.http_port)) - - def runTraffic(self): - tr = Test.AddTestRun() - tr.AddVerifierClientProcess("cached-header-verifier-client", self.replay_file, http_ports=[self.ts.Variables.port]) - tr.Processes.Default.StartBefore(self.server) - tr.Processes.Default.StartBefore(self.ts) - tr.StillRunningAfter = self.ts - tr.StillRunningAfter = self.server - - def run(self): - self.runTraffic() +Test.ContinueOnFail = True -CachedHeaderValidationTest().run() +Test.ATSReplayTest(replay_file="replays/cache-test.replay.yaml") diff --git a/tests/gold_tests/headers/cachedIMSRange.test.py b/tests/gold_tests/headers/cachedIMSRange.test.py index 64af159c9f1..6682053931a 100644 --- a/tests/gold_tests/headers/cachedIMSRange.test.py +++ b/tests/gold_tests/headers/cachedIMSRange.test.py @@ -1,5 +1,5 @@ ''' -Test cached responses and requests with bodies +Test revalidating cached objects. ''' # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -21,266 +21,6 @@ Test revalidating cached objects ''' -testName = "RevalidateCacheObject" Test.ContinueOnFail = True -# Set up Origin server -# request_header is from ATS to origin; response from Origin to ATS -# lookup_key is to make unique response in origin for header "UID" that will pass in ATS request -server = Test.MakeOriginServer("server", lookup_key="{%UID}") -# Initial request -request_header = { - "headers": "GET / HTTP/1.1\r\nHost: www.example.com\r\nUID: Fill\r\n\r\n", - "timestamp": "1469733493.993", - "body": "" -} -response_header = { - "headers": - "HTTP/1.1 200 OK\r\nConnection: close\r\nLast-Modified: Tue, 08 May 2018 15:49:41 GMT\r\nCache-Control: max-age=1\r\n\r\n", - "timestamp": "1469733493.993", - "body": "xxx" -} -server.addResponse("sessionlog.json", request_header, response_header) -# IMS revalidation request -request_IMS_header = { - "headers": "GET / HTTP/1.1\r\nUID: IMS\r\nIf-Modified-Since: Tue, 08 May 2018 15:49:41 GMT\r\nHost: www.example.com\r\n\r\n", - "timestamp": "1469733493.993", - "body": "" -} -response_IMS_header = { - "headers": "HTTP/1.1 304 Not Modified\r\nConnection: close\r\nCache-Control: max-age=1\r\n\r\n", - "timestamp": "1469733493.993", - "body": None -} -server.addResponse("sessionlog.json", request_IMS_header, response_IMS_header) - -# EtagFill -request_etagfill_header = { - "headers": "GET /etag HTTP/1.1\r\nHost: www.example.com\r\nUID: EtagFill\r\n\r\n", - "timestamp": "1469733493.993", - "body": None -} -response_etagfill_header = { - "headers": "HTTP/1.1 200 OK\r\nETag: myetag\r\nConnection: close\r\nCache-Control: max-age=1\r\n\r\n", - "timestamp": "1469733493.993", - "body": "xxx" -} -server.addResponse("sessionlog.json", request_etagfill_header, response_etagfill_header) -# INM revalidation -request_INM_header = { - "headers": "GET /etag HTTP/1.1\r\nUID: INM\r\nIf-None-Match: myetag\r\nHost: www.example.com\r\n\r\n", - "timestamp": "1469733493.993", - "body": None -} -response_INM_header = { - "headers": "HTTP/1.1 304 Not Modified\r\nConnection: close\r\nETag: myetag\r\nCache-Control: max-age=1\r\n\r\n", - "timestamp": "1469733493.993", - "body": None -} -server.addResponse("sessionlog.json", request_INM_header, response_INM_header) - -# object changed to 0 byte -request_noBody_header = { - "headers": "GET / HTTP/1.1\r\nUID: noBody\r\nHost: www.example.com\r\n\r\n", - "timestamp": "1469733493.993", - "body": "" -} -response_noBody_header = { - "headers": "HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 0\r\nCache-Control: max-age=3\r\n\r\n", - "timestamp": "1469733493.993", - "body": "" -} -server.addResponse("sessionlog.json", request_noBody_header, response_noBody_header) - -# etag object now is a 404. Yeah, 404s don't usually have Cache-Control, but, ATS's default is to cache 404s for a while. -request_etagfill_header = { - "headers": "GET /etag HTTP/1.1\r\nHost: www.example.com\r\nUID: EtagError\r\n\r\n", - "timestamp": "1469733493.993", - "body": None -} -response_etagfill_header = { - "headers": "HTTP/1.1 404 Not Found\r\nConnection: close\r\nContent-Length: 0\r\nCache-Control: max-age=3\r\n\r\n", - "timestamp": "1469733493.993", - "body": "" -} -server.addResponse("sessionlog.json", request_etagfill_header, response_etagfill_header) - -# ATS Configuration -ts = Test.MakeATSProcess("ts", enable_tls=True) -ts.Disk.plugin_config.AddLine('xdebug.so --enable=x-cache,x-cache-key,via') -ts.addDefaultSSLFiles() -ts.Disk.ssl_multicert_config.AddLine('dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key') -ts.Disk.records_config.update( - { - 'proxy.config.diags.debug.enabled': 1, - 'proxy.config.diags.debug.tags': 'http', - 'proxy.config.http.response_via_str': 3, - 'proxy.config.ssl.server.cert.path': '{0}'.format(ts.Variables.SSLDir), - 'proxy.config.ssl.server.private_key.path': '{0}'.format(ts.Variables.SSLDir), - }) - -default_304_host = 'www.default304.test' -regex_remap_conf_file = "maps.reg" -ts.Disk.remap_config.AddLines( - [ - f'map https://{default_304_host}/ http://127.0.0.1:{server.Variables.Port}/ ' - f'@plugin=regex_remap.so @pparam={regex_remap_conf_file} @pparam=no-query-string @pparam=host', - f'map http://{default_304_host}/ http://127.0.0.1:{server.Variables.Port}/ ' - f'@plugin=regex_remap.so @pparam={regex_remap_conf_file} @pparam=no-query-string @pparam=host', - f'map / http://127.0.0.1:{server.Variables.Port}', - ]) - -ts.Disk.MakeConfigFile(regex_remap_conf_file).AddLine(f'//.*/ http://127.0.0.1:{server.Variables.Port} @status=304') - -ipv4flag = "" -if not Condition.CurlUsingUnixDomainSocket(): - ipv4flag = "--ipv4" - -# Test 0 - Fill a 3 byte object with Last-Modified time into cache. -tr = Test.AddTestRun() -tr.Processes.Default.StartBefore(server) -tr.Processes.Default.StartBefore(ts) -tr.MakeCurlCommand( - '-s -D - -v {0} --http1.1 -H"UID: Fill" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "cache_and_req_body-miss.gold" -tr.StillRunningAfter = ts -tr.StillRunningAfter = server - -# Test 1 - Once it goes stale, fetch it again. We expect Origin to get IMS -# request, and serve a 304. We expect ATS to refresh the object, and give -# a 200 to user -tr = Test.AddTestRun() -tr.DelayStart = 2 -tr.MakeCurlCommand( - '-s -D - -v {0} --http1.1 -H"UID: IMS" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "cache_and_req_body-hit-stale.gold" -tr.StillRunningAfter = ts -tr.StillRunningAfter = server - -# Test 2 - Once it goes stale, fetch it via a range request. We expect -# Origin to get IMS request, and serve a 304. We expect ATS to refresh the -# object, and give a 206 to user -tr = Test.AddTestRun() -tr.DelayStart = 2 -tr.MakeCurlCommand( - '--range 0-1 -s -D - -v {0} --http1.1 -H"UID: IMS" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "cache_and_req_body-hit-stale-206.gold" -tr.StillRunningAfter = ts -tr.StillRunningAfter = server - -# Test 3 - Test 304 response served from a regex-remap rule with HTTP. -tr = Test.AddTestRun() -tr.MakeCurlCommand(f'-vs http://127.0.0.1:{ts.Variables.port}/ -H "Host: {default_304_host}"', ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.All = Testers.GoldFile("gold/http1_304.gold", case_insensitive=True) -tr.StillRunningAfter = server - -if not Condition.CurlUsingUnixDomainSocket(): - # Test 4 - Test 304 response served from a regex-remap rule with HTTPS. - tr = Test.AddTestRun() - tr.MakeCurlCommand(f'-vs -k https://127.0.0.1:{ts.Variables.ssl_port}/ -H "Host: {default_304_host}"', ts=ts) - tr.Processes.Default.ReturnCode = 0 - tr.Processes.Default.Streams.All = Testers.GoldFile("gold/http1_304.gold", case_insensitive=True) - tr.StillRunningAfter = server - - # Test 5 - Test 304 response served from a regex-remap rule with HTTP/2. - tr = Test.AddTestRun() - tr.MakeCurlCommand(f'-vs -k --http2 https://127.0.0.1:{ts.Variables.ssl_port}/ -H "Host: {default_304_host}"', ts=ts) - tr.Processes.Default.ReturnCode = 0 - tr.Processes.Default.Streams.All = Testers.GoldFile("gold/http2_304.gold", case_insensitive=True) - tr.StillRunningAfter = server - -# Test 6 - Fill a new object with an Etag. Not checking the output here. -tr = Test.AddTestRun() -tr.MakeCurlCommand( - '-s -D - -v {0} --http1.1 -H"UID: EtagFill" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/etag' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.StillRunningAfter = ts -tr.StillRunningAfter = server - -# Test 7 - Once the etag object goes stale, fetch it again. We expect -# Origin to get INM request, and serve a 304. We expect ATS to refresh the -# object, and give a 200 to user -tr = Test.AddTestRun() -tr.DelayStart = 2 -tr.MakeCurlCommand( - '-s -D - -v {0} --http1.1 -H"UID: INM" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/etag' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "cache_and_req_body-hit-stale-INM.gold" -tr.StillRunningAfter = ts -tr.StillRunningAfter = server - -# Test 8 - Once the etag object goes stale, fetch it via a range request. -# We expect Origin to get INM request, and serve a 304. We expect ATS to -# refresh the object, and give a 206 to user -tr = Test.AddTestRun() -tr.DelayStart = 2 -tr.MakeCurlCommand( - '--range 0-1 -s -D - -v {0} --http1.1 -H"UID: INM" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/etag' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "cache_and_req_body-hit-stale-206-etag.gold" -tr.StillRunningAfter = ts -tr.StillRunningAfter = server - -# Test 9 - The origin changes the initial LMT object to 0 byte. We expect ATS to fetch and serve the new 0 byte object. -tr = Test.AddTestRun() -tr.DelayStart = 3 -tr.MakeCurlCommand( - '-s -D - -v {0} --http1.1 -H"UID: noBody" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "cache_and_req_nobody-hit-stale.gold" -tr.StillRunningAfter = ts -tr.StillRunningAfter = server - -# Test 10 - Fetch the new 0 byte object again when fresh in cache to ensure its still a 0 byte object. -tr = Test.AddTestRun() -tr.DelayStart = 3 -tr.MakeCurlCommand( - '-s -D - -v {0} --http1.1 -H"UID: noBody" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "cache_and_req_nobody-hit-stale.gold" -tr.StillRunningAfter = ts -tr.StillRunningAfter = server - -# Test 11 - The origin changes the etag object to 0 byte 404. We expect ATS to fetch and serve the 404 0 byte object. -tr = Test.AddTestRun() -tr.DelayStart = 2 -tr.MakeCurlCommand( - '-s -D - -v {0} --http1.1 -H"UID: EtagError" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/etag' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "cache_and_error_nobody.gold" -tr.StillRunningAfter = ts -tr.StillRunningAfter = server - -# Test 12 - Fetch the 0 byte etag object again when fresh in cache to ensure its still a 0 byte object -tr = Test.AddTestRun() -tr.DelayStart = 2 -tr.MakeCurlCommand( - '-s -D - -v {0} --http1.1 -H"UID: EtagError" -H "x-debug: x-cache,x-cache-key,via" -H "Host: www.example.com" http://localhost:{1}/etag' - .format(ipv4flag, ts.Variables.port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "cache_and_error_nobody.gold" -tr.StillRunningAfter = ts -tr.StillRunningAfter = server +Test.ATSReplayTest(replay_file="replays/cached_ims_range.replay.yaml") diff --git a/tests/gold_tests/headers/domain-blacklist-30x.test.py b/tests/gold_tests/headers/domain-blacklist-30x.test.py index d46bdf39588..cc3ab045a44 100644 --- a/tests/gold_tests/headers/domain-blacklist-30x.test.py +++ b/tests/gold_tests/headers/domain-blacklist-30x.test.py @@ -1,5 +1,5 @@ ''' -Tests 30x responses are returned for matching domains +Test that redirect responses are returned for matching domains. ''' # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -17,94 +17,10 @@ # See the License for the specific language governing permissions and # limitations under the License. -import os -import sys - Test.Summary = ''' -Tests 30x responses are returned for matching domains +Test that redirect responses are returned for matching domains ''' -ts = Test.MakeATSProcess("ts") -server = Test.MakeOriginServer("server") - -REDIRECT_301_HOST = 'www.redirect301.test' -REDIRECT_302_HOST = 'www.redirect302.test' -REDIRECT_307_HOST = 'www.redirect307.test' -REDIRECT_308_HOST = 'www.redirect308.test' -REDIRECT_0_HOST = 'www.redirect0.test' -PASSTHRU_HOST = 'www.passthrough.test' - -ts.Disk.records_config.update( - { - 'proxy.config.diags.debug.enabled': 1, - 'proxy.config.diags.debug.tags': 'header_rewrite|dbg_header_rewrite', - 'proxy.config.body_factory.enable_logging': 1, - }) - -ts.Disk.remap_config.AddLine( - """\ -regex_map http://{0}/ http://{0}/ @plugin=header_rewrite.so @pparam=header_rewrite_rules_301.conf -regex_map http://{1}/ http://{1}/ @plugin=header_rewrite.so @pparam=header_rewrite_rules_302.conf -regex_map http://{2}/ http://{2}/ @plugin=header_rewrite.so @pparam=header_rewrite_rules_307.conf -regex_map http://{3}/ http://{3}/ @plugin=header_rewrite.so @pparam=header_rewrite_rules_308.conf -regex_map http://{4}/ http://{4}/ @plugin=header_rewrite.so @pparam=header_rewrite_rules_0.conf -""".format(REDIRECT_301_HOST, REDIRECT_302_HOST, REDIRECT_307_HOST, REDIRECT_308_HOST, REDIRECT_0_HOST)) - -for x in (0, 301, 302, 307, 308): - ts.Disk.MakeConfigFile("header_rewrite_rules_{0}.conf".format(x)).AddLine("""\ -set-redirect {0} "%{{CLIENT-URL}}" -""".format(x)) - -Test.Setup.Copy(os.path.join(os.pardir, os.pardir, 'tools', 'tcp_client.py')) -Test.Setup.Copy('data') - -redirect301tr = Test.AddTestRun(f"Test domain {REDIRECT_301_HOST}") -redirect301tr.Processes.Default.StartBefore(Test.Processes.ts) -redirect301tr.StillRunningAfter = ts -redirect301tr.Processes.Default.Command = f"{sys.executable} tcp_client.py 127.0.0.1 {ts.Variables.port} data/{REDIRECT_301_HOST}_get.txt | grep -v '^Date: '| grep -v '^Server: ATS/'" -redirect301tr.Processes.Default.TimeOut = 5 # seconds -redirect301tr.Processes.Default.ReturnCode = 0 -redirect301tr.Processes.Default.Streams.stdout = "redirect301_get.gold" - -redirect302tr = Test.AddTestRun(f"Test domain {REDIRECT_302_HOST}") -redirect302tr.StillRunningBefore = ts -redirect302tr.StillRunningAfter = ts -redirect302tr.Processes.Default.Command = f"{sys.executable} tcp_client.py 127.0.0.1 {ts.Variables.port} data/{REDIRECT_302_HOST}_get.txt | grep -v '^Date: '| grep -v '^Server: ATS/'" -redirect302tr.Processes.Default.TimeOut = 5 # seconds -redirect302tr.Processes.Default.ReturnCode = 0 -redirect302tr.Processes.Default.Streams.stdout = "redirect302_get.gold" - -redirect307tr = Test.AddTestRun(f"Test domain {REDIRECT_307_HOST}") -redirect302tr.StillRunningBefore = ts -redirect307tr.StillRunningAfter = ts -redirect307tr.Processes.Default.Command = f"{sys.executable} tcp_client.py 127.0.0.1 {ts.Variables.port} data/{REDIRECT_307_HOST}_get.txt | grep -v '^Date: '| grep -v '^Server: ATS/'" -redirect307tr.Processes.Default.TimeOut = 5 # seconds -redirect307tr.Processes.Default.ReturnCode = 0 -redirect307tr.Processes.Default.Streams.stdout = "redirect307_get.gold" - -redirect308tr = Test.AddTestRun(f"Test domain {REDIRECT_308_HOST}") -redirect308tr.StillRunningBefore = ts -redirect308tr.StillRunningAfter = ts -redirect308tr.Processes.Default.Command = f"{sys.executable} tcp_client.py 127.0.0.1 {ts.Variables.port} data/{REDIRECT_308_HOST}_get.txt | grep -v '^Date: '| grep -v '^Server: ATS/'" -redirect308tr.Processes.Default.TimeOut = 5 # seconds -redirect308tr.Processes.Default.ReturnCode = 0 -redirect308tr.Processes.Default.Streams.stdout = "redirect308_get.gold" - -redirect0tr = Test.AddTestRun(f"Test domain {REDIRECT_0_HOST}") -redirect0tr.StillRunningBefore = ts -redirect0tr.StillRunningAfter = ts -redirect0tr.Processes.Default.Command = f"{sys.executable} tcp_client.py 127.0.0.1 {ts.Variables.port} data/{REDIRECT_0_HOST}_get.txt | grep -v '^Date: '| grep -v '^Server: ATS/'" -redirect0tr.Processes.Default.TimeOut = 5 # seconds -redirect0tr.Processes.Default.ReturnCode = 0 -redirect0tr.Processes.Default.Streams.stdout = "redirect0_get.gold" - -passthroughtr = Test.AddTestRun(f"Test domain {PASSTHRU_HOST}") -passthroughtr.StillRunningBefore = ts -passthroughtr.StillRunningAfter = ts -passthroughtr.Processes.Default.Command = f"{sys.executable} tcp_client.py 127.0.0.1 {ts.Variables.port} data/{PASSTHRU_HOST}_get.txt | grep -v '^Date: '| grep -v '^Server: ATS/'" -passthroughtr.Processes.Default.TimeOut = 5 # seconds -passthroughtr.Processes.Default.ReturnCode = 0 -passthroughtr.Processes.Default.Streams.stdout = "passthrough_get.gold" +Test.ContinueOnFail = True -# Overriding the built in ERROR check since we expect some ERROR messages -ts.Disk.diags_log.Content = Testers.ContainsExpression("unsupported redirect status 0", "This test is a failure test") +Test.ATSReplayTest(replay_file="replays/domain-blacklist-30x.replay.yaml") diff --git a/tests/gold_tests/headers/gold/accept_webp.gold b/tests/gold_tests/headers/gold/accept_webp.gold deleted file mode 100644 index 9b6dcb47872..00000000000 --- a/tests/gold_tests/headers/gold/accept_webp.gold +++ /dev/null @@ -1,16 +0,0 @@ -`` -> GET /`` -> Host: www.example.com`` -> User-Agent: curl/`` -> Accept: image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5 -`` -< HTTP/1.1 200 OK -< Content-Type: image/webp -< Cache-Control: max-age=300 -< Content-Length: 3 -< Date: `` -< Age: `` -< Connection: keep-alive -< Via: http/1.1 `` (ApacheTrafficServer/`` [uScMsSfWpSeN:t cCMp sS]) -< Server: ATS/`` -`` diff --git a/tests/gold_tests/headers/gold/accept_webp_cache.gold b/tests/gold_tests/headers/gold/accept_webp_cache.gold deleted file mode 100644 index 71c016243f1..00000000000 --- a/tests/gold_tests/headers/gold/accept_webp_cache.gold +++ /dev/null @@ -1,16 +0,0 @@ -`` -> GET /`` -> Host: www.example.com`` -> User-Agent: curl/`` -> Accept: image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5 -`` -< HTTP/1.1 200 OK -< Content-Type: image/webp -< Cache-Control: max-age=300 -< Content-Length: 3 -< Date: `` -< Age: `` -< Connection: keep-alive -< Via: http/1.1 `` (ApacheTrafficServer/`` [uScRs f p eN:t cCHp s ]) -< Server: ATS/`` -`` diff --git a/tests/gold_tests/headers/gold/accept_webp_jpeg.gold b/tests/gold_tests/headers/gold/accept_webp_jpeg.gold deleted file mode 100644 index 1e76ccc3165..00000000000 --- a/tests/gold_tests/headers/gold/accept_webp_jpeg.gold +++ /dev/null @@ -1,16 +0,0 @@ -`` -> GET /`` -> Host: www.example.com`` -> User-Agent: curl/`` -> Accept: image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5 -`` -< HTTP/1.1 200 OK -< Content-Type: image/webp -< Cache-Control: max-age=300 -< Content-Length: 3 -< Date: `` -< Age: `` -< Connection: keep-alive -< Via: http/1.1 `` (ApacheTrafficServer/`` [uScMsSfWpSeN:t cCMp sS]) -< Server: ATS/`` -`` diff --git a/tests/gold_tests/headers/gold/http1_304.gold b/tests/gold_tests/headers/gold/http1_304.gold deleted file mode 100644 index 3740c2395f9..00000000000 --- a/tests/gold_tests/headers/gold/http1_304.gold +++ /dev/null @@ -1,10 +0,0 @@ -`` -> GET / HTTP/`` -> Host: www.default304.test -> User-Agent: curl/`` -> Accept: */* -`` -< HTTP/`` 304`` -< date: `` -< server: ATS/`` -`` diff --git a/tests/gold_tests/headers/gold/http2_304.gold b/tests/gold_tests/headers/gold/http2_304.gold deleted file mode 100644 index f5c3599828b..00000000000 --- a/tests/gold_tests/headers/gold/http2_304.gold +++ /dev/null @@ -1,10 +0,0 @@ -`` -> GET / HTTP/2 -> Host: www.default304.test -> User-Agent: curl/`` -> Accept: */* -`` -< HTTP/2 304`` -< date: `` -< server: ATS/`` -`` diff --git a/tests/gold_tests/headers/gold/range-200.gold b/tests/gold_tests/headers/gold/range-200.gold deleted file mode 100644 index 0aa6ac90c5e..00000000000 --- a/tests/gold_tests/headers/gold/range-200.gold +++ /dev/null @@ -1,11 +0,0 @@ -HTTP/1.1 200 OK -Server: ATS/`` -Cache-Control: max-age=`` -Last-Modified: Thu, 10 Feb 2022 00:00:00 GMT -ETag: range -Content-Length: 11 -Date: `` -Age: `` -Connection: keep-alive - -0123456789 diff --git a/tests/gold_tests/headers/gold/range-206-revalidated.gold b/tests/gold_tests/headers/gold/range-206-revalidated.gold deleted file mode 100644 index e918cc03892..00000000000 --- a/tests/gold_tests/headers/gold/range-206-revalidated.gold +++ /dev/null @@ -1,12 +0,0 @@ -HTTP/1.1 206 Partial Content -Server: ATS/`` -Cache-Control: max-age=1 -Last-Modified: Thu, 10 Feb 2022 00:00:00 GMT -Content-Length: 5 -Date: `` -Etag: range -Age: `` -Content-Range: bytes 1-5/11 -Connection: keep-alive - -12345`` diff --git a/tests/gold_tests/headers/gold/range-206.gold b/tests/gold_tests/headers/gold/range-206.gold deleted file mode 100644 index 8caff80aa1b..00000000000 --- a/tests/gold_tests/headers/gold/range-206.gold +++ /dev/null @@ -1,12 +0,0 @@ -HTTP/1.1 206 Partial Content -Server: ATS/`` -Cache-Control: max-age=300 -Last-Modified: Thu, 10 Feb 2022 00:00:00 GMT -ETag: range -Content-Length: 5 -Date: `` -Age: `` -Content-Range: bytes 1-5/11 -Connection: keep-alive - -12345`` diff --git a/tests/gold_tests/headers/gold/range-416.gold b/tests/gold_tests/headers/gold/range-416.gold deleted file mode 100644 index 1a87e708d11..00000000000 --- a/tests/gold_tests/headers/gold/range-416.gold +++ /dev/null @@ -1,6 +0,0 @@ -HTTP/1.1 416 Requested Range Not Satisfiable -Date: `` -Connection: keep-alive -Server: ATS/`` -Cache-Control: no-store -`` diff --git a/tests/gold_tests/headers/hsts.test.py b/tests/gold_tests/headers/hsts.test.py index e5438325165..147fb7cd115 100644 --- a/tests/gold_tests/headers/hsts.test.py +++ b/tests/gold_tests/headers/hsts.test.py @@ -1,5 +1,5 @@ ''' -Test the hsts response header. +Test the HSTS response header. ''' # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -18,56 +18,9 @@ # limitations under the License. Test.Summary = ''' -heck hsts header is set correctly +Check the HSTS header is set correctly ''' -Test.SkipIf(Condition.CurlUsingUnixDomainSocket()) -Test.ContinueOnFail = True - -# Define default ATS -ts = Test.MakeATSProcess("ts", enable_tls=True) -server = Test.MakeOriginServer("server") - -# **testname is required** -testName = "" -request_header = {"headers": "GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n", "timestamp": "1469733493.993", "body": ""} -response_header = {"headers": "HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", "timestamp": "1469733493.993", "body": ""} -server.addResponse("sessionlog.json", request_header, response_header) - -# ATS Configuration -ts.addDefaultSSLFiles() -ts.Disk.records_config.update( - { - 'proxy.config.diags.debug.enabled': 1, - 'proxy.config.diags.debug.tags': 'ssl', - 'proxy.config.ssl.server.cert.path': '{0}'.format(ts.Variables.SSLDir), - 'proxy.config.ssl.server.private_key.path': '{0}'.format(ts.Variables.SSLDir), - 'proxy.config.ssl.hsts_max_age': 300, - }) - -ts.Disk.remap_config.AddLine('map https://www.example.com http://127.0.0.1:{0}'.format(server.Variables.Port)) - -ts.Disk.ssl_multicert_config.AddLine('dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key') - -# Test 1 - 200 Response -tr = Test.AddTestRun() -tr.Processes.Default.StartBefore(server) -tr.Processes.Default.StartBefore(Test.Processes.ts) -tr.MakeCurlCommand( - '-s -D - --verbose --ipv4 --http1.1 --insecure --header "Host: {0}" https://localhost:{1}'.format( - 'www.example.com', ts.Variables.ssl_port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "hsts.200.gold" -tr.StillRunningAfter = ts +Test.ContinueOnFail = True -# Test 2 - 404 Not Found on Accelerator -tr = Test.AddTestRun() -tr.MakeCurlCommand( - '-s -D - --verbose --ipv4 --http1.1 --insecure --header "Host: {0}" https://localhost:{1}'.format( - 'bad_host', ts.Variables.ssl_port), - ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "hsts.404.gold" -tr.StillRunningAfter = server -tr.StillRunningAfter = ts +Test.ATSReplayTest(replay_file="replays/hsts.replay.yaml") diff --git a/tests/gold_tests/headers/invalid_range_header.test.py b/tests/gold_tests/headers/invalid_range_header.test.py index 38c36b24014..f9613e7f7ad 100644 --- a/tests/gold_tests/headers/invalid_range_header.test.py +++ b/tests/gold_tests/headers/invalid_range_header.test.py @@ -1,4 +1,5 @@ ''' +Test invalid values in the Range header. ''' # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -16,53 +17,10 @@ # See the License for the specific language governing permissions and # limitations under the License. -import os - Test.Summary = ''' -Test invalid values in range header +Test invalid values in the Range header ''' -Test.ContinueOnFail = True - - -class InvalidRangeHeaderTest: - invalidRangeRequestReplayFile = "replays/invalid_range_request.replay.yaml" - - def __init__(self): - self.setupOriginServer() - self.setupTS() - - def setupOriginServer(self): - self.server = Test.MakeVerifierServerProcess("verifier-server1", self.invalidRangeRequestReplayFile) - - def setupTS(self): - self.ts = Test.MakeATSProcess("ts1") - self.ts.Disk.records_config.update( - { - 'proxy.config.diags.debug.enabled': 1, - 'proxy.config.diags.debug.tags': 'http', - 'proxy.config.http.cache.http': 1, - 'proxy.config.http.cache.range.write': 1, - 'proxy.config.http.cache.required_headers': 0, - 'proxy.config.http.insert_age_in_response': 0 - }) - self.ts.Disk.remap_config.AddLine(f"map / http://127.0.0.1:{self.server.Variables.http_port}/",) - - def runTraffic(self): - tr = Test.AddTestRun() - tr.AddVerifierClientProcess("client1", self.invalidRangeRequestReplayFile, http_ports=[self.ts.Variables.port]) - tr.Processes.Default.StartBefore(self.server) - tr.Processes.Default.StartBefore(self.ts) - tr.StillRunningAfter = self.server - tr.StillRunningAfter = self.ts - - # verification - tr.Processes.Default.Streams.stdout += Testers.ContainsExpression( - r"Received an HTTP/1 416 response for key 2", "Verify that client receives a 416 response") - tr.Processes.Default.Streams.stdout += Testers.ContainsExpression( - r"x-responseheader: failed_response", "Verify that the response came from the server") - - def run(self): - self.runTraffic() +Test.ContinueOnFail = True -InvalidRangeHeaderTest().run() +Test.ATSReplayTest(replay_file="replays/invalid_range_request.replay.yaml") diff --git a/tests/gold_tests/headers/maps.reg b/tests/gold_tests/headers/maps.reg new file mode 100644 index 00000000000..692a82f3524 --- /dev/null +++ b/tests/gold_tests/headers/maps.reg @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +//.*/ http://127.0.0.1:1 @status=304 diff --git a/tests/gold_tests/headers/normalized_ae_match_vary_cache.test.py b/tests/gold_tests/headers/normalized_ae_match_vary_cache.test.py index de78f27571e..57f96dcfb0a 100644 --- a/tests/gold_tests/headers/normalized_ae_match_vary_cache.test.py +++ b/tests/gold_tests/headers/normalized_ae_match_vary_cache.test.py @@ -1,6 +1,6 @@ ''' Test cache matching with the normalized Accept-Encoding header field -and the Vary header field in response +and the Vary header field in response. ''' # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -18,119 +18,10 @@ # See the License for the specific language governing permissions and # limitations under the License. -import os - Test.Summary = ''' Test cache matching with the normalized Accept-Encoding and the Vary header field in response ''' Test.ContinueOnFail = True -testName = "NORMALIZE_AE_MATCH_VARY" - -replay_file = "replays/normalized_ae_varied_transactions.replay.yaml" -server = Test.MakeVerifierServerProcess("server", replay_file) - -# Verify that cache hit requests never reach the server -# Case 2 (normalize_ae:1) cache hits -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 12", "Verify empty Accept-Encoding (uuid 12) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 13", "Verify deflate request (uuid 13) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 14", "Verify br,compress request (uuid 14) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 16", "Verify br,compress,gzip request (uuid 16) is a cache hit and doesn't reach the server.") -# Case 3 (normalize_ae:2) cache hits -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 22", "Verify empty Accept-Encoding (uuid 22) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 23", "Verify deflate request (uuid 23) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 26", "Verify br,compress,gzip request (uuid 26) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 27", "Verify compress,gzip request (uuid 27) is a cache hit and doesn't reach the server.") -# Case 4 (normalize_ae:3) cache hits -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 32", "Verify empty Accept-Encoding (uuid 32) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 33", "Verify deflate request (uuid 33) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 37", "Verify compress,gzip request (uuid 37) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 38", "Verify br;q=1.1 request (uuid 38) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 39", "Verify br,gzip;q=0.8 request (uuid 39) is a cache hit and doesn't reach the server.") -# Case 5 (normalize_ae:4) cache hits -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 42", "Verify empty Accept-Encoding (uuid 42) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 43", "Verify deflate request (uuid 43) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 47", "Verify zstd,br,compress,gzip request (uuid 47) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 48", "Verify br,compress,gzip request (uuid 48) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 49", "Verify compress,zstd request (uuid 49) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 410", "Verify compress,gzip request (uuid 410) is a cache hit and doesn't reach the server.") -# Case 6 (normalize_ae:5) cache hits -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 52", "Verify empty Accept-Encoding (uuid 52) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 53", "Verify deflate request (uuid 53) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 510", "Verify compress,zstd request (uuid 510) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 511", "Verify compress,br request (uuid 511) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 512", "Verify compress,gzip request (uuid 512) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 513", "Verify zstd;q=1.1 request (uuid 513) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 514", "Verify br;q=1.1 request (uuid 514) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 515", "Verify zstd,br;q=0.8 request (uuid 515) is a cache hit and doesn't reach the server.") -server.Streams.stdout += Testers.ExcludesExpression( - "uuid: 516", "Verify zstd,gzip;q=0.8 request (uuid 516) is a cache hit and doesn't reach the server.") - -ts = Test.MakeATSProcess("ts", enable_cache=True) -ts.Disk.remap_config.AddLine( - f"map http://www.ae-0.com http://127.0.0.1:{server.Variables.http_port}" + - ' @plugin=conf_remap.so @pparam=proxy.config.http.normalize_ae=0') -ts.Disk.remap_config.AddLine( - f"map http://www.ae-1.com http://127.0.0.1:{server.Variables.http_port}" + - ' @plugin=conf_remap.so @pparam=proxy.config.http.normalize_ae=1') -ts.Disk.remap_config.AddLine( - f"map http://www.ae-2.com http://127.0.0.1:{server.Variables.http_port}" + - ' @plugin=conf_remap.so @pparam=proxy.config.http.normalize_ae=2') -# disable normalize_ae=3 on 9.1 -ts.Disk.remap_config.AddLine( - f"map http://www.ae-3.com http://127.0.0.1:{server.Variables.http_port}" + - ' @plugin=conf_remap.so @pparam=proxy.config.http.normalize_ae=3') -ts.Disk.remap_config.AddLine( - f"map http://www.ae-4.com http://127.0.0.1:{server.Variables.http_port}" + - ' @plugin=conf_remap.so @pparam=proxy.config.http.normalize_ae=4') -ts.Disk.remap_config.AddLine( - f"map http://www.ae-5.com http://127.0.0.1:{server.Variables.http_port}" + - ' @plugin=conf_remap.so @pparam=proxy.config.http.normalize_ae=5') -ts.Disk.plugin_config.AddLine('xdebug.so --enable=x-cache') -ts.Disk.records_config.update( - { - 'proxy.config.diags.debug.enabled': 1, - 'proxy.config.diags.debug.tags': 'http', - 'proxy.config.http.response_via_str': 3, - # the following variables could affect the results of alternate cache matching, - # define them with their default values explicitly - 'proxy.config.cache.limits.http.max_alts': 6, - 'proxy.config.http.cache.ignore_accept_mismatch': 2, - 'proxy.config.http.cache.ignore_accept_language_mismatch': 2, - 'proxy.config.http.cache.ignore_accept_encoding_mismatch': 2, - 'proxy.config.http.cache.ignore_accept_charset_mismatch': 2, - }) - -tr = Test.AddTestRun() -tr.Processes.Default.StartBefore(server) -tr.Processes.Default.StartBefore(ts) -tr.AddVerifierClientProcess("client", replay_file, http_ports=[ts.Variables.port]) -tr.StillRunningAfter = ts +Test.ATSReplayTest(replay_file="replays/normalized_ae_varied_transactions.replay.yaml") diff --git a/tests/gold_tests/headers/range.test.py b/tests/gold_tests/headers/range.test.py index 32b94c66645..85661b6e289 100644 --- a/tests/gold_tests/headers/range.test.py +++ b/tests/gold_tests/headers/range.test.py @@ -1,5 +1,6 @@ -""" -""" +''' +Test range requests. +''' # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information @@ -16,200 +17,10 @@ # See the License for the specific language governing permissions and # limitations under the License. -Test.Summary = """ +Test.Summary = ''' Test range requests -""" +''' Test.ContinueOnFail = True -CACHE_SHORT_MAXAGE = 1 - - -def register(microserver, request_hdr, request_body, response_hdr, response_body): - request = { - "headers": "{}\r\n\r\n".format("\r\n".join(line for line in request_hdr.split("\n") if line)), - "timestamp": "1469733493.993", - "body": request_body, - } - response = { - "headers": "{}\r\n\r\n".format("\r\n".join(line for line in response_hdr.split("\n") if line)), - "timestamp": "1469733493.993", - "body": response_body, - } - microserver.addResponse("sessionlog.json", request, response) - - -def curl_whole(ts, path=""): - return f"-sSv -D - http://127.0.0.1:{ts.Variables.port}/{path}" - - -def curl_range(ts, path="", ifrange=None, start=1, end=5): - opt = f"-H 'If-Range: {ifrange}'" if ifrange else "" - return f"-sSv -D - {opt} -H 'Range: bytes={start}-{end}' http://127.0.0.1:{ts.Variables.port}/{path}" - - -# ---- -# Setup Origin Server -# ---- -microserver = Test.MakeOriginServer("microserver") - -request_hdr = """ -GET / HTTP/1.1 -Host: 127.0.0.1 -""" - -response_hdr = """ -HTTP/1.1 200 OK -Server: microserver -Connection: close -Cache-Control: max-age=300 -Last-Modified: Thu, 10 Feb 2022 00:00:00 GMT -ETag: range -""" - -short_cache_request_hdr = """ -GET /short HTTP/1.1 -Host: 127.0.0.1 -""" - -short_cache_response_hdr = f""" -HTTP/1.1 200 OK -Server: microserver -Connection: close -Cache-Control: max-age={CACHE_SHORT_MAXAGE} -Last-Modified: Thu, 10 Feb 2022 00:00:00 GMT -ETag: range -""" - -response_body = f"{''.join(str(i) for i in range(10))}\n" - -register(microserver, request_hdr, "", response_hdr, response_body) -register(microserver, short_cache_request_hdr, "", short_cache_response_hdr, response_body) - -# The purpose here is to have a somewhat smarter origin that can respond to If-Modified-Since queries. -# We then can test how the cache server using this origin deals with stale caches. -origin = Test.MakeATSProcess("origin") - -origin.Disk.remap_config.AddLine(f"map / http://127.0.0.1:{microserver.Variables.Port}") - -origin.Disk.records_config.update( - { - "proxy.config.http.cache.http": 1, - "proxy.config.http.wait_for_cache": 1, - "proxy.config.http.insert_age_in_response": 0, - "proxy.config.http.request_via_str": 0, - "proxy.config.http.response_via_str": 0, - "proxy.config.diags.debug.enabled": 1, - "proxy.config.diags.debug.tags": "http", - }) - -# Make the origin return 304 Not Modified for stale caches -origin.Disk.cache_config.AddLine("dest_ip=127.0.0.1 pin-in-cache=1d") - -# ---- -# Setup ATS -# ---- -# HACK: Don't use a fixed port because it causes ensuing tests to fail. The problem -# appears to be microDNS not allowing address reuse. -# -# https://docs.python.org/3/library/socketserver.html#socketserver.BaseServer.allow_reuse_address -ts = Test.MakeATSProcess("ts") - -ts.Disk.remap_config.AddLine(f"map / http://127.0.0.1:{origin.Variables.port}/") - -ts.Disk.records_config.update( - { - "proxy.config.http.cache.http": 1, - "proxy.config.http.cache.range.write": 1, - "proxy.config.http.response_via_str": 3, - "proxy.config.http.wait_for_cache": 1, - "proxy.config.diags.debug.enabled": 1, - "proxy.config.diags.debug.tags": "http", - }) -# ---- -# Test Cases -# ---- - -# On cache miss -# --- - -# ATS should ignore the Range header when given an If-Range header with the incorrect etag -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_range(ts, ifrange='"should-not-match"'), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.StartBefore(microserver, ready=When.PortOpen(microserver.Variables.Port)) -tr.Processes.Default.StartBefore(origin, ready=When.PortOpen(origin.Variables.port)) -tr.Processes.Default.StartBefore(Test.Processes.ts, ready=When.PortOpen(ts.Variables.port)) -tr.Processes.Default.Streams.stdout = "gold/range-200.gold" - -# On cache hit -# --- - -# ATS should respond to Range requests with partial content -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_range(ts), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-206.gold" - -# ATS should respond to Range requests with partial content when given an If-Range header with -# the correct etag -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_range(ts, ifrange='"range"'), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-206.gold" - -# ATS should respond to Range requests with partial content when given an If-Range header -# that matches the Last-Modified header of the cached response -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_range(ts, ifrange="Thu, 10 Feb 2022 00:00:00 GMT"), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-206.gold" - -# ATS should respond to Range requests with the full content when given an If-Range header -# that doesn't match the Last-Modified header of the cached response -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_range(ts, ifrange="Thu, 10 Feb 2022 01:00:00 GMT"), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-200.gold" - -# ATS should respond to Range requests with a 416 error code when the given Range is invalid -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_range(ts, start=100, end=105), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-416.gold" - -# ATS should ignore the Range header when given an If-Range header with the incorrect etag -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_range(ts, ifrange='"should-not-match"'), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-200.gold" - -# ATS should ignore the Range header when given an If-Range header with weak etags -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_range(ts, ifrange='W/"range"'), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-200.gold" - -# ATS should ignore the Range header when given an If-Range header with a date older than the -# Last-Modified header of the cached response -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_range(ts, ifrange="Wed, 09 Feb 2022 23:00:00 GMT"), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-200.gold" - -# Write to the cache by requesting the entire content -tr = Test.AddTestRun() -tr.MakeCurlCommand(curl_whole(ts, path="short"), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-200.gold" - -# ATS should respond to range requests with partial content for stale caches in response to -# valid If-Range requests if the origin responds with 304 Not Modified. -tr = Test.AddTestRun() -tr.MakeCurlCommandMulti(f"sleep {2 * CACHE_SHORT_MAXAGE}; {{curl}} " + curl_range(ts, path="short", ifrange='"range"'), ts=ts) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/range-206-revalidated.gold" - -tr.StillRunningAfter = origin -tr.StillRunningAfter = microserver -tr.StillRunningAfter = ts +Test.ATSReplayTest(replay_file="replays/range.replay.yaml") diff --git a/tests/gold_tests/headers/replays/accept_webp.replay.yaml b/tests/gold_tests/headers/replays/accept_webp.replay.yaml new file mode 100644 index 00000000000..729d28d118d --- /dev/null +++ b/tests/gold_tests/headers/replays/accept_webp.replay.yaml @@ -0,0 +1,161 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + + blocks: + - origin_response_should_not_be_used: &origin_response_should_not_be_used + server-response: + status: 500 + reason: "Internal Server Error" + headers: + fields: + - [Content-Length, 16] + content: + encoding: plain + data: "origin-should-not" + +autest: + description: 'Verify cached image/webp responses are not reused for non-webp clients' + + server: + name: 'accept-webp-server' + + client: + name: 'accept-webp-client' + + ats: + name: 'ts-accept-webp' + process_config: + enable_cache: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http_match' + proxy.config.http.cache.ignore_accept_mismatch: 0 + proxy.config.http.insert_response_via_str: 3 + proxy.config.http.cache.http: 1 + proxy.config.http.wait_for_cache: 1 + + remap_config: + - from: "http://www.example.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + +sessions: + - transactions: + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.example.com] + - [Accept, "image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5"] + - [uuid, webp-origin] + + proxy-request: + headers: + fields: + - [Accept, { value: "image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5", as: equal }] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Connection, close] + - [Content-Type, image/webp] + - [Cache-Control, "max-age=300"] + - [Content-Length, 3] + content: + encoding: plain + data: "xxx" + + proxy-response: + status: 200 + headers: + fields: + - [Content-Type, { value: image/webp, as: equal }] + - [Cache-Control, { value: max-age=300, as: equal }] + content: + verify: + value: "xxx" + as: equal + + - client-request: + delay: 100ms + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.example.com] + - [Accept, "image/webp,image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5"] + - [uuid, webp-cache] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 200 + headers: + fields: + - [Content-Type, { value: image/webp, as: equal }] + - [Cache-Control, { value: max-age=300, as: equal }] + content: + verify: + value: "xxx" + as: equal + + - client-request: + delay: 100ms + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.example.com] + - [Accept, "image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5"] + - [uuid, no-webp-origin] + + proxy-request: + headers: + fields: + - [Accept, { value: "image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5", as: equal }] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Connection, close] + - [Content-Type, image/webp] + - [Cache-Control, "max-age=300"] + - [Content-Length, 3] + content: + encoding: plain + data: "yyy" + + proxy-response: + status: 200 + headers: + fields: + - [Content-Type, { value: image/webp, as: equal }] + - [Cache-Control, { value: max-age=300, as: equal }] + content: + verify: + value: "yyy" + as: equal diff --git a/tests/gold_tests/headers/replays/cache-test.replay.yaml b/tests/gold_tests/headers/replays/cache-test.replay.yaml index 9299ed849ea..50d5e2913f6 100644 --- a/tests/gold_tests/headers/replays/cache-test.replay.yaml +++ b/tests/gold_tests/headers/replays/cache-test.replay.yaml @@ -17,6 +17,32 @@ meta: version: "1.0" +autest: + description: 'Verify cached duplicate headers are preserved across stale revalidation' + + server: + name: 'cached-header-server' + + client: + name: 'cached-header-client' + + ats: + name: 'ts' + process_config: + enable_cache: true + + plugin_config: + - 'xdebug.so --enable=x-cache,x-cache-key,via' + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http' + proxy.config.http.response_via_str: 3 + + remap_config: + - from: "/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + # Note 1: # When testing duplicate headers here, replay files cannot # handle two seperate values. So make sure that duplicate headers diff --git a/tests/gold_tests/headers/replays/cached_ims_range.replay.yaml b/tests/gold_tests/headers/replays/cached_ims_range.replay.yaml new file mode 100644 index 00000000000..3911c76298b --- /dev/null +++ b/tests/gold_tests/headers/replays/cached_ims_range.replay.yaml @@ -0,0 +1,437 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + + blocks: + - origin_response_should_not_be_used: &origin_response_should_not_be_used + server-response: + status: 500 + reason: "Internal Server Error" + headers: + fields: + - [Content-Length, 16] + content: + encoding: plain + data: "origin-should-not" + +autest: + description: 'Verify IMS and INM cache revalidation, range revalidation, and regex-remap 304 handling' + + server: + name: 'cached-ims-range-server' + + client: + name: 'cached-ims-range-client' + + ats: + name: 'ts-cached-ims-range' + process_config: + enable_cache: true + enable_tls: true + + plugin_config: + - 'xdebug.so --enable=x-cache,x-cache-key,via' + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http' + proxy.config.http.response_via_str: 3 + + copy_to_config_dir: + - 'maps.reg' + + remap_config: + - from: "https://www.default304.test/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: "regex_remap.so" + args: + - "maps.reg" + - "no-query-string" + - "host" + - from: "http://www.default304.test/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: "regex_remap.so" + args: + - "maps.reg" + - "no-query-string" + - "host" + - from: "/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + +sessions: + - transactions: + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.example.com] + - [UID, Fill] + - [x-debug, "x-cache,x-cache-key,via"] + - [uuid, fill-lmt] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Connection, close] + - [Last-Modified, "Tue, 08 May 2018 15:49:41 GMT"] + - [Cache-Control, max-age=1] + - [Content-Length, 3] + content: + encoding: plain + data: "xxx" + + proxy-response: + status: 200 + headers: + fields: + - [X-Cache, { value: miss, as: equal }] + - [Content-Length, { value: 3, as: equal }] + content: + verify: + value: "xxx" + as: equal + + - client-request: + delay: 2s + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.example.com] + - [UID, IMS] + - [x-debug, "x-cache,x-cache-key,via"] + - [uuid, ims-refresh] + + proxy-request: + headers: + fields: + - [If-Modified-Since, { value: "Tue, 08 May 2018 15:49:41 GMT", as: equal }] + + server-response: + status: 304 + reason: Not Modified + headers: + fields: + - [Connection, close] + - [Cache-Control, max-age=1] + + proxy-response: + status: 200 + headers: + fields: + - [X-Cache, { value: hit-stale, as: equal }] + - [Content-Length, { value: 3, as: equal }] + content: + verify: + value: "xxx" + as: equal + + - client-request: + delay: 2s + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.example.com] + - [UID, IMS] + - [x-debug, "x-cache,x-cache-key,via"] + - [Range, bytes=0-1] + - [uuid, ims-range-refresh] + + server-response: + status: 304 + reason: Not Modified + headers: + fields: + - [Connection, close] + - [Cache-Control, max-age=1] + + proxy-response: + status: 206 + headers: + fields: + - [X-Cache, { value: hit-stale, as: equal }] + - [Content-Length, { value: 2, as: equal }] + - [Content-Range, { value: "bytes 0-1/3", as: equal }] + content: + verify: + value: "xx" + as: equal + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.default304.test] + - [uuid, default304-http] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 304 + + - client-request: + method: GET + version: "1.1" + url: /etag + headers: + fields: + - [Host, www.example.com] + - [UID, EtagFill] + - [x-debug, "x-cache,x-cache-key,via"] + - [uuid, fill-etag] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Connection, close] + - [ETag, myetag] + - [Cache-Control, max-age=1] + - [Content-Length, 3] + content: + encoding: plain + data: "xxx" + + proxy-response: + status: 200 + content: + verify: + value: "xxx" + as: equal + + - client-request: + delay: 2s + method: GET + version: "1.1" + url: /etag + headers: + fields: + - [Host, www.example.com] + - [UID, INM] + - [x-debug, "x-cache,x-cache-key,via"] + - [uuid, inm-refresh] + + proxy-request: + headers: + fields: + - [If-None-Match, { value: myetag, as: equal }] + + server-response: + status: 304 + reason: Not Modified + headers: + fields: + - [Connection, close] + - [ETag, myetag] + - [Cache-Control, max-age=1] + + proxy-response: + status: 200 + headers: + fields: + - [X-Cache, { value: hit-stale, as: equal }] + - [ETag, { value: myetag, as: equal }] + content: + verify: + value: "xxx" + as: equal + + - client-request: + delay: 2s + method: GET + version: "1.1" + url: /etag + headers: + fields: + - [Host, www.example.com] + - [UID, INM] + - [x-debug, "x-cache,x-cache-key,via"] + - [Range, bytes=0-1] + - [uuid, inm-range-refresh] + + server-response: + status: 304 + reason: Not Modified + headers: + fields: + - [Connection, close] + - [ETag, myetag] + - [Cache-Control, max-age=1] + + proxy-response: + status: 206 + headers: + fields: + - [X-Cache, { value: hit-stale, as: equal }] + - [ETag, { value: myetag, as: equal }] + - [Content-Length, { value: 2, as: equal }] + - [Content-Range, { value: "bytes 0-1/3", as: equal }] + content: + verify: + value: "xx" + as: equal + + - client-request: + delay: 3s + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.example.com] + - [UID, noBody] + - [x-debug, "x-cache,x-cache-key,via"] + - [uuid, nobody-refresh] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Connection, close] + - [Content-Length, 0] + - [Cache-Control, max-age=3] + + proxy-response: + status: 200 + headers: + fields: + - [Content-Length, { value: 0, as: equal }] + - [Cache-Control, { value: max-age=3, as: equal }] + + - client-request: + delay: 100ms + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.example.com] + - [UID, noBody] + - [x-debug, "x-cache,x-cache-key,via"] + - [uuid, nobody-cache] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 200 + headers: + fields: + - [Content-Length, { value: 0, as: equal }] + - [Cache-Control, { value: max-age=3, as: equal }] + + - client-request: + delay: 2s + method: GET + version: "1.1" + url: /etag + headers: + fields: + - [Host, www.example.com] + - [UID, EtagError] + - [x-debug, "x-cache,x-cache-key,via"] + - [uuid, etag-error-refresh] + + server-response: + status: 404 + reason: Not Found + headers: + fields: + - [Connection, close] + - [Content-Length, 0] + - [Cache-Control, max-age=3] + + proxy-response: + status: 404 + headers: + fields: + - [Content-Length, { value: 0, as: equal }] + - [Cache-Control, { value: max-age=3, as: equal }] + + - client-request: + delay: 100ms + method: GET + version: "1.1" + url: /etag + headers: + fields: + - [Host, www.example.com] + - [UID, EtagError] + - [x-debug, "x-cache,x-cache-key,via"] + - [uuid, etag-error-cache] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 404 + headers: + fields: + - [Content-Length, { value: 0, as: equal }] + - [Cache-Control, { value: max-age=3, as: equal }] + + - protocol: + stack: https + tls: + sni: www.default304.test + transactions: + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.default304.test] + - [uuid, default304-https] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 304 + + - protocol: + stack: http2 + tls: + sni: www.default304.test + transactions: + - client-request: + headers: + fields: + - [":method", GET] + - [":scheme", https] + - [":authority", www.default304.test] + - [":path", /] + - [uuid, default304-h2] + + <<: *origin_response_should_not_be_used + + proxy-response: + headers: + fields: + - [":status", { value: "304", as: equal }] diff --git a/tests/gold_tests/headers/replays/domain-blacklist-30x.replay.yaml b/tests/gold_tests/headers/replays/domain-blacklist-30x.replay.yaml new file mode 100644 index 00000000000..32f2616be0d --- /dev/null +++ b/tests/gold_tests/headers/replays/domain-blacklist-30x.replay.yaml @@ -0,0 +1,215 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + + blocks: + - origin_response_should_not_be_used: &origin_response_should_not_be_used + server-response: + status: 500 + reason: "Internal Server Error" + headers: + fields: + - [Content-Length, 16] + content: + encoding: plain + data: "origin-should-not" + +autest: + description: 'Verify header_rewrite redirects are returned for configured hosts' + + server: + name: 'redirect-server' + + client: + name: 'redirect-client' + + ats: + name: 'ts-domain-redirects' + + process_config: + disable_log_checks: true + + copy_to_config_dir: + - 'rewrite_rules' + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'header_rewrite|dbg_header_rewrite' + proxy.config.body_factory.enable_logging: 1 + + remap_config: + - >- + regex_map http://www.redirect301.test/ http://www.redirect301.test/ + @plugin=header_rewrite.so + @pparam=rewrite_rules/header_rewrite_rules_301.conf + - >- + regex_map http://www.redirect302.test/ http://www.redirect302.test/ + @plugin=header_rewrite.so + @pparam=rewrite_rules/header_rewrite_rules_302.conf + - >- + regex_map http://www.redirect307.test/ http://www.redirect307.test/ + @plugin=header_rewrite.so + @pparam=rewrite_rules/header_rewrite_rules_307.conf + - >- + regex_map http://www.redirect308.test/ http://www.redirect308.test/ + @plugin=header_rewrite.so + @pparam=rewrite_rules/header_rewrite_rules_308.conf + - >- + regex_map http://www.redirect0.test/ http://www.redirect0.test/ + @plugin=header_rewrite.so + @pparam=rewrite_rules/header_rewrite_rules_0.conf + + log_validation: + diags_log: + contains: + - expression: 'unsupported redirect status 0' + description: 'Verify the failure case for redirect status 0 is logged' + +sessions: + - transactions: + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.redirect301.test] + - [uuid, redirect-301] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 301 + headers: + fields: + - [Location, { value: "http://www.redirect301.test/", as: equal }] + - [Cache-Control, { value: no-store, as: equal }] + content: + verify: + value: 'The new location is "http://www.redirect301.test/".' + as: contains + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.redirect302.test] + - [uuid, redirect-302] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 302 + headers: + fields: + - [Location, { value: "http://www.redirect302.test/", as: equal }] + - [Cache-Control, { value: no-store, as: equal }] + content: + verify: + value: 'The new location is "http://www.redirect302.test/".' + as: contains + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.redirect307.test] + - [uuid, redirect-307] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 307 + headers: + fields: + - [Location, { value: "http://www.redirect307.test/", as: equal }] + - [Cache-Control, { value: no-store, as: equal }] + content: + verify: + value: 'The new location is "http://www.redirect307.test/".' + as: contains + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.redirect308.test] + - [uuid, redirect-308] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 308 + headers: + fields: + - [Location, { value: "http://www.redirect308.test/", as: equal }] + - [Cache-Control, { value: no-store, as: equal }] + content: + verify: + value: 'The new location is "http://www.redirect308.test/".' + as: contains + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.redirect0.test] + - [uuid, redirect-0] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 302 + headers: + fields: + - [Location, { value: "http://www.redirect0.test/", as: equal }] + - [Cache-Control, { value: no-store, as: equal }] + content: + verify: + value: 'The new location is "http://www.redirect0.test/".' + as: contains + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.passthrough.test] + - [uuid, passthrough-404] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 404 + headers: + fields: + - [Cache-Control, { value: no-store, as: equal }] + - [Content-Type, { value: text/html, as: contains }] + content: + verify: + value: "Not Found on Accelerator" + as: contains diff --git a/tests/gold_tests/headers/replays/hsts.replay.yaml b/tests/gold_tests/headers/replays/hsts.replay.yaml new file mode 100644 index 00000000000..477a2ff973a --- /dev/null +++ b/tests/gold_tests/headers/replays/hsts.replay.yaml @@ -0,0 +1,106 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + + blocks: + - origin_response_should_not_be_used: &origin_response_should_not_be_used + server-response: + status: 500 + reason: "Internal Server Error" + headers: + fields: + - [Content-Length, 0] + +autest: + description: 'Verify HSTS is added on mapped HTTPS traffic but not accelerator 404 responses' + + server: + name: 'hsts-server' + + client: + name: 'hsts-client' + + ats: + name: 'ts-hsts' + process_config: + enable_tls: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'ssl' + proxy.config.ssl.hsts_max_age: 300 + + remap_config: + - from: "https://www.example.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + +sessions: + - protocol: + stack: https + tls: + sni: www.example.com + transactions: + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, www.example.com] + - [uuid, hsts-200] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Connection, close] + - [Content-Length, 0] + + proxy-response: + status: 200 + headers: + fields: + - [Strict-Transport-Security, { value: max-age=300, as: equal }] + + - protocol: + stack: https + tls: + sni: bad_host + transactions: + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, bad_host] + - [uuid, hsts-404] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 404 + headers: + fields: + - [Strict-Transport-Security, { as: absent }] + - [Content-Type, { value: text/html, as: contains }] + content: + verify: + value: "Not Found on Accelerator" + as: contains diff --git a/tests/gold_tests/headers/replays/invalid_range_request.replay.yaml b/tests/gold_tests/headers/replays/invalid_range_request.replay.yaml index 5f6702bf0d1..2e68f97296b 100644 --- a/tests/gold_tests/headers/replays/invalid_range_request.replay.yaml +++ b/tests/gold_tests/headers/replays/invalid_range_request.replay.yaml @@ -17,6 +17,32 @@ meta: version: "1.0" +autest: + description: 'Verify invalid Range requests are forwarded to origin and return 416' + + server: + name: 'invalid-range-server' + + client: + name: 'invalid-range-client' + + ats: + name: 'ts' + process_config: + enable_cache: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http' + proxy.config.http.cache.http: 1 + proxy.config.http.cache.range.write: 1 + proxy.config.http.cache.required_headers: 0 + proxy.config.http.insert_age_in_response: 0 + + remap_config: + - from: "/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + sessions: # Populate cache entry - transactions: @@ -55,3 +81,10 @@ sessions: headers: fields: - [X-ResponseHeader, failed_response] + + proxy-response: + status: 416 + reason: Range Not Satisfiable + headers: + fields: + - [X-ResponseHeader, { value: failed_response, as: equal }] diff --git a/tests/gold_tests/headers/replays/normalized_ae_varied_transactions.replay.yaml b/tests/gold_tests/headers/replays/normalized_ae_varied_transactions.replay.yaml index f69edb718fb..bb0f010ae83 100644 --- a/tests/gold_tests/headers/replays/normalized_ae_varied_transactions.replay.yaml +++ b/tests/gold_tests/headers/replays/normalized_ae_varied_transactions.replay.yaml @@ -26,6 +26,71 @@ meta: fields: - [ Content-Length, 0 ] +autest: + description: 'Verify cache matching with normalized Accept-Encoding and Vary' + + server: + name: 'normalized-ae-server' + + client: + name: 'normalized-ae-client' + + ats: + name: 'ts' + process_config: + enable_cache: true + + plugin_config: + - 'xdebug.so --enable=x-cache' + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http' + proxy.config.http.response_via_str: 3 + proxy.config.cache.limits.http.max_alts: 6 + proxy.config.http.cache.ignore_accept_mismatch: 2 + proxy.config.http.cache.ignore_accept_language_mismatch: 2 + proxy.config.http.cache.ignore_accept_encoding_mismatch: 2 + proxy.config.http.cache.ignore_accept_charset_mismatch: 2 + + remap_config: + - from: "http://www.ae-0.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: "conf_remap.so" + args: + - "proxy.config.http.normalize_ae=0" + - from: "http://www.ae-1.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: "conf_remap.so" + args: + - "proxy.config.http.normalize_ae=1" + - from: "http://www.ae-2.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: "conf_remap.so" + args: + - "proxy.config.http.normalize_ae=2" + - from: "http://www.ae-3.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: "conf_remap.so" + args: + - "proxy.config.http.normalize_ae=3" + - from: "http://www.ae-4.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: "conf_remap.so" + args: + - "proxy.config.http.normalize_ae=4" + - from: "http://www.ae-5.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: "conf_remap.so" + args: + - "proxy.config.http.normalize_ae=5" + sessions: - transactions: diff --git a/tests/gold_tests/headers/replays/range.replay.yaml b/tests/gold_tests/headers/replays/range.replay.yaml new file mode 100644 index 00000000000..0355f8c6366 --- /dev/null +++ b/tests/gold_tests/headers/replays/range.replay.yaml @@ -0,0 +1,332 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + + blocks: + - full_object_response: &full_object_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [Server, microserver] + - [Connection, close] + - [Cache-Control, max-age=300] + - [Last-Modified, "Thu, 10 Feb 2022 00:00:00 GMT"] + - [ETag, range] + - [Content-Length, 11] + content: + encoding: plain + data: "0123456789\n" + + - short_object_response: &short_object_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [Server, microserver] + - [Connection, close] + - [Cache-Control, max-age=1] + - [Last-Modified, "Thu, 10 Feb 2022 00:00:00 GMT"] + - [ETag, range] + - [Content-Length, 11] + content: + encoding: plain + data: "0123456789\n" + + - origin_response_should_not_be_used: &origin_response_should_not_be_used + server-response: + status: 500 + reason: "Internal Server Error" + headers: + fields: + - [Content-Length, 16] + content: + encoding: plain + data: "origin-should-not" + +autest: + description: 'Verify range request handling for hits, misses, and stale cache revalidation' + + server: + name: 'range-server' + + client: + name: 'range-client' + + ats: + name: 'ts-range' + process_config: + enable_cache: true + + records_config: + proxy.config.http.cache.http: 1 + proxy.config.http.cache.range.write: 1 + proxy.config.http.response_via_str: 3 + proxy.config.http.wait_for_cache: 1 + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http' + + remap_config: + - from: "http://127.0.0.1/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + +sessions: + - transactions: + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=1-5] + - [If-Range, '"should-not-match"'] + - [uuid, miss-ifrange] + + <<: *full_object_response + + proxy-response: + status: 200 + headers: + fields: + - [ETag, { value: range, as: equal }] + - [Content-Length, { value: 11, as: equal }] + content: + verify: + value: "0123456789\n" + as: equal + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=1-5] + - [uuid, hit-range] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 206 + headers: + fields: + - [ETag, { value: range, as: equal }] + - [Content-Length, { value: 5, as: equal }] + - [Content-Range, { value: "bytes 1-5/11", as: equal }] + content: + verify: + value: "12345" + as: equal + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=1-5] + - [If-Range, '"range"'] + - [uuid, hit-ifrange-etag] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 206 + headers: + fields: + - [Content-Range, { value: "bytes 1-5/11", as: equal }] + content: + verify: + value: "12345" + as: equal + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=1-5] + - [If-Range, "Thu, 10 Feb 2022 00:00:00 GMT"] + - [uuid, hit-ifrange-date] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 206 + headers: + fields: + - [Content-Range, { value: "bytes 1-5/11", as: equal }] + content: + verify: + value: "12345" + as: equal + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=1-5] + - [If-Range, "Thu, 10 Feb 2022 01:00:00 GMT"] + - [uuid, miss-ifrange-newer-date] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 200 + content: + verify: + value: "0123456789\n" + as: equal + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=100-105] + - [uuid, invalid-range] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 416 + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=1-5] + - [If-Range, '"should-not-match"'] + - [uuid, hit-ifrange-bad-etag] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 200 + content: + verify: + value: "0123456789\n" + as: equal + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=1-5] + - [If-Range, 'W/"range"'] + - [uuid, hit-ifrange-weak-etag] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 200 + content: + verify: + value: "0123456789\n" + as: equal + + - client-request: + method: GET + version: "1.1" + url: / + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=1-5] + - [If-Range, "Wed, 09 Feb 2022 23:00:00 GMT"] + - [uuid, hit-ifrange-older-date] + + <<: *origin_response_should_not_be_used + + proxy-response: + status: 200 + content: + verify: + value: "0123456789\n" + as: equal + + - client-request: + method: GET + version: "1.1" + url: /short + headers: + fields: + - [Host, 127.0.0.1] + - [uuid, fill-short] + + <<: *short_object_response + + proxy-response: + status: 200 + headers: + fields: + - [Cache-Control, { value: max-age=1, as: equal }] + content: + verify: + value: "0123456789\n" + as: equal + + - client-request: + delay: 2s + method: GET + version: "1.1" + url: /short + headers: + fields: + - [Host, 127.0.0.1] + - [Range, bytes=1-5] + - [If-Range, '"range"'] + - [uuid, revalidate-short] + + server-response: + status: 304 + reason: Not Modified + headers: + fields: + - [Cache-Control, max-age=1] + - [ETag, range] + + proxy-response: + status: 206 + headers: + fields: + - [Cache-Control, { value: max-age=1, as: equal }] + - [ETag, { value: range, as: equal }] + - [Content-Range, { value: "bytes 1-5/11", as: equal }] + content: + verify: + value: "12345" + as: equal diff --git a/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_0.conf b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_0.conf new file mode 100644 index 00000000000..4c9074d1556 --- /dev/null +++ b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_0.conf @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set-redirect 0 "%{CLIENT-URL}" diff --git a/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_301.conf b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_301.conf new file mode 100644 index 00000000000..1048164884f --- /dev/null +++ b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_301.conf @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set-redirect 301 "%{CLIENT-URL}" diff --git a/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_302.conf b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_302.conf new file mode 100644 index 00000000000..fb64f255bf8 --- /dev/null +++ b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_302.conf @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set-redirect 302 "%{CLIENT-URL}" diff --git a/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_307.conf b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_307.conf new file mode 100644 index 00000000000..cfdde954ca9 --- /dev/null +++ b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_307.conf @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set-redirect 307 "%{CLIENT-URL}" diff --git a/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_308.conf b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_308.conf new file mode 100644 index 00000000000..106a4ac7314 --- /dev/null +++ b/tests/gold_tests/headers/rewrite_rules/header_rewrite_rules_308.conf @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set-redirect 308 "%{CLIENT-URL}" diff --git a/tests/gold_tests/ip_allow/ip_allow_subjects.test.py b/tests/gold_tests/ip_allow/ip_allow_subjects.test.py new file mode 100644 index 00000000000..81790e5092e --- /dev/null +++ b/tests/gold_tests/ip_allow/ip_allow_subjects.test.py @@ -0,0 +1,30 @@ +''' +Verify that exceeding MAX_SUBJECTS for proxy.config.acl.subjects logs an error and does not crash. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify that exceeding MAX_SUBJECTS for proxy.config.acl.subjects logs an error and does not crash. +''' + +# Scenario 1: Configuring 4 ACL subjects (more than MAX_SUBJECTS=3) should log +# an error but not crash. Request should still succeed with 200 OK. +Test.ATSReplayTest(replay_file="replay/ip_allow_subjects_overflow.replay.yaml") + +# Scenario 2: Configuring exactly 3 ACL subjects (equal to MAX_SUBJECTS) should +# work without error. Request should succeed with 200 OK. +Test.ATSReplayTest(replay_file="replay/ip_allow_subjects_valid.replay.yaml") diff --git a/tests/gold_tests/ip_allow/replay/ip_allow_subjects_overflow.replay.yaml b/tests/gold_tests/ip_allow/replay/ip_allow_subjects_overflow.replay.yaml new file mode 100644 index 00000000000..1d66dce38a8 --- /dev/null +++ b/tests/gold_tests/ip_allow/replay/ip_allow_subjects_overflow.replay.yaml @@ -0,0 +1,72 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +autest: + description: 'Verify that exceeding MAX_SUBJECTS (4 subjects) logs an error but does not crash' + + server: + name: 'server-overflow' + + client: + name: 'client-overflow' + + ats: + name: 'ts-overflow' + + process_config: + enable_cache: false + disable_log_checks: true + + records_config: + proxy.config.acl.subjects: 'PEER,PROXY,PLUGIN,PEER' + + remap_config: + - from: "http://www.example.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + + log_validation: + diags_log: + contains: + - expression: "Too many ACL subjects were provided" + description: "Should log error when more than MAX_SUBJECTS are configured" + +sessions: +- transactions: + + - client-request: + method: GET + url: /get + version: '1.1' + headers: + fields: + - [Host, www.example.com] + - [uuid, overflow-subjects-request] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Content-Length, "3"] + - [Connection, close] + content: + data: "xxx" + + proxy-response: + status: 200 diff --git a/tests/gold_tests/ip_allow/replay/ip_allow_subjects_valid.replay.yaml b/tests/gold_tests/ip_allow/replay/ip_allow_subjects_valid.replay.yaml new file mode 100644 index 00000000000..6e9351aed1f --- /dev/null +++ b/tests/gold_tests/ip_allow/replay/ip_allow_subjects_valid.replay.yaml @@ -0,0 +1,71 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +autest: + description: 'Verify that exactly MAX_SUBJECTS (3 subjects) works without error' + + server: + name: 'server-valid' + + client: + name: 'client-valid' + + ats: + name: 'ts-valid' + + process_config: + enable_cache: false + + records_config: + proxy.config.acl.subjects: 'PEER,PROXY,PLUGIN' + + remap_config: + - from: "http://www.example.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + + log_validation: + diags_log: + excludes: + - expression: "Too many ACL subjects were provided" + description: "Should not log error when exactly MAX_SUBJECTS are configured" + +sessions: +- transactions: + + - client-request: + method: GET + url: /get + version: '1.1' + headers: + fields: + - [Host, www.example.com] + - [uuid, valid-subjects-request] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Content-Length, "3"] + - [Connection, close] + content: + data: "xxx" + + proxy-response: + status: 200 diff --git a/tests/gold_tests/logging/custom-log.test.py b/tests/gold_tests/logging/custom-log.test.py index 58c036f9b39..754557f745b 100644 --- a/tests/gold_tests/logging/custom-log.test.py +++ b/tests/gold_tests/logging/custom-log.test.py @@ -83,9 +83,10 @@ tr.MakeCurlCommand('"http://127.123.32.243:{0}" --verbose'.format(ts.Variables.port), ts=ts) tr.Processes.Default.ReturnCode = 0 -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. -test_run = Test.AddTestRun() -test_run.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + - os.path.join(ts.Variables.LOGDIR, 'test_log_field.log')) -test_run.Processes.Default.ReturnCode = 0 +# Wait for all expected log lines to be written. +Test.AddAwaitFileContainsTestRun( + 'Await custom log lines.', + os.path.join(ts.Variables.LOGDIR, 'test_log_field.log'), + r'^127\.', + 8, +) diff --git a/tests/gold_tests/logging/log-field-json.test.py b/tests/gold_tests/logging/log-field-json.test.py index 5853f5b80be..9d8f96caa65 100644 --- a/tests/gold_tests/logging/log-field-json.test.py +++ b/tests/gold_tests/logging/log-field-json.test.py @@ -111,9 +111,10 @@ '--verbose --header "Host: test-2" --header "Foo: ab\x80d/ef" http://localhost:{0}/test-4'.format(ts.Variables.port), ts=ts) tr.Processes.Default.ReturnCode = 0 -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. -test_run = Test.AddTestRun() -test_run.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + - os.path.join(ts.Variables.LOGDIR, 'field-json-test.log')) -test_run.Processes.Default.ReturnCode = 0 +# Wait for all expected log lines to be written. +Test.AddAwaitFileContainsTestRun( + 'Await field-json log lines.', + os.path.join(ts.Variables.LOGDIR, 'field-json-test.log'), + r'^\{"foo":', + 4, +) diff --git a/tests/gold_tests/logging/log-field.test.py b/tests/gold_tests/logging/log-field.test.py index 55e5003a6e1..4268ec5cc73 100644 --- a/tests/gold_tests/logging/log-field.test.py +++ b/tests/gold_tests/logging/log-field.test.py @@ -112,8 +112,10 @@ tr.MakeCurlCommand('--verbose --ipv4 --header "Host: test-3" http://127.0.0.1:{0}/test-3'.format(ts.Variables.port), ts=ts) tr.Processes.Default.ReturnCode = 0 -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. -test_run = Test.AddTestRun() -test_run.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + os.path.join(ts.Variables.LOGDIR, 'field-test.log')) -test_run.Processes.Default.ReturnCode = 0 +# Wait for all expected log lines to be written. +Test.AddAwaitFileContainsTestRun( + 'Await field-test log lines.', + os.path.join(ts.Variables.LOGDIR, 'field-test.log'), + r'^Transfer-Encoding:', + 3, +) diff --git a/tests/gold_tests/logging/log-filenames.test.py b/tests/gold_tests/logging/log-filenames.test.py index 5ec6d6d685d..40781a74100 100644 --- a/tests/gold_tests/logging/log-filenames.test.py +++ b/tests/gold_tests/logging/log-filenames.test.py @@ -100,16 +100,17 @@ def __configure_traffic_server(self, log_data): def __configure_await_TestRun(self, log_path): ''' Configure a TestRun that awaits upon the provided log_path to - exist. + contain the sentinel log entry. Args: log_path (str): The log file upon which we will wait. ''' description = self.__description - tr = Test.AddTestRun(f'Awaiting log files to be written for: {description}') - condwait_path = os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') - tr.Processes.Default.Command = f'{condwait_path} 60 1 -f {log_path}' - tr.Processes.Default.ReturnCode = 0 + Test.AddAwaitFileContainsTestRun( + f'Awaiting log files to be written for: {description}', + log_path, + r'^http://127\.0\.0\.1:\d+/: 502$', + ) def __configure_traffic_TestRun(self, description): ''' Configure a TestRun to run the expected transactions. @@ -160,7 +161,7 @@ def set_log_expectations(self): diags_path = self.ts.Disk.diags_log.AbsPath self.ts.Disk.diags_log.Content += Testers.ContainsExpression( - "Traffic Server is fully initialized", f"{diags_path} should contain traffic_server diag messages") + "logging.yaml finished loading", f"{diags_path} should contain traffic_server diag messages") error_log_path = self.ts.Disk.error_log.AbsPath self.ts.Disk.error_log.Content += Testers.ContainsExpression( diff --git a/tests/gold_tests/logging/log-filter.test.py b/tests/gold_tests/logging/log-filter.test.py index 9de7d74146a..e9dd9483966 100644 --- a/tests/gold_tests/logging/log-filter.test.py +++ b/tests/gold_tests/logging/log-filter.test.py @@ -84,11 +84,13 @@ tr.Processes.Default.StartBefore(ts) tr.AddVerifierClientProcess("client-1", replay_file, http_ports=[ts.Variables.port]) -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. -test_run = Test.AddTestRun() -test_run.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + os.path.join(ts.Variables.LOGDIR, 'filter-test.log')) -test_run.Processes.Default.ReturnCode = 0 +# Wait for all expected log lines to be written. +Test.AddAwaitFileContainsTestRun( + 'Await filtered log lines.', + os.path.join(ts.Variables.LOGDIR, 'filter-test.log'), + r'^http://example\.com/test-', + 5, +) # We already waited for the above, so we don't have to wait for this one. test_run = Test.AddTestRun() diff --git a/tests/gold_tests/logging/log-milestone-fields.test.py b/tests/gold_tests/logging/log-milestone-fields.test.py new file mode 100644 index 00000000000..72f5f8617b0 --- /dev/null +++ b/tests/gold_tests/logging/log-milestone-fields.test.py @@ -0,0 +1,176 @@ +''' +Verify that msdms milestone difference fields produce valid output for both +cache-miss and cache-hit transactions. This exercises the Phase 1 timing +fields proposed for the squid.log local_disk format. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +Test.Summary = 'Verify msdms milestone logging fields for cache miss and cache hit paths' +Test.ContinueOnFail = True + + +class MilestoneFieldsTest: + """ + Sends two requests for the same cacheable URL: the first is a cache miss + that populates the cache, the second is a cache hit served from RAM cache. + A custom log format records all Phase 1 milestone timing fields plus the + cache result code. A validation script then parses the log and checks: + + - Every expected key=value pair is present on each line + - All values are integers, with "-" for unset milestones + - Cache miss line: ms > 0, origin-phase fields present + - Cache hit line: hit_proc >= 0 and hit_xfer >= 0 + - No epoch-length garbage values (> 1_000_000_000) + """ + + # All Phase 1 msdms fields plus ms, cache result code, and cache key hash. + LOG_FORMAT = ( + 'crc=% ckh=% ms=%' + ' c_ttfb=%<{TS_MILESTONE_UA_BEGIN_WRITE-TS_MILESTONE_SM_START}msdms>' + ' c_tls=%<{TS_MILESTONE_TLS_HANDSHAKE_END-TS_MILESTONE_TLS_HANDSHAKE_START}msdms>' + ' c_hdr=%<{TS_MILESTONE_UA_READ_HEADER_DONE-TS_MILESTONE_SM_START}msdms>' + ' c_proc=%<{TS_MILESTONE_CACHE_OPEN_READ_BEGIN-TS_MILESTONE_UA_READ_HEADER_DONE}msdms>' + ' cache=%<{TS_MILESTONE_CACHE_OPEN_READ_END-TS_MILESTONE_CACHE_OPEN_READ_BEGIN}msdms>' + ' dns=%<{TS_MILESTONE_SERVER_FIRST_CONNECT-TS_MILESTONE_CACHE_OPEN_READ_END}msdms>' + ' o_tcp=%<{TS_MILESTONE_SERVER_CONNECT_END-TS_MILESTONE_SERVER_FIRST_CONNECT}msdms>' + ' o_wait=%<{TS_MILESTONE_SERVER_FIRST_READ-TS_MILESTONE_SERVER_CONNECT_END}msdms>' + ' o_hdr=%<{TS_MILESTONE_SERVER_READ_HEADER_DONE-TS_MILESTONE_SERVER_FIRST_READ}msdms>' + ' o_proc=%<{TS_MILESTONE_UA_BEGIN_WRITE-TS_MILESTONE_SERVER_READ_HEADER_DONE}msdms>' + ' o_body=%<{TS_MILESTONE_SERVER_CLOSE-TS_MILESTONE_UA_BEGIN_WRITE}msdms>' + ' c_xfer=%<{TS_MILESTONE_SM_FINISH-TS_MILESTONE_SERVER_CLOSE}msdms>' + ' hit_proc=%<{TS_MILESTONE_UA_BEGIN_WRITE-TS_MILESTONE_CACHE_OPEN_READ_END}msdms>' + ' hit_xfer=%<{TS_MILESTONE_SM_FINISH-TS_MILESTONE_UA_BEGIN_WRITE}msdms>') + + def __init__(self): + self._server = Test.MakeOriginServer("server") + self._nameserver = Test.MakeDNServer("dns", default='127.0.0.1') + self._setupOriginServer() + self._setupTS() + + def _setupOriginServer(self): + self._server.addResponse( + "sessionlog.json", + { + 'timestamp': 100, + "headers": "GET /cacheable HTTP/1.1\r\nHost: example.com\r\n\r\n", + "body": "", + }, + { + 'timestamp': 100, + "headers": + ( + "HTTP/1.1 200 OK\r\n" + "Content-Type: text/plain\r\n" + "Cache-Control: max-age=300\r\n" + "Connection: close\r\n" + "\r\n"), + "body": "This is a cacheable response body for milestone testing.", + }, + ) + + def _setupTS(self): + self._ts = Test.MakeATSProcess("ts", enable_cache=True) + + self._ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'http|log', + 'proxy.config.dns.nameservers': f'127.0.0.1:{self._nameserver.Variables.Port}', + 'proxy.config.dns.resolv_conf': 'NULL', + 'proxy.config.http.cache.http': 1, + 'proxy.config.log.max_secs_per_buffer': 1, + }) + + self._ts.Disk.remap_config.AddLine(f'map / http://127.0.0.1:{self._server.Variables.Port}/') + + self._ts.Disk.logging_yaml.AddLines( + f''' +logging: + formats: + - name: milestone_test + format: '{self.LOG_FORMAT}' + logs: + - filename: milestone_fields + format: milestone_test + mode: ascii +'''.split("\n")) + + @property + def _log_path(self) -> str: + return os.path.join(self._ts.Variables.LOGDIR, 'milestone_fields.log') + + @property + def _validate_script(self) -> str: + return os.path.join(Test.TestDirectory, 'verify_milestone_fields.py') + + def run(self): + self._sendCacheMiss() + self._waitForCacheIO() + self._sendCacheHit() + self._waitForLog() + self._validateLog() + + def _sendCacheMiss(self): + tr = Test.AddTestRun('Cache miss request') + tr.Processes.Default.StartBefore(self._server) + tr.Processes.Default.StartBefore(self._nameserver) + tr.Processes.Default.StartBefore(self._ts) + tr.MakeCurlCommand( + f'--verbose --header "Host: example.com" ' + f'http://127.0.0.1:{self._ts.Variables.port}/cacheable', ts=self._ts) + tr.Processes.Default.ReturnCode = 0 + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + def _waitForCacheIO(self): + tr = Test.AddTestRun('Wait for cache write to complete') + tr.Processes.Default.Command = 'sleep 1' + tr.Processes.Default.ReturnCode = 0 + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + def _sendCacheHit(self): + tr = Test.AddTestRun('Cache hit request') + tr.MakeCurlCommand( + f'--verbose --header "Host: example.com" ' + f'http://127.0.0.1:{self._ts.Variables.port}/cacheable', ts=self._ts) + tr.Processes.Default.ReturnCode = 0 + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + def _waitForLog(self): + tr = Test.AddAwaitFileContainsTestRun( + 'Wait for milestone log lines to be written', + self._log_path, + r'^crc=.* hit_xfer=', + desired_count=2, + ) + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + def _validateLog(self): + tr = Test.AddTestRun('Validate milestone fields in log') + tr.Processes.Default.Command = f'python3 {self._validate_script} {self._log_path}' + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.TimeOut = 10 + tr.Processes.Default.Streams.stdout += Testers.ContainsExpression('PASS', 'Validation script should report PASS') + tr.Processes.Default.Streams.stdout += Testers.ExcludesExpression('FAIL', 'Validation script should not report FAIL') + + +MilestoneFieldsTest().run() diff --git a/tests/gold_tests/logging/log-mstsms.test.py b/tests/gold_tests/logging/log-mstsms.test.py new file mode 100644 index 00000000000..cd3e71b515c --- /dev/null +++ b/tests/gold_tests/logging/log-mstsms.test.py @@ -0,0 +1,119 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +Test.Summary = ''' +Test log fields. +''' + +ts = Test.MakeATSProcess("ts", enable_cache=True) +server = Test.MakeOriginServer("server") + +request_header = {'timestamp': 100, "headers": "GET /test-1 HTTP/1.1\r\nHost: test-1\r\n\r\n", "body": ""} +response_header = { + 'timestamp': 100, + "headers": + "HTTP/1.1 200 OK\r\nTest: 1\r\nContent-Type: application/json\r\nConnection: close\r\nContent-Type: application/json\r\nTransfer-Encoding: chunked\r\n\r\n", + "body": "Test 1" +} +server.addResponse("sessionlog.json", request_header, response_header) +server.addResponse( + "sessionlog.json", { + 'timestamp': 101, + "headers": "GET /test-2 HTTP/1.1\r\nHost: test-2\r\n\r\n", + "body": "" + }, { + 'timestamp': 101, + "headers": + "HTTP/1.1 200 OK\r\nTest: 2\r\nContent-Type: application/jason\r\nConnection: close\r\nContent-Type: application/json\r\n\r\n", + "body": "Test 2" + }) +server.addResponse( + "sessionlog.json", { + 'timestamp': 102, + "headers": "GET /test-3 HTTP/1.1\r\nHost: test-3\r\n\r\n", + "body": "" + }, { + 'timestamp': 102, + "headers": "HTTP/1.1 200 OK\r\nTest: 3\r\nConnection: close\r\nContent-Type: application/json\r\n\r\n", + "body": "Test 3" + }) + +nameserver = Test.MakeDNServer("dns", default='127.0.0.1') + +ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'http|log', + # 'proxy.config.net.connections_throttle': 100, + 'proxy.config.dns.nameservers': f"127.0.0.1:{nameserver.Variables.Port}", + 'proxy.config.dns.resolv_conf': 'NULL', + }) +# setup some config file for this server +ts.Disk.remap_config.AddLine('map / http://localhost:{}/'.format(server.Variables.Port)) + +ts.Disk.logging_yaml.AddLines( + ''' +logging: + formats: + - name: custom + format: 'mstsms:%' + logs: + - filename: field-mstsms + format: custom +'''.split("\n")) + +# first test is a miss for default +tr = Test.AddTestRun() +# Wait for the micro server +tr.Processes.Default.StartBefore(server) +tr.Processes.Default.StartBefore(nameserver) +# Delay on readiness of our ssl ports +tr.Processes.Default.StartBefore(Test.Processes.ts) + +tr.MakeCurlCommand('--verbose --header "Host: test-1" http://localhost:{0}/test-1'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 + +tr = Test.AddTestRun() +tr.MakeCurlCommand('--verbose --header "Host: test-2" http://localhost:{0}/test-2'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 + +tr = Test.AddTestRun() +tr.MakeCurlCommand('--verbose --header "Host: test-3" http://localhost:{0}/test-3'.format(ts.Variables.port), ts=ts) +tr.Processes.Default.ReturnCode = 0 + + +# check comma count and ensure last character is a digit +def check_lines(path): + with open(path, 'r') as file: + for line_num, line in enumerate(file, 1): + line = line.rstrip('\n') + comma_count = line.count(',') + if comma_count != 19: + return False, "Check comma count", f"Expected 19 commas, got {comma_count}" + if not line[-1].isdigit(): + return False, "Check last char", f"Expected last character to be a digit got '{line[-1]}'" + return True, "", "" + + +logpath = os.path.join(ts.Variables.LOGDIR, 'field-mstsms.log') + +# Wait for all expected log lines to be written. +tr = Test.AddAwaitFileContainsTestRun('Await mstsms log lines.', logpath, r'^mstsms:', 3) +tr.Streams.All.Content = Testers.Lambda(lambda info, tester: check_lines(logpath)) diff --git a/tests/gold_tests/logging/log_retention.test.py b/tests/gold_tests/logging/log_retention.test.py index 1ac46931bd5..8b21343d0e7 100644 --- a/tests/gold_tests/logging/log_retention.test.py +++ b/tests/gold_tests/logging/log_retention.test.py @@ -488,7 +488,7 @@ def get_command_to_rotate_thrice(self): tr.StillRunningAfter = test.server tr = Test.AddTestRun("Get the log to rotate.") -test.tr.MakeCurlCommandMulti(test.get_command_to_rotate_once(), ts=test.ts) +tr.MakeCurlCommandMulti(test.get_command_to_rotate_once(), ts=test.ts) tr.Processes.Default.ReturnCode = 0 tr.StillRunningAfter = test.ts tr.StillRunningAfter = test.server diff --git a/tests/gold_tests/logging/new_log_flds.test.py b/tests/gold_tests/logging/new_log_flds.test.py index 94be8baec7d..91e91d1fa73 100644 --- a/tests/gold_tests/logging/new_log_flds.test.py +++ b/tests/gold_tests/logging/new_log_flds.test.py @@ -96,13 +96,13 @@ ts=ts) tr.Processes.Default.ReturnCode = 0 -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. +# Wait for the final log line to be written. # -test_run = Test.AddTestRun() -test_run.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + - os.path.join(ts.Variables.LOGDIR, 'test_new_log_flds.log')) -test_run.Processes.Default.ReturnCode = 0 +Test.AddAwaitFileContainsTestRun( + 'Await new log field output.', + os.path.join(ts.Variables.LOGDIR, 'test_new_log_flds.log'), + r'reallyreallyreallyreallylong\.com$', +) # Validate generated log. # diff --git a/tests/gold_tests/logging/pqsi-pqsp.test.py b/tests/gold_tests/logging/pqsi-pqsp.test.py index 4bbc504c8db..6dc33ea68ea 100644 --- a/tests/gold_tests/logging/pqsi-pqsp.test.py +++ b/tests/gold_tests/logging/pqsi-pqsp.test.py @@ -78,10 +78,8 @@ log_filespec = os.path.join(ts.Variables.LOGDIR, 'field-test.log') -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. -tr = Test.AddTestRun() -tr.Processes.Default.Command = (os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + log_filespec) -tr.Processes.Default.ReturnCode = 0 +# Wait for the cache-hit line to be written. +Test.AddAwaitFileContainsTestRun('Await pqsi/pqsp cache-hit line.', log_filespec, r'^0 0$') tr = Test.AddTestRun() tr.Processes.Default.Command = "sed '1s/^127.0.0.1 [1-6][0-9]*$$/abc/' < " + log_filespec diff --git a/tests/gold_tests/logging/sigusr2.test.py b/tests/gold_tests/logging/sigusr2.test.py index 397119ab2b5..67f85c5a27f 100644 --- a/tests/gold_tests/logging/sigusr2.test.py +++ b/tests/gold_tests/logging/sigusr2.test.py @@ -18,9 +18,12 @@ # limitations under the License. import os +import shlex import sys TS_PID_SCRIPT = 'ts_process_handler.py' +ROTATE_DIAGS_SCRIPT = 'sigusr2_rotate_diags.sh' +ROTATE_CUSTOM_LOG_SCRIPT = 'sigusr2_rotate_custom_log.sh' class Sigusr2Test: @@ -29,17 +32,29 @@ class Sigusr2Test: """ __ts_counter = 1 - __server = None - - def __init__(self): - self.server = self.__configure_server() - self.ts = self.__configure_traffic_server() - - def __configure_traffic_server(self): - self._ts_name = "sigusr2_ts{}".format(Sigusr2Test.__ts_counter) - Sigusr2Test.__ts_counter += 1 - self.ts = Test.MakeATSProcess(self._ts_name) - self.ts.Disk.records_config.update( + __server_counter = 1 + + @classmethod + def _next_ts_name(cls): + ts_name = f"sigusr2_ts{cls.__ts_counter}" + cls.__ts_counter += 1 + return ts_name + + @classmethod + def _next_server_name(cls): + server_name = f"sigusr2_server{cls.__server_counter}" + cls.__server_counter += 1 + return server_name + + @staticmethod + def _make_script_command(script_name, *args): + quoted_args = ' '.join(shlex.quote(str(arg)) for arg in args) + return f"bash ./{script_name} {quoted_args}" + + def _configure_traffic_server(self, tr): + ts_name = self._next_ts_name() + ts = tr.MakeATSProcess(ts_name) + ts.Disk.records_config.update( { 'proxy.config.http.wait_for_cache': 1, 'proxy.config.diags.debug.enabled': 1, @@ -50,18 +65,54 @@ def __configure_traffic_server(self): 'proxy.config.log.rolling_enabled': 0, 'proxy.config.log.auto_delete_rolled_files': 0, }) + return ts, ts_name - self.diags_log = self.ts.Disk.diags_log.AbsPath + def _configure_server(self): + server = Test.MakeOriginServer(self._next_server_name()) - # Add content handles for the rotated logs. - self.rotated_diags_log = self.diags_log + "_old" - self.ts.Disk.File(self.rotated_diags_log, id="diags_log_old") + for path in ['/first', '/second', '/third']: + request_header = { + 'headers': f'GET {path} HTTP/1.1\r\nHost: does.not.matter\r\n\r\n', + 'timestamp': '1469733493.993', + 'body': '' + } + response_header = { + 'headers': 'HTTP/1.1 200 OK\r\n' + 'Connection: close\r\n' + 'Cache-control: max-age=85000\r\n\r\n', + 'timestamp': '1469733493.993', + 'body': 'xxx' + } + server.addResponse('sessionlog.json', request_header, response_header) + + return server - self.log_dir = os.path.dirname(self.diags_log) + def add_system_log_test(self): + tr = Test.AddTestRun('Verify system logs can be rotated') + ts, ts_name = self._configure_traffic_server(tr) - self.ts.Disk.remap_config.AddLine( - 'map http://127.0.0.1:{0} http://127.0.0.1:{1}'.format(self.ts.Variables.port, self.server.Variables.Port)) - self.ts.Disk.logging_yaml.AddLine( + rotated_diags_log = f'{ts.Disk.diags_log.AbsPath}_old' + ts.Disk.File(rotated_diags_log, id='diags_log_old') + + tr.Processes.Default.Command = self._make_script_command( + ROTATE_DIAGS_SCRIPT, sys.executable, f'./{TS_PID_SCRIPT}', ts_name, ts.Disk.diags_log.AbsPath, rotated_diags_log) + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.StartBefore(ts) + + ts.Disk.diags_log.Content += Testers.ExcludesExpression( + 'traffic server running', 'The new diags.log should not reference the running traffic server') + ts.Disk.diags_log.Content += Testers.ContainsExpression( + 'Reseated diags.log', 'The new diags.log should indicate the newly opened diags.log') + ts.Disk.diags_log_old.Content += Testers.ContainsExpression( + 'traffic server running', 'The rotated diags.log should keep the original startup message') + + def add_configured_log_test(self): + tr = Test.AddTestRun('Verify yaml.log logs can be rotated') + ts, ts_name = self._configure_traffic_server(tr) + server = self._configure_server() + + ts.Disk.remap_config.AddLine(f'map http://127.0.0.1:{ts.Variables.port} http://127.0.0.1:{server.Variables.Port}') + ts.Disk.logging_yaml.AddLine( ''' logging: formats: @@ -71,155 +122,43 @@ def __configure_traffic_server(self): - filename: test_rotation format: has_path ''') - self.configured_log = os.path.join(self.log_dir, "test_rotation.log") - self.ts.Disk.File(self.configured_log, id="configured_log") - - self.rotated_configured_log = self.configured_log + "_old" - self.ts.Disk.File(self.rotated_configured_log, id="configured_log_old") - self.ts.StartBefore(self.server) - return self.ts - - def __configure_server(self): - if Sigusr2Test.__server: - return Sigusr2Test.__server - server = Test.MakeOriginServer("server") - Sigusr2Test.__server = server - for path in ['/first', '/second', '/third']: - request_header = { - "headers": "GET {} HTTP/1.1\r\n" - "Host: does.not.matter\r\n\r\n".format(path), - "timestamp": "1469733493.993", - "body": "" - } - response_header = { - "headers": "HTTP/1.1 200 OK\r\n" - "Connection: close\r\n" - "Cache-control: max-age=85000\r\n\r\n", - "timestamp": "1469733493.993", - "body": "xxx" - } - server.addResponse("sessionlog.json", request_header, response_header) - return server - - def get_sigusr2_signal_command(self): - """ - Return the command that will send a USR2 signal to the traffic server - process. - """ - return (f"{sys.executable} {TS_PID_SCRIPT} " - f"--signal SIGUSR2 {self._ts_name}") - - -Test.Summary = ''' -Verify support of external log rotation via SIGUSR2. -''' -Test.Setup.CopyAs(TS_PID_SCRIPT, Test.RunDirectory) + configured_log = os.path.join(ts.Variables.LOGDIR, 'test_rotation.log') + ts.Disk.File(configured_log, id='configured_log') -# -# Test 1: Verify SIGUSR2 behavior for system logs. -# -tr1 = Test.AddTestRun("Verify system logs can be rotated") + rotated_configured_log = f'{configured_log}_old' + ts.Disk.File(rotated_configured_log, id='configured_log_old') -# Configure Server. -diags_test = Sigusr2Test() + ts.StartBefore(server) + tr.Processes.Default.Command = self._make_script_command( + ROTATE_CUSTOM_LOG_SCRIPT, sys.executable, f'./{TS_PID_SCRIPT}', ts_name, ts.Variables.port, configured_log, + rotated_configured_log) + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.StartBefore(ts) -# Configure our rotation processes. -rotate_diags_log = tr1.Processes.Process("rotate_diags_log", "mv {} {}".format(diags_test.diags_log, diags_test.rotated_diags_log)) + ts.Disk.configured_log.Content += Testers.ExcludesExpression( + '/first', 'The new test_rotation.log should not have the first GET retrieval in it.') + ts.Disk.configured_log.Content += Testers.ExcludesExpression( + '/second', 'The new test_rotation.log should not have the second GET retrieval in it.') + ts.Disk.configured_log.Content += Testers.ContainsExpression( + '/third', 'The new test_rotation.log should have the third GET retrieval in it.') -# Configure the signaling of SIGUSR2 to traffic_server. -tr1.Processes.Default.Command = diags_test.get_sigusr2_signal_command() -tr1.Processes.Default.Return = 0 -tr1.Processes.Default.Ready = When.FileExists(diags_test.diags_log) + ts.Disk.configured_log_old.Content += Testers.ContainsExpression( + '/first', 'test_rotation.log_old should have the first GET retrieval in it.') + ts.Disk.configured_log_old.Content += Testers.ContainsExpression( + '/second', 'test_rotation.log_old should have the second GET retrieval in it.') + ts.Disk.configured_log_old.Content += Testers.ExcludesExpression( + '/third', 'test_rotation.log_old should not have the third GET retrieval in it.') -# Configure process order. -tr1.Processes.Default.StartBefore(rotate_diags_log) -rotate_diags_log.StartBefore(diags_test.ts) -tr1.StillRunningAfter = diags_test.ts -tr1.StillRunningAfter = diags_test.server -# diags.log should have been rotated. The old one had the reference to traffic -# server running, this new one shouldn't. But it should indicate that the new -# diags.log was opened. -diags_test.ts.Disk.diags_log.Content += Testers.ExcludesExpression( - "traffic server running", "The new diags.log should not reference the running traffic server") +Test.Summary = ''' +Verify support of external log rotation via SIGUSR2. +''' -diags_test.ts.Disk.diags_log.Content += Testers.ContainsExpression( - "Reseated diags.log", "The new diags.log should indicate the newly opened diags.log") +Test.Setup.Copy(TS_PID_SCRIPT) +Test.Setup.Copy(ROTATE_DIAGS_SCRIPT) +Test.Setup.Copy(ROTATE_CUSTOM_LOG_SCRIPT) -# -# Test 2: Verify SIGUSR2 isn't needed for rotated configured logs. -# -tr2 = Test.AddTestRun("Verify yaml.log logs can be rotated") -configured_test = Sigusr2Test() - -first_curl = tr2.Processes.Process( - "first_curl", 'curl "http://127.0.0.1:{0}/first" --verbose'.format(configured_test.ts.Variables.port)) -# Note that for each of these processes, aside from the final Default one, they -# are all treated like long-running servers to AuTest. Thus the long sleeps -# only allow us to wait until the logs get populated with the desired content, -# the test will not wait the entire time for them to complete. -first_curl_ready = tr2.Processes.Process("first_curl_ready", 'sleep 30') -# In the autest environment, it can take more than 10 seconds for the log file to be created. -first_curl_ready.StartupTimeout = 30 -first_curl_ready.Ready = When.FileContains(configured_test.configured_log, "/first") - -rotate_log = tr2.Processes.Process( - "rotate_log_file", "mv {} {}".format(configured_test.configured_log, configured_test.rotated_configured_log)) - -second_curl = tr2.Processes.Process( - "second_curl", 'curl "http://127.0.0.1:{0}/second" --verbose'.format(configured_test.ts.Variables.port)) - -second_curl_ready = tr2.Processes.Process("second_curl_ready", 'sleep 30') -# In the autest environment, it can take more than 10 seconds for the log file to be created. -second_curl_ready.StartupTimeout = 30 -second_curl_ready.Ready = When.FileContains(configured_test.rotated_configured_log, "/second") - -send_pkill = tr2.Processes.Process("Send_SIGUSR2", configured_test.get_sigusr2_signal_command()) -send_pkill_ready = tr2.Processes.Process("send_pkill_ready", 'sleep 30') -send_pkill_ready.StartupTimeout = 30 -send_pkill_ready.Ready = When.FileExists(configured_test.configured_log) - -third_curl = tr2.Processes.Process( - "third_curl", 'curl "http://127.0.0.1:{0}/third" --verbose'.format(configured_test.ts.Variables.port)) -third_curl_ready = tr2.Processes.Process("third_curl_ready", 'sleep 30') -# In the autest environment, it can take more than 10 seconds for the log file to be created. -third_curl_ready.StartupTimeout = 30 -third_curl_ready.Ready = When.FileContains(configured_test.configured_log, "/third") - -tr2.Processes.Default.Command = "echo waiting for test processes to be done" -tr2.Processes.Default.Return = 0 - -# Configure process order: -# 1. curl /first. The entry should be logged to current log which will be _old. -# 2. mv the log to _old. -# 3. curl /second. The entry should end up in _old log. -# 4. Send a SIGUSR2 to traffic_server. The log should be recreated. -# 5. curl /third. The entry should end up in the new, non-old, log file. -# -tr2.Processes.Default.StartBefore(third_curl_ready) -third_curl_ready.StartBefore(third_curl) -third_curl.StartBefore(send_pkill_ready) -send_pkill_ready.StartBefore(send_pkill) -send_pkill.StartBefore(second_curl_ready) -second_curl_ready.StartBefore(second_curl) -second_curl.StartBefore(rotate_log) -rotate_log.StartBefore(first_curl_ready) -first_curl_ready.StartBefore(first_curl) -first_curl.StartBefore(configured_test.ts) -tr2.StillRunningAfter = configured_test.ts - -# Verify that the logs are in the correct files. -configured_test.ts.Disk.configured_log.Content += Testers.ExcludesExpression( - "/first", "The new test_rotation.log should not have the first GET retrieval in it.") -configured_test.ts.Disk.configured_log.Content += Testers.ExcludesExpression( - "/second", "The new test_rotation.log should not have the second GET retrieval in it.") -configured_test.ts.Disk.configured_log.Content += Testers.ContainsExpression( - "/third", "The new test_rotation.log should have the third GET retrieval in it.") - -configured_test.ts.Disk.configured_log_old.Content += Testers.ContainsExpression( - "/first", "test_rotation.log_old should have the first GET retrieval in it.") -configured_test.ts.Disk.configured_log_old.Content += Testers.ContainsExpression( - "/second", "test_rotation.log_old should have the second GET retrieval in it.") -configured_test.ts.Disk.configured_log_old.Content += Testers.ExcludesExpression( - "/third", "test_rotation.log_old should not have the third GET retrieval in it.") +sigusr2_test = Sigusr2Test() +sigusr2_test.add_system_log_test() +sigusr2_test.add_configured_log_test() diff --git a/tests/gold_tests/logging/sigusr2_rotate_custom_log.sh b/tests/gold_tests/logging/sigusr2_rotate_custom_log.sh new file mode 100644 index 00000000000..80b915f998b --- /dev/null +++ b/tests/gold_tests/logging/sigusr2_rotate_custom_log.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +if [ "$#" -ne 6 ]; then + echo "Usage: $0 " >&2 + exit 2 +fi + +python_bin="$1" +handler="$2" +ts_name="$3" +ats_port="$4" +configured_log="$5" +rotated_configured_log="$6" +base_url="http://127.0.0.1:${ats_port}" + +wait_for_file() { + path="$1" + tries="$2" + attempt=0 + + while [ "$attempt" -lt "$tries" ]; do + if [ -f "$path" ]; then + return 0 + fi + sleep 1 + attempt=$((attempt + 1)) + done + + echo "Timed out waiting for file: $path" >&2 + return 1 +} + +wait_for_contains() { + path="$1" + needle="$2" + tries="$3" + attempt=0 + + while [ "$attempt" -lt "$tries" ]; do + if [ -f "$path" ] && grep -F "$needle" "$path" >/dev/null 2>&1; then + return 0 + fi + sleep 1 + attempt=$((attempt + 1)) + done + + echo "Timed out waiting for '$needle' in $path" >&2 + return 1 +} + +request_path() { + path="$1" + attempt=0 + + while [ "$attempt" -lt 60 ]; do + if curl --fail --silent --show-error --output /dev/null --max-time 5 "${base_url}${path}"; then + return 0 + fi + sleep 1 + attempt=$((attempt + 1)) + done + + echo "Timed out requesting ${base_url}${path}" >&2 + return 1 +} + +rm -f "$configured_log" "$rotated_configured_log" + +request_path "/first" +wait_for_contains "$configured_log" "/first" 60 + +mv "$configured_log" "$rotated_configured_log" + +request_path "/second" +wait_for_contains "$rotated_configured_log" "/second" 60 + +# Send the SIGUSR2 signal to the handler to trigger ATS to rotate the log. +"$python_bin" "$handler" --signal SIGUSR2 "$ts_name" + +wait_for_file "$configured_log" 60 + +request_path "/third" +wait_for_contains "$configured_log" "/third" 60 diff --git a/tests/gold_tests/logging/sigusr2_rotate_diags.sh b/tests/gold_tests/logging/sigusr2_rotate_diags.sh new file mode 100644 index 00000000000..b45bf04ed79 --- /dev/null +++ b/tests/gold_tests/logging/sigusr2_rotate_diags.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +if [ "$#" -ne 5 ]; then + echo "Usage: $0 " >&2 + exit 2 +fi + +python_bin="$1" +handler="$2" +ts_name="$3" +diags_log="$4" +rotated_diags_log="$5" + +wait_for_file() { + path="$1" + tries="$2" + attempt=0 + + while [ "$attempt" -lt "$tries" ]; do + if [ -f "$path" ]; then + return 0 + fi + sleep 1 + attempt=$((attempt + 1)) + done + + echo "Timed out waiting for file: $path" >&2 + return 1 +} + +wait_for_contains() { + path="$1" + needle="$2" + tries="$3" + attempt=0 + + while [ "$attempt" -lt "$tries" ]; do + if [ -f "$path" ] && grep -F "$needle" "$path" >/dev/null 2>&1; then + return 0 + fi + sleep 1 + attempt=$((attempt + 1)) + done + + echo "Timed out waiting for '$needle' in $path" >&2 + return 1 +} + +rm -f "$rotated_diags_log" + +wait_for_file "$diags_log" 60 +wait_for_contains "$diags_log" "traffic server running" 60 + +mv "$diags_log" "$rotated_diags_log" + +# Send the SIGUSR2 signal to the handler to trigger ATS to rotate the log. +"$python_bin" "$handler" --signal SIGUSR2 "$ts_name" + +wait_for_file "$diags_log" 60 +wait_for_contains "$diags_log" "Reseated diags.log" 60 +wait_for_contains "$rotated_diags_log" "traffic server running" 60 diff --git a/tests/gold_tests/logging/verify_milestone_fields.py b/tests/gold_tests/logging/verify_milestone_fields.py new file mode 100644 index 00000000000..01e91ac00ac --- /dev/null +++ b/tests/gold_tests/logging/verify_milestone_fields.py @@ -0,0 +1,237 @@ +#!/usr/bin/env python3 +''' +Validate milestone timing fields and cache key hash in an ATS log file. + +Parses key=value log lines and checks: + - All expected fields are present + - All values are integers + - No epoch-length garbage (> 1 billion) from the difference_msec bug + - Cache miss lines have ms > 0 and origin-phase fields populated + - Cache hit lines have hit_proc and hit_xfer populated + - The miss-path chain sums to approximately c_ttfb + - Cache key hash (ckh) is a valid base64 string on every line + - Cache key hash is identical between miss and hit for the same URL +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import base64 +import sys + +ALL_FIELDS = [ + 'crc', + 'ckh', + 'ms', + 'c_ttfb', + 'c_tls', + 'c_hdr', + 'c_proc', + 'cache', + 'dns', + 'o_tcp', + 'o_wait', + 'o_hdr', + 'o_proc', + 'o_body', + 'c_xfer', + 'hit_proc', + 'hit_xfer', +] + +TIMING_FIELDS = [f for f in ALL_FIELDS if f not in ('crc', 'ckh')] + +# Fields that form the contiguous miss-path chain to c_ttfb: +# c_ttfb = c_hdr + c_proc + cache + dns + o_conn + o_wait + o_hdr + o_proc +MISS_CHAIN = ['c_hdr', 'c_proc', 'cache', 'dns', 'o_tcp', 'o_wait', 'o_hdr', 'o_proc'] + +EPOCH_THRESHOLD = 1_000_000_000 +# Each msdms field is truncated independently to integer milliseconds. +# Allow one truncated millisecond per component in the miss chain. +CHAIN_TOLERANCE = len(MISS_CHAIN) + + +def parse_line(line: str) -> dict[str, str]: + """Parse a space-separated key=value log line into a dict.""" + fields = {} + for token in line.strip().split(): + if '=' in token: + key, val = token.split('=', 1) + fields[key] = val + return fields + + +def validate_line(fields: dict[str, str], line_num: int) -> list[str]: + """Return a list of error strings (empty = pass).""" + errors = [] + + for name in ALL_FIELDS: + if name not in fields: + errors.append(f'line {line_num}: missing field "{name}"') + + ckh = fields.get('ckh') + if ckh is not None: + if ckh == '-': + errors.append(f'line {line_num}: ckh should not be "-" (cache lookup was performed)') + else: + try: + raw = base64.b64decode(ckh, validate=True) + if len(raw) not in (16, 32): + errors.append(f'line {line_num}: ckh decoded to {len(raw)} bytes (expected 16 or 32)') + except Exception as e: + errors.append(f'line {line_num}: ckh is not valid base64: {ckh!r} ({e})') + + for name in TIMING_FIELDS: + val_str = fields.get(name) + if val_str is None: + continue + # Accept '-' as a valid sentinel for unset milestones. + if val_str == '-': + continue + try: + val = int(val_str) + except ValueError: + errors.append(f'line {line_num}: field "{name}" is not an integer: {val_str!r}') + continue + + if val > EPOCH_THRESHOLD: + errors.append(f'line {line_num}: field "{name}" looks like an epoch leak: {val} ' + f'(> {EPOCH_THRESHOLD})') + + crc = fields.get('crc', '') + is_miss = 'MISS' in crc or 'NONE' in crc + is_hit = 'HIT' in crc and 'MISS' not in crc + + ms_str = fields.get('ms', '0') + try: + ms_val = int(ms_str) + except ValueError: + ms_val = -1 + + if ms_val < 0 and ms_str != '-': + errors.append(f'line {line_num}: ms should be >= 0, got {ms_val}') + + if is_miss: + for name in MISS_CHAIN: + val_str = fields.get(name) + if val_str is None or val_str == '-': + continue + try: + val = int(val_str) + except ValueError: + continue + if val < -10: + errors.append(f'line {line_num}: miss field "{name}" has unexpected value: {val}') + + # Verify the signed chain sum approximates c_ttfb within a tolerance + # for per-field millisecond truncation. + chain_vals = [] + for name in MISS_CHAIN: + val_str = fields.get(name) + if val_str is None or val_str == '-': + chain_vals.append(0) + continue + try: + chain_vals.append(int(val_str)) + except ValueError: + chain_vals.append(0) + + chain_sum = sum(chain_vals) + c_ttfb_str = fields.get('c_ttfb') + if c_ttfb_str and c_ttfb_str != '-': + try: + c_ttfb_val = int(c_ttfb_str) + if c_ttfb_val >= 0 and abs(chain_sum - c_ttfb_val) > CHAIN_TOLERANCE: + errors.append( + f'line {line_num}: chain sum ({chain_sum}) != c_ttfb ({c_ttfb_val}), ' + f'diff={abs(chain_sum - c_ttfb_val)}ms') + except ValueError: + pass + + if is_hit: + for name in ['hit_proc', 'hit_xfer']: + val_str = fields.get(name) + if val_str is None: + errors.append(f'line {line_num}: cache hit missing field "{name}"') + continue + if val_str == '-': + errors.append(f'line {line_num}: cache hit field "{name}" should not be "-"') + continue + try: + val = int(val_str) + if val < 0: + errors.append(f'line {line_num}: cache hit field "{name}" should be >= 0, got {val}') + except ValueError: + errors.append(f'line {line_num}: cache hit field "{name}" is not an integer: {val_str!r}') + + return errors + + +def main(): + if len(sys.argv) != 2: + print(f'Usage: {sys.argv[0]} ', file=sys.stderr) + sys.exit(1) + + log_path = sys.argv[1] + try: + with open(log_path) as f: + lines = [l for l in f.readlines() if l.strip()] + except FileNotFoundError: + print(f'FAIL: log file not found: {log_path}') + sys.exit(1) + + if len(lines) < 2: + print(f'FAIL: expected at least 2 log lines (miss + hit), got {len(lines)}') + sys.exit(1) + + all_errors = [] + miss_found = False + hit_found = False + cache_key_hashes = set() + + for i, line in enumerate(lines, start=1): + fields = parse_line(line) + crc = fields.get('crc', '') + if 'MISS' in crc: + miss_found = True + if 'HIT' in crc and 'MISS' not in crc: + hit_found = True + ckh = fields.get('ckh') + if ckh and ckh != '-': + cache_key_hashes.add(ckh) + errors = validate_line(fields, i) + all_errors.extend(errors) + + if not miss_found: + all_errors.append('No cache miss line found in log') + if not hit_found: + all_errors.append('No cache hit line found in log') + if len(cache_key_hashes) != 1: + all_errors.append( + f'Expected identical cache key hash on all lines, got {len(cache_key_hashes)} ' + f'distinct values: {cache_key_hashes}') + + if all_errors: + for err in all_errors: + print(f'FAIL: {err}') + sys.exit(1) + else: + print(f'PASS: validated {len(lines)} log lines ' + f'(miss={miss_found}, hit={hit_found}), all fields correct') + sys.exit(0) + + +if __name__ == '__main__': + main() diff --git a/tests/gold_tests/pluginTest/money_trace/money_trace.test.py b/tests/gold_tests/pluginTest/money_trace/money_trace.test.py index 25bca70dd1b..1783db061f6 100644 --- a/tests/gold_tests/pluginTest/money_trace/money_trace.test.py +++ b/tests/gold_tests/pluginTest/money_trace/money_trace.test.py @@ -197,9 +197,10 @@ def maketrace(name): tr.StillRunningAfter = ts tr.StillRunningAfter = server -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. +# Wait for the final log line to be written. # 11 Test -tr = Test.AddTestRun() -ps = tr.Processes.Default -ps.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + os.path.join(ts.Variables.LOGDIR, 'remap.log')) +Test.AddAwaitFileContainsTestRun( + 'Await final money_trace remap log line.', + os.path.join(ts.Variables.LOGDIR, 'remap.log'), + r'^cqh: not a trace header - trace-id=.* pqh: trace-id=.* - psh: not a trace header -$', +) diff --git a/tests/gold_tests/pluginTest/money_trace/money_trace_global.test.py b/tests/gold_tests/pluginTest/money_trace/money_trace_global.test.py index 7f91c29d061..a8de1876cde 100644 --- a/tests/gold_tests/pluginTest/money_trace/money_trace_global.test.py +++ b/tests/gold_tests/pluginTest/money_trace/money_trace_global.test.py @@ -98,9 +98,9 @@ tr.StillRunningAfter = ts tr.StillRunningAfter = server -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. -tr = Test.AddTestRun() -ps = tr.Processes.Default -ps.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + os.path.join(ts.Variables.LOGDIR, 'global.log')) -#ps.ReturnCode = 0 +# Wait for the final log line to be written. +Test.AddAwaitFileContainsTestRun( + 'Await final money_trace global log line.', + os.path.join(ts.Variables.LOGDIR, 'global.log'), + r'^cqh: - trace-id=.* pqh: trace-id=.* psh: -$', +) diff --git a/tests/gold_tests/pluginTest/prefetch/prefetch_cmcd.test.py b/tests/gold_tests/pluginTest/prefetch/prefetch_cmcd.test.py index cca600daadc..40dde89f739 100644 --- a/tests/gold_tests/pluginTest/prefetch/prefetch_cmcd.test.py +++ b/tests/gold_tests/pluginTest/prefetch/prefetch_cmcd.test.py @@ -273,8 +273,6 @@ tr.MakeCurlCommand(f"--verbose --proxy 127.0.0.1:{ts0.Variables.port} \'http://ts0/{crr_name}\' -H \'{crr_header}\'", ts=ts0) tr.Processes.Default.ReturnCode = 0 -condwaitpath = os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') - ts0log = os.path.join(ts0.Variables.LOGDIR, 'transaction.log') Test.AddAwaitFileContainsTestRun('Await ts transactions to finish logging.', ts0log, 'crr.txt') diff --git a/tests/gold_tests/pluginTest/slice/replay/slice_long_etag.replay.yaml b/tests/gold_tests/pluginTest/slice/replay/slice_long_etag.replay.yaml new file mode 100644 index 00000000000..5ed5de8b28d --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/replay/slice_long_etag.replay.yaml @@ -0,0 +1,111 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +autest: + description: 'Verify slice plugin handles long ETag values without crashing' + + server: + name: 'server-long-etag' + process_config: + other_args: '-f "{url}"' + + client: + name: 'client-long-etag' + + ats: + name: 'ts-long-etag' + + process_config: + enable_cache: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'slice' + + remap_config: + - from: "http://preload/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + + - from: "http://slice/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: "slice.so" + args: + - "--blockbytes-test=11" + +# Body is 28 bytes ("lets go surfin now everybody"), block size is 11. +# This means 3 blocks: bytes 0-10, 11-21, 22-27. +# The slice plugin copies the etag from block 0 into data->m_etag +# for validation on blocks 1 and 2. A 4000-char etag exercises the +# null-termination and bounds-checking code path. + +sessions: + +# Session 1: Preload the asset into cache (bypassing slice plugin) +- transactions: + + - client-request: + method: GET + url: /long_etag + version: '1.1' + headers: + fields: + - [Host, preload] + - [uuid, preload-long-etag] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Content-Length, "28"] + - [Connection, close] + - [Cache-Control, "max-age=500"] + - [Etag, '"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"'] + - [Last-Modified, "Fri, 07 Mar 2025 18:06:58 GMT"] + content: + data: "lets go surfin now everybody" + + proxy-response: + status: 200 + +# Session 2: Fetch through slice plugin (should serve from cache in blocks) +- transactions: + + - client-request: + delay: 100ms + method: GET + url: /long_etag + version: '1.1' + headers: + fields: + - [Host, slice] + - [uuid, slice-long-etag] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [Content-Length, "28"] + content: + data: "lets go surfin now everybody" + + proxy-response: + status: 200 diff --git a/tests/gold_tests/pluginTest/slice/slice_crr_ident.test.py b/tests/gold_tests/pluginTest/slice/slice_crr_ident.test.py index 1dfaeb278df..db803deb187 100644 --- a/tests/gold_tests/pluginTest/slice/slice_crr_ident.test.py +++ b/tests/gold_tests/pluginTest/slice/slice_crr_ident.test.py @@ -210,8 +210,6 @@ ps.Streams.stdout.Content = Testers.ContainsExpression("404", "expected 404 Not Found response") tr.StillRunningAfter = ts -condwaitpath = os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') - tslog = os.path.join(ts.Variables.LOGDIR, 'transaction.log') Test.AddAwaitFileContainsTestRun('Await ts transactions to finish logging.', tslog, '404.txt') diff --git a/tests/gold_tests/pluginTest/slice/slice_ident.test.py b/tests/gold_tests/pluginTest/slice/slice_ident.test.py index 7babe58ea39..faabb4f9f22 100644 --- a/tests/gold_tests/pluginTest/slice/slice_ident.test.py +++ b/tests/gold_tests/pluginTest/slice/slice_ident.test.py @@ -177,7 +177,6 @@ tr.StillRunningAfter = ts # 7 - wait for logs -condwaitpath = os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') tslog = os.path.join(ts.Variables.LOGDIR, 'transaction.log') Test.AddAwaitFileContainsTestRun('Await ts transactions to finish logging.', tslog, '404.txt') diff --git a/tests/gold_tests/pluginTest/slice/slice_long_etag.test.py b/tests/gold_tests/pluginTest/slice/slice_long_etag.test.py new file mode 100644 index 00000000000..447822220b0 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/slice_long_etag.test.py @@ -0,0 +1,27 @@ +''' +Verify slice plugin handles long ETag values without crashing. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify the slice plugin handles long ETag values (4000 chars) without crashing +when copying between internal buffers during multi-block range requests. +''' + +Test.SkipUnless(Condition.PluginExists('slice.so'),) + +Test.ATSReplayTest(replay_file="replay/slice_long_etag.replay.yaml") diff --git a/tests/gold_tests/pluginTest/slice/slice_prefetch.test.py b/tests/gold_tests/pluginTest/slice/slice_prefetch.test.py index 5b544e9d091..baf9cbb1eca 100644 --- a/tests/gold_tests/pluginTest/slice/slice_prefetch.test.py +++ b/tests/gold_tests/pluginTest/slice/slice_prefetch.test.py @@ -174,8 +174,10 @@ # 6 Test - All requests (client & slice internal) logs to see background fetches cache_file = os.path.join(ts.Variables.LOGDIR, 'cache.log') -# Wait for log file to appear, then wait one extra second to make sure TS is done writing it. -test_run = Test.AddTestRun("Checking debug logs for background fetches") -test_run.Processes.Default.Command = (os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + cache_file) +# Wait for the final cache log line to be written. +test_run = Test.AddAwaitFileContainsTestRun( + "Checking debug logs for background fetches", + cache_file, + r'\*/18 hit-fresh, none$', +) ts.Disk.File(cache_file).Content = "gold/slice_prefetch.gold" -test_run.Processes.Default.ReturnCode = 0 diff --git a/tests/gold_tests/post/post-early-return.test.py b/tests/gold_tests/post/post-early-return.test.py index 82879eb4015..a8e85e2c06c 100644 --- a/tests/gold_tests/post/post-early-return.test.py +++ b/tests/gold_tests/post/post-early-return.test.py @@ -61,18 +61,27 @@ 'proxy.config.diags.debug.tags': 'http', }) +mock_origin = os.path.join(Test.Variables.AtsTestToolsDir, 'mock_origin.py') +mock_origin_args = '--status 420 --reason "Be Calm"' + server1 = Test.Processes.Process( - "server1", "bash -c '" + Test.TestDirectory + "/server1.sh {} outserver1'".format(Test.Variables.upstream_port1)) + "server1", f"python3 {mock_origin} {Test.Variables.upstream_port1} {mock_origin_args} --output outserver1") server2 = Test.Processes.Process( - "server2", "bash -c '" + Test.TestDirectory + "/server1.sh {} outserver1'".format(Test.Variables.upstream_port2)) + "server2", f"python3 {mock_origin} {Test.Variables.upstream_port2} {mock_origin_args} --output outserver1") server3 = Test.Processes.Process( - "server3", "bash -c '" + Test.TestDirectory + "/server1.sh {} outserver1'".format(Test.Variables.upstream_port3)) + "server3", f"python3 {mock_origin} {Test.Variables.upstream_port3} {mock_origin_args} --output outserver1") server4 = Test.Processes.Process( - "server4", "bash -c '" + Test.TestDirectory + "/server1.sh {} outserver1'".format(Test.Variables.upstream_port4)) + "server4", f"python3 {mock_origin} {Test.Variables.upstream_port4} {mock_origin_args} --output outserver1") server5 = Test.Processes.Process( - "server5", "bash -c '" + Test.TestDirectory + "/server1.sh {} outserver1'".format(Test.Variables.upstream_port5)) + "server5", f"python3 {mock_origin} {Test.Variables.upstream_port5} {mock_origin_args} --output outserver1") server6 = Test.Processes.Process( - "server6", "bash -c '" + Test.TestDirectory + "/server1.sh {} outserver1'".format(Test.Variables.upstream_port6)) + "server6", f"python3 {mock_origin} {Test.Variables.upstream_port6} {mock_origin_args} --output outserver1") +server1.Ready = When.PortOpen(Test.Variables.upstream_port1) +server2.Ready = When.PortOpen(Test.Variables.upstream_port2) +server3.Ready = When.PortOpen(Test.Variables.upstream_port3) +server4.Ready = When.PortOpen(Test.Variables.upstream_port4) +server5.Ready = When.PortOpen(Test.Variables.upstream_port5) +server6.Ready = When.PortOpen(Test.Variables.upstream_port6) big_post_body = "0123456789" * 231070 big_post_body_file = open(os.path.join(Test.RunDirectory, "big_post_body"), "w") diff --git a/tests/gold_tests/proxy_protocol/proxy_protocol.test.py b/tests/gold_tests/proxy_protocol/proxy_protocol.test.py index c596e25110c..fc675b9f424 100644 --- a/tests/gold_tests/proxy_protocol/proxy_protocol.test.py +++ b/tests/gold_tests/proxy_protocol/proxy_protocol.test.py @@ -90,13 +90,12 @@ def checkAccessLog(self): """ Test.Disk.File(os.path.join(self.ts.Variables.LOGDIR, 'access.log'), exists=True, content=f"gold/access-{self.name}.gold") - # Wait for log file to appear, then wait one extra second to make sure - # TS is done writing it. - tr = Test.AddTestRun() - tr.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + - os.path.join(self.ts.Variables.LOGDIR, 'access.log')) - tr.Processes.Default.ReturnCode = 0 + Test.AddAwaitFileContainsTestRun( + f'Await PROXY protocol access log lines. {self.name}', + os.path.join(self.ts.Variables.LOGDIR, 'access.log'), + r'^127\.0\.0\.1 0 127\.0\.0\.1$', + 2, + ) def run(self): self.runTraffic() diff --git a/tests/gold_tests/records/records_runroot_precedence.test.py b/tests/gold_tests/records/records_runroot_precedence.test.py new file mode 100644 index 00000000000..49077c174cc --- /dev/null +++ b/tests/gold_tests/records/records_runroot_precedence.test.py @@ -0,0 +1,163 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +Test.Summary = ''' +Verify that when runroot is active, path records from records.yaml are +overridden by the resolved Layout paths (precedence: env var > runroot > records.yaml). + +When --run-root (or TS_RUNROOT) is set and the PROXY_CONFIG_* environment +variables for path records are unset, RecConfigOverrideFromEnvironment() +returns the actual Layout path (e.g. Layout::bindir, Layout::logdir) which +was populated from runroot.yaml — effectively making runroot.yaml override +records.yaml for these path records. +''' +Test.ContinueOnFail = True +Test.SkipUnless( + Test.Variables.BINDIR.startswith(Test.Variables.PREFIX), "need to guarantee bin path starts with prefix for runroot") + +ts = Test.MakeATSProcess("ts") +ts_dir = os.path.join(Test.RunDirectory, "ts") + +# Set deliberately WRONG values in records.yaml for all 4 runroot-managed +# path records. If runroot override works, these values must NOT be used. +ts.Disk.records_config.append_to_document( + ''' + bin_path: wrong_bin_path + local_state_dir: wrong_runtime + log: + logfile_dir: wrong_log + plugin: + plugin_dir: wrong_plugin +''') + +# Test 3 setup: env var that must win over both runroot and records.yaml. +ts.Env['PROXY_CONFIG_DIAGS_DEBUG_TAGS'] = 'env_wins' +ts.Disk.records_config.update(''' + diags: + debug: + enabled: 0 + tags: config_value + ''') + +# Build the ATS command: +# - Unset the 4 path env vars (the test framework always sets them, +# which masks the runroot code path). +# - Set TS_RUNROOT to the sandbox dir so the runroot mechanism activates. +original_cmd = ts.Command +ts.Command = ( + "env" + " -u PROXY_CONFIG_BIN_PATH" + " -u PROXY_CONFIG_LOCAL_STATE_DIR" + " -u PROXY_CONFIG_LOG_LOGFILE_DIR" + " -u PROXY_CONFIG_PLUGIN_PLUGIN_DIR" + f" TS_RUNROOT={ts_dir}" + f" {original_cmd}") + +# --------------------------------------------------------------------------- +# Test 0: Create runroot.yaml that maps to the sandbox layout, then start ATS. +# +# The runroot.yaml must exist before ATS starts because TS_RUNROOT triggers +# Layout::runroot_setup() during initialization. We write a runroot.yaml +# whose paths match the sandbox structure the test framework already created +# (traffic_layout init would create a different FHS-style layout that does +# not match the sandbox, so we write it manually). +# --------------------------------------------------------------------------- +runroot_yaml = os.path.join(ts_dir, 'runroot.yaml') + +runroot_lines = [ + f"prefix: {ts_dir}", + f"bindir: {os.path.join(ts_dir, 'bin')}", + f"sbindir: {os.path.join(ts_dir, 'bin')}", + f"sysconfdir: {os.path.join(ts_dir, 'config')}", + f"logdir: {os.path.join(ts_dir, 'log')}", + f"libexecdir: {os.path.join(ts_dir, 'plugin')}", + f"localstatedir: {os.path.join(ts_dir, 'runtime')}", + f"runtimedir: {os.path.join(ts_dir, 'runtime')}", + f"cachedir: {os.path.join(ts_dir, 'cache')}", +] +runroot_content = "\\n".join(runroot_lines) + "\\n" + +tr = Test.AddTestRun("Create runroot.yaml") +tr.Processes.Default.Command = f"mkdir -p {ts_dir} && printf '{runroot_content}' > {runroot_yaml}" +tr.Processes.Default.ReturnCode = 0 + +# --------------------------------------------------------------------------- +# Test 1: Start ATS with runroot active +# --------------------------------------------------------------------------- +tr = Test.AddTestRun("Start ATS with runroot") +tr.Processes.Default.Command = 'echo start' +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.StartBefore(ts) +tr.StillRunningAfter = ts + +# ATS must not crash (the original nullptr bug) and must complete startup. +ts.Disk.traffic_out.Content = Testers.ExcludesExpression( + "basic_string", "must not crash with 'basic_string: construction from null'") +ts.Disk.traffic_out.Content += Testers.ContainsExpression("records parsing completed", "ATS should complete records parsing") + +# Verify the override log messages appear in traffic.out. +# The errata notes from RecYAMLDecoder are printed by RecCoreInit(). +ts.Disk.traffic_out.Content += Testers.ContainsExpression( + "'proxy.config.bin_path' overridden with .* by runroot", "bin_path override by runroot must be logged") +ts.Disk.traffic_out.Content += Testers.ContainsExpression( + "'proxy.config.local_state_dir' overridden with .* by runroot", "local_state_dir override by runroot must be logged") +ts.Disk.traffic_out.Content += Testers.ContainsExpression( + "'proxy.config.log.logfile_dir' overridden with .* by runroot", "logfile_dir override by runroot must be logged") +ts.Disk.traffic_out.Content += Testers.ContainsExpression( + "'proxy.config.plugin.plugin_dir' overridden with .* by runroot", "plugin_dir override by runroot must be logged") +ts.Disk.traffic_out.Content += Testers.ContainsExpression( + "'proxy.config.diags.debug.tags' overridden with 'env_wins' by environment variable", + "diags.debug.tags override by environment variable must be logged") + +# --------------------------------------------------------------------------- +# Test 2: Verify path records do NOT contain the records.yaml values. +# +# Because runroot is active and env vars are unset, the records should hold +# the resolved Layout paths from runroot.yaml, not the records.yaml values. +# --------------------------------------------------------------------------- +tr = Test.AddTestRun("Verify runroot overrides records.yaml for path records") +tr.Processes.Default.Command = ( + 'traffic_ctl config get' + ' proxy.config.bin_path' + ' proxy.config.local_state_dir' + ' proxy.config.log.logfile_dir' + ' proxy.config.plugin.plugin_dir') +tr.Processes.Default.Env = ts.Env +tr.Processes.Default.ReturnCode = 0 +tr.StillRunningAfter = ts + +# The deliberately wrong records.yaml values must NOT appear. +tr.Processes.Default.Streams.stdout = Testers.ExcludesExpression( + 'wrong_bin_path', 'bin_path must be overridden by runroot, not records.yaml') +tr.Processes.Default.Streams.stdout += Testers.ExcludesExpression( + 'wrong_runtime', 'local_state_dir must be overridden by runroot, not records.yaml') +tr.Processes.Default.Streams.stdout += Testers.ExcludesExpression( + 'wrong_log', 'logfile_dir must be overridden by runroot, not records.yaml') +tr.Processes.Default.Streams.stdout += Testers.ExcludesExpression( + 'wrong_plugin', 'plugin_dir must be overridden by runroot, not records.yaml') + +# --------------------------------------------------------------------------- +# Test 3: Verify env vars still take highest precedence over runroot. +# --------------------------------------------------------------------------- +tr = Test.AddTestRun("Verify env var overrides both runroot and records.yaml") +tr.Processes.Default.Command = 'traffic_ctl config get proxy.config.diags.debug.tags' +tr.Processes.Default.Env = ts.Env +tr.Processes.Default.ReturnCode = 0 +tr.StillRunningAfter = ts +tr.Processes.Default.Streams.stdout = Testers.ContainsExpression( + 'proxy.config.diags.debug.tags: env_wins', 'Env var must override both runroot and records.yaml') diff --git a/tests/gold_tests/slow_post/server_abort.test.py b/tests/gold_tests/slow_post/server_abort.test.py index 1004bcdb6a3..05867859839 100644 --- a/tests/gold_tests/slow_post/server_abort.test.py +++ b/tests/gold_tests/slow_post/server_abort.test.py @@ -47,4 +47,5 @@ tr.StillRunningAfter = server tr.StillRunningAfter = ts server.Streams.stderr += Testers.ContainsExpression( - "UnicodeDecodeError", "Verify that the server raises an exception when processing the request.") + "(UnicodeDecodeError|IndexError: list index out of range)", + "Verify that the server raises an exception when processing the request.") diff --git a/tests/gold_tests/thread_config/check_threads.py b/tests/gold_tests/thread_config/check_threads.py index 1a7de52cb92..e0347b7dddc 100755 --- a/tests/gold_tests/thread_config/check_threads.py +++ b/tests/gold_tests/thread_config/check_threads.py @@ -36,13 +36,16 @@ def _count_threads_once(ts_path, etnet_threads, accept_threads, task_threads, ai # Find the pid corresponding to the ats process we started in autest. # It needs to match the process name and the binary path. # If autest can expose the pid of the process this is not needed anymore. + # Match by CWD or command line containing ts_path, since under + # ASAN the CWD may differ from the expected path. process_name = p.name() process_cwd = p.cwd() process_exe = p.exe() - if process_cwd != ts_path: - continue - if process_name != '[TS_MAIN]' and process_name != 'traffic_server' and os.path.basename( - process_exe) != 'traffic_server': + is_ts = process_name == '[TS_MAIN]' or process_name == 'traffic_server' or os.path.basename( + process_exe) == 'traffic_server' + match_by_cwd = process_cwd == ts_path + match_by_cmdline = any(ts_path in arg for arg in (p.cmdline() or [])) + if not is_ts or not (match_by_cwd or match_by_cmdline): continue except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess): continue diff --git a/tests/gold_tests/timeout/ssl-delay-server.cc b/tests/gold_tests/timeout/ssl-delay-server.cc index d993696229e..3f509f0e369 100644 --- a/tests/gold_tests/timeout/ssl-delay-server.cc +++ b/tests/gold_tests/timeout/ssl-delay-server.cc @@ -39,6 +39,7 @@ #include #include #include +#include char req_buf[10000]; char post_buf[1000]; @@ -156,6 +157,10 @@ main(int argc, char *argv[]) ttfb_delay = atoi(argv[3]); const char *pem_file = argv[4]; + // Ignore SIGPIPE which can be raised when a client disconnects during + // the handshake delay, killing the process unexpectedly. + signal(SIGPIPE, SIG_IGN); + fprintf(stderr, "Listen on %d connect delay=%d ttfb delay=%d\n", listen_port, connect_delay, ttfb_delay); int listenfd = socket(AF_INET, SOCK_STREAM, 0); @@ -198,9 +203,11 @@ main(int argc, char *argv[]) for (;;) { sfd = accept(listenfd, (struct sockaddr *)nullptr, nullptr); - if (sfd <= 0) { - // Failure - printf("Listen failure\n"); + if (sfd < 0) { + if (errno == EINTR) { + continue; + } + printf("Listen failure errno=%d\n", errno); exit(1); } diff --git a/tests/gold_tests/post/server1.sh b/tests/gold_tests/tls/replay/tls_cert_compression.replay.yaml old mode 100755 new mode 100644 similarity index 61% rename from tests/gold_tests/post/server1.sh rename to tests/gold_tests/tls/replay/tls_cert_compression.replay.yaml index c4461b308ee..bc05706ec25 --- a/tests/gold_tests/post/server1.sh +++ b/tests/gold_tests/tls/replay/tls_cert_compression.replay.yaml @@ -14,23 +14,28 @@ # See the License for the specific language governing permissions and # limitations under the License. -# A very simple cleartext server for one HTTP transaction. Does no validation of the Request message. -# Sends a fixed response message +meta: + version: "1.0" -response() { - # Wait for end of Request message. - # - while ((1 == 1)); do - if [[ -f $outfile ]]; then - if tr '\r\n' '=!' <$outfile | grep '=!=!' >/dev/null; then - break - fi - fi - sleep 1 - done +sessions: +- transactions: - printf "HTTP/1.1 420 Be Calm\r\nContent-Length: 0\r\n\r\n" + - client-request: + method: GET + url: /cert-compression-test + version: '1.1' + headers: + fields: + - [Host, example.com] + - [uuid, cert-compression-request] -} -outfile=$2 -response | nc -l $1 >"$outfile" + server-response: + status: 200 + reason: OK + headers: + fields: + - [Content-Length, "0"] + - [X-Response, cert-compression-response] + + proxy-response: + status: 200 diff --git a/tests/gold_tests/tls/tls_cert_comp.test.py b/tests/gold_tests/tls/tls_cert_comp.test.py new file mode 100644 index 00000000000..a2d6aa61e55 --- /dev/null +++ b/tests/gold_tests/tls/tls_cert_comp.test.py @@ -0,0 +1,163 @@ +''' +Verify TLS Certificate Compression (RFC 8879) between two ATS processes. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify TLS Certificate Compression (RFC 8879) works between two ATS +instances. An edge ATS (client) connects via HTTPS to a mid ATS (server) +with cert compression enabled. The test verifies compression and +decompression succeed by checking the ssl cert compression metrics. +''' + +Test.SkipUnless(Condition.HasATSFeature('TS_HAS_CERT_COMPRESSION_CALLBACKS')) + +REPLAY_FILE = 'replay/tls_cert_compression.replay.yaml' + + +class TestCertCompression: + server_counter: int = 0 + ts_counter: int = 0 + client_counter: int = 0 + + def __init__(self, algorithm: str) -> None: + self._algorithm = algorithm + self._server = self._configure_server() + self._ts_mid = self._configure_ts_mid() + self._ts_edge = self._configure_ts_edge() + + def _configure_server(self) -> 'Process': + name = f'server-{TestCertCompression.server_counter}' + TestCertCompression.server_counter += 1 + server = Test.MakeVerifierServerProcess(name, REPLAY_FILE) + return server + + def _configure_ts_mid(self) -> 'Process': + """Mid-tier ATS that terminates TLS and forwards to origin.""" + name = f'm{TestCertCompression.ts_counter}' + TestCertCompression.ts_counter += 1 + ts = Test.MakeATSProcess(name, enable_tls=True, enable_cache=False) + + ts.addDefaultSSLFiles() + ts.Disk.ssl_multicert_config.AddLine('dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key') + + ts.Disk.remap_config.AddLine(f'map / http://127.0.0.1:{self._server.Variables.http_port}/') + + ts.Disk.records_config.update( + { + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.cert_compression.algorithms': self._algorithm, + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'ssl_cert_compress', + }) + + return ts + + def _configure_ts_edge(self) -> 'Process': + """Edge ATS that connects to mid-tier via HTTPS.""" + name = f'e{TestCertCompression.ts_counter}' + TestCertCompression.ts_counter += 1 + ts = Test.MakeATSProcess(name, enable_tls=True, enable_cache=False) + + ts.addDefaultSSLFiles() + ts.Disk.ssl_multicert_config.AddLine('dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key') + + ts.Disk.remap_config.AddLine(f'map / https://127.0.0.1:{self._ts_mid.Variables.ssl_port}/') + + ts.Disk.records_config.update( + { + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, + 'proxy.config.ssl.client.verify.server.policy': 'PERMISSIVE', + 'proxy.config.ssl.client.cert_compression.algorithms': self._algorithm, + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'ssl_cert_compress', + }) + + return ts + + def run(self) -> None: + # Test run 1: Send traffic through the proxy chain. + tr = Test.AddTestRun(f'Send request through edge->mid with {self._algorithm} cert compression') + tr.Processes.Default.StartBefore(self._server) + tr.Processes.Default.StartBefore(self._ts_mid) + tr.Processes.Default.StartBefore(self._ts_edge) + + name = f'client-{TestCertCompression.client_counter}' + TestCertCompression.client_counter += 1 + tr.AddVerifierClientProcess(name, REPLAY_FILE, http_ports=[self._ts_edge.Variables.port]) + + # Test run 2: Check compression metric on the mid-tier (server side). + tr = Test.AddTestRun(f'Verify {self._algorithm} compression metric on mid-tier') + tr.Processes.Default.Command = (f'traffic_ctl metric get' + f' proxy.process.ssl.cert_compress.{self._algorithm}') + tr.Processes.Default.Env = self._ts_mid.Env + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.All = Testers.ContainsExpression( + f'proxy.process.ssl.cert_compress.{self._algorithm} 1', + f'Certificate should have been compressed with {self._algorithm}') + tr.StillRunningAfter = self._ts_mid + tr.StillRunningAfter = self._ts_edge + tr.StillRunningAfter = self._server + + # Test run 3: Check decompression metric on the edge (client side). + tr = Test.AddTestRun(f'Verify {self._algorithm} decompression metric on edge') + tr.Processes.Default.Command = (f'traffic_ctl metric get' + f' proxy.process.ssl.cert_decompress.{self._algorithm}') + tr.Processes.Default.Env = self._ts_edge.Env + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.All = Testers.ContainsExpression( + f'proxy.process.ssl.cert_decompress.{self._algorithm} 1', + f'Certificate should have been decompressed with {self._algorithm}') + tr.StillRunningAfter = self._ts_mid + tr.StillRunningAfter = self._ts_edge + tr.StillRunningAfter = self._server + + # Test run 4: Verify no failures on either side. + tr = Test.AddTestRun(f'Verify no {self._algorithm} compression failures on mid-tier') + tr.Processes.Default.Command = (f'traffic_ctl metric get' + f' proxy.process.ssl.cert_compress.{self._algorithm}_failure') + tr.Processes.Default.Env = self._ts_mid.Env + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.All = Testers.ContainsExpression( + f'proxy.process.ssl.cert_compress.{self._algorithm}_failure 0', + f'There should be no {self._algorithm} compression failures') + tr.StillRunningAfter = self._ts_mid + tr.StillRunningAfter = self._ts_edge + tr.StillRunningAfter = self._server + + tr = Test.AddTestRun(f'Verify no {self._algorithm} decompression failures on edge') + tr.Processes.Default.Command = (f'traffic_ctl metric get' + f' proxy.process.ssl.cert_decompress.{self._algorithm}_failure') + tr.Processes.Default.Env = self._ts_edge.Env + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.All = Testers.ContainsExpression( + f'proxy.process.ssl.cert_decompress.{self._algorithm}_failure 0', + f'There should be no {self._algorithm} decompression failures') + tr.StillRunningAfter = self._ts_mid + tr.StillRunningAfter = self._ts_edge + tr.StillRunningAfter = self._server + + +algorithms = ['zlib'] +if Condition.HasATSFeature('TS_HAS_BROTLI'): + algorithms.append('brotli') +if Condition.HasATSFeature('TS_HAS_ZSTD'): + algorithms.append('zstd') +for algorithm in algorithms: + TestCertCompression(algorithm).run() diff --git a/tests/gold_tests/tls/tls_client_versions.test.py b/tests/gold_tests/tls/tls_client_versions.test.py index 1f0343d9cd1..134124eabfb 100644 --- a/tests/gold_tests/tls/tls_client_versions.test.py +++ b/tests/gold_tests/tls/tls_client_versions.test.py @@ -25,6 +25,7 @@ Test.SkipUnless(Condition.HasOpenSSLVersion("1.1.1")) Test.SkipUnless(Condition.HasLegacyTLSSupport()) +has_curl_tlsv1 = Condition.HasCurlTLSVersionSupport("1.0") # Define default ATS ts = Test.MakeATSProcess("ts", enable_tls=True) @@ -85,22 +86,24 @@ tr.StillRunningAfter = ts # Target foo.com for TLSv1. Should succeed -tr = Test.AddTestRun("foo.com TLSv1") -tr.MakeCurlCommand( - "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.0 --tlsv1 --resolve 'foo.com:{0}:127.0.0.1' -k https://foo.com:{0}".format( - ts.Variables.ssl_port), - ts=ts) -tr.ReturnCode = 0 -tr.StillRunningAfter = ts +if has_curl_tlsv1: + tr = Test.AddTestRun("foo.com TLSv1") + tr.MakeCurlCommand( + "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.0 --tlsv1 --resolve 'foo.com:{0}:127.0.0.1' -k https://foo.com:{0}".format( + ts.Variables.ssl_port), + ts=ts) + tr.ReturnCode = 0 + tr.StillRunningAfter = ts # Target bar.com for TLSv1. Should fail -tr = Test.AddTestRun("bar.com TLSv1") -tr.MakeCurlCommand( - "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.0 --tlsv1 --resolve 'bar.com:{0}:127.0.0.1' -k https://bar.com:{0}".format( - ts.Variables.ssl_port), - ts=ts) -tr.ReturnCode = 35 -tr.StillRunningAfter = ts +if has_curl_tlsv1: + tr = Test.AddTestRun("bar.com TLSv1") + tr.MakeCurlCommand( + "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.0 --tlsv1 --resolve 'bar.com:{0}:127.0.0.1' -k https://bar.com:{0}".format( + ts.Variables.ssl_port), + ts=ts) + tr.ReturnCode = 35 + tr.StillRunningAfter = ts # Target bar.com for TLSv1_2. Should succeed tr = Test.AddTestRun("bar.com TLSv1_2") diff --git a/tests/gold_tests/tls/tls_client_versions_minmax.test.py b/tests/gold_tests/tls/tls_client_versions_minmax.test.py index 9c63d0700d8..4c0d1742e86 100644 --- a/tests/gold_tests/tls/tls_client_versions_minmax.test.py +++ b/tests/gold_tests/tls/tls_client_versions_minmax.test.py @@ -25,6 +25,8 @@ Test.SkipUnless(Condition.HasOpenSSLVersion("1.1.1")) Test.SkipUnless(Condition.HasLegacyTLSSupport()) +has_curl_tlsv1 = Condition.HasCurlTLSVersionSupport("1.0") +has_curl_tlsv1_1 = Condition.HasCurlTLSVersionSupport("1.1") # Define default ATS ts = Test.MakeATSProcess("ts", enable_tls=True) @@ -90,31 +92,34 @@ tr.StillRunningAfter = ts # Target foo.com for TLSv1. Should succeed -tr = Test.AddTestRun("foo.com TLSv1") -tr.MakeCurlCommand( - "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.0 --tlsv1 --resolve 'foo.com:{0}:127.0.0.1' -k https://foo.com:{0}".format( - ts.Variables.ssl_port), - ts=ts) -tr.ReturnCode = 0 -tr.StillRunningAfter = ts +if has_curl_tlsv1: + tr = Test.AddTestRun("foo.com TLSv1") + tr.MakeCurlCommand( + "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.0 --tlsv1 --resolve 'foo.com:{0}:127.0.0.1' -k https://foo.com:{0}".format( + ts.Variables.ssl_port), + ts=ts) + tr.ReturnCode = 0 + tr.StillRunningAfter = ts # Target foo.com for TLSv1_1. Should succeed -tr = Test.AddTestRun("foo.com TLSv1_1") -tr.MakeCurlCommand( - "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.1 --tlsv1.1 --resolve 'foo.com:{0}:127.0.0.1' -k https://foo.com:{0}".format( - ts.Variables.ssl_port), - ts=ts) -tr.ReturnCode = 0 -tr.StillRunningAfter = ts +if has_curl_tlsv1_1: + tr = Test.AddTestRun("foo.com TLSv1_1") + tr.MakeCurlCommand( + "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.1 --tlsv1.1 --resolve 'foo.com:{0}:127.0.0.1' -k https://foo.com:{0}".format( + ts.Variables.ssl_port), + ts=ts) + tr.ReturnCode = 0 + tr.StillRunningAfter = ts # Target bar.com for TLSv1. Should fail -tr = Test.AddTestRun("bar.com TLSv1") -tr.MakeCurlCommand( - "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.0 --tlsv1 --resolve 'bar.com:{0}:127.0.0.1' -k https://bar.com:{0}".format( - ts.Variables.ssl_port), - ts=ts) -tr.ReturnCode = 35 -tr.StillRunningAfter = ts +if has_curl_tlsv1: + tr = Test.AddTestRun("bar.com TLSv1") + tr.MakeCurlCommand( + "-v --ciphers DEFAULT@SECLEVEL=0 --tls-max 1.0 --tlsv1 --resolve 'bar.com:{0}:127.0.0.1' -k https://bar.com:{0}".format( + ts.Variables.ssl_port), + ts=ts) + tr.ReturnCode = 35 + tr.StillRunningAfter = ts # Target bar.com for TLSv1_2. Should succeed tr = Test.AddTestRun("bar.com TLSv1_2") diff --git a/tests/gold_tests/tls/tls_sni_host_policy.test.py b/tests/gold_tests/tls/tls_sni_host_policy.test.py index 03a6e172c0b..6a2e7477e61 100644 --- a/tests/gold_tests/tls/tls_sni_host_policy.test.py +++ b/tests/gold_tests/tls/tls_sni_host_policy.test.py @@ -181,10 +181,12 @@ tr.Processes.Default.ReturnCode = 0 tr.Processes.Default.Streams.All = Testers.ExcludesExpression("Access Denied", "Check response") -# Wait for the error.log to appaer. -test_run = Test.AddTestRun() -test_run.Processes.Default.Command = ( - os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + os.path.join(ts.Variables.LOGDIR, 'error.log')) +# Wait for the error.log entry to be written. +test_run = Test.AddAwaitFileContainsTestRun( + 'Await SNI mismatch error log entry.', + os.path.join(ts.Variables.LOGDIR, 'error.log'), + "SNI/hostname mismatch: connecting to .* for host='bob' sni='dave', returning a 403", +) ts.Disk.diags_log.Content += Testers.ContainsExpression( "WARNING: SNI/hostname mismatch sni=dave host=bob action=terminate", "Should have warning on mismatch") diff --git a/tests/gold_tests/tls/tls_verify_ca_override.test.py b/tests/gold_tests/tls/tls_verify_ca_override.test.py index 2de7f424d79..be2704d2324 100644 --- a/tests/gold_tests/tls/tls_verify_ca_override.test.py +++ b/tests/gold_tests/tls/tls_verify_ca_override.test.py @@ -17,7 +17,7 @@ # limitations under the License. Test.Summary = ''' -Test tls server certificate verification options. Exercise conf_remap for ca bundle +Test tls server certificate verification options. Exercise conf_remap for ca bundle path and file. ''' # Define default ATS @@ -58,18 +58,25 @@ ts.addSSLfile("ssl/signer2.pem") ts.addSSLfile("ssl/signer2.key") + +def ca_cert_overrides(filename): + return ( + f'@pparam=proxy.config.ssl.client.CA.cert.path={ts.Variables.SSLDir} ' + f'@pparam=proxy.config.ssl.client.CA.cert.filename={filename}') + + ts.Disk.remap_config.AddLine( - 'map /case1 https://127.0.0.1:{0}/ @plugin=conf_remap.so @pparam=proxy.config.ssl.client.CA.cert.filename={1}/{2}'.format( - server1.Variables.SSL_Port, ts.Variables.SSLDir, "signer.pem")) + f'map /case1 https://127.0.0.1:{server1.Variables.SSL_Port}/ ' + f'@plugin=conf_remap.so {ca_cert_overrides("signer.pem")}') ts.Disk.remap_config.AddLine( - 'map /badcase1 https://127.0.0.1:{0}/ @plugin=conf_remap.so @pparam=proxy.config.ssl.client.CA.cert.filename={1}/{2}'.format( - server1.Variables.SSL_Port, ts.Variables.SSLDir, "signer2.pem")) + f'map /badcase1 https://127.0.0.1:{server1.Variables.SSL_Port}/ ' + f'@plugin=conf_remap.so {ca_cert_overrides("signer2.pem")}') ts.Disk.remap_config.AddLine( - 'map /case2 https://127.0.0.1:{0}/ @plugin=conf_remap.so @pparam=proxy.config.ssl.client.CA.cert.filename={1}/{2}'.format( - server2.Variables.SSL_Port, ts.Variables.SSLDir, "signer2.pem")) + f'map /case2 https://127.0.0.1:{server2.Variables.SSL_Port}/ ' + f'@plugin=conf_remap.so {ca_cert_overrides("signer2.pem")}') ts.Disk.remap_config.AddLine( - 'map /badcase2 https://127.0.0.1:{0}/ @plugin=conf_remap.so @pparam=proxy.config.ssl.client.CA.cert.filename={1}/{2}'.format( - server2.Variables.SSL_Port, ts.Variables.SSLDir, "signer.pem")) + f'map /badcase2 https://127.0.0.1:{server2.Variables.SSL_Port}/ ' + f'@plugin=conf_remap.so {ca_cert_overrides("signer.pem")}') ts.Disk.ssl_multicert_config.AddLine('dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key') diff --git a/tests/gold_tests/traffic_ctl/traffic_ctl_set_read_only.test.py b/tests/gold_tests/traffic_ctl/traffic_ctl_set_read_only.test.py new file mode 100644 index 00000000000..b03a548d05a --- /dev/null +++ b/tests/gold_tests/traffic_ctl/traffic_ctl_set_read_only.test.py @@ -0,0 +1,113 @@ +''' +Verify that the management RPC refuses to write records registered as +RECA_READ_ONLY. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from jsonrpc import Request, Response + +Test.Summary = ''' +Verify that records registered with RECA_READ_ONLY cannot be modified through +the management RPC backing "traffic_ctl config set" +(admin_config_set_records). +''' + +Test.ContinueOnFail = True + +# proxy.config.thread.max_heartbeat_mseconds is registered as RECA_READ_ONLY +# in src/records/RecordsConfig.cc with a default of 60. RECA_READ_ONLY = 2 +# in the RecAccessT enum (see include/records/RecDefs.h). +READ_ONLY_RECORD = "proxy.config.thread.max_heartbeat_mseconds" +DEFAULT_VALUE = "60" +ATTEMPTED_VALUE = "999" +RECA_READ_ONLY = "2" +RECT_CONFIG_BIT = "1" # bit value in the rec_types filter + +# RecordError::RECORD_READ_ONLY in src/mgmt/rpc/handlers/common/RecordsUtils.h +# (assigned as Codes::RECORD + offset). This is the per-tier code that the +# JSONRPC response must surface in error.data[*].code so programmatic +# clients can branch on the access tier without parsing the message text. +RECORD_READ_ONLY_CODE = 2009 + +ts = Test.MakeATSProcess("ts") + + +def lookup_request(): + """Build a JSONRPC request that fetches the read-only record.""" + return Request.admin_lookup_records([{"record_name": READ_ONLY_RECORD, "rec_types": [RECT_CONFIG_BIT]}]) + + +def assert_record_at_default(resp: Response): + """Validate the looked-up record is RECA_READ_ONLY and at its default value.""" + if resp.is_error(): + return (False, f"unexpected error: {resp.error_as_str()}") + + records = resp.result.get('recordList', []) + if len(records) != 1: + return (False, f"expected exactly 1 record, got {len(records)}") + + rec = records[0]['record'] + if rec.get('record_name') != READ_ONLY_RECORD: + return (False, f"record_name {rec.get('record_name')!r} != {READ_ONLY_RECORD!r}") + if rec.get('current_value') != DEFAULT_VALUE: + return (False, f"current_value {rec.get('current_value')!r} != {DEFAULT_VALUE!r}") + + access = rec.get('config_meta', {}).get('access_type') + if str(access) != RECA_READ_ONLY: + return (False, f"access_type {access!r} != {RECA_READ_ONLY!r} (record is not RECA_READ_ONLY)") + + return (True, "record is RECA_READ_ONLY and at the registered default") + + +def assert_set_was_rejected(resp: Response): + """Validate the set attempt produced the per-tier not-writable error code.""" + if not resp.is_error(): + return (False, f"set should have failed but returned a result: {resp.result!r}") + # Validate the structured error code rather than the message text so the + # test stays meaningful if the error wording is ever rephrased and so + # that any regression to the generic Codes::RECORD (2000) is caught. + if not resp.contains_nested_error(code=RECORD_READ_ONLY_CODE): + return (False, f"expected nested error code {RECORD_READ_ONLY_CODE} (RECORD_READ_ONLY); got: {resp.error_as_str()}") + return (True, f"set was refused with code {RECORD_READ_ONLY_CODE} (RECORD_READ_ONLY)") + + +# Step 0: confirm the record is registered as RECA_READ_ONLY and starts at +# its registered default value. This anchors the rest of the test against +# the registration in RecordsConfig.cc -- if someone reclassifies the record +# the test fails noisily here instead of silently exercising a permissive +# write path. +tr = Test.AddTestRun("Confirm record is RECA_READ_ONLY and at default") +tr.Processes.Default.StartBefore(ts) +tr.AddJsonRPCClientRequest(ts, lookup_request()) +tr.Processes.Default.Streams.stdout = Testers.CustomJSONRPCResponse(assert_record_at_default) + +# Step 1: attempt to write the record via the management RPC. The handler +# must reject the request with the "Record is not writable" error. +tr = Test.AddTestRun("Attempt to set the RECA_READ_ONLY record (must be refused)") +tr.AddJsonRPCClientRequest( + ts, Request.admin_config_set_records([{ + "record_name": READ_ONLY_RECORD, + "record_value": ATTEMPTED_VALUE, + }])) +tr.Processes.Default.Streams.stdout = Testers.CustomJSONRPCResponse(assert_set_was_rejected) + +# Step 2: re-look-up the record. Even if step 1's response had been +# misleading, the record must still hold its default value -- this is the +# assertion that catches the underlying bug at the storage level. +tr = Test.AddTestRun("Confirm record was not modified") +tr.AddJsonRPCClientRequest(ts, lookup_request()) +tr.Processes.Default.Streams.stdout = Testers.CustomJSONRPCResponse(assert_record_at_default) diff --git a/tests/tools/mock_origin.py b/tests/tools/mock_origin.py new file mode 100644 index 00000000000..f2390ed7fdc --- /dev/null +++ b/tests/tools/mock_origin.py @@ -0,0 +1,167 @@ +#!/usr/bin/env python3 +''' +A reusable mock origin server for ATS autests. + +Replaces the various ad-hoc nc-based shell scripts (post/server1.sh, +chunked_encoding/server2..4.sh, post_slow_server/server.sh) with a single +Python tool that: + + - Handles When.PortOpen() readiness probes gracefully (nc -l cannot). + - Accepts one real HTTP request, optionally saves it to a file, and sends + a configurable response. + - Drains remaining request data after responding so that ATS does not see + a connection reset while still forwarding a POST body (avoids HTTP/2 502). + - Supports Content-Length bodies, chunked transfer encoding, and arbitrary + response delays. + +Usage examples mapping to the original shell scripts: + + # post/server1.sh PORT OUTFILE + mock_origin.py PORT --output OUTFILE --status 420 --reason "Be Calm" + + # chunked_encoding/server2.sh PORT OUTFILE (Content-Length body) + mock_origin.py PORT --output OUTFILE --body "123456789012345" + + # chunked_encoding/server3.sh PORT OUTFILE (Chunked body) + mock_origin.py PORT --output OUTFILE --body "123456789012345" --chunked + + # post_slow_server/server.sh PORT (Delayed 200KB response) + mock_origin.py PORT --output rcv_file --delay 120 --body-size 204800 +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import socket +import sys +import time + +FILLER_LINE_WIDTH = 8 + + +def build_response(args): + '''Build the complete HTTP response bytes from CLI arguments.''' + + body = b'' + if args.body is not None: + body = args.body.encode() + elif args.body_size and args.body_size > 0: + lines = [] + offset = 0 + while offset < args.body_size: + offset += FILLER_LINE_WIDTH + lines.append(f'{offset:07d}\n'.encode()) + body = b''.join(lines)[:args.body_size] + + status_line = f'HTTP/1.1 {args.status} {args.reason}\r\n'.encode() + + if args.chunked: + headers = b'Transfer-Encoding: chunked\r\n' + for h in (args.header or []): + headers += h.encode() + b'\r\n' + headers += b'\r\n' + chunk = f'{len(body):X}\r\n'.encode() + body + b'\r\n' + terminator = b'0\r\n\r\n' + return status_line + headers + chunk + terminator + else: + headers = f'Content-Length: {len(body)}\r\n'.encode() + for h in (args.header or []): + headers += h.encode() + b'\r\n' + headers += b'\r\n' + return status_line + headers + body + + +def serve_one(args): + '''Listen, absorb readiness probes, serve one real HTTP transaction, exit.''' + + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + sock.bind(('', args.port)) + sock.listen(1) + + response = build_response(args) + + while True: + conn, addr = sock.accept() + data = b'' + try: + while True: + chunk = conn.recv(65536) + if not chunk: + break + data += chunk + if b'\r\n\r\n' in data: + break + except ConnectionError: + pass + + if not data: + # Readiness probe (e.g. When.PortOpen) -- connected and + # disconnected without sending data. Go back to waiting. + conn.close() + continue + + # Real HTTP request arrived. + if args.output: + with open(args.output, 'wb') as f: + f.write(data) + + if args.delay > 0: + time.sleep(args.delay) + + try: + conn.sendall(response) + except ConnectionError: + pass + + # Drain remaining request data (e.g. a large POST body that is still + # being forwarded by ATS). Closing without draining causes a TCP RST + # which makes ATS return 502 on HTTP/2 streams. + try: + while True: + if not conn.recv(65536): + break + except ConnectionError: + pass + + conn.close() + break + + sock.close() + + +def main(): + parser = argparse.ArgumentParser( + description='Mock origin server for ATS autests. ' + 'Listens on PORT, serves one HTTP transaction, then exits. ' + 'Compatible with When.PortOpen() readiness probes.') + + parser.add_argument('port', type=int, help='TCP port to listen on') + parser.add_argument('--output', '-o', help='Write received request data to FILE') + parser.add_argument('--status', '-s', type=int, default=200, help='HTTP status code (default: 200)') + parser.add_argument('--reason', '-r', default='OK', help='HTTP reason phrase (default: OK)') + parser.add_argument('--header', action='append', help='Additional response header (repeatable), e.g. "X-Foo: bar"') + parser.add_argument('--body', '-b', help='Response body string') + parser.add_argument('--body-size', type=int, default=0, help='Generate N bytes of filler body data') + parser.add_argument('--chunked', action='store_true', help='Use chunked transfer encoding') + parser.add_argument('--delay', '-d', type=float, default=0, help='Seconds to delay before sending response') + + args = parser.parse_args() + serve_one(args) + + +if __name__ == '__main__': + main() diff --git a/tools/benchmark/CMakeLists.txt b/tools/benchmark/CMakeLists.txt index 6a062bfaad8..49f25fad1c1 100644 --- a/tools/benchmark/CMakeLists.txt +++ b/tools/benchmark/CMakeLists.txt @@ -35,3 +35,15 @@ target_link_libraries(benchmark_SharedMutex PRIVATE Catch2::Catch2 ts::tscore li add_executable(benchmark_Random benchmark_Random.cc) target_link_libraries(benchmark_Random PRIVATE Catch2::Catch2WithMain ts::tscore) + +add_executable(benchmark_HostDB benchmark_HostDB.cc) +target_link_libraries( + benchmark_HostDB + PRIVATE ts::tscore + ts::tsutil + ts::inkevent + ts::http + ts::http_remap + ts::inkcache + ts::inkhostdb +) diff --git a/src/iocore/hostdb/benchmark_HostDB.cc b/tools/benchmark/benchmark_HostDB.cc similarity index 99% rename from src/iocore/hostdb/benchmark_HostDB.cc rename to tools/benchmark/benchmark_HostDB.cc index d11c5b03ea4..f1f41a1c4dc 100644 --- a/src/iocore/hostdb/benchmark_HostDB.cc +++ b/tools/benchmark/benchmark_HostDB.cc @@ -149,7 +149,7 @@ struct StartDNS : Continuation { HostList::iterator it; ResultList results; Clock::time_point start_time; - bool is_callback; + bool is_callback{false}; StartDNS(const std::vector &hlist, int id, latch &l) : Continuation(new_ProxyMutex()), hostlist(hlist), id(id), done_latch(l) diff --git a/tools/build_boringssl_h3_tools.sh b/tools/build_boringssl_h3_tools.sh index 286e5d15098..d8118e1063a 100755 --- a/tools/build_boringssl_h3_tools.sh +++ b/tools/build_boringssl_h3_tools.sh @@ -112,7 +112,8 @@ else OS="linux" fi -go_version=1.24.12 +go_version=1.26.2 +BORINGSSL_COMMIT=${BORINGSSL_COMMIT:-"c3ffc3300a9450cf8e396c7880be7c6cadc16a4a"} wget https://go.dev/dl/go${go_version}.${OS}-${ARCH}.tar.gz rm -rf ${BASE}/go && tar -C ${BASE} -xf go${go_version}.${OS}-${ARCH}.tar.gz rm go${go_version}.${OS}-${ARCH}.tar.gz @@ -121,7 +122,7 @@ GO_BINARY_PATH=${BASE}/go/bin/go if [ ! -d boringssl ]; then git clone https://boringssl.googlesource.com/boringssl cd boringssl - git checkout 02bc0949e5cac0e1ee82c6f365f5a6c3cfd0cfa9 + git checkout ${BORINGSSL_COMMIT} cd .. fi cd boringssl @@ -179,14 +180,14 @@ echo "Building quiche" QUICHE_BASE="${BASE:-/opt}/quiche" [ ! -d quiche ] && git clone https://github.com/cloudflare/quiche.git cd quiche -git checkout 0.23.2 +git checkout 0.28.0 QUICHE_BSSL_PATH=${BORINGSSL_LIB_PATH} QUICHE_BSSL_LINK_KIND=dylib cargo build -j4 --package quiche --release --features ffi,pkg-config-meta,qlog sudo mkdir -p ${QUICHE_BASE}/lib/pkgconfig sudo mkdir -p ${QUICHE_BASE}/include sudo cp target/release/libquiche.a ${QUICHE_BASE}/lib/ [ -f target/release/libquiche.so ] && sudo cp target/release/libquiche.so ${QUICHE_BASE}/lib/ # Why a link? https://github.com/cloudflare/quiche/issues/1808#issuecomment-2196233378 -sudo ln -s ${QUICHE_BASE}/lib/libquiche.so ${QUICHE_BASE}/lib/libquiche.so.0 +sudo ln -sf ${QUICHE_BASE}/lib/libquiche.so ${QUICHE_BASE}/lib/libquiche.so.0 sudo cp quiche/include/quiche.h ${QUICHE_BASE}/include/ sudo cp target/release/quiche.pc ${QUICHE_BASE}/lib/pkgconfig sudo chmod -R a+rX ${BASE} @@ -196,7 +197,7 @@ LDFLAGS=${LDFLAGS:-"-Wl,-rpath,${BORINGSSL_LIB_PATH}"} # Then nghttp3 echo "Building nghttp3..." -[ ! -d nghttp3 ] && git clone --depth 1 -b v1.8.0 https://github.com/ngtcp2/nghttp3.git +[ ! -d nghttp3 ] && git clone --depth 1 -b v1.15.0 https://github.com/ngtcp2/nghttp3.git cd nghttp3 git submodule update --init autoreconf -if @@ -214,7 +215,7 @@ cd .. # Now ngtcp2 echo "Building ngtcp2..." -[ ! -d ngtcp2 ] && git clone --depth 1 -b v1.11.0 https://github.com/ngtcp2/ngtcp2.git +[ ! -d ngtcp2 ] && git clone --depth 1 -b v1.22.1 https://github.com/ngtcp2/ngtcp2.git cd ngtcp2 autoreconf -if ./configure \ @@ -234,7 +235,7 @@ cd .. # Then nghttp2, with support for H3 echo "Building nghttp2 ..." -[ ! -d nghttp2 ] && git clone --depth 1 -b v1.65.0 https://github.com/tatsuhiro-t/nghttp2.git +[ ! -d nghttp2 ] && git clone --depth 1 -b v1.69.0 https://github.com/nghttp2/nghttp2.git cd nghttp2 git submodule update --init autoreconf -if @@ -264,7 +265,7 @@ cd .. # Then curl echo "Building curl ..." -[ ! -d curl ] && git clone --depth 1 -b curl-8_12_1 https://github.com/curl/curl.git +[ ! -d curl ] && git clone --depth 1 -b curl-8_20_0 https://github.com/curl/curl.git cd curl # On mac autoreconf fails on the first attempt with an issue finding ltmain.sh. # The second runs fine. diff --git a/tools/build_openssl_h3_tools.sh b/tools/build_openssl_h3_tools.sh index 3e22cf69e06..550523e613d 100755 --- a/tools/build_openssl_h3_tools.sh +++ b/tools/build_openssl_h3_tools.sh @@ -28,7 +28,7 @@ readonly WORKDIR cd "${WORKDIR}" # Update this as the draft we support updates. -OPENSSL_BRANCH=${OPENSSL_BRANCH:-"openssl-3.1.4+quic"} +OPENSSL_BRANCH=${OPENSSL_BRANCH:-"openssl-3.1.7+quic"} # Set these, if desired, to change these to your preferred installation # directory @@ -120,7 +120,7 @@ echo "Building quiche" QUICHE_BASE="${BASE:-/opt}/quiche" [ ! -d quiche ] && git clone https://github.com/cloudflare/quiche.git cd quiche -git checkout 0.23.2 +git checkout 0.28.0 PKG_CONFIG_PATH="$OPENSSL_LIB"/pkgconfig LD_LIBRARY_PATH="$OPENSSL_LIB" \ cargo build -j4 --package quiche --release --features ffi,pkg-config-meta,qlog,openssl @@ -130,7 +130,7 @@ sudo mkdir -p ${QUICHE_BASE}/include sudo cp target/release/libquiche.a ${QUICHE_BASE}/lib/ [ -f target/release/libquiche.so ] && sudo cp target/release/libquiche.so ${QUICHE_BASE}/lib/ # Why a link? https://github.com/cloudflare/quiche/issues/1808#issuecomment-2196233378 -sudo ln -s ${QUICHE_BASE}/lib/libquiche.so ${QUICHE_BASE}/lib/libquiche.so.0 +sudo ln -sf ${QUICHE_BASE}/lib/libquiche.so ${QUICHE_BASE}/lib/libquiche.so.0 sudo cp quiche/include/quiche.h ${QUICHE_BASE}/include/ sudo cp target/release/quiche.pc ${QUICHE_BASE}/lib/pkgconfig sudo chmod -R a+rX ${BASE} @@ -139,7 +139,7 @@ cd .. # Then nghttp3 echo "Building nghttp3..." -[ ! -d nghttp3 ] && git clone --depth 1 -b v1.8.0 https://github.com/ngtcp2/nghttp3.git +[ ! -d nghttp3 ] && git clone --depth 1 -b v1.15.0 https://github.com/ngtcp2/nghttp3.git cd nghttp3 git submodule update --init autoreconf -if @@ -157,7 +157,7 @@ cd .. # Now ngtcp2 echo "Building ngtcp2..." -[ ! -d ngtcp2 ] && git clone --depth 1 -b v1.11.0 https://github.com/ngtcp2/ngtcp2.git +[ ! -d ngtcp2 ] && git clone --depth 1 -b v1.22.1 https://github.com/ngtcp2/ngtcp2.git cd ngtcp2 autoreconf -if ./configure \ @@ -174,7 +174,7 @@ cd .. # Then nghttp2, with support for H3 echo "Building nghttp2 ..." -[ ! -d nghttp2 ] && git clone --depth 1 -b v1.65.0 https://github.com/tatsuhiro-t/nghttp2.git +[ ! -d nghttp2 ] && git clone --depth 1 -b v1.69.0 https://github.com/nghttp2/nghttp2.git cd nghttp2 git submodule update --init autoreconf -if @@ -202,7 +202,7 @@ cd .. # Then curl echo "Building curl ..." -[ ! -d curl ] && git clone --depth 1 -b curl-8_12_1 https://github.com/curl/curl.git +[ ! -d curl ] && git clone --depth 1 -b curl-8_20_0 https://github.com/curl/curl.git cd curl # On mac autoreconf fails on the first attempt with an issue finding ltmain.sh. # The second runs fine.