Integration test

.github/copilot-instructions.md

@@ -0,0 +1,46 @@
+Copilot instructions for EventGate
+
+Purpose
+AWS Lambda event gateway that receives messages via API Gateway and dispatches them to Kafka, EventBridge, and PostgreSQL.
+
+Structure
+- Entry point: `src/event_gate_lambda.py`
+- Handlers: `src/handlers/` (HandlerApi, HandlerToken, HandlerTopic, HandlerHealth)
+- Writers: `src/writers/` (inherit from `Writer` base class)
+- Config: `conf/config.json`, `conf/access.json`, `conf/topic_schemas/*.json`
+- Production Terraform scripts are not part of this repository; `terraform_examples/` for reference configurations only
+
+Python style
+- Python 3.13
+- Type hints for public functions and classes
+- Use `logging.getLogger(__name__)`, not print
+- Lazy % formatting in logging: `logger.info("msg %s", var)`
+- F-strings in exceptions: `raise ValueError(f"Error {var}")`
+- All imports at top of file (never inside functions)
+- Apache 2.0 license header in every .py file (including `__init__.py`)
+- Docstrings must start with a short summary line
+- End all log messages with a period: `logger.info("Message.")`
+
+Patterns
+- `__init__` methods must not raise exceptions; defer validation and connection to first use (lazy init)
+- Writers: inherit from `Writer(ABC)`, implement `write(topic, message) -> (bool, str|None)` and `check_health() -> (bool, str)`
+- Route dispatch via `ROUTE_MAP` dict mapping routes to handler functions in `event_gate_lambda.py`
+- Separate business logic from environment access (env vars, file I/O, network calls)
+- No duplicate validation; centralize parsing in one layer where practical
+- Preserve existing formatting and conventions
+- Keep API Gateway response structure stable: `{"statusCode": int, "headers": {...}, "body": "..."}`
+- Keep error response format stable: `{"success": false, "statusCode": int, "errors": [...]}`
+
+Testing
+- Mirror src structure: `src/handlers/` -> `tests/handlers/`
+- Unit tests: mock external services via `conftest.py` (Kafka, EventBridge, PostgreSQL, S3)
+- Integration tests: call `lambda_handler` directly with real containers (testcontainers-python for Kafka, PostgreSQL, LocalStack)
+- No real API/DB calls in unit tests
+- Use `mocker.patch("module.dependency")` or `mocker.patch.object(Class, "method")`
+- Assert pattern: `assert expected == actual`
+
+Quality gates (run after changes, fix only if below threshold)
+- black .
+- mypy .
+- pylint $(git ls-files '*.py') >= 9.5
+- pytest tests/ >= 80% coverage

.github/workflows/check_python.yml

@@ -111,7 +111,32 @@ jobs:
         id: check-format
         run: black --check $(git ls-files '*.py')
 
-  pytest-test:
+  mypy-check:
+    name: Mypy Type Check
+    needs: detect
+    if: needs.detect.outputs.python_changed == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548
+        with:
+          python-version: '3.13'
+          cache: 'pip'
+
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+
+      - name: Check types with Mypy
+        id: check-types
+        run: mypy .
+
+  unit-tests:
     name: Pytest Unit Tests with Coverage
     needs: detect
     if: needs.detect.outputs.python_changed == 'true'
@@ -123,7 +148,8 @@ jobs:
           persist-credentials: false
           fetch-depth: 0
 
-      - uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548
+      - name: Set up Python
+        uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548
         with:
           python-version: '3.13'
           cache: 'pip'
@@ -132,13 +158,14 @@ jobs:
         run: pip install -r requirements.txt
 
       - name: Check code coverage with Pytest
-        run: pytest --cov=. -v tests/ --cov-fail-under=80
+        run: pytest --cov=. -v tests/unit/ --cov-fail-under=80
 
-  mypy-check:
-    name: Mypy Type Check
+  integration-tests:
+    name: Pytest Integration Tests
     needs: detect
     if: needs.detect.outputs.python_changed == 'true'
     runs-on: ubuntu-latest
+    timeout-minutes: 15
     steps:
       - name: Checkout repository
         uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
@@ -155,9 +182,8 @@ jobs:
       - name: Install dependencies
         run: pip install -r requirements.txt
 
-      - name: Check types with Mypy
-        id: check-types
-        run: mypy .
+      - name: Run integration tests
+        run: pytest tests/integration/ -v --tb=short --log-cli-level=INFO
 
   noop:
     name: No Operation

DEVELOPER.md

@@ -5,8 +5,9 @@
 - [Run Pylint Tool Locally](#run-pylint-tool-locally)
 - [Run Black Tool Locally](#run-black-tool-locally)
 - [Run mypy Tool Locally](#run-mypy-tool-locally)
-- [Run Unit Test](#running-unit-test)
+- [Run Unit Test Locally](#run-unit-test-locally)
 - [Code Coverage](#code-coverage)
+- [Run Integration Test Locally](#run-integration-test-locally)
 
 ## Get Started
 
@@ -45,7 +46,7 @@ To run Pylint on a specific file, follow the pattern `pylint <path_to_file>/<nam
 
 Example:
 ```shell
-pylint src/writer_kafka.py
+pylint src/event_gate_lambda.py
 ``` 
 
 ## Run Black Tool Locally
@@ -68,7 +69,7 @@ To run Black on a specific file, follow the pattern `black <path_to_file>/<name_
 
 Example:
 ```shell
-black src/writer_kafka.py
+black src/writers/writer_kafka.py
 ``` 
 
 ### Expected Output
@@ -100,39 +101,39 @@ To run mypy on a specific file, follow the pattern `mypy <path_to_file>/<name_of
 
 Example:
 ```shell
-mypy src/writer_kafka.py
+mypy src/handlers/handler_token.py
 ```
 
-## Running Unit Test
+## Run Unit Test Locally
 
 Unit tests are written using pytest. To run the tests, use the following command:
 
 ```shell
-pytest tests/
+pytest tests/unit/
 ```
 
 This will execute all tests located in the tests directory.
 
 ### Focused / Selective Test Runs
 Run a single test file:
 ```shell
-pytest tests/test_writer_kafka.py -q
+pytest tests/unit/writers/test_writer_kafka.py
 ```
 Filter by keyword expression:
 ```shell
-pytest -k kafka -q
+pytest -k kafka
 ```
 Run a single test function (node id):
 ```shell
-pytest tests/test_event_gate_lambda.py::test_post_multiple_writer_failures -q
+pytest tests/unit/writers/test_writer_eventbridge.py::test_write_success
 ```
 
 ## Code Coverage
 
 Code coverage is collected using the pytest-cov coverage tool. To run the tests and collect coverage information, use the following command:
 
 ```shell
-pytest --cov=. -v tests/ --cov-fail-under=80 --cov-report=term-missing
+pytest --cov=. -v tests/unit/ --cov-fail-under=80 --cov-report=html
 ```
 
 This will execute all tests in the tests directory and generate a code coverage report with missing line details and enforce a minimum 80% threshold.
@@ -141,3 +142,65 @@ Open the HTML coverage report:
 ```shell
 open htmlcov/index.html
 ```
+
+## Run Integration Test Locally
+
+Integration tests validate EventGate against real service dependencies using testcontainers-python.
+
+### Integration Test Approach
+
+EventGate uses a **direct invocation approach** for integration testing:
+- **Lambda handler is called directly** in Python (not run in a container)
+- **External dependencies run in Docker containers**: Kafka, PostgreSQL, LocalStack (EventBridge)
+- **Mock JWT provider runs in-process** as a background thread (no container)
+- Test configuration is dynamically generated and injected via environment variables
+
+This approach is faster (~8s vs 30s+), more reliable, and easier to debug than container-based Lambda testing.
+
+### Prerequisites
+- Docker running (Docker Desktop on macOS/Windows, or Docker Engine on Linux)
+- Python 3.13 with dependencies installed
+
+### Run Integration Tests
+
+Containers start and stop automatically:
+```shell
+pytest tests/integration/ -v
+```
+
+With detailed logging:
+```shell
+pytest tests/integration/ -v --log-cli-level=INFO
+```
+
+### Run Specific Integration Tests
+
+Run a single test file:
+```shell
+pytest tests/integration/test_health_endpoint.py -v
+```
+
+Run a specific test function:
+```shell
+pytest tests/integration/test_topics_endpoint.py::TestPostEventEndpoint::test_post_event_with_valid_token_returns_202 -v
+```
+
+### Troubleshooting
+
+If containers fail to start, check Docker is running:
+```shell
+docker info
+```
+
+If image pulls fail with TLS or timeout errors, pre-pull the required images manually:
+```shell
+docker pull testcontainers/ryuk:0.8.1
+docker pull postgres:16
+docker pull confluentinc/cp-kafka:7.6.0
+docker pull localstack/localstack:latest
+```
+
+View container logs in pytest output by increasing log level:
+```shell
+pytest tests/integration/ -v --log-cli-level=DEBUG
+```

README.md

@@ -146,10 +146,9 @@ Use when Kafka access needs Kerberos / SASL_SSL or custom `librdkafka` build.
 | Static code analysis (Pylint) | [Run Pylint Tool Locally](./DEVELOPER.md#run-pylint-tool-locally)     |
 | Formatting (Black)            | [Run Black Tool Locally](./DEVELOPER.md#run-black-tool-locally)       |
 | Type checking (mypy)          | [Run mypy Tool Locally](./DEVELOPER.md#run-mypy-tool-locally)         |
-| Terraform Linter (TFLint)     | [Run TFLint Tool Locally](./DEVELOPER.md#run-tflint-tool-locally)     |
-| Security Scanner (Trivy)      | [Run Trivy Tool Locally](./DEVELOPER.md#run-trivy-tool-locally)       |
-| Unit tests                    | [Running Unit Test](./DEVELOPER.md#running-unit-test)                 |
+| Unit tests                    | [Run Unit Test Locally](./DEVELOPER.md#run-unit-test-locally)         |
 | Code coverage                 | [Code Coverage](./DEVELOPER.md#code-coverage)                         |
+| Integration tests             | [Run Integration Test Locally](./DEVELOPER.md#run-integration-test-locally) |
 
 ## Security & Authorization
 - JWT tokens must be RS256 signed; current and previous public keys are fetched at cold start from `token_public_keys_url` as DER base64 values (list `keys[*].key`, with single-key fallback `{ "key": "..." }`).

conf/access.json

@@ -7,6 +7,7 @@
         "BarUser"
     ],
     "public.cps.za.test": [
-        "TestUser"
+        "TestUser",
+        "IntegrationTestUser"
     ]
 }

requirements.txt

@@ -1,4 +1,4 @@
-pytest==8.4.2
+pytest==9.0.2
 pytest-cov==6.3.0
 pytest-mock==3.15.0
 pylint==3.3.8
@@ -12,5 +12,7 @@ PyJWT==2.10.1
 requests==2.32.5
 boto3==1.40.25
 confluent-kafka==2.12.1
+testcontainers==4.9.0
+docker==7.1.0
 # psycopg2-binary==2.9.10 # Ideal for local development, but not for long-term production use
 psycopg2==2.9.10