Detection Engineering Lab

A small runnable detection engineering lab that analyzes lab-generated security events with Sigma-style rules, Splunk SPL equivalents, MITRE ATT&CK mappings, and a Python-based detection engine.

Overview

Detection Engineering Lab is a pull-and-run project for testing simple detection logic against small, lab-generated datasets.

The lab includes sample Windows Security, Sysmon, DNS, and email events. It loads Sigma-style YAML rules, matches them against the events, and generates a SOC-style findings report in Markdown.

The project is intentionally lightweight so the detection logic, event fields, and matching behavior are easy to inspect.

Quick Start

Docker

docker compose up --build

Local Python

python -m pip install -r requirements.txt
python src/main.py

Interactive Dataset Selection

python src/main.py --interactive

Tests

python -m pytest -q

Architecture

detection-engineering-lab/
|-- datasets/
|   |-- windows/
|   |-- network/
|   |-- email/
|   `-- custom/
|-- rules/
|   |-- sigma-style/
|   `-- splunk/
|-- src/
|-- reports/
|-- tests/
|-- Dockerfile
|-- docker-compose.yml
`-- requirements.txt

• datasets/ contains lab-generated security events.
• rules/sigma-style/ contains YAML detection rules.
• rules/splunk/ contains equivalent Splunk SPL searches.
• src/ contains the rule loader, dataset loader, normalizer, detection engine, and report writer.
• reports/ contains the generated findings report.
• tests/ contains pytest coverage for rule loading, normalization, and detection output.

Included Detection Rules

Detection	Data Source	Severity	MITRE ATT&CK
Suspicious PowerShell Encoded Command	Sysmon process creation	High	T1059.001
New Local User Created	Windows Security Event 4720	Medium	T1136
Robocopy Data Staging	Sysmon process creation	Medium	T1074
nslookup Spawned by PowerShell	Sysmon process creation	Medium	T1059, T1071.004
Scheduled Task Persistence	Sysmon process creation	High	T1053.005
Office Application Spawning PowerShell	Sysmon process creation	High	T1204, T1059.001

How It Works

The main workflow is:

Load dataset files
        |
        v
Normalize common event fields
        |
        v
Load Sigma-style rules
        |
        v
Match rules against events
        |
        v
Print detections and write reports/findings.md

The detection engine supports simple matching behavior:

• exact field matching
• case-insensitive string matching
• keyword matching in command lines
• parent-child process relationships
• Windows EventCode matching

Using Your Own Dataset

Custom datasets can be placed under:

datasets/custom/

Then run:

python src/main.py --dataset datasets/custom

To analyze one specific file:

python src/main.py --dataset datasets/custom/my-file.jsonl

JSONL is recommended, with one JSON event per line. Standard JSON files may contain either one event object or a list of events.

The engine normalizes common field names from internal lab events, Elastic/ECS-style events, Splunk JSON exports, and Sysmon-style JSON.

Common canonical fields used by the rules include:

EventCode
host
user.name
process.name
process.command_line
process.parent.name
dns.question.name
message

Example Elastic/ECS-style JSONL event:

{"@timestamp":"2026-01-15T10:00:00Z","host":{"name":"ECS-LAB-01"},"event":{"code":"1"},"user":{"name":"lab_user"},"process":{"name":"powershell.exe","command_line":"powershell.exe -EncodedCommand RkFLRQ==","parent":{"name":"cmd.exe"}}}

Example Sysmon-style JSONL event:

{"UtcTime":"2026-01-15T10:05:00Z","Computer":"SYSMON-LAB-01","EventID":1,"User":"lab_user","Image":"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe","CommandLine":"powershell.exe -EncodedCommand RkFLRQ==","ParentImage":"C:\\Windows\\System32\\cmd.exe"}

Interactive Dataset Selection

Interactive mode lists available JSON and JSONL files and lets the user choose which files to analyze:

python src/main.py --interactive

To list files only under datasets/custom/:

python src/main.py --dataset datasets/custom --interactive

The selector supports:

Enter       use the default dataset path
1           analyze one file
1,2,4       analyze multiple files
all         analyze all listed files
q           quit

Invalid input is handled without a Python traceback.

How to Run Locally on Windows PowerShell

python -m venv .venv
.venv\Scripts\activate
python -m pip install -r requirements.txt
python src/main.py

How to Run Locally on Linux/macOS

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python src/main.py

Run Tests

python -m pytest -q

Example Output

Detection Engineering Lab

Dataset path: datasets
Rules loaded: 6
Logs analyzed: 20
Detections triggered: 6

[HIGH] Suspicious PowerShell Encoded Command
Host: LAB-WIN-01
User: test_admin
MITRE: attack.t1059.001
Evidence: powershell.exe -NoProfile -EncodedCommand RkFLRV9MQUJfUEFZTE9BRA==

Report Output

Each run writes a Markdown report to:

reports/findings.md

The report includes:

• total logs analyzed
• total detections triggered
• severity
• host
• user
• source file
• MITRE ATT&CK tags
• evidence
• recommended response steps

Technical Areas Covered

This lab touches several practical blue-team engineering areas:

• detection rule writing
• MITRE ATT&CK mapping
• Splunk SPL translation
• Windows Security Event and Sysmon-style log analysis
• Python-based log processing
• event normalization
• Dockerized execution
• pytest-based validation
• Markdown findings reporting

Limitations

• This is a lab project, not a production SIEM.
• The rules are intentionally simplified for readability.
• Matching logic is designed for small JSON/JSONL datasets.
• External datasets with uncommon fields may require additional normalization.
• The sample logs are lab-generated and should not be treated as real incident data.

Disclaimer

All logs, users, hosts, domains, IP addresses, and findings are lab-generated placeholders. This repository does not contain real customer data, real company data, real malware, or real attack infrastructure.