kv-db

A lightweight, persistent key-value database built from scratch in C++, containerized with Docker. This project was built as a self-learning exercise to deeply understand systems programming, networking, persistence, and concurrency in C++.

---

About

kv-db is a Redis-inspired key-value store that supports basic CRUD operations over a TCP connection. It is designed around a phonebook use case where string keys map to string values.

This is a personal learning project. The goal is not to build a production database, but to understand how databases, servers, networking, and concurrency actually work under the hood.

The server is designed to run cross-platform on both Windows and Linux using platform-specific networking abstractions.

---

Features

• Custom HashMap — hand-rolled hash table with separate chaining for collision resolution and dynamic growth, protected by a std::shared_mutex for concurrent read/write safety
• Cross-Platform TCP Server — raw socket server supporting:
  • Windows networking through Winsock2
  • Linux networking through POSIX sockets
• Command Parser — parses INSERT, GET, and DELETE commands from raw TCP bytes
• Thread Pool — fixed pool of 3 worker threads handling client sessions concurrently using condition variables and mutexes
• Snapshot Scheduler — background thread that automatically triggers RDB snapshots on a fixed interval
• Persistence — hybrid durability layer combining:
  • AOF (Append-Only File) — logs every write operation in real time, reset after each snapshot
  • RDB Snapshots — full point-in-time dumps of the database every 5 minutes
• Rate Limiting — per-IP and global request throttling using the Token Bucket algorithm
• Graceful Shutdown — type stop to cleanly flush data and join all threads
• Docker Support — multi-stage containerized build targeting Linux
• Authentication — (in progress)

---

Commands

Connect to the server using ncat or any TCP client:

ncat 127.0.0.1 6625

Command	Description	Example
INSERT key value	Inserts or updates a key-value pair	INSERT Ahmed 51020651
GET key	Retrieves the value for a key	GET Ahmed
DELETE key	Removes a key-value pair	DELETE Ahmed

---

Architecture

Client (ncat / custom client)
        │
        │ TCP
        ▼
 Cross-Platform TCP Server
 (Winsock2 / Linux sockets)
        │
        ├──▶ Rate Limiter (Token Bucket)
        │    per-IP + global request cap
        │         │
        │         ▼
        │    Command Parser
        │         │
        │         ▼
        │    Thread Pool (3 workers)
        │         │
        │         ▼
        │    HashMap (in-memory store)
        │    shared_mutex: readers run concurrently,
        │    writers get exclusive access
        │
        └──▶ Persistence Layer
                  ├── appendonly.log  (real-time AOF log)
                  └── snapshot.log    (periodic RDB snapshot)
                            ▲
                   SnapshotScheduler
                   (background thread, every 5 min)

---

Concurrency Model

Component	Protection	Strategy
HashMap	std::shared_mutex	Multiple readers, exclusive writers
AOF stream	std::mutex	Single writer at a time
Client jobs	ThreadPool + std::condition_variable	Workers sleep until job arrives
Snapshot	SnapshotScheduler thread	Sleeps on interval, wakes on shutdown
Rate limiter	std::mutex per map + global	Per-IP and global window isolated

---

Rate Limiting

kv-db uses the Token Bucket algorithm for rate limiting — the same approach used by Stripe, GitHub, and AWS.

• Each IP gets a bucket of tokens (default: 10)
• Each request consumes one token
• Tokens refill at a fixed rate (default: 5/sec)
• A global cap limits total requests per second across all IPs (default: 1000)
• Blocked requests are dropped immediately without consuming a worker thread

---

Getting Started

Prerequisites

• C++17 compiler
• CMake 3.15+
• Docker (optional)
• nmap/ncat for testing

For local builds:

• Windows: MSVC / MinGW with Winsock2
• Linux: GCC / Clang with POSIX sockets

---

Build Locally

cmake -S . -B build
cmake --build build

Run:

./build/Debug/KV_Database.exe   # Windows
./build/KV_Database             # Linux

The server starts on port 6625 by default. Type stop to shut it down cleanly.

---

Run With Docker

Build the image:

docker build -t kv-db:latest .

Run the container:

docker run -d -p 6625:6625 --name my-kv-store kv-db:latest

Connect using:

ncat 127.0.0.1 6625

---

Project Structure

kv-db/
├── src/
│   ├── main.cpp
│   ├── Server/
│   │   ├── Server.h
│   │   └── Server.cpp
│   ├── RAM/
│   │   ├── HashMap.h
│   │   └── HashMap.cpp
│   ├── Storage/
│   │   ├── Persistence.h
│   │   └── Persistence.cpp
│   ├── Worker/
│   │   ├── ThreadPool.h
│   │   ├── ThreadPool.cpp
│   │   ├── SnapshotScheduler.h
│   │   └── SnapshotScheduler.cpp
│   ├── Limit/
│   │   ├── Limiter.h
│   │   └── Limiter.cpp
│   └── Tests/
│       ├── test_hashmap.cpp
│       └── test_threadpool.cpp
├── Dockerfile
├── .dockerignore
├── CMakeLists.txt
└── README.md

---

Roadmap

• Custom HashMap with separate chaining
• TCP server with Winsock2
• Cross-platform networking support (Windows/Linux)
• Command parser (INSERT, GET, DELETE)
• AOF persistence with real-time logging
• RDB snapshots every 5 minutes
• AOF reset after each snapshot
• Thread pool for concurrent client sessions
• HashMap thread safety with shared_mutex
• Persistence thread safety with mutex
• Snapshot scheduler background thread
• Clean server shutdown
• Data recovery on restart (RDB + AOF replay)
• Unit tests with GoogleTest (HashMap + ThreadPool)
• Docker containerization (multi-stage Linux build)
• Rate limiting with Token Bucket algorithm
• Authentication (username + password)

---

License

This project is licensed under the MIT License. See LICENSE for details.

---

Built from scratch as a self-learning project to understand C++, networking, persistence, and systems programming.