Mini Search Engine

A compact search engine prototype with crawling, indexing, ranking, and a small API + UI for experimentation.

Key features

• Full-text search with TF-IDF and optional BM25-style ranking and tuning
• Phrase search and exact-match queries
• Metadata filtering (search by document fields)
• Synonyms support and tuning for query expansion
• Result caching (local persistent cache under data/)
• Rate limiting (optional Redis-backed limiter via REDIS_URL)
• Clustering / result grouping for exploratory views
• Ranking benchmark tools for tuning and evaluation (tools/ranking_benchmark.py)
• Small web UI and FastAPI endpoints for programmatic access

Quick start

1. Create and activate a virtual environment and install dependencies:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt

2. Crawl and build the index (creates data/index.pkl and search cache files):

python -m src.crawl_and_index

3. Start the API server:

uvicorn src.api:app --reload --port 8000

4. Open the web UI at web/index.html or call the API at http://localhost:8000/search.

Running tests

pytest -q

Tools

• Run ranking experiments: python tools/ranking_benchmark.py
• Quick smoke check: python tools/smoke_check.py

Docker

docker build -t mini-search .
docker run -p 8000:8000 mini-search

Files of interest

• src/crawler.py
• src/indexer.py
• src/crawl_and_index.py
• src/api.py
• tools/ranking_benchmark.py
• web/index.html
• requirements.txt

Notes & next steps

• Improve ranking (BM25 parameters, learning-to-rank experiments)
• Add pagination, richer web UI, and CI for tests/benchmarks
• Populate data/ with a representative corpus for benchmarking