Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,4 @@ pnpm-lock.yaml
.expo

# Data
books.jsonl
books.jsonl
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ This is a monorepo containing multiple standalone projects. Each project lives i
code-samples/
├── typesense-angular-search-bar/ # Angular + Typesense search implementation
├── typesense-astro-search/ # Astro + Typesense search implementation
├── typesense-django-full-text-search/ # Python (Django) + Typesense backend implementation
├── typesense-gin-full-text-search/ # Go (Gin) + Typesense backend implementation
├── typesense-kotlin/ # Kotlin (Android) + Typesense search implementation
├── typesense-langchain-ai-search/ # Python + LangChain + Typesense AI search implementation
Expand All @@ -34,6 +35,7 @@ code-samples/
| ---------------------------------------------------------------------------- | ------------- | --------------------------------------------------------------- |
| [typesense-angular-search-bar](./typesense-angular-search-bar) | Angular | A modern search bar with instant search capabilities |
| [typesense-astro-search](./typesense-astro-search) | Astro | A modern search bar with instant search capabilities |
| [typesense-django-full-text-search](./typesense-django-full-text-search) | Python (Django) | Backend API with full-text search using Typesense |
| [typesense-gin-full-text-search](./typesense-gin-full-text-search) | Go (Gin) | Backend API with full-text search using Typesense |
| [typesense-kotlin](./typesense-kotlin) | Kotlin (Android) | A native Android search bar with instant search capabilities |
| [typesense-langchain-ai-search](./typesense-langchain-ai-search) | LangChain (Python) | Conversational AI-powered movie search and recommendation app |
Expand Down
15 changes: 15 additions & 0 deletions typesense-django-full-text-search/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
PORT=8000

DB_HOST=localhost
DB_USER=postgres
DB_PASSWORD=password
DB_NAME=typesense_books
DB_PORT=5432

TYPESENSE_HOST=localhost
TYPESENSE_PORT=8108
TYPESENSE_PROTOCOL=http
TYPESENSE_API_KEY=xyz

# Django
DJANGO_SECRET_KEY=change-me-in-production
31 changes: 31 additions & 0 deletions typesense-django-full-text-search/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so

# Virtual environments
.venv/
venv/
ENV/

# Django
*.log
db.sqlite3
db.sqlite3-journal
staticfiles/
media/

# Environment
.env

# IDE
.idea/
.vscode/
*.swp
*.swo

# OS
.DS_Store
Thumbs.db

281 changes: 281 additions & 0 deletions typesense-django-full-text-search/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,281 @@
# Django Full-Text Search with Typesense

A production-ready RESTful search API built with Python, Django, PostgreSQL, and Typesense. Features full-text search, CRUD operations, real-time async indexing, soft deletes, paginated sync, and background workers.

## Tech Stack

- Python 3.10+
- Django 5.0+
- PostgreSQL (`psycopg2-binary`)
- Typesense (`typesense-python`)
- APScheduler (background periodic sync)
- Docker

## Prerequisites

- Python 3.10+ installed
- Docker (for Typesense and PostgreSQL)
- Basic knowledge of Python, Django, REST APIs, and SQL

## Quick Start

### 1. Clone the repository

```bash
git clone https://github.com/typesense/code-samples.git
cd typesense-django-full-text-search
```

### 2. Set up virtual environment & install dependencies

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

### 3. Start Typesense and PostgreSQL

Run Typesense and PostgreSQL using Docker:

```bash
# Create local data directory for Typesense
mkdir -p typesense-data

# Start Typesense
docker run -d \
-p 8108:8108 \
-v "$(pwd)"/typesense-data:/data \
typesense/typesense:27.1 \
--data-dir /data \
--api-key=xyz \
--enable-cors

# Start PostgreSQL
docker run -d \
-p 5432:5432 \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=password \
-e POSTGRES_DB=typesense_books \
postgres:15
```

### 4. Set up environment variables

Copy `.env.example` to `.env`:

```bash
cp .env.example .env
```

The `.env` file should look like this:

```env
PORT=8000

# PostgreSQL Configuration
DB_HOST=localhost
DB_USER=postgres
DB_PASSWORD=password
DB_NAME=typesense_books
DB_PORT=5432

# Typesense Configuration
TYPESENSE_HOST=localhost
TYPESENSE_PORT=8108
TYPESENSE_PROTOCOL=http
TYPESENSE_API_KEY=xyz

# Django Configuration
DJANGO_SECRET_KEY=change-me-in-production
```

### 5. Run Database Migrations

```bash
python manage.py migrate
```

### 6. Project Structure

```text
typesense-django-full-text-search/
├── books/
│ ├── search/
│ │ ├── __init__.py # Exports search client and sync functions
│ │ ├── client.py # Typesense Python client initialization
│ │ ├── collections.py # Typesense collection schema definition
│ │ ├── sync.py # Paginated full & incremental sync logic
│ │ └── worker.py # APScheduler background periodic sync worker
│ ├── tests/
│ │ ├── test_models.py # Tests for soft-delete & model managers
│ │ ├── test_sync.py # Tests for document mapping & sync logic
│ │ └── test_views.py # Tests for API endpoints & search proxy
│ ├── apps.py # Starts background sync worker on Django startup
│ ├── models.py # Book Django model with ActiveBookManager
│ ├── urls.py # App routing
│ └── views.py # CRUD & search proxy API views
├── typesensedjango/
│ ├── settings.py # Django project settings
│ └── urls.py # Root URL configuration
├── .env.example
├── manage.py
└── requirements.txt
```

### 7. Start the Development Server

```bash
python manage.py runserver 8000
```

The server starts at `http://localhost:8000`. Upon startup, Django automatically initializes the Typesense collection schema and launches the background periodic sync thread.

### 8. API Endpoints

#### Search

```bash
GET /search/?q=<query>
```

Example:

```bash
curl "http://localhost:8000/search/?q=harry"
```

Response:

```json
{
"query": "harry",
"found": 1,
"results": [...],
"facet_counts": []
}
```

#### CRUD Operations

**List books (Paginated):**

```bash
GET /books/?page=1&limit=10
```

**Create a book:**

```bash
POST /books/
Content-Type: application/json

{
"title": "The Hobbit",
"authors": ["J.R.R. Tolkien"],
"publication_year": 1937,
"average_rating": 4.8,
"image_url": "https://example.com/hobbit.jpg",
"ratings_count": 1250000
}
```

**Get a book:**

```bash
GET /books/:id/
```

**Update a book:**

```bash
PUT /books/:id/
Content-Type: application/json

{
"title": "The Hobbit: There and Back Again",
"average_rating": 4.9
}
```

**Delete a book (soft delete):**

```bash
DELETE /books/:id/
```

Returns `204 No Content` and soft-deletes the row in PostgreSQL while deleting the document from Typesense.

#### Sync Operations

**Trigger manual sync:**

```bash
POST /sync/
```

Response:

```json
{
"message": "Sync completed",
"syncedAt": "2026-07-23T16:30:00+00:00"
}
```

**Check sync status:**

```bash
GET /sync/status/
```

Response:

```json
{
"lastSyncTime": "2026-07-23T16:30:00+00:00",
"syncWorkerRunning": true,
"syncJobActive": false
}
```

### 9. How It Works

#### Architecture

```plaintext
User Request
Django REST Views (CRUD)
PostgreSQL (Source of Truth)
Real-Time / Periodic Sync → Typesense (Search Index)
Background Worker (APScheduler thread running every 60s)
```

#### Sync Strategies

1. **Real-time Sync**: On every POST/PUT/DELETE API request, the change hits PostgreSQL first, and is then immediately mirrored to Typesense.
2. **Periodic Background Sync**: Runs every 60 seconds via `APScheduler` in a thread. Uses `updated_at > last_sync_time` to catch any database changes that happened outside the API or during server downtime.
3. **Startup Sync**: On boot, if Typesense is empty, a full sync runs. If Typesense has documents, an incremental sync runs starting from epoch so missing records are backfilled.
4. **Manual Sync**: `POST /sync/` triggers an on-demand full sync.

#### Memory-Safe Keyset Pagination

Both full sync and incremental sync fetch records from PostgreSQL using keyset pagination (`id__gt=last_id`) in batches of 1,000 to prevent memory allocation spikes when syncing large tables.

### 10. Running Tests

Run the complete test suite using Django's test runner:

```bash
python manage.py test books.tests
```

```text
Ran 19 tests in 0.15s

OK
```
Empty file.
3 changes: 3 additions & 0 deletions typesense-django-full-text-search/books/admin.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from django.contrib import admin

# Register your models here.
41 changes: 41 additions & 0 deletions typesense-django-full-text-search/books/apps.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
import sys
import logging
from django.apps import AppConfig

logger = logging.getLogger(__name__)

class BooksConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'books'

def ready(self):
# Avoid running during management commands (like migrate, makemigrations)
if 'manage.py' in sys.argv and 'runserver' not in sys.argv:
return

from .search import (
initialize_typesense,
determine_and_run_startup_sync,
start_background_sync_worker
)
import threading

def _startup_sequence():
try:
logger.info('Initializing Typesense...')
initialize_typesense()

logger.info('Running startup sync...')
try:
determine_and_run_startup_sync()
except Exception as sync_err:
logger.warning('Startup sync skipped (database might not be migrated yet): %s', sync_err)

start_background_sync_worker()
except Exception as e:
logger.error('Failed to start background sync worker: %s', e)

import os
if os.environ.get('RUN_MAIN') == 'true' or not sys.argv[0].endswith('manage.py'):
thread = threading.Thread(target=_startup_sequence, daemon=True)
thread.start()
Loading