Give it a goal in plain English. It figures out what to search, which pages to scrape, what data to extract, and where to send it.
Most scraping tools require you to know the exact URL, the exact CSS selector, and the exact schema upfront. This platform takes a different approach — you describe what you want in natural language and a chain of agents handles everything else.
Example goals:
Find top 10 Python libraries for data visualization with GitHub stars and last updated dateGet all ML engineer job listings in Bangalore with company name and salaryFind AI startup funding rounds announced this month with amount and investor namesScrape top 5 FastAPI tutorials with URLs and descriptions
The system searches the web, visits relevant pages, reads them, extracts structured data, validates it, and returns clean JSON or CSV — no selectors, no manual configuration.
User Goal
│
▼
┌─────────────┐
│ Planner │ LLM breaks the goal into: search queries, data fields, expected count
└──────┬──────┘
│
▼
┌─────────────┐
│ Search │ DuckDuckGo search returns relevant URLs
└──────┬──────┘
│
▼
┌─────────────┐
│ Scraper │ httpx fetches page content → Playwright fallback for JS-heavy pages
└──────┬──────┘
│
▼
┌─────────────┐
│ Extractor │ LLM reads raw text and pulls out structured JSON matching the schema
└──────┬──────┘
│
▼
┌─────────────┐
│ Validator │ Checks completeness, deduplicates, retries if too few results (max 2x)
└──────┬──────┘
│
▼
Structured Output (JSON / CSV)
│
▼
n8n Webhook (optional) → Email / Sheets / Slack / Database
Each node is a LangGraph state machine node. The validator can loop back to the search node if results are insufficient, creating a self-correcting retry loop.
Agentic pipeline
- LangGraph-based multi-node agent with typed state passing between nodes
- Planner dynamically determines search strategy and output schema from the goal
- Self-correcting retry loop — if extracted results are below the expected count, the agent re-searches with the same queries and merges new results
Intelligent scraping
- Primary scraper uses httpx for fast, lightweight fetching
- Automatic Playwright fallback for pages that block httpx or require JavaScript rendering
- HTML cleaning strips nav, footer, scripts, and ads before passing to the LLM
- Content truncated to 6000 chars per page to stay within LLM context limits
LLM extraction
- Groq (LLaMA 4 Scout) reads cleaned page text and extracts structured JSON
- System prompt dynamically built from the planner's field list
- Handles markdown-wrapped responses, extracts raw JSON arrays robustly
- Falls back gracefully per-URL — one failed page doesn't break the whole run
Workflow automation with n8n
- After any scrape completes, optionally POST results to an n8n webhook
- n8n receives the full payload: goal, result count, and data array
- Connect any downstream action visually: Gmail, Google Sheets, Slack, Postgres, HTTP requests, conditional logic, etc.
- Works for both one-off runs and scheduled monitors
Scheduled monitoring
- Create monitors with a goal + cron expression (e.g.
0 9 * * *= every day at 9am) - APScheduler runs them in the background automatically
- Each monitor run saves to SQLite history and triggers the n8n webhook if configured
- Useful for: job market tracking, competitor monitoring, funding round alerts, price tracking
Run history
- Every run (manual or scheduled) is saved to SQLite with goal, status, result count, timestamp, and full data
- History tab in the UI lets you browse and re-open any past result set
React frontend
- Real-time SSE streaming — UI updates as each agent node completes
- Animated vertical step tracker shows which node is running with a sliding progress bar
- Results open in a glassmorphism modal with JSON and CSV download
- History tab for past runs, Monitors tab to create/delete scheduled jobs
- Optional n8n webhook URL field on the agent form
| Layer | Technology |
|---|---|
| Agent framework | LangGraph |
| LLM | Groq — LLaMA 4 Scout 17B |
| Web search | DuckDuckGo (ddgs) |
| Scraping | httpx + BeautifulSoup4 + Playwright |
| Schema validation | Pydantic v2 |
| Backend API | FastAPI (async) |
| Streaming | Server-Sent Events (SSE) |
| Database | SQLite via aiosqlite |
| Scheduler | APScheduler |
| Workflow automation | n8n (self-hosted via Docker) |
| Frontend | React + Vite |
├── backend/
│ ├── main.py # FastAPI app, all endpoints, lifespan
│ ├── db.py # SQLite helpers (runs, monitors)
│ ├── n8n.py # n8n webhook trigger
│ ├── scheduler.py # APScheduler cron job management
│ ├── requirements.txt
│ ├── .env # GROQ_API_KEY
│ └── agent/
│ ├── graph.py # LangGraph state graph definition
│ ├── nodes.py # planner, search, scrape, extract, validate nodes
│ ├── tools.py # search_web, scrape_url (httpx + Playwright)
│ └── schemas.py # Pydantic models, AgentState
└── frontend/
├── src/
│ ├── App.jsx # Main UI — agent, history, monitors tabs
│ └── index.css
├── index.html
├── vite.config.js
└── package.json
- Python 3.11+
- Node.js 18+
- A free Groq API key from console.groq.com
- Docker (optional, for n8n)
cd backend
python -m venv venv
venv\Scripts\activate # Windows
# source venv/bin/activate # Mac/Linux
pip install -r requirements.txt
playwright install chromium
cp .env.example .env
# Add your GROQ_API_KEY to .env
uvicorn main:app --reloadcd frontend
npm install
npm run devdocker run -it --rm -p 5678:5678 -v n8n_data:/home/node/.n8n n8nio/n8nOpen http://localhost:5678, create a Webhook node, copy the URL, paste it into the app.
| Method | Endpoint | Description |
|---|---|---|
| POST | /scrape |
Run agent, return results |
| POST | /scrape/stream |
Run agent, stream SSE progress |
| GET | /runs |
List past runs |
| GET | /runs/{id} |
Get full run data |
| POST | /monitors |
Create a scheduled monitor |
| GET | /monitors |
List monitors |
| DELETE | /monitors/{id} |
Delete a monitor |
| GET | /health |
Health check |
{
"goal": "Find top 10 Python libraries for data visualization with GitHub stars",
"n8n_webhook": "http://localhost:5678/webhook-test/abc123"
}{
"run_id": 1,
"goal": "Find top 10 Python libraries for data visualization with GitHub stars",
"result_count": 10,
"data": [
{ "name": "Matplotlib", "github_stars": 20000, "last_updated": "2024-01-01" },
...
]
}Once the webhook receives data, you can chain any of these in n8n:
- Google Sheets — append each result as a new row
- Gmail / SMTP — email yourself when new results arrive
- Slack — post a summary message to a channel
- IF node — only trigger if
result_count > 0or a specific field changed - Postgres / MySQL — insert rows into a database
- Compare datasets — diff against previous run, alert only on new items
- HTTP Request — forward to any other API or webhook
| Goal | Use case |
|---|---|
| ML engineer jobs in Bangalore | Daily job market monitoring |
| AI startup funding rounds this month | Investor/competitor intelligence |
| Top Python libraries by GitHub stars | Tech landscape research |
| FastAPI tutorials with URLs | Content aggregation |
| Product prices on e-commerce sites | Price alert monitoring |