47 lines (35 loc) · 2.5 KB

SQD Data Processing Crates

This repo contains Rust crates for blockchain data processing, indexing, and querying. They are used in various SQD components.

Crate Overview

Core Primitives & Types

Crate	Description
`sqd-primitives`	Basic types, range types, and SID (sequence ID) primitives used throughout the system
`sqd-dataset`	Dataset descriptions, table schemas, and schema options for defining data structures
`sqd-array`	Arrow array operations including building, slicing, sorting, I/O, and chunking utilities
`sqd-bloom-filter`	Bloom filter implementation for arrow buffers

Data Models & Processing

Crate	Description
`sqd-data`	Data models for various blockchain protocols. Used in the query engine and parquet writers
`sqd-data-core`	Core data processing utilities: chunk building, table processing, serialization, and sorting algorithms
`sqd-data-client`	HTTP client for fetching data from SQD data sources with streaming support
`sqd-data-source`	Abstractions for standard and mapped data sources

Query & Storage

Crate	Description
`sqd-query`	Query engine used by the Worker
`sqd-query-example`	Example demonstrating how to use the query engine with Parquet files
`sqd-storage`	RocksDB-based storage layer with key-value store and table management
`sqd-polars`	Polars DataFrame integration for high-performance data analysis

Services & Ingestion

Crate	Description
`sqd-archive`	Archive service for ingesting and storing data to S3 with layout management, progress tracking, and Prometheus metrics
`sqd-bds`	Big Data Service (WIP) - Cassandra-based data storage
`sqd-hotblocks`	Hotblocks database with portal-like API

Building Docker image

The "docker" workflow should be triggered manually with the following inputs:

target — either hotblocks or sqd-archive.
tag — a 8-byte hash of the commit. This will be the published docker tag.
platforms — platforms to build for. Using only linux/amd64 instead of default values can save a lot of building time.