diff --git a/README.md b/README.md index 44f2538..926798b 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,21 @@ # Sample Sender with Multiple Hosts for HA Ingestion +## Client implementations + +This project is a QuestDB high-availability sender. The **Java** implementation in this +repository is the **reference**. It is ported, in parity, to the three clients built on the +same C/Rust core (the "CRusty" clients): + +- [`rust/`](./rust) - Rust port. +- [`python/`](./python) - Python port, plus a pandas/polars ingestion and egress demo. +- [`c/`](./c) - C/C++ port. + +Each mirrors the Java design (CSV replay loop, per-worker senders, the `qwp` / `qwpudp` / +`ilp` transports, failover, store-and-forward, and the probe). Per-language build steps and +client differences (for example, auto-flush is present in Java and Python but not in Rust or +C/C++) are documented in that folder's `README.md`. The rest of this document describes the +reference Java implementation. + ## Compile `mvn -DskipTests clean package` @@ -46,6 +62,98 @@ java -jar target/ilp_sender-1.0-SNAPSHOT.jar \ QWP. That is a different timeout from `--connect-timeout-ms` (below), which bounds a single connect attempt. +## Pacing: `--delay-ms` vs `--rate` + +Two mutually exclusive ways to throttle generation: + +- `--delay-ms` (default `50`): a fixed `Thread.sleep(delay)` after **every row**, per worker. + Simple, but a poor throttle at speed: `sleep()` has millisecond granularity (and often + over-sleeps), and the fixed per-row cost caps throughput. `--delay-ms 1` yields well under + 1000 rows/s; `--delay-ms 0` runs flat out (millions/s). +- `--rate` (default `0` = off): a **target aggregate rate in rows/second across all workers**. + Each worker paces itself to its share (`rate / num-senders`) against a deadline schedule, + sending rows back-to-back and sleeping only when it runs *ahead* of schedule (coalescing many + rows into one sleep). This hits high targets a per-row delay never could: `--rate 300000` + holds ~300k rows/s regardless of `--num-senders`. + +When `--rate > 0` it **takes precedence** and `--delay-ms` is ignored (a warning is printed if +both are set). Measured on a local QuestDB: `--rate 5000` → 4.03 s for 20k rows (~4966/s); +`--rate 300000` → 2.03 s for 600k rows (~296k/s), steady per-second progress at the target. + +### Regenerating the replay CSV + +Two scripts regenerate the replay CSV. **Both always export from the public demo box** +(`https://demo.questdb.io/exp`) on purpose: the demo is *continuously ingesting live market +data*, so every export is a fresh, recent-price snapshot rather than a stale file. You then +**recreate the table locally** from that CSV and replay it for internal demos. The sender +always writes into a table named **`trades`** locally, regardless of which demo table the CSV +came from. + +- **`regenerate_csv.sh`** — pulls the crypto **`trades`** table (BTC-USDT, ...). Its + timestamp is microseconds. + ``` + ./regenerate_csv.sh # -> trades.csv, 1,000,000 most-recent rows + ./regenerate_csv.sh trades.csv.gz # gzip output (matches the --csv default's .gz) + ./regenerate_csv.sh trades.csv 250000 # custom row count + ``` +- **`regenerate_csv_fx.sh`** — pulls the FX **`fx_trades`** table (EURCHF, AUDNZD, ...). + Two schema differences from the crypto table, both handled in the query + `select timestamp, symbol, side, price, quantity as amount from fx_trades order by timestamp desc limit N`: + its size column is **`quantity`**, aliased to **`amount`** so it matches the `trades` schema + the sender writes; and its designated timestamp is **`TIMESTAMP_NS`** (nanoseconds), not + micros (see [Nanosecond vs microsecond timestamps](#nanosecond-vs-microsecond-timestamps)). + ``` + ./regenerate_csv_fx.sh # -> fx_trades.csv, 1,000,000 rows + ./regenerate_csv_fx.sh fx_trades.csv.gz # gzipped + ./regenerate_csv_fx.sh fx_trades.csv 250000 # custom row count + ``` + +Both export exactly `symbol, side, price, amount, timestamp` (the columns the sender reads), +download to a temp file first so a failed fetch never clobbers a good CSV, and gzip-compress +when the output path ends in `.gz` (the loader auto-detects `.gz`). Override `DEMO_HOST` only if +you mirror the demo tables elsewhere. + +### Nanosecond vs microsecond timestamps + +The crypto `trades` table exports microsecond timestamps; the FX `fx_trades` table exports +**nanosecond** timestamps (`TIMESTAMP_NS`, e.g. `...192508297Z`). This only ever matters with +`--timestamp-from-file`; in the default replay mode the sender stamps `now()` and ignores the +file timestamp entirely, so precision is irrelevant. + +**The generator sends timestamps at nanosecond resolution** (via `at(long, NANOS)`), and +QuestDB stores them at the **target column's** resolution: a micros `TIMESTAMP` column silently +truncates the extra digits, a `TIMESTAMP_NS` column keeps them. Cross-resolution replay is +therefore safe in both directions and lands every row. Validated against a local QuestDB, 20k +crypto + 20k FX rows each way (all 40k landing, no errors): + +| Local `trades` column | Replayed source | Stored as | +| --- | --- | --- | +| `timestamp` (micros) | crypto micros `...790999Z` | `...790999Z` | +| `timestamp` (micros) | FX nanos `...192508297Z` | `...192508Z` (truncated to micros) | +| `timestamp_ns` (nanos) | crypto micros `...790999Z` | `...790999000Z` | +| `timestamp_ns` (nanos) | FX nanos `...192508297Z` | `...192508297Z` (**full nanos preserved**) | + +So if your local table is `TIMESTAMP_NS`, the FX source's nanosecond precision is preserved +end-to-end; if it is micros, the sub-microsecond digits are truncated on store (lossless to the +column's resolution, never an error). To recreate the local table at a given resolution, +pre-create it before ingesting, e.g.: + +```sql +-- microsecond trades table +create table trades (symbol symbol, side symbol, price double, amount double, + trade_id varchar, timestamp timestamp) timestamp(timestamp) partition by day wal; + +-- nanosecond trades table (swap the timestamp type) +create table trades (symbol symbol, side symbol, price double, amount double, + trade_id varchar, timestamp timestamp_ns) timestamp(timestamp) partition by day wal; +``` + +**If the table does not exist, the server auto-creates it as `TIMESTAMP_NS`** (nanosecond), +because the generator sends nanosecond timestamps: the designated timestamp column is created +at the resolution of the incoming value. Verified end-to-end: dropping `trades` and replaying +the FX chunk with `--timestamp-from-file` recreated it with a `TIMESTAMP_NS` column holding the +full `...192508297Z` value. Pre-create the table (above) if you want micros instead. + ## Transport (`--protocol`) - `--protocol qwp` (default): QWP over WebSocket. Adds store-and-forward (un-acked diff --git a/boost_tcp.sh b/boost_tcp.sh new file mode 100755 index 0000000..4355344 --- /dev/null +++ b/boost_tcp.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# boost_tcp.sh - lift the socket-buffer clamp for QuestDB QWP ingestion. +# +# The Rust/C/Python QWP clients hardcode a 4 MiB socket buffer, which default +# Linux silently clamps to ~416 KB (net.core.wmem_max) AND disables autotuning, +# pinning each connection at ~426 KB per RTT. On a real network this caps +# throughput hard (e.g. ~210k rows/s at 20 ms RTT). Raising the max buffers lets +# the client's 4 MiB request through. Apply on BOTH the sender and server boxes. +# +# SAFE any way you run it: `sudo bash boost_tcp.sh`, `./boost_tcp.sh`, +# `. boost_tcp.sh`, `source boost_tcp.sh`. No set -e, no exec, no exit -> it can +# never kill or exit your shell. Runtime-only: values reset on reboot, so run it +# once per boot (or persist via /etc/sysctl.d/ - see README). +boost_tcp() { + local S="" + [ "$(id -u)" -ne 0 ] && S="sudo" + $S sysctl -w net.core.rmem_max=8388608 + $S sysctl -w net.core.wmem_max=8388608 + $S sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216" + $S sysctl -w net.ipv4.tcp_rmem="4096 65536 16777216" + $S sysctl -w net.core.somaxconn=32768 + $S sysctl -w net.core.netdev_max_backlog=65536 + echo "boost_tcp: applied (holds until reboot)" + $S sysctl net.core.wmem_max net.core.rmem_max +} +boost_tcp diff --git a/c/.gitignore b/c/.gitignore new file mode 100644 index 0000000..796b96d --- /dev/null +++ b/c/.gitignore @@ -0,0 +1 @@ +/build diff --git a/c/CMakeLists.txt b/c/CMakeLists.txt new file mode 100644 index 0000000..72239d2 --- /dev/null +++ b/c/CMakeLists.txt @@ -0,0 +1,30 @@ +cmake_minimum_required(VERSION 3.15) +project(questdb_ha_sender_cpp CXX) + +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +if(NOT CMAKE_BUILD_TYPE) + set(CMAKE_BUILD_TYPE Release) +endif() + +# Path to a c-questdb-client checkout (v7.0.0+ / main, with QWP + egress). +# Override with -DQUESTDB_CLIENT_DIR=/path/to/c-questdb-client. +set(QUESTDB_CLIENT_DIR "/Users/j/prj/questdb/c-questdb-client" + CACHE PATH "c-questdb-client checkout") + +# The egress reader (query client / probe) needs the reader surface, and the probe +# uses tls_verify=unsafe_off; enable both cargo features via the client's CMake options. +set(QUESTDB_ENABLE_READER ON CACHE BOOL "" FORCE) +set(QUESTDB_ENABLE_INSECURE_SKIP_VERIFY ON CACHE BOOL "" FORCE) + +add_subdirectory(${QUESTDB_CLIENT_DIR} ${CMAKE_BINARY_DIR}/questdb_client_build) + +find_package(ZLIB REQUIRED) + +add_executable(csv_parallel_sender csv_parallel_sender.cpp) +target_include_directories(csv_parallel_sender PRIVATE ${QUESTDB_CLIENT_DIR}/include) +target_link_libraries(csv_parallel_sender PRIVATE questdb_client ZLIB::ZLIB) + +find_package(Threads REQUIRED) +target_link_libraries(csv_parallel_sender PRIVATE Threads::Threads) diff --git a/c/README.md b/c/README.md new file mode 100644 index 0000000..18e3dcb --- /dev/null +++ b/c/README.md @@ -0,0 +1,119 @@ +# C++ HA sender + +A C++ port of the Java/Rust/Python `CsvParallelSender`. It replays a CSV of trades in a loop +across N worker threads over QuestDB's QWP (WebSocket/UDP) and ILP/HTTP transports, with a +probe that reads back the latest ingested timestamp and the serving node's live role. + +Built on the C/C++ client headers from +[`c-questdb-client`](https://github.com/questdb/c-questdb-client) (`line_sender.hpp` for +ingestion, `line_reader.hpp` for the egress query client) — the same C/Rust core the Rust and +Python ports use. C++17. + +## Build + +Requires a `c-questdb-client` checkout (v7.0.0+ / `main`, which has QWP + the egress reader), a +Rust toolchain (`cargo`, pulled in via the bundled Corrosion), a C++17 compiler, and zlib. The +project builds the client as a CMake subdirectory (Corrosion compiles the Rust FFI), so point it +at your checkout: + +```bash +cmake -B build -S . -DQUESTDB_CLIENT_DIR=/path/to/c-questdb-client +cmake --build build -j +``` + +`QUESTDB_CLIENT_DIR` defaults to `~/prj/questdb/c-questdb-client`. The CMake forces +`QUESTDB_ENABLE_READER=ON` (the query client / probe) and `QUESTDB_ENABLE_INSECURE_SKIP_VERIFY=ON` +(`tls_verify=unsafe_off`). The binary is `build/csv_parallel_sender`. The QWP wire protocol must +match the target server build. + +## Transports (`--protocol`) + +- `qwp` (default): QWP over WebSocket. Per-worker store-and-forward, transactional commit, + multi-host failover, and the probe. Use the WebSocket/HTTP port (`:9000`). +- `qwpudp`: QWP over UDP, fire-and-forget datagrams to `:9007`. Ingest-only, unauthenticated, + single-endpoint (no failover), best-effort. +- `ilp`: the legacy ILP/HTTP transport. + +## Run + +```bash +./build/csv_parallel_sender \ + --protocol qwp \ + --addrs localhost:9000 \ + --total-events 100000 \ + --num-senders 4 \ + --delay-ms 0 \ + --batch-size 10000 \ + --batches-per-transaction 10 \ + --csv ../trades20250728.csv.gz +``` + +Flags mirror the other ports: `--addrs`, `--token`/`--username`/`--password`, `--total-events`, +`--delay-ms`, `--rate`, `--num-senders`, `--retry-timeout`, `--csv`, `--timestamp-from-file`, +`--seconds-offset`, `--protocol`, `--sender-id`, `--store-forward-dir`, `--batch-size`, +`--batches-per-transaction`, `--probe-interval-ms`, `--enterprise`, `--zone`. +(`--timestamp-from-file` and `--enterprise` are presence flags.) Confirm results server-side with +`SELECT count() FROM trades`. + +## Pacing: `--delay-ms` vs `--rate` + +- `--delay-ms` (default `50`): a fixed sleep after **every row**, per worker. +- `--rate` (default `0` = off): a **target aggregate rate in rows/second across all workers**. + Each worker paces to its share (`rate / num-senders`) against a deadline schedule, sending + rows back-to-back and sleeping only when it runs *ahead* — reaching high targets a per-row + sleep never could. When `> 0` it takes precedence over `--delay-ms` (a warning prints if both + are set). Measured: `--rate 5000`, 20k rows, 4 workers → ~5.0 s, steady ~5000 rows/s. + +## Timestamps + +The generator sends timestamps at **nanosecond** resolution (`timestamp_nanos`), and QuestDB +stores them at the **target column's** resolution: a `TIMESTAMP_NS` column keeps full nanos, a +micros `TIMESTAMP` column truncates the extra digits (silently, never an error). **If the table +does not exist, the server auto-creates it as `TIMESTAMP_NS`.** Pre-create `trades` with a +`timestamp` (micros) column if you want micros. The CSV parser already reads fractional seconds +to full nanosecond precision. Verified: replaying the nanosecond FX chunk with +`--timestamp-from-file` preserved `...192508297Z` in an auto-created `TIMESTAMP_NS` table. + +## Probe (QWP/WebSocket only) + +For `qwp`, a background thread uses the egress `reader` to run `select timestamp from trades +limit -1` each interval and prints the latest ingested timestamp, then `switch status` for the +serving node's live role. Unlike the Python port, the C++ `reader` exposes `server_info()` +(role/node/cluster), so the probe **does** fall back to the QWP handshake role when +`switch status` is unavailable (OSS / missing SYSTEM ADMIN), labelled `(handshake role; ...)` — +matching Java/Rust. The reader uses `target=any` (replica-fallback reads) and fails over across +the `--addrs` hosts. + +## Differences from the other ports + +- **No auto-flush** (like Rust; unlike Java/Python) — the C/C++ client has none, so batching is + driven purely by explicit `flush()` at the same boundaries as the other ports. +- **Reader scheme is `ws`/`wss`** (like Rust; Python required `qwpws`). +- Row values from the CSV are validated UTF-8 via `utf8_view` at build time (throws on invalid + input); fixed column names use the `_cn` / `_tn` literals. +- Threads via `std::thread`, one `line_sender` per worker (unique `sender_id` + spill dir). + +## Validated + +Against a local QuestDB (OSS), all rows accounted for server-side: + +| What | Result | +| --- | --- | +| `qwp`, 4000 rows, 1 worker | 4000 / 4000, distinct per-row timestamps; probe reported live timestamps + handshake role fallback | +| `ilp`, 2000 rows, 2 workers | 2000 / 2000 | +| `qwpudp` (paced) | rows land; best-effort, lossy under fast bursts | +| `qwp`, `--rate 5000`, nanos FX, `--timestamp-from-file`, 4 workers | steady ~5000 rows/s; auto-created `TIMESTAMP_NS` table preserved `...192508297Z` | +| QWP failover (primary crash) | see below | + +### QWP failover (primary crash) + +Two servers from the same build (primary `:9100`, fallback `:9000`), a single-worker `qwp` +run of 40000 rows with `--addrs :9100,:9000` and the probe active. The primary was +hard-killed mid-run: + +- The sender **completed all 40000 rows with exit 0**, despite its primary crashing. +- The primary took `trade_id` seq 1–7500; the fallback took 7501–40000 (32500 rows, all + distinct). Combined: **contiguous 1–40000, zero loss, zero duplication** — store-and-forward + replayed the in-flight transaction to the new host, handing off exactly on a transaction + boundary. +- The probe failed over from the primary to the fallback. diff --git a/c/csv_parallel_sender.cpp b/c/csv_parallel_sender.cpp new file mode 100644 index 0000000..d137b17 --- /dev/null +++ b/c/csv_parallel_sender.cpp @@ -0,0 +1,568 @@ +// Parallel CSV replay sender for QuestDB, a C++ port of the Java/Rust ha_sender. +// +// Replays a CSV of trades in a loop across N worker threads over one of three transports +// (--protocol): +// * qwp - QWP over WebSocket: store-and-forward (un-acked frames spill to disk and +// replay after an outage), transactional commit, multi-host failover. A probe +// thread polls the latest ingested timestamp and the serving node's live role +// over a QWP query client (the egress reader). +// * qwpudp - QWP over UDP: fire-and-forget datagrams to :9007. Ingest-only, +// unauthenticated, single-endpoint, best-effort. +// * ilp - the legacy ILP/HTTP transport. +// +// Like the Rust client, the C/C++ client has no auto-flush: batching is driven by explicit +// flush() calls at the same boundaries as the Java/Rust ports. + +#include +#include + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace qi = questdb::ingress; +namespace eg = questdb::egress; +using namespace questdb::ingress::literals; + +namespace { + +constexpr std::string_view PROBE_QUERY = "select timestamp from trades limit -1"; +// Live lifecycle role of whichever node the query client is connected to (current_role / +// target_role), unlike the QWP handshake which only refreshes on reconnect. +constexpr std::string_view STATUS_QUERY = "switch status"; + +struct Args { + std::string addrs = "localhost:9000"; + std::string token, username, password; + uint64_t total_events = 1'000'000; + uint64_t delay_ms = 50; + uint64_t rate = 0; // target aggregate rows/s across all workers (0 = off, use delay_ms) + unsigned num_senders = 10; + uint64_t retry_timeout = 360'000; + std::string csv = "./trades20250728.csv.gz"; + bool timestamp_from_file = false; + int64_t seconds_offset = 0; + std::string protocol = "qwp"; + std::string sender_id = "ha_sender"; + std::string store_forward_dir = "/tmp/qdb-sf"; + uint64_t batch_size = 10'000; + uint64_t batches_per_transaction = 10; + uint64_t probe_interval_ms = 1000; + bool enterprise = false; + std::string zone = "eu-west-1"; +}; + +struct TradeRow { + std::string symbol; + std::string side; + double price; + double amount; + int64_t ts_nanos; // only used with --timestamp-from-file +}; + +int64_t now_nanos() { + return std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()) + .count(); +} + +std::vector split_addrs(const std::string& csv) { + std::vector out; + std::stringstream ss(csv); + std::string item; + while (std::getline(ss, item, ',')) { + size_t b = item.find_first_not_of(" \t"); + size_t e = item.find_last_not_of(" \t"); + if (b != std::string::npos) out.push_back(item.substr(b, e - b + 1)); + } + return out; +} + +bool has_token(const Args& a) { return !a.token.empty(); } +bool has_basic(const Args& a) { return !a.username.empty() && !a.password.empty(); } +bool use_tls(const Args& a) { return has_token(a) || has_basic(a); } + +void append_auth(std::string& c, const Args& a) { + if (has_token(a)) + c += "token=" + a.token + ";"; + else if (has_basic(a)) + c += "username=" + a.username + ";password=" + a.password + ";"; +} + +std::string build_ingest_conf(const Args& a, unsigned worker_id) { + auto addrs = split_addrs(a.addrs); + std::string joined; + for (size_t i = 0; i < addrs.size(); ++i) joined += (i ? "," : "") + addrs[i]; + + if (a.protocol == "qwpudp") { + return "qwpudp::addr=" + addrs[0] + ";"; + } + if (a.protocol == "ilp") { + std::string c = (use_tls(a) ? "https" : "http"); + c += "::addr=" + joined + ";"; + append_auth(c, a); + if (use_tls(a)) c += "tls_verify=unsafe_off;"; + c += "retry_timeout=" + std::to_string(a.retry_timeout) + ";"; + return c; + } + // qwp (WebSocket) + std::string who = a.sender_id + "-" + std::to_string(worker_id); + std::string sf = a.store_forward_dir + "/" + who; + std::error_code ec; + std::filesystem::create_directories(sf, ec); + std::string c = (use_tls(a) ? "qwpwss" : "qwpws"); + c += "::addr=" + joined + ";"; + append_auth(c, a); + if (use_tls(a)) c += "tls_verify=unsafe_off;"; + c += "sender_id=" + who + ";"; + c += "sf_dir=" + sf + ";"; + c += "reconnect_max_duration_millis=" + std::to_string(a.retry_timeout) + ";"; + c += "reconnect_initial_backoff_millis=100;"; + c += "reconnect_max_backoff_millis=5000;"; + if (a.enterprise) c += "request_durable_ack=true;"; + return c; +} + +std::string build_reader_conf(const Args& a) { + auto addrs = split_addrs(a.addrs); + std::string joined; + for (size_t i = 0; i < addrs.size(); ++i) joined += (i ? "," : "") + addrs[i]; + std::string c = (use_tls(a) ? "wss" : "ws"); + c += "::addr=" + joined + ";target=any;failover=true;"; + append_auth(c, a); + if (use_tls(a)) c += "tls_verify=unsafe_off;"; + if (!a.zone.empty()) c += "zone=" + a.zone + ";"; + return c; +} + +std::string format_micros(int64_t micros) { + std::time_t secs = static_cast(micros / 1'000'000); + int64_t frac = micros % 1'000'000; + std::tm tm{}; + gmtime_r(&secs, &tm); + char buf[32]; + std::strftime(buf, sizeof(buf), "%Y-%m-%dT%H:%M:%S", &tm); + char out[64]; + std::snprintf(out, sizeof(out), "%s.%06lldZ", buf, static_cast(frac)); + return out; +} + +const char* role_name(eg::server_role r) { + switch (r) { + case eg::server_role::standalone: return "STANDALONE"; + case eg::server_role::primary: return "PRIMARY"; + case eg::server_role::replica: return "REPLICA"; + case eg::server_role::primary_catchup: return "PRIMARY_CATCHUP"; + default: return "UNKNOWN"; + } +} + +// Read a string cell from a Symbol or Varchar column; empty for null / non-string kinds. +std::optional col_string(const eg::column& col, size_t row) { + std::optional out; + col.visit(eg::overload{ + [&](eg::varlen_view v) { + if (auto x = v.as_string_view(row)) out = std::string(*x); + }, + [&](eg::symbol_view v) { + if (auto x = v.resolve(row)) out = std::string(*x); + }, + [](auto) {}, + }); + return out; +} + +void load_csv(const std::string& path, bool need_timestamp, std::vector& out) { + gzFile f = gzopen(path.c_str(), "rb"); // transparently reads .gz and plain files + if (!f) throw std::runtime_error("cannot open CSV: " + path); + char line[4096]; + char* header = gzgets(f, line, sizeof(line)); + if (!header) { gzclose(f); return; } + + // Minimal CSV field split: honours double-quoted fields (quotes are dropped and a + // comma inside quotes does not split), then trims whitespace on unquoted fields. + auto fields = [](const char* s) { + std::vector v; + std::string cur; + bool in_quotes = false; + for (const char* p = s; *p && *p != '\n' && *p != '\r'; ++p) { + char ch = *p; + if (ch == '"') { in_quotes = !in_quotes; continue; } + if (ch == ',' && !in_quotes) { v.push_back(cur); cur.clear(); } + else cur += ch; + } + v.push_back(cur); + for (auto& x : v) { + size_t b = x.find_first_not_of(" \t"); + size_t e = x.find_last_not_of(" \t"); + x = (b == std::string::npos) ? "" : x.substr(b, e - b + 1); + } + return v; + }; + + auto cols = fields(header); + auto idx = [&](const std::string& name) -> int { + for (size_t i = 0; i < cols.size(); ++i) + if (cols[i] == name) return static_cast(i); + throw std::runtime_error("CSV missing required column: " + name); + }; + int i_sym = idx("symbol"), i_side = idx("side"), i_price = idx("price"), + i_amt = idx("amount"); + int i_ts = need_timestamp ? idx("timestamp") : -1; + + while (gzgets(f, line, sizeof(line))) { + auto r = fields(line); + if (r.size() <= static_cast(i_amt)) continue; + TradeRow tr; + tr.symbol = r[i_sym]; + tr.side = r[i_side]; + tr.price = std::stod(r[i_price]); + tr.amount = std::stod(r[i_amt]); + tr.ts_nanos = 0; + if (need_timestamp) { + // ISO-8601 like 2025-07-28T12:34:56.789012Z + std::tm tm{}; + const std::string& s = r[i_ts]; + int y, mo, d, h, mi, se; + long frac = 0; + if (std::sscanf(s.c_str(), "%d-%d-%dT%d:%d:%d", &y, &mo, &d, &h, &mi, &se) == 6) { + tm.tm_year = y - 1900; tm.tm_mon = mo - 1; tm.tm_mday = d; + tm.tm_hour = h; tm.tm_min = mi; tm.tm_sec = se; + std::time_t secs = timegm(&tm); + size_t dot = s.find('.'); + if (dot != std::string::npos) { + std::string fs = s.substr(dot + 1); + while (!fs.empty() && !std::isdigit((unsigned char)fs.back())) fs.pop_back(); + fs.resize(9, '0'); // pad to nanoseconds + frac = std::stol(fs.substr(0, 9)); + } + tr.ts_nanos = static_cast(secs) * 1'000'000'000 + frac; + } + } + out.push_back(std::move(tr)); + } + gzclose(f); +} + +void run_worker(unsigned worker_id, uint64_t total_events, const Args& args, + const std::vector& rows, std::atomic& total_sent) { + std::printf("Sender %u will send %llu events\n", worker_id, + static_cast(total_events)); + const bool is_qwp = args.protocol == "qwp"; + const bool is_udp = args.protocol == "qwpudp"; + // Single worker on a QWP transport stamps rows client-side (monotonic, no O3). + const bool per_row = (is_qwp || is_udp) && args.num_senders == 1; + const uint64_t commit_every = + is_qwp ? args.batch_size * args.batches_per_transaction : args.batch_size; + + // Rate limiting (when --rate > 0): pace this worker to its share of the aggregate target, + // rate / num-senders rows/second. interval_nanos is the ideal spacing between this worker's + // rows; we sleep only when running ahead of a deadline schedule, so rows go out back-to-back + // until we get ahead. Reaches high targets a fixed per-row sleep cannot. + const double interval_nanos = + args.rate > 0 ? 1'000'000'000.0 * args.num_senders / static_cast(args.rate) : 0.0; + const bool rate_limited = interval_nanos > 0.0; + + auto sender = qi::line_sender::from_conf(qi::utf8_view{build_ingest_conf(args, worker_id)}); + auto buffer = sender.new_buffer(); + const size_t n = rows.size(); + uint64_t sent = 0; + const auto pace_start = std::chrono::steady_clock::now(); + + for (uint64_t i = 0; i < total_events; ++i) { + const TradeRow& r = rows[i % n]; + std::string trade_id = std::to_string(worker_id) + "-" + std::to_string(i + 1); + buffer.table("trades"_tn) + .symbol("symbol"_cn, qi::utf8_view{r.symbol}) + .symbol("side"_cn, qi::utf8_view{r.side}) + .column("price"_cn, r.price) + .column("amount"_cn, r.amount) + .column("trade_id"_cn, qi::utf8_view{trade_id}); + + if (args.timestamp_from_file) + buffer.at(qi::timestamp_nanos{r.ts_nanos + args.seconds_offset * 1'000'000'000}); + else if (args.seconds_offset != 0) + buffer.at(qi::timestamp_nanos{now_nanos() + args.seconds_offset * 1'000'000'000}); + else if (per_row) + buffer.at(qi::timestamp_nanos::now()); + else + buffer.at_now(); + + ++sent; + total_sent.fetch_add(1, std::memory_order_relaxed); + if (sent % commit_every == 0) sender.flush(buffer); + if (rate_limited) { + // Deadline for the row just sent (sent is 1-based). Sleep only while ahead of + // schedule; the 1ms floor coalesces many rows into one sleep at high rates. + const uint64_t target_nanos = static_cast(sent * interval_nanos); + const uint64_t elapsed_nanos = static_cast( + std::chrono::duration_cast( + std::chrono::steady_clock::now() - pace_start) + .count()); + if (target_nanos > elapsed_nanos) { + const uint64_t sleep_nanos = target_nanos - elapsed_nanos; + if (sleep_nanos > 1'000'000) + std::this_thread::sleep_for(std::chrono::nanoseconds(sleep_nanos)); + } + } else if (args.delay_ms > 0) { + std::this_thread::sleep_for(std::chrono::milliseconds(args.delay_ms)); + } + } + sender.flush(buffer); + if (is_qwp) sender.close_drain(); // drain un-acked store-and-forward frames + std::printf("Sender %u finished sending %llu events\n", worker_id, + static_cast(sent)); +} + +// Pull current_role / target_role out of `switch status` by column name. +void read_switch_status(eg::reader& reader, std::optional& current, + std::optional& target) { + current.reset(); + target.reset(); + auto cur = reader.execute(qi::utf8_view{STATUS_QUERY}); + while (auto batch_opt = cur.next_batch()) { + auto& batch = *batch_opt; + const size_t cols = batch.column_count(); + for (size_t r = 0; r < batch.row_count(); ++r) { + for (size_t c = 0; c < cols; ++c) { + std::string name(batch.column_name(c)); + std::string lower = name; + for (auto& ch : lower) ch = std::tolower((unsigned char)ch); + if (lower.find("role") == std::string::npos) continue; + auto v = col_string(batch.column(c), r); + if (lower.find("current") != std::string::npos) current = v; + else if (lower.find("target") != std::string::npos) target = v; + else if (!current) current = v; + } + } + } +} + +void run_probe(const Args& args, std::atomic& stop) { + std::unique_ptr reader; + try { + reader = std::make_unique(qi::utf8_view{build_reader_conf(args)}); + } catch (const std::exception& e) { + std::printf("[query client] connect failed: %s\n", e.what()); + return; + } + // server_info() is a non-owning view whose role is only populated at the handshake; it + // reads as "other" after queries run. Capture the handshake node/role once, at connect, + // and use it as the (labelled, possibly-stale) fallback when `switch status` is absent. + std::string hs_node, hs_role; + { + auto si = reader->server_info(); + hs_node = std::string(si.node_id()); + if (hs_node.empty()) hs_node = "(none)"; + hs_role = role_name(si.role()); + std::string cluster(si.cluster_id()); + std::printf("[query client] connected, serving node=%s role=%s cluster=%s\n", + hs_node.c_str(), hs_role.c_str(), cluster.empty() ? "(none)" : cluster.c_str()); + } + + bool was_down = false; + while (!stop.load(std::memory_order_relaxed)) { + try { + int64_t latest = 0; + bool have = false; + bool is_nanos = false; + auto cur = reader->execute(qi::utf8_view{PROBE_QUERY}); + while (auto batch_opt = cur.next_batch()) { + auto& batch = *batch_opt; + auto col = batch.column(0); + is_nanos = col.kind() == eg::column_kind::timestamp_nanos; + for (size_t r = 0; r < batch.row_count(); ++r) + if (auto v = col.get(r)) { latest = *v; have = true; } + } + if (was_down) { std::printf("[query client] connection restored\n"); was_down = false; } + if (have) { + std::optional current, target; + try { read_switch_status(*reader, current, target); } catch (...) {} + std::string served; + if (current) { + bool switching = target && *target != *current; + served = " served by role=" + *current + + (switching ? " (switching -> " + *target + ")" : "") + + " node=" + hs_node; + } else { + served = " served by role=" + hs_role + " node=" + hs_node + + " (handshake role; live 'switch status' unavailable, may be stale)"; + } + int64_t micros = is_nanos ? latest / 1000 : latest; + std::printf("[probe] latest trades timestamp = %s (raw=%lld)%s\n", + format_micros(micros).c_str(), static_cast(latest), + served.c_str()); + } + } catch (const std::exception& e) { + if (!was_down) { + std::printf("[query client] connection lost (%s), will retry\n", e.what()); + was_down = true; + } + } + for (uint64_t slept = 0; slept < args.probe_interval_ms && !stop.load(); slept += 50) + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + } +} + +std::string need_value(int& i, int argc, char** argv) { + if (i + 1 >= argc) throw std::runtime_error(std::string("missing value for ") + argv[i]); + return argv[++i]; +} + +Args parse_args(int argc, char** argv) { + Args a; + for (int i = 1; i < argc; ++i) { + std::string k = argv[i]; + if (k == "--addrs") a.addrs = need_value(i, argc, argv); + else if (k == "--token") a.token = need_value(i, argc, argv); + else if (k == "--username") a.username = need_value(i, argc, argv); + else if (k == "--password") a.password = need_value(i, argc, argv); + else if (k == "--total-events") a.total_events = std::stoull(need_value(i, argc, argv)); + else if (k == "--delay-ms") a.delay_ms = std::stoull(need_value(i, argc, argv)); + else if (k == "--rate") a.rate = std::stoull(need_value(i, argc, argv)); + else if (k == "--num-senders") a.num_senders = std::stoul(need_value(i, argc, argv)); + else if (k == "--retry-timeout") a.retry_timeout = std::stoull(need_value(i, argc, argv)); + else if (k == "--csv") a.csv = need_value(i, argc, argv); + else if (k == "--timestamp-from-file") a.timestamp_from_file = true; + else if (k == "--seconds-offset") a.seconds_offset = std::stoll(need_value(i, argc, argv)); + else if (k == "--protocol") a.protocol = need_value(i, argc, argv); + else if (k == "--sender-id") a.sender_id = need_value(i, argc, argv); + else if (k == "--store-forward-dir") a.store_forward_dir = need_value(i, argc, argv); + else if (k == "--batch-size") a.batch_size = std::stoull(need_value(i, argc, argv)); + else if (k == "--batches-per-transaction") + a.batches_per_transaction = std::stoull(need_value(i, argc, argv)); + else if (k == "--probe-interval-ms") + a.probe_interval_ms = std::stoull(need_value(i, argc, argv)); + else if (k == "--enterprise") a.enterprise = true; + else if (k == "--zone") a.zone = need_value(i, argc, argv); + else throw std::runtime_error("unknown argument: " + k); + } + return a; +} + +} // namespace + +int main(int argc, char** argv) { + Args args; + try { + args = parse_args(argc, argv); + } catch (const std::exception& e) { + std::fprintf(stderr, "%s\n", e.what()); + return 2; + } + + if (args.protocol != "qwp" && args.protocol != "qwpudp" && args.protocol != "ilp") { + std::fprintf(stderr, "--protocol must be 'qwp', 'qwpudp', or 'ilp', got: %s\n", + args.protocol.c_str()); + return 2; + } + auto addrs = split_addrs(args.addrs); + if (addrs.empty()) { std::fprintf(stderr, "--addrs must contain a host:port\n"); return 2; } + if (args.protocol == "qwpudp") { + if (addrs.size() > 1) { + std::fprintf(stderr, "--protocol qwpudp supports a single host:port only (UDP is " + "connectionless, no failover); got %zu addresses\n", addrs.size()); + return 2; + } + if (has_token(args) || !args.username.empty() || !args.password.empty()) + std::fprintf(stderr, "[warn] --protocol qwpudp is unauthenticated (UDP accepts any " + "connection); ignoring --token/--username/--password\n"); + } + + // --rate takes precedence: when set it drives pacing and --delay-ms is ignored. + if (args.rate > 0 && args.delay_ms > 0) + std::fprintf(stderr, "[warn] --rate %llu overrides --delay-ms %llu (rate limiting drives " + "pacing; the per-row delay is ignored)\n", + (unsigned long long)args.rate, (unsigned long long)args.delay_ms); + + std::vector rows; + try { + load_csv(args.csv, args.timestamp_from_file, rows); + } catch (const std::exception& e) { + std::fprintf(stderr, "Failed to read CSV: %s\n", e.what()); + return 2; + } + if (rows.empty()) { std::fprintf(stderr, "CSV has no data rows: %s\n", args.csv.c_str()); return 2; } + + if (args.rate > 0) + std::printf("Pacing: rate=%llu rows/s (aggregate across %u workers)\n", + (unsigned long long)args.rate, args.num_senders); + else + std::printf("Pacing: delay-ms=%llu\n", (unsigned long long)args.delay_ms); + + if (args.protocol == "qwp") + std::printf("Ingestion started. Protocol: qwp (WebSocket, sender-id=%s, " + "store-and-forward=%s, batch-size=%llu, batches-per-transaction=%llu, " + "retry-timeout-ms=%llu) | addrs: %s\n", + args.sender_id.c_str(), args.store_forward_dir.c_str(), + (unsigned long long)args.batch_size, + (unsigned long long)args.batches_per_transaction, + (unsigned long long)args.retry_timeout, args.addrs.c_str()); + else if (args.protocol == "qwpudp") + std::printf("Ingestion started. Protocol: qwpudp (QWP/UDP datagrams, ingest-only, " + "unauthenticated, no store-and-forward, no failover; query client disabled, " + "batch-size=%llu) | addr: %s\n", + (unsigned long long)args.batch_size, addrs[0].c_str()); + else + std::printf("Ingestion started. Protocol: ilp (HTTP, retry-timeout-ms=%llu) | addrs: %s\n", + (unsigned long long)args.retry_timeout, args.addrs.c_str()); + + std::atomic total_sent{0}; + std::atomic stop{false}; + auto start = std::chrono::steady_clock::now(); + + std::thread reporter([&] { + uint64_t last = 0; + while (!stop.load()) { + std::this_thread::sleep_for(std::chrono::seconds(1)); + uint64_t now = total_sent.load(); + std::printf("[progress] sent=%llu rate=%llu rows/s\n", + (unsigned long long)now, (unsigned long long)(now - last)); + last = now; + } + }); + + std::thread probe; + if (args.protocol == "qwp" && args.probe_interval_ms > 0) + probe = std::thread(run_probe, std::cref(args), std::ref(stop)); + + const uint64_t base = args.total_events / args.num_senders; + const uint64_t rem = args.total_events % args.num_senders; + std::atomic failures{0}; + std::vector workers; + for (unsigned id = 0; id < args.num_senders; ++id) { + uint64_t events = base + (id < rem ? 1 : 0); + workers.emplace_back([&, id, events] { + try { + run_worker(id, events, args, rows, total_sent); + } catch (const std::exception& e) { + std::fprintf(stderr, "Worker failed: Sender %u: %s\n", id, e.what()); + failures.fetch_add(1); + } + }); + } + for (auto& w : workers) w.join(); + stop.store(true); + if (probe.joinable()) probe.join(); + reporter.join(); + + if (failures.load() > 0) return 1; + + double elapsed = std::chrono::duration(std::chrono::steady_clock::now() - start).count(); + double rate = elapsed > 0 ? args.total_events / elapsed : 0; + std::printf("All workers completed. protocol=%s events=%llu elapsed=%.3f s throughput=%.0f rows/s\n", + args.protocol.c_str(), (unsigned long long)args.total_events, elapsed, rate); + return 0; +} diff --git a/python/README.md b/python/README.md new file mode 100644 index 0000000..d109053 --- /dev/null +++ b/python/README.md @@ -0,0 +1,216 @@ +# Python HA sender + +A Python port of the Java/Rust `CsvParallelSender`, plus a pandas/polars dataframe demo. +It replays a CSV of trades in a loop across N worker threads over QuestDB's QWP +(WebSocket/UDP) and ILP/HTTP transports, with a probe that reads back the latest ingested +timestamp and the serving node's live role. + +Built on the QuestDB Python client (`questdb.ingress`), which wraps the same C/Rust client +as the other ports. + +## Which script to use + +This port has grown a few scripts. Pick by what you need: + +| Script | Use it for | Path / speed | +| --- | --- | --- | +| `csv_columnar_sender.py` | **High-throughput ingestion** - bulk CSV replay at volume | Columnar QWP (`Client.dataframe`, Arrow/polars). Fastest Python path; flat memory. | +| `csv_parallel_sender.py` | **HA / failover demos** - store-and-forward durability, the live probe, `qwpudp`/`ilp` comparison, and the per-event `row()` API | Row-by-row QWP/UDP/ILP. Correct but slow for volume - keep `--total-events` and the batch window small. | +| `read_bench.py` | Measuring read/scan throughput (rows/s) of a table | Streaming `iter_arrow()` egress. | +| `enrich_polars_demo.py` | Read a table as polars, enrich, write it back as polars | Streaming read + columnar write. | +| `dataframe_demo.py` | pandas/polars ingestion + egress round-trip showcase | `Sender.dataframe` / `Client.dataframe`. | + +**Row-by-row vs columnar.** `sender.row()` in a Python loop is the *slowest* path: every cell +crosses the Python/Cython boundary under the GIL (the client's own perf notes call it "~16x +slower" than the columnar bulk path). `Client.dataframe` ships whole Arrow columns to the +native client - the only Python path that reaches Java/Rust-class throughput. So use +`csv_columnar_sender.py` for volume, and reach for `csv_parallel_sender.py` **only** when you +need store-and-forward failover, which the columnar path deliberately bypasses (dataframe +ingestion uses the direct column sender, not store-and-forward). Do not push high volume +(e.g. 100M rows) through the row-by-row sender - its per-row Python cost plus a large flush +window will pin the buffer in memory and can OOM the host. + +```bash +# Fast columnar ingestion (recommended for volume): +python csv_columnar_sender.py \ + --addrs host:9000 --total-events 100000000 \ + --num-senders 2 --chunk-rows 100000 --csv ../trades.csv \ + --token "$QDB_TOKEN" --tls-verify unsafe_off +``` + +## Network tuning for throughput (important on real networks) + +The Rust/C/Python QWP clients hardcode a 4 MiB socket buffer, which default Linux silently +clamps to ~416 KB and pins each connection at ~426 KB **per RTT** - a hard throughput cap on +a real network (e.g. ~210k rows/s at 20 ms RTT, while Java, which does not touch socket +buffers, does ~52 MB/s). Two levers: + +- Run [`boost_tcp.sh`](../boost_tcp.sh) on **both** the sender and server hosts to raise + `net.core.wmem_max`/`rmem_max` (and friends) so the client's 4 MiB buffer request actually + goes through instead of being clamped. It is runtime-only (resets on reboot); persist via + `/etc/sysctl.d/` if you want it permanent. +- Prefer **multiple senders** (`--num-senders`): the cap is per connection, so parallel + connections multiply throughput (~11M rows/s with 2 connections same-zone EC2 in the + client team's tests, vs ~4M with 1). The tuning affects egress too, so it also speeds up + `read_bench.py` on a high-RTT link. + +## Important: the client must be built from source + +The QWP transports (`qwpws`/`qwpwss`/`qwpudp`) **and the query client / egress reader are not +in the PyPI release** (`questdb` 4.x on PyPI is ILP-only: `Protocol` has just +`Tcp/Tcps/Http/Https`, and there is no `Client`/`QueryResult`). They live on the +**`sm_qwp_dataframe_bench`** branch of +[`py-questdb-client`](https://github.com/questdb/py-questdb-client), which bumps the bundled +C client to the QWP + egress build. You must build that branch. + +Requirements: **Python ≥ 3.10** (the branch is 3.10+; the PyPI-era 3.9 venv will not work), +a Rust toolchain (`cargo`), and a C compiler. Build steps (a git worktree keeps your +`main` checkout untouched): + +```bash +cd ~/prj/python/py-questdb-client +git worktree add --detach /tmp/pyqdb_qwp origin/sm_qwp_dataframe_bench +cd /tmp/pyqdb_qwp +git submodule update --init # bundled c-questdb-client (QWP + egress) + +python3.12 -m venv /tmp/pyqdb_venv +/tmp/pyqdb_venv/bin/pip install -U pip "cython>=3.1.2" "setuptools>=80.9.0" numpy pandas polars pyarrow +/tmp/pyqdb_venv/bin/pip install -e . # compiles Cython + the Rust FFI +``` + +Verify: + +```python +from questdb.ingress import Protocol, Client # Client only exists on the QWP build +print([p for p in dir(Protocol) if not p.startswith('_')]) +# -> ['Http', 'Https', 'QwpUdp', 'QwpWs', 'QwpWss', 'Tcp', 'Tcps'] +``` + +Then run the scripts with that interpreter (`/tmp/pyqdb_venv/bin/python`). `pandas`, `polars`, +and `pyarrow` are only needed for the dataframe paths. + +## Transports (`--protocol`) + +- `qwp` (default): QWP over WebSocket. Per-worker store-and-forward, transactional commit, + multi-host failover, and the probe. Use the WebSocket/HTTP port (`:9000`). +- `qwpudp`: QWP over UDP, fire-and-forget datagrams to `:9007`. Ingest-only, + unauthenticated, single-endpoint (no failover), best-effort. **Especially lossy in + Python**: the Cython `row()` loop emits datagrams so fast that even a single worker can + overrun the server's UDP receive buffer (a full run can vanish). Pace it with `--delay-ms` + and keep `--batch-size` small, and treat results as best-effort. +- `ilp`: the legacy ILP/HTTP transport. + +## Run + +```bash +/tmp/pyqdb_venv/bin/python csv_parallel_sender.py \ + --protocol qwp \ + --addrs localhost:9000 \ + --total-events 100000 \ + --num-senders 4 \ + --delay-ms 0 \ + --batch-size 10000 \ + --batches-per-transaction 10 \ + --csv ../trades20250728.csv.gz +``` + +Flags mirror the Java/Rust ports: `--addrs`, `--token`/`--username`/`--password`, +`--total-events`, `--delay-ms`, `--rate`, `--num-senders`, `--retry-timeout`, `--csv`, +`--timestamp-from-file`, `--seconds-offset`, `--protocol`, `--sender-id`, +`--store-forward-dir`, `--batch-size`, `--batches-per-transaction`, `--probe-interval-ms`, +`--enterprise`, `--zone`. Confirm results server-side with `SELECT count() FROM trades`. + +## Pacing: `--delay-ms` vs `--rate` + +- `--delay-ms` (default `50`): a fixed sleep after **every row**, per worker. +- `--rate` (default `0` = off): a **target aggregate rate in rows/second across all workers**. + Each worker paces to its share (`rate / num-senders`) against a deadline schedule, sending + rows back-to-back and sleeping only when it runs *ahead*. When `> 0` it takes precedence over + `--delay-ms` (a warning prints if both are set). Note Python's per-row throughput ceiling is + lower than Java/Rust (the GIL / Python-level row building), so very high `--rate` targets may + not be reached; the pacing simply never sleeps in that case. + +## Timestamps + +The generator sends timestamps at **nanosecond** resolution (`TimestampNanos`), and QuestDB +stores them at the **target column's** resolution: a `TIMESTAMP_NS` column keeps full nanos, a +micros `TIMESTAMP` column truncates the extra digits (silently, never an error). **If the table +does not exist, the server auto-creates it as `TIMESTAMP_NS`.** Pre-create `trades` with a +`timestamp` (micros) column if you want micros. The CSV timestamp is parsed to full nanoseconds +(the fractional-seconds string is read directly, since Python's `datetime` is microsecond-only +and would otherwise drop the last three digits of a `TIMESTAMP_NS` source). + +## Probe (QWP/WebSocket only) + +For `qwp`, a background thread uses the pooled query client (`Client`) to run +`select timestamp from trades limit -1` each interval and prints the latest ingested +timestamp, then `switch status` for the serving node's live role. `switch status` is +Enterprise-only (needs SYSTEM ADMIN); on OSS the probe prints +`(live 'switch status' unavailable, ...)`. The client uses `target=any` (replica-fallback +reads) and fails over across the `--addrs` hosts. + +## Dataframe demo (pandas / polars ingestion + egress) + +`dataframe_demo.py` shows the columnar paths the row sender does not: + +```bash +/tmp/pyqdb_venv/bin/python dataframe_demo.py --addr localhost:9000 --csv ../trades20250728.csv.gz --rows 5000 +``` + +- **pandas ingestion** via `Sender.dataframe(df, ...)` — the numpy-backed pandas planner. +- **polars ingestion** via `Client.dataframe(df, ...)` — the pooled QWP Arrow-columnar path, + which takes polars / pyarrow / any Arrow C Stream source natively. (Numpy-backed pandas is + **not** accepted by `Client.dataframe` — it raises `UnsupportedDataFrameShapeError` — so + pandas goes through `Sender.dataframe`, or convert with `pyarrow.Table.from_pandas`.) +- **egress** via `Client.query(sql).to_pandas()` and `.to_polars()` (also `.to_arrow()` / + `iter_pandas()`; the result exposes the Arrow PyCapsule interface for zero-copy consumers). + +## Differences from the Java/Rust ports + +- **The client is pre-release / build-from-source** (see above) — not `pip install questdb`. +- **Ingestion and querying are split across two classes**: `Sender` (ingest) and `Client` + (pooled QWP: query + Arrow-columnar dataframe ingest). The probe uses `Client`. +- **Auto-flush exists** in the Python client (unlike the Rust client) and is verified to work + over QWP (row-threshold flush fires mid-stream; see Validated). This port sets + `auto_flush=off` and flushes at the same batch boundaries as the Java/Rust ports for + identical cadence. +- **Config-string scheme is `qwpws`/`qwpwss` for both the sender and the query client** + (the Rust reader used `ws`/`wss`; Python only accepts the `qwp*` schemes). +- **The probe has no handshake-role fallback** — because the Python binding does not expose it. + Java/Rust fall back to the QWP handshake role when `switch status` is unavailable. That role + exists in the underlying C client (`line_reader_server_info_role`/`_role_byte`) and the Rust + reader (`server_info()`), but the Python `Client` on this branch does not wrap it, so this + port prints "unavailable" on OSS. Likewise the query path's `on_failover_reset` callback is + wired internally in Cython but not surfaced to Python, so there is no user-facing failover + narration. +- Threads (like Java/Rust). Python's GIL is released during client I/O, but row-building is + Python-level, so row-by-row throughput is well below Java/Rust — use the dataframe path + for volume. + +## Validated + +Against a local QuestDB (OSS), all rows accounted for server-side: + +| What | Result | +| --- | --- | +| `qwp`, 3000 rows, 1 worker | 3000 / 3000, distinct per-row timestamps; probe reported live timestamps | +| `qwp`, `--rate 5000`, nanos FX, `--timestamp-from-file`, 4 workers | 20000 / 20000, steady ~5000 rows/s; auto-created `TIMESTAMP_NS` table preserved `...192508297Z` | +| `ilp`, 2000 rows, 2 workers | 2000 / 2000 | +| `qwpudp` (paced) | rows land; best-effort, lossy under fast bursts | +| `dataframe_demo` | pandas + polars ingest (10000 rows), egress to pandas and polars | +| auto-flush | `auto_flush_rows=1000`, 2500 rows, no manual flush: 2000 landed mid-stream (2 auto-flushes), close drained to 2500 | +| QWP failover (primary crash) | see below | + +### QWP failover (primary crash) + +Two servers from the same build (primary `:9100`, fallback `:9000`), a single-worker `qwp` +run of 40000 rows with `--addrs :9100,:9000` and the probe active. The primary was +hard-killed mid-run: + +- The sender **completed all 40000 rows with exit 0**, despite its primary crashing. +- The primary took `trade_id` seq 1–18500; the fallback took 18501–40000 (21500 rows, all + distinct). Combined: **contiguous 1–40000, zero loss, zero duplication** — store-and-forward + replayed the in-flight transaction to the new host, handing off exactly on a transaction + boundary. +- The probe failed over from the primary to the fallback (visible as a second + `connection lost -> restored` cycle once post-failover data committed on the new host). diff --git a/python/csv_columnar_sender.py b/python/csv_columnar_sender.py new file mode 100644 index 0000000..a23a40e --- /dev/null +++ b/python/csv_columnar_sender.py @@ -0,0 +1,305 @@ +#!/usr/bin/env python3 +"""Fast CSV replay sender for QuestDB - the COLUMNAR QWP path. + +This is the throughput-oriented Python sender. It replays a CSV to a QuestDB +``trades`` table over the QWP WebSocket transport using the *column-major* client +path (``Client.dataframe``), which ships whole Arrow columns to the native Rust +client in bulk. That is the ONLY Python path that gets near Java/Rust throughput. + +Why not row-by-row? ``Sender.row(...)`` in a Python loop is the SLOWEST path: every +cell crosses the Python/Cython boundary under the GIL. The client's own perf notes +call the row API a "per-cell call ... ~16x slower" than the columnar bulk path. +The column path (``Client.dataframe``) stores raw pointers into the Arrow buffers +("no data copy at append time") and does one memcpy per column at flush. Use it. + +Memory stays flat: the CSV is loaded once into a polars DataFrame, and rows are +sent in bounded, zero-copy slices. ``Client.dataframe`` additionally streams each +send internally at ``max_rows_per_batch`` (16384) rows, so nothing balloons - unlike +the row-by-row sender, which buffered up to 100k rows before its first flush and +grew without bound when Python could not reach that threshold fast enough. + +Timestamps: + * ``--timestamp-from-file`` uses the CSV's ``timestamp`` column (parsed to ns). + * otherwise each batch is stamped with a strictly-increasing "now" (nanoseconds), + so the target looks live and there is no out-of-order within a worker. + +Auth / TLS (QuestDB Enterprise): ``--token`` (bearer) or ``--username``/``--password`` +(basic) turn on TLS automatically (scheme ``qwpwss``); ``--tls`` forces TLS with no +auth. Certificate verification is on by default; ``--tls-verify unsafe_off`` for +self-signed certs. Multiple ``--addrs`` give QWP failover. + +Usage: + python csv_columnar_sender.py \ + --addrs host:9000[,host2:9000] \ + --total-events 100000000 \ + --num-senders 1 \ + --chunk-rows 100000 \ + --rate 0 \ + --csv ../trades.csv \ + [--timestamp-from-file] \ + [--token TOK | --username U --password P] [--tls-verify on|unsafe_off] + +Requires the QWP/egress build of the client (see README.md). +""" + +import argparse +import gzip +import io +import os +import sys +import threading +import time + +import numpy as np +import polars as pl + +from questdb.ingress import Client + +TABLE = "trades" + + +def use_tls(args): + return bool(args.tls or args.token or (args.username and args.password)) + + +def build_conf(args): + """QWP connect string for the columnar client, with optional auth + TLS + failover.""" + scheme = "qwpwss" if use_tls(args) else "qwpws" + addrs = ",".join(a.strip() for a in args.addrs.split(",") if a.strip()) + parts = [f"{scheme}::addr={addrs};"] + if args.token: + parts.append(f"token={args.token};") + elif args.username and args.password: + parts.append(f"username={args.username};password={args.password};") + if use_tls(args) and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def preflight(conf, timeout_s): + """Check the server answers within timeout_s over the full path (TCP+TLS+auth+query), + in a daemon thread so a hung native connect can't block us. Returns (ok, message). + Without this, a down/unreachable/misconfigured server makes the ingest client retry + forever and sit at sent=0.""" + outcome = {} + + def probe(): + try: + with Client.from_conf(conf) as c: + c.query("select 1").to_polars() + outcome["ok"] = True + except Exception as e: # noqa: BLE001 + outcome["err"] = str(e) + + t = threading.Thread(target=probe, daemon=True) + t.start() + t.join(timeout_s) + if t.is_alive(): + return False, f"no response within {timeout_s:g}s (server down / unreachable / wrong host or port?)" + if outcome.get("ok"): + return True, None + return False, outcome.get("err", "unknown error") + + +def load_base(path, need_timestamp): + """Load the CSV once into a compact polars DataFrame. symbol/side become + Categorical (-> SYMBOL), price/amount Float64, and (if needed) timestamp is + parsed to nanosecond Datetime. trade_id is synthesised per batch at send time.""" + if path.endswith(".gz"): + with gzip.open(path, "rb") as fh: + df = pl.read_csv(io.BytesIO(fh.read())) + else: + df = pl.read_csv(path) + for col in ("symbol", "side", "price", "amount"): + if col not in df.columns: + raise ValueError(f"CSV missing required column: {col}") + exprs = [ + pl.col("symbol").cast(pl.Utf8).cast(pl.Categorical), + pl.col("side").cast(pl.Utf8).cast(pl.Categorical), + pl.col("price").cast(pl.Float64), + pl.col("amount").cast(pl.Float64), + ] + if need_timestamp: + if "timestamp" not in df.columns: + raise ValueError("CSV missing required column: timestamp") + exprs.append( + pl.col("timestamp").cast(pl.Utf8).str.to_datetime(time_unit="ns").alias("timestamp") + ) + return df.select(exprs) + + +def run_worker(wid, events, base, args, counts): + """Send `events` rows for this worker: cycle over the base frame in bounded + slices, attach trade_id (+ a live timestamp unless reading it from file), and + ship each slice columnar via Client.dataframe.""" + m = base.height + from_file = args.timestamp_from_file + conf = build_conf(args) + + # Rate pacing (aggregate rows/s across all workers): deadline schedule, sleep + # only when ahead. interval = ns per row for this worker's share. + interval_ns = (1_000_000_000.0 * args.num_senders / args.rate) if args.rate > 0 else 0.0 + + sent = 0 + pos = 0 + last_ts = 0 + pace_start = time.perf_counter_ns() + with Client.from_conf(conf) as client: + while sent < events: + n = min(args.chunk_rows, events - sent, m - pos) + sl = base.slice(pos, n) # zero-copy view into the base frame + + # trade_id, vectorised (native polars, no Python per-row loop). + add = [(pl.lit(f"{wid}-") + pl.int_range(sent, sent + n).cast(pl.Utf8)).alias("trade_id")] + if not from_file: + base_ns = max(time.time_ns(), last_ts + 1) + ts = np.arange(n, dtype=np.int64) + base_ns # strictly increasing -> no O3 + last_ts = int(ts[-1]) + add.append(pl.Series("timestamp", ts).cast(pl.Datetime("ns", "UTC")).alias("timestamp")) + chunk = sl.with_columns(add) + + client.dataframe(chunk, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") + + sent += n + counts[wid] = sent + pos = (pos + n) % m + + if interval_ns > 0.0: + target = pace_start + int(sent * interval_ns) + sleep_ns = target - (time.perf_counter_ns() - pace_start) + if sleep_ns > 1_000_000: + time.sleep(sleep_ns / 1_000_000_000) + print(f"Sender {wid} finished sending {sent} events") + + +def main(argv): + ap = argparse.ArgumentParser(description="Fast columnar CSV replay sender for QuestDB") + ap.add_argument("--addrs", default="localhost:9000", + help="comma-separated host:port (QWP/WebSocket port, usually :9000)") + ap.add_argument("--total-events", type=int, default=1_000_000) + ap.add_argument("--num-senders", type=int, default=1, + help="worker threads, each with its own columnar client") + ap.add_argument("--chunk-rows", type=int, default=100_000, + help="rows per Client.dataframe call (pacing granularity + peak-memory bound)") + ap.add_argument("--rate", type=int, default=0, + help="target aggregate rows/second across ALL workers (0 = flat out)") + ap.add_argument("--csv", default="../trades20250728.csv.gz") + ap.add_argument("--timestamp-from-file", action="store_true", + help="use the CSV timestamp column instead of a live 'now' stamp") + # Enterprise auth / TLS + ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") + ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") + ap.add_argument("--password", default=None, help="basic-auth password") + ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", + help="certificate verification; use unsafe_off for self-signed") + ap.add_argument("--connect-timeout", type=float, default=10.0, + help="seconds to wait for the server on a preflight check before failing " + "loudly (0 = skip; the ingest client would otherwise retry forever)") + args = ap.parse_args(argv) + + for name, val in (("--total-events", args.total_events), ("--num-senders", args.num_senders), + ("--chunk-rows", args.chunk_rows)): + if val <= 0: + print(f"{name} must be > 0", file=sys.stderr) + return 2 + if args.rate < 0: + print("--rate must be >= 0", file=sys.stderr) + return 2 + if not os.path.exists(args.csv): + print(f"CSV file not found: {args.csv}", file=sys.stderr) + return 2 + # An empty --token (typically an unset env var, e.g. --token "$ILP_TOKEN" with + # ILP_TOKEN unset) silently disables auth+TLS. Against a TLS/auth server the client + # then hangs retrying the handshake at sent=0, so fail loudly instead. + if args.token is not None and not args.token.strip(): + print("[error] --token is empty (is $ILP_TOKEN set and exported?). It would connect " + "with NO auth/TLS and hang against a secured server. Set the token, or drop " + "--token to connect plaintext on purpose.", file=sys.stderr) + return 2 + if args.username and not (args.password or "").strip(): + print("[error] --username given but --password is empty.", file=sys.stderr) + return 2 + + base = load_base(args.csv, args.timestamp_from_file) + if base.height == 0: + print("CSV has no data rows.", file=sys.stderr) + return 2 + + conf = build_conf(args) + print(f"[conf] {conf.split('::')[0]} tls={use_tls(args)} " + f"auth={'token' if args.token else 'basic' if args.username else 'none'} " + f"| addrs: {args.addrs}") + + if args.connect_timeout > 0: + t0 = time.monotonic() + ok, msg = preflight(conf, args.connect_timeout) + if not ok: + print(f"[error] preflight failed: {msg}. Is QuestDB up and the token/addr correct? " + f"(set --connect-timeout 0 to skip this check)", file=sys.stderr) + return 2 + print(f"[preflight] server reachable ({time.monotonic() - t0:.2f}s)") + + print(f"Ingestion started (columnar QWP). base={base.height:,} rows, " + f"total={args.total_events:,}, workers={args.num_senders}, chunk-rows={args.chunk_rows:,}, " + f"timestamps={'file' if args.timestamp_from_file else 'live-now'}, " + f"pacing={'rate ' + str(args.rate) + ' rows/s' if args.rate else 'flat out'}") + + counts = [0] * args.num_senders + stop = threading.Event() + start = time.monotonic() + + def reporter(): + last = 0 + while not stop.is_set(): + stop.wait(1.0) + now = sum(counts) + print(f"[progress] sent={now:,} rate={now - last:,} rows/s") + last = now + + rep = threading.Thread(target=reporter, daemon=True) + rep.start() + + base_events = args.total_events // args.num_senders + rem = args.total_events % args.num_senders + errors = [] + + def wrapper(wid, ev): + try: + run_worker(wid, ev, base, args, counts) + except Exception as e: # noqa: BLE001 + errors.append(f"Sender {wid}: {e}") + + # Daemon workers + a polling join so Ctrl+C is honoured even while a worker is + # blocked in a native connect/flush: KeyboardInterrupt fires between join timeouts, + # and daemon threads don't hold the process open once main returns. + threads = [] + for wid in range(args.num_senders): + ev = base_events + (1 if wid < rem else 0) + t = threading.Thread(target=wrapper, args=(wid, ev), daemon=True) + t.start() + threads.append(t) + try: + while any(t.is_alive() for t in threads): + for t in threads: + t.join(0.2) + except KeyboardInterrupt: + stop.set() + print(f"\nInterrupted. Sent ~{sum(counts):,} rows; exiting.", file=sys.stderr) + return 130 + stop.set() + + if errors: + for e in errors: + print(f"Worker failed: {e}", file=sys.stderr) + return 1 + + elapsed = time.monotonic() - start + rate = args.total_events / elapsed if elapsed > 0 else 0 + print(f"All workers completed. events={args.total_events:,} elapsed={elapsed:.3f}s " + f"throughput={rate:,.0f} rows/s") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/python/csv_parallel_sender.py b/python/csv_parallel_sender.py new file mode 100644 index 0000000..5f7833e --- /dev/null +++ b/python/csv_parallel_sender.py @@ -0,0 +1,424 @@ +#!/usr/bin/env python3 +"""Parallel CSV replay sender for QuestDB, a Python port of the Java/Rust ha_sender. + +Replays a CSV of trades in a loop across N worker threads over one of three transports +(``--protocol``): + + * ``qwp`` - QWP over WebSocket: store-and-forward (un-acked frames spill to disk and + replay after an outage), transactional commit, multi-host failover. A + background probe polls the latest ingested timestamp and the serving + node's live role over a QWP query client (``Client``). + * ``qwpudp`` - QWP over UDP: fire-and-forget datagrams to :9007. Ingest-only, + unauthenticated, single-endpoint, best-effort (no ack, no failover). + * ``ilp`` - the legacy ILP/HTTP transport. + +Requires the QWP/egress build of the questdb Python client (see README.md); the PyPI +release does not expose QWP or the query client. +""" + +import argparse +import gzip +import csv as csvmod +import os +import sys +import threading +import time +from datetime import datetime, timezone + +from questdb.ingress import Sender, Client, TimestampNanos, ServerTimestamp + +PROBE_QUERY = "select timestamp from trades limit -1" +# Enterprise lifecycle status of whichever node the query client is connected to. Returns +# the LIVE role (current_role / target_role), unlike the QWP handshake which only refreshes +# on reconnect and so goes stale after an in-place primary<->replica switch. +STATUS_QUERY = "switch status" + + +def parse_args(argv): + p = argparse.ArgumentParser(description="Parallel CSV replay sender for QuestDB") + p.add_argument("--addrs", default="localhost:9000", + help="Comma-separated host:port. QWP/WebSocket + ILP use :9000; UDP uses :9007.") + p.add_argument("--token", default=None, help="Bearer token (QWP/WebSocket + ILP only).") + p.add_argument("--username", default=None, help="Basic-auth username.") + p.add_argument("--password", default=None, help="Basic-auth password.") + p.add_argument("--total-events", type=int, default=1_000_000) + p.add_argument("--delay-ms", type=int, default=50, + help="Per-row sleep in ms (0 = flat out). Ignored when --rate > 0.") + p.add_argument("--rate", type=int, default=0, + help="Target aggregate rows/second across ALL workers (0 = off, use " + "--delay-ms). Takes precedence over --delay-ms: each worker paces to " + "rate/num-senders against a deadline schedule.") + p.add_argument("--num-senders", type=int, default=10) + p.add_argument("--retry-timeout", type=int, default=360_000, + help="retry_timeout (ILP) / reconnect_max_duration_millis (QWP), in ms.") + p.add_argument("--csv", default="./trades20250728.csv.gz") + p.add_argument("--timestamp-from-file", action="store_true") + p.add_argument("--seconds-offset", type=int, default=0) + p.add_argument("--protocol", default="qwp", choices=["qwp", "qwpudp", "ilp"]) + p.add_argument("--sender-id", default="ha_sender") + p.add_argument("--store-forward-dir", default="/tmp/qdb-sf") + p.add_argument("--batch-size", type=int, default=10_000) + p.add_argument("--batches-per-transaction", type=int, default=10) + p.add_argument("--probe-interval-ms", type=int, default=1000) + p.add_argument("--enterprise", action="store_true") + p.add_argument("--zone", default="eu-west-1") + return p.parse_args(argv) + + +def addr_list(args): + return [a.strip() for a in args.addrs.split(",") if a.strip()] + + +def has_token(args): + return bool(args.token) + + +def has_basic(args): + return bool(args.username) and bool(args.password) + + +def tls(args): + return has_token(args) or has_basic(args) + + +def _append_auth(parts, args): + if has_token(args): + parts.append(f"token={args.token};") + elif has_basic(args): + parts.append(f"username={args.username};password={args.password};") + + +def build_ingest_conf(args, worker_id): + """Ingestion connect string for the configured transport (same keys as the Rust port).""" + addrs = addr_list(args) + if args.protocol == "qwpudp": + return f"qwpudp::addr={addrs[0]};auto_flush=off;" + + if args.protocol == "ilp": + scheme = "https" if tls(args) else "http" + parts = [f"{scheme}::addr={','.join(addrs)};", "auto_flush=off;"] + _append_auth(parts, args) + if tls(args): + parts.append("tls_verify=unsafe_off;") + parts.append(f"retry_timeout={args.retry_timeout};") + return "".join(parts) + + # qwp (WebSocket) + scheme = "qwpwss" if tls(args) else "qwpws" + who = f"{args.sender_id}-{worker_id}" + sf = os.path.join(args.store_forward_dir, who) + os.makedirs(sf, exist_ok=True) + parts = [f"{scheme}::addr={','.join(addrs)};", "auto_flush=off;"] + _append_auth(parts, args) + if tls(args): + parts.append("tls_verify=unsafe_off;") + parts.append(f"sender_id={who};") + parts.append(f"sf_dir={sf};") + parts.append(f"reconnect_max_duration_millis={args.retry_timeout};") + parts.append("reconnect_initial_backoff_millis=100;") + parts.append("reconnect_max_backoff_millis=5000;") + if args.enterprise: + parts.append("request_durable_ack=true;") + return "".join(parts) + + +def build_client_conf(args): + """Connect string for the QWP query client (probe): target=any (replica-fallback + reads), failover on, zone bias. Same hosts/auth as the senders.""" + addrs = addr_list(args) + scheme = "qwpwss" if tls(args) else "qwpws" + parts = [f"{scheme}::addr={','.join(addrs)};", "target=any;", "failover=true;"] + _append_auth(parts, args) + if tls(args): + parts.append("tls_verify=unsafe_off;") + if args.zone: + parts.append(f"zone={args.zone};") + return "".join(parts) + + +def iso_to_nanos(raw): + """Parse an ISO-8601 timestamp to epoch nanoseconds, preserving full nanosecond + precision. datetime is microsecond-limited and would drop the last 3 digits of a + TIMESTAMP_NS value (e.g. ...192508297Z -> ...192508000), so the fractional seconds are + split out and parsed as an integer, and only the whole-second part goes through datetime.""" + raw = raw.strip() + frac_nanos = 0 + dot = raw.find(".") + if dot != -1: + end = dot + 1 + while end < len(raw) and raw[end].isdigit(): + end += 1 + frac = raw[dot + 1:end] + frac_nanos = int(frac[:9].ljust(9, "0")) + raw = raw[:dot] + raw[end:] + dt = datetime.fromisoformat(raw.replace("Z", "+00:00")) + if dt.tzinfo is None: + dt = dt.replace(tzinfo=timezone.utc) + return int(dt.timestamp()) * 1_000_000_000 + frac_nanos + + +def load_csv(path, need_timestamp): + """Load the CSV (optionally gzipped). Returns a list of (symbol, side, price, amount, + ts_nanos) tuples; ts_nanos is 0 unless need_timestamp.""" + opener = gzip.open if path.endswith(".gz") else open + rows = [] + with opener(path, "rt", encoding="utf-8", newline="") as fh: + reader = csvmod.reader(fh) + header = next(reader, None) + if header is None: + return rows + idx = {name.strip(): i for i, name in enumerate(header)} + for col in ("symbol", "side", "price", "amount"): + if col not in idx: + raise ValueError(f"CSV missing required column: {col}") + if need_timestamp and "timestamp" not in idx: + raise ValueError("CSV missing required column: timestamp") + for rec in reader: + if not rec: + continue + ts_nanos = 0 + if need_timestamp: + ts_nanos = iso_to_nanos(rec[idx["timestamp"]].strip()) + rows.append(( + rec[idx["symbol"]].strip(), + rec[idx["side"]].strip(), + float(rec[idx["price"]].strip()), + float(rec[idx["amount"]].strip()), + ts_nanos, + )) + return rows + + +def run_worker(worker_id, total_events, args, rows, counts): + print(f"Sender {worker_id} will send {total_events} events") + is_qwp = args.protocol == "qwp" + is_udp = args.protocol == "qwpudp" + # Single worker on a QWP transport stamps rows client-side (monotonic, no O3); ILP or + # more than one worker use server-side timestamps (ServerTimestamp). + per_row_ts = (is_qwp or is_udp) and args.num_senders == 1 + # Flush cadence: qwp commits every batch-size*batches-per-transaction; qwpudp/ilp flush + # every batch-size rows (auto_flush is off, so the buffer is drained explicitly). + commit_every = args.batch_size * args.batches_per_transaction if is_qwp else args.batch_size + + # Rate limiting (when --rate > 0): pace this worker to its share of the aggregate target, + # rate / num-senders rows/second. interval_nanos is the ideal spacing between this worker's + # rows; we sleep only when running ahead of a deadline schedule, so rows go out back-to-back + # until we get ahead. Reaches high targets a fixed per-row sleep cannot. + interval_nanos = (1_000_000_000 * args.num_senders / args.rate) if args.rate > 0 else 0.0 + rate_limited = interval_nanos > 0.0 + + conf = build_ingest_conf(args, worker_id) + n = len(rows) + sent = 0 + pace_start = time.perf_counter_ns() + with Sender.from_conf(conf) as sender: + for i in range(total_events): + symbol, side, price, amount, ts_nanos = rows[i % n] + if args.timestamp_from_file: + at = TimestampNanos(ts_nanos + args.seconds_offset * 1_000_000_000) + elif args.seconds_offset != 0: + at = TimestampNanos(time.time_ns() + args.seconds_offset * 1_000_000_000) + elif per_row_ts: + at = TimestampNanos.now() + else: + at = ServerTimestamp + sender.row( + "trades", + symbols={"symbol": symbol, "side": side}, + columns={"price": price, "amount": amount, "trade_id": f"{worker_id}-{i + 1}"}, + at=at, + ) + sent += 1 + counts[worker_id] = sent + if sent % commit_every == 0: + sender.flush() + if rate_limited: + # Deadline for the row just sent (sent is 1-based). Sleep only while ahead of + # schedule; the 1ms floor coalesces many rows into one sleep at high rates. + target_nanos = int(sent * interval_nanos) + sleep_nanos = target_nanos - (time.perf_counter_ns() - pace_start) + if sleep_nanos > 1_000_000: + time.sleep(sleep_nanos / 1_000_000_000) + elif args.delay_ms > 0: + time.sleep(args.delay_ms / 1000.0) + sender.flush() + # QWP/WebSocket: drain any un-acked store-and-forward frames before closing. + if is_qwp: + sender.close_drain() + print(f"Sender {worker_id} finished sending {sent} events") + + +def _switch_status_roles(client): + """Return (current_role, target_role) from `switch status`, or (None, None).""" + try: + df = client.query(STATUS_QUERY).to_pandas() + except Exception: + return None, None + if len(df) == 0: + return None, None + current = target = None + last = df.iloc[-1] + for col in df.columns: + lc = col.lower() + if "role" not in lc: + continue + val = last[col] + if "current" in lc: + current = val + elif "target" in lc: + target = val + elif current is None: + current = val + return current, target + + +def run_probe(args, stop): + conf = build_client_conf(args) + try: + client = Client.from_conf(conf) + except Exception as e: + print(f"[query client] connect failed: {e}") + return + print("[query client] connected") + was_down = False + try: + while not stop.is_set(): + try: + df = client.query(PROBE_QUERY).to_pandas() + if was_down: + print("[query client] connection restored") + was_down = False + if len(df): + latest = df.iloc[-1, 0] + current, target = _switch_status_roles(client) + if current is not None: + switching = target is not None and str(target).lower() != str(current).lower() + sw = f" (switching -> {target})" if switching else "" + served = f" served by role={current}{sw}" + else: + served = " (live 'switch status' unavailable, e.g. OSS or missing SYSTEM ADMIN)" + print(f"[probe] latest trades timestamp = {latest}{served}") + except Exception as e: + if not was_down: + print(f"[query client] connection lost ({e}), will retry") + was_down = True + stop.wait(args.probe_interval_ms / 1000.0) + finally: + client.close() + + +def main(argv): + args = parse_args(argv) + + for name, val in (("--num-senders", args.num_senders), ("--total-events", args.total_events), + ("--batch-size", args.batch_size), + ("--batches-per-transaction", args.batches_per_transaction)): + if val <= 0: + print(f"{name} must be > 0", file=sys.stderr) + return 2 + + addrs = addr_list(args) + if not addrs: + print("--addrs must contain at least one host:port", file=sys.stderr) + return 2 + if args.protocol == "qwpudp": + if len(addrs) > 1: + print("--protocol qwpudp supports a single host:port only (UDP is connectionless, " + f"there is no failover); got {len(addrs)} addresses", file=sys.stderr) + return 2 + if args.token or args.username or args.password: + print("[warn] --protocol qwpudp is unauthenticated (UDP accepts any connection); " + "ignoring --token/--username/--password", file=sys.stderr) + + if args.rate < 0: + print("--rate must be >= 0 (0 disables rate limiting, falling back to --delay-ms)", + file=sys.stderr) + return 2 + # --rate takes precedence: when set it drives pacing and --delay-ms is ignored. + if args.rate > 0 and args.delay_ms > 0: + print(f"[warn] --rate {args.rate} overrides --delay-ms {args.delay_ms} (rate limiting " + "drives pacing; the per-row delay is ignored)", file=sys.stderr) + + if not os.path.exists(args.csv): + print(f"CSV file not found: {args.csv}", file=sys.stderr) + return 2 + rows = load_csv(args.csv, args.timestamp_from_file) + if not rows: + print("CSV has no data rows.", file=sys.stderr) + return 2 + + if args.protocol == "qwp": + print(f"Ingestion started. Protocol: qwp (WebSocket, sender-id={args.sender_id}, " + f"store-and-forward={args.store_forward_dir}, batch-size={args.batch_size}, " + f"batches-per-transaction={args.batches_per_transaction}, " + f"retry-timeout-ms={args.retry_timeout}) | addrs: {','.join(addrs)}") + elif args.protocol == "qwpudp": + print(f"Ingestion started. Protocol: qwpudp (QWP/UDP datagrams, ingest-only, " + f"unauthenticated, no store-and-forward, no failover; query client disabled, " + f"batch-size={args.batch_size}) | addr: {addrs[0]}") + else: + print(f"Ingestion started. Protocol: ilp (HTTP, retry-timeout-ms={args.retry_timeout}) " + f"| addrs: {','.join(addrs)}") + + if args.rate > 0: + print(f"Pacing: rate={args.rate} rows/s (aggregate across {args.num_senders} workers)") + else: + print(f"Pacing: delay-ms={args.delay_ms}") + + counts = [0] * args.num_senders + stop = threading.Event() + start = time.monotonic() + + def reporter(): + last = 0 + while not stop.is_set(): + stop.wait(1.0) + now = sum(counts) + print(f"[progress] sent={now} rate={now - last} rows/s") + last = now + + threads = [] + rep = threading.Thread(target=reporter, daemon=True) + rep.start() + + probe_thread = None + if args.protocol == "qwp" and args.probe_interval_ms > 0: + probe_thread = threading.Thread(target=run_probe, args=(args, stop), daemon=True) + probe_thread.start() + + base = args.total_events // args.num_senders + rem = args.total_events % args.num_senders + errors = [] + + def worker_wrapper(wid, events): + try: + run_worker(wid, events, args, rows, counts) + except Exception as e: # noqa: BLE001 + errors.append(f"Sender {wid}: {e}") + + for wid in range(args.num_senders): + events = base + (1 if wid < rem else 0) + t = threading.Thread(target=worker_wrapper, args=(wid, events)) + t.start() + threads.append(t) + + for t in threads: + t.join() + stop.set() + if probe_thread: + probe_thread.join(timeout=2.0) + + if errors: + for e in errors: + print(f"Worker failed: {e}", file=sys.stderr) + return 1 + + elapsed = time.monotonic() - start + rate = args.total_events / elapsed if elapsed > 0 else 0 + print(f"All workers completed. protocol={args.protocol} events={args.total_events} " + f"elapsed={elapsed:.3f} s throughput={rate:,.0f} rows/s") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/python/dataframe_demo.py b/python/dataframe_demo.py new file mode 100644 index 0000000..163958d --- /dev/null +++ b/python/dataframe_demo.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +"""Pandas and polars ingestion + egress round-trip over QWP/WebSocket. + +Showcases the columnar/dataframe capabilities the row-by-row sender does not: + + * pandas ingestion via ``Sender.dataframe`` (the numpy-backed pandas planner). + * polars ingestion via ``Client.dataframe`` (the pooled QWP Arrow-columnar path; + it takes polars / pyarrow / any Arrow C Stream source natively — numpy-backed + pandas is not accepted there, so pandas goes through ``Sender.dataframe``). + * egress via ``Client.query(...).to_pandas()`` and ``.to_polars()``. + +Usage: + python dataframe_demo.py [--addr localhost:9000] [--csv ../trades20250728.csv.gz] [--rows 5000] + +Requires the QWP/egress build of the questdb client (see README.md). +""" + +import argparse +import sys +import time +import urllib.parse +import urllib.request + +import pandas as pd +import polars as pl + +from questdb.ingress import Sender, Client + +TABLE = "df_demo" + + +def exec_sql(addr, sql): + """Run a statement over the HTTP /exec endpoint (same host, HTTP port).""" + url = f"http://{addr}/exec?" + urllib.parse.urlencode({"query": sql}) + with urllib.request.urlopen(url, timeout=10) as resp: + resp.read() + + +def wait_for_count(client, table, at_least, timeout_s=30): + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + try: + n = int(client.query(f"select count() from {table}").to_pandas().iloc[0, 0]) + if n >= at_least: + return n + except Exception: + pass + time.sleep(0.5) + return -1 + + +def load_pandas(csv_path, rows): + """Load the CSV into a pandas DataFrame with dtypes the ingestion path accepts + (object strings, float64, tz-aware nanosecond timestamps).""" + raw = pd.read_csv(csv_path, nrows=rows) + return pd.DataFrame({ + "symbol": raw["symbol"].astype("object"), + "side": raw["side"].astype("object"), + "price": raw["price"].astype("float64"), + "amount": raw["amount"].astype("float64"), + "timestamp": pd.to_datetime(raw["timestamp"], utc=True).dt.as_unit("ns"), + }) + + +def main(argv): + ap = argparse.ArgumentParser(description="Pandas/polars ingestion + egress demo") + ap.add_argument("--addr", default="localhost:9000") + ap.add_argument("--csv", default="../trades20250728.csv.gz") + ap.add_argument("--rows", type=int, default=5000) + args = ap.parse_args(argv) + + pdf = load_pandas(args.csv, args.rows) + print(f"loaded pandas DataFrame: {pdf.shape[0]} rows\n{pdf.dtypes}\n") + pldf = pl.from_pandas(pdf) + print(f"built polars DataFrame: {pldf.height} rows\n") + + exec_sql(args.addr, f"drop table if exists {TABLE}") + + # --- INGESTION --- + # pandas -> Sender.dataframe (numpy-backed pandas planner) over QWP/WebSocket. + with Sender.from_conf(f"qwpws::addr={args.addr};auto_flush=off;") as sender: + sender.dataframe(pdf, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") + sender.flush() + print(f"[ingest] pandas via Sender.dataframe: {pdf.shape[0]} rows") + + with Client.from_conf(f"qwpws::addr={args.addr};") as client: + # polars -> Client.dataframe (Arrow-columnar path, polars is native). + client.dataframe(pldf, table_name=TABLE, symbols=["symbol", "side"], at="timestamp") + print(f"[ingest] polars via Client.dataframe: {pldf.height} rows") + + expected = pdf.shape[0] + pldf.height + n = wait_for_count(client, TABLE, expected) + print(f"[ingest] server applied {n} rows (expected {expected})\n") + + # --- EGRESS --- + pd_out = client.query( + f"select symbol, count() n, round(avg(price), 2) avg_price " + f"from {TABLE} order by n desc limit 5" + ).to_pandas() + print("[egress] Client.query(...).to_pandas():") + print(pd_out.to_string(index=False)) + print() + + pl_out = client.query( + f"select side, count() n, round(sum(amount), 4) total_amount " + f"from {TABLE} group by side order by side" + ).to_polars() + print("[egress] Client.query(...).to_polars():") + print(pl_out) + + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/python/enrich_polars_demo.py b/python/enrich_polars_demo.py new file mode 100644 index 0000000..7e55db2 --- /dev/null +++ b/python/enrich_polars_demo.py @@ -0,0 +1,196 @@ +#!/usr/bin/env python3 +"""Stream a table through polars, enrich it, and write it back - chunked. + +Memory-efficient polars round-trip over the QWP query client. Rather than +materializing the whole result set, it streams the read in Arrow record batches +and ingests each enriched batch before pulling the next, so peak memory is one +batch (not all ``--limit`` rows): + + * READ - ``Client.query(sql).iter_arrow()`` yields ``pyarrow.RecordBatch`` chunks + one at a time from a server-side cursor (verified streaming in the client: + ``iter_arrow`` pulls via ``line_reader_cursor_next_batch``; ``to_polars`` by + contrast is "materialise-whole" and would buffer every row in RAM); + ``pl.from_arrow(batch)`` wraps each into a polars DataFrame (zero-copy). + * ENRICH - add an ``enriched_rnd`` column: a random ``A``-``Z`` value per row. + * WRITE - ``Client.dataframe(chunk, ...)`` ingests each polars chunk into + ``enriched__demo`` over the QWP Arrow-columnar path. + +Both the read and the write are polars DataFrames. No pandas in the round-trip. +The read and the write use separate ``Client`` connections so the open read +stream and the ingest never share a socket. + +Column typing is automatic: QuestDB SYMBOL columns come back as polars ``Categorical`` +and VARCHAR as ``String``, and ``Client.dataframe``'s default ``symbols='auto'`` maps +``Categorical -> SYMBOL`` and ``String -> VARCHAR`` on the way back in. ``enriched_rnd`` +is built as a ``Categorical`` so it too lands as a SYMBOL. + +Auth / TLS (QuestDB Enterprise): pass ``--token`` (bearer) or ``--username``/``--password`` +(basic). Either turns on TLS automatically (scheme ``qwpwss``); ``--tls`` forces TLS with +no auth. Certificate verification is on by default; use ``--tls-verify unsafe_off`` for +self-signed certs. The same credentials are applied to the HTTP ``/exec`` calls used for +drop/verify. + +Usage: + python enrich_polars_demo.py TABLE \ + [--addr host:9000] [--limit 200000000] [--timestamp-col timestamp] \ + [--seed 42] [--keep] \ + [--token TOK | --username U --password P] [--tls] [--tls-verify on|unsafe_off] + +By default the target table is dropped and recreated on each run; pass ``--keep`` +to append instead. ``--limit`` defaults to the latest 200 million rows by designated +timestamp, streamed ascending so re-ingest stays in timestamp order. Requires the +QWP/egress build of the client (see README.md). +""" + +import argparse +import base64 +import json +import ssl +import sys +import time +import urllib.parse +import urllib.request + +import numpy as np +import polars as pl + +from questdb.ingress import Client + +LETTERS = np.array(list("ABCDEFGHIJKLMNOPQRSTUVWXYZ")) + + +def use_tls(args): + return bool(args.tls or args.token or (args.username and args.password)) + + +def build_conf(args): + """QWP connect string for the reader/writer, with optional auth + TLS.""" + scheme = "qwpwss" if use_tls(args) else "qwpws" + parts = [f"{scheme}::addr={args.addr};"] + if args.token: + parts.append(f"token={args.token};") + elif args.username and args.password: + parts.append(f"username={args.username};password={args.password};") + if use_tls(args) and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def exec_sql(args, sql): + """Run a statement over the HTTP(S) /exec endpoint (auth-aware), return JSON.""" + scheme = "https" if use_tls(args) else "http" + host = args.addr.split(",")[0] + url = f"{scheme}://{host}/exec?" + urllib.parse.urlencode({"query": sql}) + req = urllib.request.Request(url) + if args.token: + req.add_header("Authorization", f"Bearer {args.token}") + elif args.username and args.password: + cred = base64.b64encode(f"{args.username}:{args.password}".encode()).decode() + req.add_header("Authorization", f"Basic {cred}") + ctx = None + if use_tls(args) and args.tls_verify == "unsafe_off": + ctx = ssl._create_unverified_context() + with urllib.request.urlopen(req, timeout=60, context=ctx) as resp: + return json.load(resp) + + +def wait_for_count(args, table, at_least, timeout_s=60): + """Poll until the table holds >= at_least rows (QWP commits asynchronously).""" + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + try: + n = int(exec_sql(args, f"select count() from {table}")["dataset"][0][0]) + if n >= at_least: + return n + except Exception: + pass + time.sleep(0.25) + return -1 + + +def enrich(chunk, rng): + """Add an `enriched_rnd` column of random A-Z letters (polars Categorical, so + it lands as a QuestDB SYMBOL). `rng` carries state across chunks so the random + values are not identical batch-to-batch.""" + rnd = LETTERS[rng.integers(0, 26, size=chunk.height)] + return chunk.with_columns(pl.Series("enriched_rnd", rnd, dtype=pl.Categorical)) + + +def main(argv): + ap = argparse.ArgumentParser(description="Chunked polars read -> enrich -> write") + ap.add_argument("table", help="source table name") + ap.add_argument("--addr", default="localhost:9000", help="host:port (QWP/HTTP port)") + ap.add_argument("--limit", type=int, default=200_000_000, + help="max rows to read (latest N by timestamp); default 200,000,000") + ap.add_argument("--timestamp-col", default="timestamp", + help="designated timestamp column of the source table") + ap.add_argument("--seed", type=int, default=42, help="RNG seed for enriched_rnd") + ap.add_argument("--keep", action="store_true", + help="append to the target instead of dropping it first") + # Enterprise auth / TLS + ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") + ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") + ap.add_argument("--password", default=None, help="basic-auth password") + ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", + help="certificate verification; use unsafe_off for self-signed") + args = ap.parse_args(argv) + + ts_col = args.timestamp_col + target = f"enriched_{args.table}_demo" + rng = np.random.default_rng(args.seed) + conf = build_conf(args) + print(f"[conf] {conf.split('::')[0]} tls={use_tls(args)} " + f"auth={'token' if args.token else 'basic' if args.username else 'none'}") + + if not args.keep: + try: + exec_sql(args, f"drop table if exists {target}") + except Exception as e: + print(f"[warn] could not drop '{target}' via /exec ({e}); " + f"continuing (rows will append)", file=sys.stderr) + + # `limit -N` returns the latest N rows in designated-timestamp (ascending) + # order already, so the stream re-ingests in order with no explicit sort. + sql = f"select * from {args.table} limit -{args.limit}" + + total = 0 + t0 = time.monotonic() + # Separate connections: the read stream stays open while each chunk is written. + with Client.from_conf(conf) as reader, Client.from_conf(conf) as writer: + result = reader.query(sql) + for i, batch in enumerate(result.iter_arrow()): + chunk = pl.from_arrow(batch) # RecordBatch -> polars, zero-copy + if chunk.height == 0: + continue + chunk = enrich(chunk, rng) + writer.dataframe(chunk, table_name=target, at=ts_col) + total += chunk.height + print(f"[chunk {i}] {chunk.height:,} rows enriched + written " + f"(running total {total:,})") + + if total == 0: + print("[read] source table is empty - nothing to enrich", file=sys.stderr) + return 1 + print(f"[done] streamed {total:,} rows -> '{target}' in " + f"{time.monotonic() - t0:.1f}s (peak memory ~one batch)") + + # --- VERIFY server-side (QWP applies the commit asynchronously) --- + expected = total if not args.keep else 0 + n = wait_for_count(args, target, expected) + print(f"[verify] '{target}' now holds {n:,} rows") + try: + dist = exec_sql( + args, + f"select enriched_rnd, count() n from {target} " + f"group by enriched_rnd order by enriched_rnd" + )["dataset"] + print(f"[verify] enriched_rnd distinct values: {len(dist)} (sample: {dist[:5]})") + except Exception as e: + print(f"[warn] verify query failed: {e}", file=sys.stderr) + + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/python/read_bench.py b/python/read_bench.py new file mode 100644 index 0000000..b6fcbbe --- /dev/null +++ b/python/read_bench.py @@ -0,0 +1,208 @@ +#!/usr/bin/env python3 +"""Read the last N rows of a table as fast as Python allows, reporting rows/sec and +throughput (MB/s, Gb/s) live. + +Throughput is measured as the decoded Arrow payload (``batch.nbytes``), i.e. the data +volume handed to the reader, not the compressed QWP wire bytes (which the client does +not expose) - so it is a data-throughput figure, generally larger than the on-wire rate. + +Fastest path, by design: consume the QWP query cursor's Arrow batches directly +(``Client.query(sql).iter_arrow()``) and just tally ``batch.num_rows``. No pandas +or polars conversion - the Rust client already decodes the wire into Arrow inside +``iter_arrow`` (that decode is the unavoidable "read" work, done in native code), +so the Python loop does nothing but add row counts. Converting each batch to +polars (``pl.from_arrow``) or pandas (``iter_pandas``) would only add a per-batch +materialisation step on top and measure a slower number. (``to_arrow``/``to_polars`` +are "materialise-whole" - they buffer the entire result; ``iter_arrow`` streams.) + +A single connection is capped by the client's socket buffer at ~426 KB per RTT (the +QWP client hardcodes a 4 MiB buffer that default Linux clamps; see ../boost_tcp.sh), +so on a real network one reader is round-trip bound no matter how lean the loop is. +``--readers N`` splits the row range across N parallel connections (each scans a +timestamp slice) and reports the aggregate rows/sec, which is the way past the +per-connection wall. ``--readers 1`` (default) is the plain single stream. + +Auth / TLS (QuestDB Enterprise): pass ``--token`` (bearer) or ``--username``/``--password`` +(basic). Either turns on TLS automatically (scheme ``qwpwss``); ``--tls`` forces TLS with +no auth. Certificate verification is on by default; use ``--tls-verify unsafe_off`` for +self-signed certs. + +Usage: + python read_bench.py TABLE \ + [--addr host:9000] [--limit 10000000] [--readers 1] \ + [--token TOK | --username U --password P] [--tls] [--tls-verify on|unsafe_off] + +Requires the QWP/egress build of the client (see README.md). +""" + +import argparse +import sys +import threading +import time +from datetime import datetime, timezone + +from questdb.ingress import Client + + +def use_tls(args): + return bool(args.tls or args.token or (args.username and args.password)) + + +def build_conf(args): + """QWP connect string for the reader(s), with optional auth + TLS.""" + scheme = "qwpwss" if use_tls(args) else "qwpws" + parts = [f"{scheme}::addr={args.addr};"] + if args.token: + parts.append(f"token={args.token};") + elif args.username and args.password: + parts.append(f"username={args.username};password={args.password};") + if use_tls(args) and args.tls_verify == "unsafe_off": + parts.append("tls_verify=unsafe_off;") + return "".join(parts) + + +def ns_to_iso(ns): + """Epoch nanoseconds -> ISO-8601 string with nanosecond precision (unit-safe + for both TIMESTAMP and TIMESTAMP_NS columns in a SQL literal).""" + sec, frac = divmod(int(ns), 1_000_000_000) + dt = datetime.fromtimestamp(sec, tz=timezone.utc) + return dt.strftime("%Y-%m-%dT%H:%M:%S") + f".{frac:09d}Z" + + +def split_queries(args, conf): + """Build one SELECT per reader. For a single reader, the exact `limit -N` + query; for N readers, split the last-N rows' timestamp range into N slices.""" + table, ts, limit, readers = args.table, args.timestamp_col, args.limit, args.readers + if readers <= 1: + return [f"select * from {table} limit -{limit}"] + # Range of the latest `limit` rows, as epoch nanoseconds. + with Client.from_conf(conf) as c: + mm = c.query( + f"select min({ts}) lo, max({ts}) hi " + f"from (select {ts} from {table} limit -{limit})" + ).to_polars() + if mm.height == 0 or mm["lo"][0] is None: + return [f"select * from {table} limit -{limit}"] + lo = mm["lo"].dt.epoch("ns")[0] + hi = mm["hi"].dt.epoch("ns")[0] + if hi <= lo: # single instant / one row - can't split by time + print("[warn] timestamp range too narrow to split; using 1 reader", file=sys.stderr) + return [f"select * from {table} limit -{limit}"] + span = hi - lo + edges = [lo + (span * i) // readers for i in range(readers + 1)] + edges[-1] = hi + sqls = [] + for i in range(readers): + a = ns_to_iso(edges[i]) + b = ns_to_iso(edges[i + 1]) + op = "<=" if i == readers - 1 else "<" # last slice includes the max row + sqls.append(f"select * from {table} where {ts} >= '{a}' and {ts} {op} '{b}'") + return sqls + + +def run_reader(idx, sql, conf, counts, byts, errors): + try: + with Client.from_conf(conf) as client: + result = client.query(sql) + n = 0 + b = 0 + for batch in result.iter_arrow(): + n += batch.num_rows # count rows and decoded Arrow bytes + b += batch.nbytes # (buffer bytes of this batch's columns) + counts[idx] = n + byts[idx] = b + counts[idx] = n + byts[idx] = b + except Exception as e: # noqa: BLE001 + errors.append(f"reader {idx}: {e}") + + +def main(argv): + ap = argparse.ArgumentParser(description="Read last N rows as fast as possible, report rows/sec") + ap.add_argument("table", help="source table name") + ap.add_argument("--addr", default="localhost:9000", help="host:port (QWP/HTTP port)") + ap.add_argument("--limit", type=int, default=10_000_000, + help="number of most-recent rows to read; default 10,000,000") + ap.add_argument("--readers", type=int, default=1, + help="parallel reader connections (split by timestamp); default 1") + ap.add_argument("--timestamp-col", default="timestamp", + help="designated timestamp column (used to split across readers)") + ap.add_argument("--report-interval", type=float, default=0.5, + help="seconds between progress lines; default 0.5") + # Enterprise auth / TLS + ap.add_argument("--token", default=None, help="bearer token (turns on TLS)") + ap.add_argument("--username", default=None, help="basic-auth username (turns on TLS)") + ap.add_argument("--password", default=None, help="basic-auth password") + ap.add_argument("--tls", action="store_true", help="force TLS (qwpwss) with no auth") + ap.add_argument("--tls-verify", choices=["on", "unsafe_off"], default="on", + help="certificate verification; use unsafe_off for self-signed") + args = ap.parse_args(argv) + + if args.readers < 1: + print("--readers must be >= 1", file=sys.stderr) + return 2 + + conf = build_conf(args) + print(f"[conf] {conf.split('::')[0]} tls={use_tls(args)} " + f"auth={'token' if args.token else 'basic' if args.username else 'none'}") + sqls = split_queries(args, conf) + readers = len(sqls) + print(f"[scan] reading last {args.limit:,} rows of '{args.table}' " + f"across {readers} reader(s) ...") + + counts = [0] * readers + byts = [0] * readers + errors = [] + stop = threading.Event() + + def reporter(): + last_r, last_b = 0, 0 + # stop.wait() returns True once the event is set -> exit without a final + # stray line after [done]; it returns False on timeout -> print a tick. + while not stop.wait(args.report_interval): + r, b = sum(counts), sum(byts) + rps = (r - last_r) / args.report_interval + mbps = (b - last_b) / args.report_interval / 1e6 + print(f"[scan] {r:>14,} rows | {rps:>13,.0f} rows/s | {mbps:>9,.1f} MB/s") + last_r, last_b = r, b + + rep = threading.Thread(target=reporter, daemon=True) + rep.start() + + t0 = time.monotonic() + threads = [] + for i, sql in enumerate(sqls): + t = threading.Thread(target=run_reader, args=(i, sql, conf, counts, byts, errors)) + t.start() + threads.append(t) + for t in threads: + t.join() + elapsed = time.monotonic() - t0 + stop.set() + + if errors: + for e in errors: + print(f"reader failed: {e}", file=sys.stderr) + return 1 + + total = sum(counts) + tbytes = sum(byts) + if total == 0: + print(f"[done] '{args.table}' returned 0 rows", file=sys.stderr) + return 1 + rate = total / elapsed if elapsed > 0 else float("inf") + mbps = tbytes / elapsed / 1e6 if elapsed > 0 else float("inf") + gbps = tbytes * 8 / elapsed / 1e9 if elapsed > 0 else float("inf") + gib = tbytes / (1024 ** 3) + print(f"[done] {total:,} rows, {gib:.2f} GiB decoded in {elapsed:.3f}s " + f"across {readers} reader(s)") + print(f"[done] {rate:,.0f} rows/s | {mbps:,.1f} MB/s | {gbps:.2f} Gb/s " + f"(decoded Arrow payload, not wire bytes)") + if readers > 1: + per = " ".join(f"r{i}={c:,}" for i, c in enumerate(counts)) + print(f"[done] per-reader rows: {per}") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/regenerate_csv.sh b/regenerate_csv.sh new file mode 100755 index 0000000..daf3d81 --- /dev/null +++ b/regenerate_csv.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# +# Regenerate the replay CSV from QuestDB's public demo instance. +# +# Pulls the most recent trades from https://demo.questdb.io via the /exp CSV +# export endpoint, so the sender replays fresh data instead of a stale snapshot. +# +# Usage: +# ./regenerate_csv.sh [output-file] [limit] +# +# output-file destination path (default: trades.csv). If it ends in .gz the +# result is gzip-compressed, matching the sender's default +# --csv ./trades20250728.csv.gz (loadCsv auto-detects .gz). +# limit number of most-recent rows to fetch (default: 1000000). +# +# Env overrides: +# DEMO_HOST base URL (default: https://demo.questdb.io) +# +# Examples: +# ./regenerate_csv.sh # -> trades.csv, 1,000,000 rows +# ./regenerate_csv.sh trades.csv.gz # -> gzipped +# ./regenerate_csv.sh trades.csv 250000 # -> 250k rows +set -euo pipefail + +OUT="${1:-trades.csv}" +LIMIT="${2:-1000000}" +DEMO_HOST="${DEMO_HOST:-https://demo.questdb.io}" + +# select * gives symbol, side, price, amount, timestamp: exactly the columns the +# sender reads (timestamp only needed with --timestamp-from-file). Ordering desc +# gives the most recent rows; the sender stamps now() by default so order does +# not affect ingestion unless --timestamp-from-file is set. +QUERY="select * from trades order by timestamp desc limit ${LIMIT}" + +# Download to a temp file first so a failed/partial fetch never clobbers a good CSV. +TMP="$(mktemp "${TMPDIR:-/tmp}/trades.XXXXXX.csv")" +trap 'rm -f "$TMP"' EXIT + +echo "Fetching ${LIMIT} rows from ${DEMO_HOST}/exp ..." +# -G + --data-urlencode builds the encoded query string; --fail turns HTTP errors +# into a non-zero exit; --retry rides out transient network blips. +curl --fail --show-error --silent --retry 3 --retry-delay 1 --retry-all-errors \ + -G "${DEMO_HOST}/exp" \ + --data-urlencode "query=${QUERY}" \ + -o "$TMP" + +# Sanity: a valid export has a header line plus data. +LINES="$(wc -l < "$TMP" | tr -d ' ')" +if [ "$LINES" -lt 2 ]; then + echo "ERROR: export returned only ${LINES} line(s); leaving ${OUT} untouched." >&2 + echo "Response was:" >&2 + head -c 500 "$TMP" >&2 + exit 1 +fi + +case "$OUT" in + *.gz) + gzip -c "$TMP" > "$OUT" + ;; + *) + cp "$TMP" "$OUT" + ;; +esac + +echo "Wrote ${OUT} ($((LINES - 1)) data rows, header included)." diff --git a/regenerate_csv_fx.sh b/regenerate_csv_fx.sh new file mode 100755 index 0000000..b5eb23a --- /dev/null +++ b/regenerate_csv_fx.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# +# Regenerate the replay CSV from the FX table on QuestDB's public demo instance. +# +# Companion to regenerate_csv.sh (which pulls the crypto `trades` table). This one +# pulls `fx_trades`, so you replay FX symbols (EUR-USD, ...) instead of crypto. +# +# We ALWAYS export from the demo box (https://demo.questdb.io): it is continuously +# ingesting, so every export is a fresh, recent-price snapshot. You then recreate +# the table locally from the CSV for internal demos. +# +# Two schema notes about fx_trades vs the sender's fixed `trades` schema: +# * Its size column is `quantity`, so the query aliases `quantity AS amount` to +# match the `amount` column the sender writes. +# * Its designated `timestamp` is TIMESTAMP_NS (nanoseconds), where the crypto +# `trades` table is micros. This only matters with --timestamp-from-file; see +# the README ("Nanosecond vs microsecond timestamps"). In the default replay +# mode the sender stamps now() and ignores the file timestamp entirely. +# +# Usage: +# ./regenerate_csv_fx.sh [output-file] [limit] +# +# output-file destination path (default: fx_trades.csv). If it ends in .gz the +# result is gzip-compressed (loadCsv auto-detects .gz). +# limit number of most-recent rows to fetch (default: 1000000). +# +# Env overrides: +# DEMO_HOST base URL (default: https://demo.questdb.io). Intentionally the +# demo box; override only if you mirror fx_trades elsewhere. +# +# Examples: +# ./regenerate_csv_fx.sh # -> fx_trades.csv, 1,000,000 rows +# ./regenerate_csv_fx.sh fx_trades.csv.gz # -> gzipped +# ./regenerate_csv_fx.sh fx_trades.csv 250000 # -> 250k rows +set -euo pipefail + +OUT="${1:-fx_trades.csv}" +LIMIT="${2:-1000000}" +DEMO_HOST="${DEMO_HOST:-https://demo.questdb.io}" + +# Explicit column list (not select *) so we can alias quantity -> amount and drop +# the fx-only columns (ecn, counterparty, order_id, ...) the sender does not read. +# The result has exactly symbol, side, price, amount, timestamp: the sender's schema. +QUERY="select timestamp, symbol, side, price, quantity as amount from fx_trades order by timestamp desc limit ${LIMIT}" + +# Download to a temp file first so a failed/partial fetch never clobbers a good CSV. +TMP="$(mktemp "${TMPDIR:-/tmp}/fx_trades.XXXXXX.csv")" +trap 'rm -f "$TMP"' EXIT + +echo "Fetching ${LIMIT} rows from ${DEMO_HOST}/exp (fx_trades) ..." +# -G + --data-urlencode builds the encoded query string; --fail turns HTTP errors +# into a non-zero exit; --retry rides out transient network blips. +curl --fail --show-error --silent --retry 3 --retry-delay 1 --retry-all-errors \ + -G "${DEMO_HOST}/exp" \ + --data-urlencode "query=${QUERY}" \ + -o "$TMP" + +# Sanity: a valid export has a header line plus data. +LINES="$(wc -l < "$TMP" | tr -d ' ')" +if [ "$LINES" -lt 2 ]; then + echo "ERROR: export returned only ${LINES} line(s); leaving ${OUT} untouched." >&2 + echo "Response was:" >&2 + head -c 500 "$TMP" >&2 + exit 1 +fi + +case "$OUT" in + *.gz) + gzip -c "$TMP" > "$OUT" + ;; + *) + cp "$TMP" "$OUT" + ;; +esac + +echo "Wrote ${OUT} ($((LINES - 1)) data rows, header included)." diff --git a/rust/.gitignore b/rust/.gitignore new file mode 100644 index 0000000..ea8c4bf --- /dev/null +++ b/rust/.gitignore @@ -0,0 +1 @@ +/target diff --git a/rust/Cargo.lock b/rust/Cargo.lock new file mode 100644 index 0000000..7421cb0 --- /dev/null +++ b/rust/Cargo.lock @@ -0,0 +1,1857 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "aes" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +dependencies = [ + "cfg-if", + "cipher", + "cpufeatures 0.2.17", +] + +[[package]] +name = "android_system_properties" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" +dependencies = [ + "libc", +] + +[[package]] +name = "anstream" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" +dependencies = [ + "anstyle", + "anstyle-parse", + "anstyle-query", + "anstyle-wincon", + "colorchoice", + "is_terminal_polyfill", + "utf8parse", +] + +[[package]] +name = "anstyle" +version = "1.0.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" + +[[package]] +name = "anstyle-parse" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" +dependencies = [ + "utf8parse", +] + +[[package]] +name = "anstyle-query" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "anstyle-wincon" +version = "3.0.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" +dependencies = [ + "anstyle", + "once_cell_polyfill", + "windows-sys 0.61.2", +] + +[[package]] +name = "asn1-rs" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f6fd5ddaf0351dff5b8da21b2fb4ff8e08ddd02857f0bf69c47639106c0fff0" +dependencies = [ + "asn1-rs-derive 0.4.0", + "asn1-rs-impl 0.1.0", + "displaydoc", + "nom", + "num-traits", + "rusticata-macros", + "thiserror 1.0.69", +] + +[[package]] +name = "asn1-rs" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7f43a50ac4fdca5df8e885c21b835997f0a1cdee65494a6847694a98652d9d8" +dependencies = [ + "asn1-rs-derive 0.6.0", + "asn1-rs-impl 0.2.0", + "displaydoc", + "nom", + "num-traits", + "rusticata-macros", + "thiserror 2.0.18", + "time", +] + +[[package]] +name = "asn1-rs-derive" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "726535892e8eae7e70657b4c8ea93d26b8553afb1ce617caee529ef96d7dee6c" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", + "synstructure 0.12.6", +] + +[[package]] +name = "asn1-rs-derive" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3109e49b1e4909e9db6515a30c633684d68cdeaa252f215214cb4fa1a5bfee2c" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", + "synstructure 0.13.2", +] + +[[package]] +name = "asn1-rs-impl" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2777730b2039ac0f95f093556e61b6d26cebed5393ca6f152717777cec3a42ed" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", +] + +[[package]] +name = "asn1-rs-impl" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64ct" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" + +[[package]] +name = "block-buffer" +version = "0.10.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" +dependencies = [ + "generic-array", +] + +[[package]] +name = "block-padding" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a8894febbff9f758034a5b8e12d87918f56dfc64a8e1fe757d65e29041538d93" +dependencies = [ + "generic-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cbc" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "26b52a9543ae338f279b96b0b9fed9c8093744685043739079ce85cd58f289a6" +dependencies = [ + "cipher", +] + +[[package]] +name = "cc" +version = "1.2.66" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f5d6cac793997bd970000024b2934968efe83b382de4fdcf4fcb46b6ee4ad996" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "chacha20" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", +] + +[[package]] +name = "chrono" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "cipher" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +dependencies = [ + "crypto-common", + "inout", +] + +[[package]] +name = "clap" +version = "4.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51" +dependencies = [ + "clap_builder", + "clap_derive", +] + +[[package]] +name = "clap_builder" +version = "4.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" +dependencies = [ + "anstream", + "anstyle", + "clap_lex", + "strsim", +] + +[[package]] +name = "clap_derive" +version = "4.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "clap_lex" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" + +[[package]] +name = "cms" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b77c319abfd5219629c45c34c89ba945ed3c5e49fcde9d16b6c3885f118a730" +dependencies = [ + "const-oid", + "der", + "spki", + "x509-cert", +] + +[[package]] +name = "colorchoice" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" + +[[package]] +name = "const-oid" +version = "0.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "cpufeatures" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" +dependencies = [ + "libc", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc32c" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a47af21622d091a8f0fb295b88bc886ac74efcc613efc19f5d0b21de5c89e47" +dependencies = [ + "rustc_version", +] + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "csv" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" +dependencies = [ + "csv-core", + "itoa", + "ryu", + "serde_core", +] + +[[package]] +name = "csv-core" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" +dependencies = [ + "memchr", +] + +[[package]] +name = "data-encoding" +version = "2.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4ae5f15dda3c708c0ade84bfee31ccab44a3da4f88015ed22f63732abe300c8" + +[[package]] +name = "der" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" +dependencies = [ + "const-oid", + "der_derive", + "flagset", + "pem-rfc7468", + "zeroize", +] + +[[package]] +name = "der-parser" +version = "10.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07da5016415d5a3c4dd39b11ed26f915f52fc4e0dc197d87908bc916e51bc1a6" +dependencies = [ + "asn1-rs 0.7.2", + "displaydoc", + "nom", + "num-bigint", + "num-traits", + "rusticata-macros", +] + +[[package]] +name = "der_derive" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8034092389675178f570469e6c3b0465d3d30b4505c294a6550db47f3c17ad18" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" + +[[package]] +name = "des" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffdd80ce8ce993de27e9f063a444a4d53ce8e8db4c1f00cc03af5ad5a9867a1e" +dependencies = [ + "cipher", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "block-buffer", + "crypto-common", + "subtle", +] + +[[package]] +name = "displaydoc" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "dns-lookup" +version = "3.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e39034cee21a2f5bbb66ba0e3689819c4bb5d00382a282006e802a7ffa6c41d" +dependencies = [ + "cfg-if", + "libc", + "socket2", + "windows-sys 0.60.2", +] + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "flagset" +version = "0.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7ac824320a75a52197e8f2d787f6a38b6718bb6897a35142d749af3c0e8f4fe" + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-core", + "futures-task", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi 5.3.0", + "wasip2", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", + "rand_core 0.10.1", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + +[[package]] +name = "http" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "indoc" +version = "2.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "79cf5c93f93228cf8efb3ba362535fb11199ac548a09ce117c9b1adc3030d706" +dependencies = [ + "rustversion", +] + +[[package]] +name = "inout" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +dependencies = [ + "block-padding", + "generic-array", +] + +[[package]] +name = "is_terminal_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jks" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e03966fd15eea3cb2886320a78d01e77f8aaeabd3fb01504ee6a2238876c23bc" +dependencies = [ + "asn1-rs 0.5.2", + "sha1", + "thiserror 1.0.69", +] + +[[package]] +name = "jobserver" +version = "0.1.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" +dependencies = [ + "getrandom 0.4.3", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "memmap2" +version = "0.9.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0" +dependencies = [ + "libc", +] + +[[package]] +name = "minimal-lexical" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "nom" +version = "7.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" +dependencies = [ + "memchr", + "minimal-lexical", +] + +[[package]] +name = "num-bigint" +version = "0.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "oid-registry" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12f40cff3dde1b6087cc5d5f5d4d65712f34016a03ed60e9c08dcc392736b5b7" +dependencies = [ + "asn1-rs 0.7.2", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "once_cell_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" + +[[package]] +name = "p12-keystore" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffb9bf5222606eb712d3bb30e01bc9420545b00859970897e70c682353a034f2" +dependencies = [ + "base64", + "cbc", + "cms", + "der", + "des", + "hex", + "hmac", + "pkcs12", + "pkcs5", + "rand 0.10.2", + "rc2", + "sha1", + "sha2", + "thiserror 2.0.18", + "x509-parser", +] + +[[package]] +name = "pbkdf2" +version = "0.12.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" +dependencies = [ + "digest", + "hmac", +] + +[[package]] +name = "pem-rfc7468" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" +dependencies = [ + "base64ct", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pkcs12" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "695b3df3d3cc1015f12d70235e35b6b79befc5fa7a9b95b951eab1dd07c9efc2" +dependencies = [ + "cms", + "const-oid", + "der", + "digest", + "spki", + "x509-cert", + "zeroize", +] + +[[package]] +name = "pkcs5" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e847e2c91a18bfa887dd028ec33f2fe6f25db77db3619024764914affe8b69a6" +dependencies = [ + "aes", + "cbc", + "der", + "pbkdf2", + "scrypt", + "sha2", + "spki", +] + +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "questdb-confstr" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7aceffde1cbf8e67f34cdfd70d2436396176d6ff648fa719e0231fb9856ef3e9" + +[[package]] +name = "questdb-rs" +version = "7.0.0" +dependencies = [ + "base64ct", + "bytes", + "crc32c", + "dns-lookup", + "indoc", + "itoa", + "jks", + "libc", + "log", + "memmap2", + "p12-keystore", + "questdb-confstr", + "rand 0.9.4", + "ring", + "rustls", + "rustls-pki-types", + "ryu", + "serde", + "serde_json", + "slugify", + "socket2", + "ureq", + "webpki-roots", + "windows-sys 0.60.2", + "zstd", +] + +[[package]] +name = "questdb_ha_sender" +version = "1.0.0" +dependencies = [ + "chrono", + "clap", + "csv", + "flate2", + "questdb-rs", +] + +[[package]] +name = "quote" +version = "1.0.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.9.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" +dependencies = [ + "rand_chacha", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" +dependencies = [ + "chacha20", + "getrandom 0.4.3", + "rand_core 0.10.1", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + +[[package]] +name = "rc2" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "62c64daa8e9438b84aaae55010a93f396f8e60e3911590fcba770d04643fc1dd" +dependencies = [ + "cipher", +] + +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted", + "windows-sys 0.52.0", +] + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "rusticata-macros" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632" +dependencies = [ + "nom", +] + +[[package]] +name = "rustls" +version = "0.23.41" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b92b125634d9b795e7beca796cc790df15a7fb38323bf3196fda83292d06b1f" +dependencies = [ + "log", + "once_cell", + "ring", + "rustls-pki-types", + "rustls-webpki", + "subtle", + "zeroize", +] + +[[package]] +name = "rustls-pki-types" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "764899a24af3980067ee14bc143654f297b22eaebfe3c7b6b211920a5a59b046" +dependencies = [ + "zeroize", +] + +[[package]] +name = "rustls-webpki" +version = "0.103.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +dependencies = [ + "ring", + "rustls-pki-types", + "untrusted", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "salsa20" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97a22f5af31f73a954c10289c93e8a50cc23d971e80ee446f1f6f7137a088213" +dependencies = [ + "cipher", +] + +[[package]] +name = "scrypt" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0516a385866c09368f0b5bcd1caff3366aace790fcd46e2bb032697bb172fd1f" +dependencies = [ + "pbkdf2", + "salsa20", + "sha2", +] + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "sha1" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest", +] + +[[package]] +name = "sha2" +version = "0.10.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "digest", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simd-adler32" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "slugify" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6b8cf203d2088b831d7558f8e5151bfa420c57a34240b28cee29d0ae5f2ac8b" +dependencies = [ + "unidecode", +] + +[[package]] +name = "socket2" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + +[[package]] +name = "spki" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +dependencies = [ + "base64ct", + "der", +] + +[[package]] +name = "strsim" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" + +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + +[[package]] +name = "syn" +version = "1.0.109" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "2.0.118" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.12.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f36bdaa60a83aca3921b5259d5400cbf5e90fc51931376a9bd4a0eb79aa7210f" +dependencies = [ + "proc-macro2", + "quote", + "syn 1.0.109", + "unicode-xid", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl 2.0.18", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "time" +version = "0.3.53" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "18dfaaeddcb932337b5e7866ee7d0ce9b76d2fd092997146f187ec09b4558a50" +dependencies = [ + "deranged", + "num-conv", + "powerfmt", + "serde_core", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" + +[[package]] +name = "time-macros" +version = "0.2.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c431b87111666e491a90baa837f914fb45cd5dc3c268591b0220ff5057f2085f" +dependencies = [ + "num-conv", + "time-core", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + +[[package]] +name = "unidecode" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "402bb19d8e03f1d1a7450e2bd613980869438e0666331be3e073089124aa1adc" + +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + +[[package]] +name = "ureq" +version = "3.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d39cb1dbab692d82a977c0392ffac19e188bd9186a9f32806f0aaa859d75585a" +dependencies = [ + "base64", + "log", + "percent-encoding", + "ureq-proto", + "utf-8", +] + +[[package]] +name = "ureq-proto" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d81f9efa9df032be5934a46a068815a10a042b494b6a58cb0a1a97bb5467ed6f" +dependencies = [ + "base64", + "http", + "httparse", + "log", +] + +[[package]] +name = "utf-8" +version = "0.7.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9" + +[[package]] +name = "utf8parse" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.118", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "webpki-roots" +version = "1.0.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf85cb06032201fa7c6f829d7db5a7e5aa45bcc0655327713065f6f0576731bf" +dependencies = [ + "rustls-pki-types", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets 0.52.6", +] + +[[package]] +name = "windows-sys" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" +dependencies = [ + "windows-targets 0.53.5", +] + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm 0.52.6", + "windows_aarch64_msvc 0.52.6", + "windows_i686_gnu 0.52.6", + "windows_i686_gnullvm 0.52.6", + "windows_i686_msvc 0.52.6", + "windows_x86_64_gnu 0.52.6", + "windows_x86_64_gnullvm 0.52.6", + "windows_x86_64_msvc 0.52.6", +] + +[[package]] +name = "windows-targets" +version = "0.53.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" +dependencies = [ + "windows-link", + "windows_aarch64_gnullvm 0.53.1", + "windows_aarch64_msvc 0.53.1", + "windows_i686_gnu 0.53.1", + "windows_i686_gnullvm 0.53.1", + "windows_i686_msvc 0.53.1", + "windows_x86_64_gnu 0.53.1", + "windows_x86_64_gnullvm 0.53.1", + "windows_x86_64_msvc 0.53.1", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" + +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" + +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + +[[package]] +name = "windows_i686_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "x509-cert" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1301e935010a701ae5f8655edc0ad17c44bad3ac5ce8c39185f75453b720ae94" +dependencies = [ + "const-oid", + "der", + "spki", +] + +[[package]] +name = "x509-parser" +version = "0.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d43b0f71ce057da06bc0851b23ee24f3f86190b07203dd8f567d0b706a185202" +dependencies = [ + "asn1-rs 0.7.2", + "data-encoding", + "der-parser", + "lazy_static", + "nom", + "oid-registry", + "rusticata-macros", + "thiserror 2.0.18", + "time", +] + +[[package]] +name = "zerocopy" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7cbbc0a705a0fd05cc3676525980d2bf5a9bc4adac6d6475209a7887cf59d19" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2e817b7b52d0c7358d3246da9d69935ebb18116b2b102b4230dac079b4862f5" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "zeroize" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" + +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/rust/Cargo.toml b/rust/Cargo.toml new file mode 100644 index 0000000..27aa59f --- /dev/null +++ b/rust/Cargo.toml @@ -0,0 +1,27 @@ +[package] +name = "questdb_ha_sender" +version = "1.0.0" +edition = "2021" +description = "Parallel CSV replay sender for QuestDB over QWP (WebSocket/UDP) and ILP/HTTP, a Rust port of the Java ha_sender" + +[[bin]] +name = "csv_parallel_sender" +path = "src/main.rs" + +[dependencies] +# The Rust/C client, built locally from the c-questdb-client checkout. Default features +# already enable every sync sender (TCP, HTTP, QWP/UDP, QWP/WebSocket) plus webpki roots +# and the ring crypto backend; we add the egress reader (for the probe / query client), +# zstd result decompression, and insecure-skip-verify (for tls_verify=unsafe_off). +questdb-rs = { path = "../../../questdb/c-questdb-client/questdb-rs", features = [ + "sync-reader-ws", + "compression-zstd", + "insecure-skip-verify", +] } +clap = { version = "4", features = ["derive"] } +csv = "1" +flate2 = "1" +chrono = "0.4" + +[profile.release] +opt-level = 3 diff --git a/rust/README.md b/rust/README.md new file mode 100644 index 0000000..8b927c3 --- /dev/null +++ b/rust/README.md @@ -0,0 +1,171 @@ +# Rust HA sender + +A Rust port of the Java `CsvParallelSender`. It replays a CSV of trades in a loop across +N worker threads over one of three QuestDB transports, and (on QWP/WebSocket) runs a probe +that reports the latest ingested timestamp and the serving node's live role. + +It is built on the Rust/C client (`questdb-rs`) from +[`c-questdb-client`](https://github.com/questdb/c-questdb-client), which powers the C, C++, +and Python clients too. + +## Transports (`--protocol`) + +- `qwp` (default): **QWP over WebSocket**. Store-and-forward (un-acked frames spill to disk + and replay after an outage), transactional commit, and multi-host failover. A background + probe polls the latest ingested timestamp and the serving node's live role over a QWP + query client (the `Reader`). Use the server's WebSocket/HTTP port (`:9000`). +- `qwpudp`: **QWP over UDP**, fire-and-forget datagrams to `:9007`. Ingest-only, + unauthenticated, single-endpoint (no failover), best-effort (no ack, no store-and-forward). + Keep `--batch-size` small: UDP has no backpressure, so a large flush burst overflows the + server's receive buffer and drops datagrams wholesale. +- `ilp`: the legacy **ILP over HTTP** transport. + +## Build + +The client is a **path dependency** on a local `c-questdb-client` checkout, expected at +`../../../questdb/c-questdb-client/questdb-rs` (i.e. `~/prj/questdb/c-questdb-client` +alongside `~/prj/java/questdb_java_ha_sender`). Adjust the path in `Cargo.toml` if your +checkout is elsewhere. The QWP wire protocol must match the target server build. + +``` +cargo build --release +``` + +Requires a Rust toolchain new enough for the client's 2024 edition (1.85+; tested on 1.93). + +## Run + +``` +./target/release/csv_parallel_sender \ + --protocol qwp \ + --addrs localhost:9000 \ + --total-events 5000000 \ + --num-senders 4 \ + --delay-ms 0 \ + --batch-size 10000 \ + --batches-per-transaction 10 \ + --csv ../trades20250728.csv.gz +``` + +UDP (single host, small batch, no auth, no probe): + +``` +./target/release/csv_parallel_sender \ + --protocol qwpudp --addrs localhost:9007 \ + --total-events 100000 --num-senders 1 --batch-size 500 \ + --csv ../trades20250728.csv.gz +``` + +QWP batch errors and UDP loss are asynchronous or silent, so always confirm the result with +a server-side `SELECT count() FROM trades` after a run. + +## Flags + +| Flag | Default | Notes | +| --- | --- | --- | +| `--addrs` | `localhost:9000` | Comma-separated `host:port`. QWP/WebSocket + ILP use `:9000`; UDP uses `:9007` and accepts one host only. | +| `--token` | | Bearer token (QWP/WebSocket + ILP). UDP rejects auth. | +| `--username` / `--password` | | Basic auth (QWP/WebSocket + ILP). | +| `--total-events` | `1000000` | Rows across all workers. | +| `--delay-ms` | `50` | Per-row sleep. Ignored when `--rate > 0`. | +| `--rate` | `0` | Target **aggregate** rows/second across all workers (`0` = off). Takes precedence over `--delay-ms`; see [Pacing](#pacing-delay-ms-vs-rate). | +| `--num-senders` | `10` | Worker threads (one `Sender` each). | +| `--retry-timeout` | `360000` | `retry_timeout` (ILP) / `reconnect_max_duration_millis` (QWP). | +| `--csv` | `./trades20250728.csv.gz` | `.csv` or `.csv.gz`; needs `symbol,side,price,amount[,timestamp]`. | +| `--timestamp-from-file` | `false` | Use the CSV timestamp column instead of "now". | +| `--seconds-offset` | `0` | Shift each timestamp by N seconds. | +| `--protocol` | `qwp` | `qwp` \| `qwpudp` \| `ilp`. | +| `--sender-id` | `ha_sender` | Store-and-forward key base; each worker gets `-`. | +| `--store-forward-dir` | `/tmp/qdb-sf` | Spill directory (QWP/WebSocket). | +| `--batch-size` | `10000` | Rows per flush. QWP commits every `batch-size × batches-per-transaction`; UDP/ILP flush every `batch-size`. | +| `--batches-per-transaction` | `10` | QWP transaction size, in batches. | +| `--probe-interval-ms` | `1000` | Probe poll interval (QWP/WebSocket only; `0` disables). | +| `--enterprise` | `false` | Request durable acks (Enterprise only). | +| `--zone` | `eu-west-1` | Preferred zone for the query client / probe. | + +## Pacing: `--delay-ms` vs `--rate` + +- `--delay-ms` (default `50`): a fixed sleep after **every row**, per worker. A poor throttle + at speed — the fixed per-row cost caps throughput well under 1000 rows/s at `1`ms. +- `--rate` (default `0` = off): a **target aggregate rate in rows/second across all workers**. + Each worker paces to its share (`rate / num-senders`) against a deadline schedule, sending + rows back-to-back and sleeping only when it runs *ahead*. This reaches high targets a per-row + sleep never could (e.g. `--rate 300000`). When `> 0` it takes precedence over `--delay-ms` + (a warning prints if both are set). Measured: `--rate 5000`, 20k rows, 4 workers → ~5.0 s, + steady ~5000 rows/s. + +## Timestamps + +The generator sends timestamps at **nanosecond** resolution (`TimestampNanos`), and QuestDB +stores them at the **target column's** resolution: a `TIMESTAMP_NS` column keeps full nanos, a +micros `TIMESTAMP` column truncates the extra digits (silently, never an error). **If the table +does not exist, the server auto-creates it as `TIMESTAMP_NS`.** Pre-create `trades` with a +`timestamp` (micros) column if you want micros. Verified: replaying the nanosecond FX chunk with +`--timestamp-from-file` preserved `...192508297Z` end-to-end in an auto-created `TIMESTAMP_NS` +table. + +- **Single worker on a QWP transport** (`qwp` or `qwpudp`, `--num-senders 1`): each row is + stamped with the current time client-side (monotonic, so no out-of-order), giving distinct + per-row timestamps. +- **ILP, or more than one worker**: rows use server-side `at_now()` (O3-safe across workers; + a whole batch shares one timestamp). +- `--timestamp-from-file` / `--seconds-offset` stamp each row explicitly from the CSV. + +## Probe (QWP/WebSocket only) + +A background thread runs `select timestamp from trades limit -1` each interval and prints +the latest ingested timestamp, then asks the serving node for its live role with +`switch status`: + +``` +[query client] connected, serving node=n1 role=PRIMARY zone=eu-west-1 cluster=... +[probe] latest trades timestamp = 2026-07-01T14:31:40.591545Z (raw=1782916300591545) served by role=PRIMARY node=n1 zone=eu-west-1 +``` + +`switch status` gives the authoritative live role (`current_role`, plus `target_role` while +a switch is in flight, shown as `(switching -> ROLE)`). It is Enterprise-only and needs +SYSTEM ADMIN; on OSS the probe falls back to the QWP handshake role, labelled +`(handshake role; live 'switch status' unavailable, may be stale)`. The query client uses +`target=any` (so reads fall back to a replica when no primary is available) and fails over +across the `--addrs` hosts automatically. + +## Differences from the Java client + +- **No auto-flush.** The Rust/C client deliberately has no auto-flush (it rejects every + `auto_flush*` key except `off`); batching is driven purely by explicit `flush()` calls. + This port reproduces the Java cadence by flushing at the same row boundaries, so behavior + matches; there is simply no background flush timer or byte cap. +- **No `--connect-timeout-ms`.** The Rust QWP transport exposes no single-connect timeout + knob (only the overall `reconnect_max_duration`), so that flag is omitted here. +- **Single bearer token only.** QWP (WebSocket/HTTP) auth is a single `token` (or + username/password). The split `x`/`y` key components only exist for legacy TCP-ILP ECDSA + auth, which this tool does not use. + +## Validated + +Smoke-tested against a local QuestDB (OSS) with all rows accounted for server-side: + +| Transport | Run | Result | +| --- | --- | --- | +| `ilp` | 2000 rows, 1 worker | 2000 / 2000 | +| `qwpudp` | 2000 rows, 1 worker | 2000 / 2000 | +| `qwp` | 5000 rows, 1 worker | 5000 / 5000, distinct per-row timestamps; probe reported live timestamps + role | +| `qwp` | 20000 rows, 4 workers | 20000 / 20000, one store-and-forward dir per worker | +| `qwp` | `--rate 5000`, nanos FX, `--timestamp-from-file`, 4 workers | steady ~5000 rows/s; auto-created `TIMESTAMP_NS` table preserved `...192508297Z` | + +### QWP failover (primary crash) + +Two servers from the same build (primary on `:9100`, fallback on `:9000`), a single-worker +`qwp` run of 60000 rows with `--addrs :9100,:9000` and the probe active. The primary was +hard-killed mid-run: + +- The sender **completed all 60000 rows with exit 0**, despite its primary crashing. +- The primary took `trade_id` `0-1`…`0-20000`; the fallback took `0-20001`…`0-60000` + (40000 rows, all distinct). Combined: a **contiguous 1–60000 with zero loss and zero + duplication** — store-and-forward replayed the in-flight transaction to the new host and + the handoff landed exactly on a transaction boundary. +- The probe kept reporting across the crash and reconnected to the fallback transparently. + +Note: the probe's `on_failover_reset` callback fires only on *mid-query* failover; its +`limit -1` polls return in microseconds, so a between-poll reconnect is silent (no +`failed over` line). Comparing `reader.current_addr()` across polls would surface it. diff --git a/rust/src/main.rs b/rust/src/main.rs new file mode 100644 index 0000000..4f445a9 --- /dev/null +++ b/rust/src/main.rs @@ -0,0 +1,690 @@ +//! Parallel CSV replay sender for QuestDB, a Rust port of the Java `CsvParallelSender`. +//! +//! It replays a CSV of trades in a loop across N worker threads over one of three +//! transports, selected with `--protocol`: +//! +//! * `qwp` - QWP over WebSocket: store-and-forward (un-acked frames spill to disk and +//! replay after an outage), transactional commit, multi-host failover. A background probe +//! thread polls the latest ingested timestamp and the serving node's live role via the +//! query client (the `Reader`). +//! * `qwpudp` - QWP over UDP: fire-and-forget datagrams to :9007. Ingest-only, +//! unauthenticated, single-endpoint, best-effort (no ack, no failover). +//! * `ilp` - the legacy ILP/HTTP transport. +//! +//! Unlike the Java client, the Rust/C client has no auto-flush: batching is driven purely +//! by when we call `flush()`. We reproduce the Java cadence explicitly (see `run_worker`). + +use std::io::Read; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::thread; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use clap::Parser; +use questdb::egress::column::ColumnView; +use questdb::egress::reader::{BatchView, Reader}; +use questdb::egress::FailoverEvent; +use questdb::ingress::{Sender, TimestampNanos}; + +const PROBE_QUERY: &str = "select timestamp from trades limit -1"; +// Enterprise lifecycle status of whichever node the query client is connected to. Returns +// the LIVE role (current_role / target_role), unlike the QWP handshake SERVER_INFO which is +// only refreshed on a reconnect and so goes stale after an in-place primary<->replica switch. +const STATUS_QUERY: &str = "switch status"; + +#[derive(Parser, Debug)] +#[command( + name = "csv_parallel_sender", + about = "Parallel CSV replay sender for QuestDB (QWP WebSocket/UDP, ILP/HTTP)" +)] +struct Args { + /// Comma-separated host:port list. QWP/WebSocket + ILP use :9000; QWP/UDP uses :9007. + #[arg(long, default_value = "localhost:9000")] + addrs: String, + + /// Bearer token (QWP/WebSocket + ILP only; QWP/UDP is unauthenticated). + #[arg(long)] + token: Option, + + /// Basic-auth username (with --password). + #[arg(long)] + username: Option, + + /// Basic-auth password (with --username). + #[arg(long)] + password: Option, + + /// Total rows to send across all workers. + #[arg(long, default_value_t = 1_000_000)] + total_events: u64, + + /// Per-row sleep in milliseconds (0 = as fast as possible). Ignored when --rate > 0. + #[arg(long, default_value_t = 50)] + delay_ms: u64, + + /// Target aggregate rows/second across ALL workers (0 = off, use --delay-ms). Takes + /// precedence over --delay-ms: each worker paces to rate/num-senders against a deadline + /// schedule, reaching high targets a per-row sleep cannot. + #[arg(long, default_value_t = 0)] + rate: u64, + + /// Number of worker threads (each is its own Sender). + #[arg(long, default_value_t = 10)] + num_senders: usize, + + /// Overall "keep retrying" budget in ms: retry_timeout (ILP) / reconnect_max_duration (QWP). + #[arg(long, default_value_t = 360_000)] + retry_timeout: u64, + + /// CSV path (.csv or .csv.gz) with symbol,side,price,amount[,timestamp] columns. + #[arg(long, default_value = "./trades20250728.csv.gz")] + csv: String, + + /// Use the CSV's own timestamp column instead of server/client "now". + #[arg(long, default_value_t = false)] + timestamp_from_file: bool, + + /// Shift each timestamp by this many seconds (works with or without --timestamp-from-file). + #[arg(long, default_value_t = 0)] + seconds_offset: i64, + + /// Transport: qwp | qwpudp | ilp. + #[arg(long, default_value = "qwp")] + protocol: String, + + /// Store-and-forward key base (QWP/WebSocket). Each worker gets -. + #[arg(long, default_value = "ha_sender")] + sender_id: String, + + /// Store-and-forward spill directory (QWP/WebSocket). + #[arg(long, default_value = "/tmp/qdb-sf")] + store_forward_dir: String, + + /// Rows per flush. QWP commits every batch-size*batches-per-transaction; UDP flushes + /// every batch-size rows (keep small for UDP; large bursts overflow the receive buffer). + #[arg(long, default_value_t = 10_000)] + batch_size: u64, + + /// QWP/WebSocket transaction size, in batches. + #[arg(long, default_value_t = 10)] + batches_per_transaction: u64, + + /// Probe poll interval in ms (QWP/WebSocket only; 0 disables). + #[arg(long, default_value_t = 1000)] + probe_interval_ms: u64, + + /// Enterprise only: request durable acks (QWP/WebSocket). + #[arg(long, default_value_t = false)] + enterprise: bool, + + /// Preferred zone for the query client / probe (QWP/WebSocket). + #[arg(long, default_value = "eu-west-1")] + zone: String, +} + +impl Args { + fn has_token(&self) -> bool { + self.token.as_deref().is_some_and(|t| !t.is_empty()) + } + + fn has_basic(&self) -> bool { + self.username.as_deref().is_some_and(|u| !u.is_empty()) + && self.password.as_deref().is_some_and(|p| !p.is_empty()) + } + + fn tls(&self) -> bool { + self.has_token() || self.has_basic() + } + + fn addr_list(&self) -> Vec { + self.addrs + .split(',') + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()) + .collect() + } +} + +struct TradeRow { + symbol: String, + side: String, + price: f64, + amount: f64, + /// Nanoseconds since epoch, parsed at load time; only used with --timestamp-from-file. + ts_nanos: i64, +} + +fn now_nanos() -> i64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.as_nanos() as i64) + .unwrap_or(0) +} + +fn main() { + let args = Args::parse(); + + if !matches!(args.protocol.as_str(), "qwp" | "qwpudp" | "ilp") { + eprintln!("--protocol must be 'qwp', 'qwpudp', or 'ilp', got: {}", args.protocol); + std::process::exit(2); + } + if args.num_senders == 0 { + eprintln!("--num-senders must be > 0"); + std::process::exit(2); + } + if args.total_events == 0 { + eprintln!("--total-events must be > 0"); + std::process::exit(2); + } + if args.batch_size == 0 { + eprintln!("--batch-size must be > 0"); + std::process::exit(2); + } + if args.batches_per_transaction == 0 { + eprintln!("--batches-per-transaction must be > 0"); + std::process::exit(2); + } + // --rate takes precedence: when set it drives pacing and --delay-ms is ignored. + if args.rate > 0 && args.delay_ms > 0 { + eprintln!( + "[warn] --rate {} overrides --delay-ms {} (rate limiting drives pacing; the per-row \ + delay is ignored)", + args.rate, args.delay_ms + ); + } + println!( + "Pacing: {}", + if args.rate > 0 { + format!("rate={} rows/s (aggregate across {} workers)", args.rate, args.num_senders) + } else { + format!("delay-ms={}", args.delay_ms) + } + ); + + let addrs = args.addr_list(); + if addrs.is_empty() { + eprintln!("--addrs must contain at least one host:port"); + std::process::exit(2); + } + // QWP/UDP is single-endpoint (no failover) and unauthenticated. Enforce both upfront so + // the failure is a clear message here, not a per-worker exception mid-run. + if args.protocol == "qwpudp" { + if addrs.len() > 1 { + eprintln!( + "--protocol qwpudp supports a single host:port only (UDP is connectionless, \ + there is no failover); got {} addresses", + addrs.len() + ); + std::process::exit(2); + } + if args.has_token() || args.username.is_some() || args.password.is_some() { + eprintln!( + "[warn] --protocol qwpudp is unauthenticated (UDP accepts any connection); \ + ignoring --token/--username/--password" + ); + } + } + + let rows = match load_csv(&args.csv, args.timestamp_from_file) { + Ok(rows) if !rows.is_empty() => rows, + Ok(_) => { + eprintln!("CSV has no data rows: {}", args.csv); + std::process::exit(2); + } + Err(e) => { + eprintln!("Failed to read CSV {}: {e}", args.csv); + std::process::exit(2); + } + }; + + match args.protocol.as_str() { + "qwp" => println!( + "Ingestion started. Protocol: qwp (WebSocket, sender-id={}, store-and-forward={}, \ + batch-size={}, batches-per-transaction={}, retry-timeout-ms={}) | addrs: {}", + args.sender_id, args.store_forward_dir, args.batch_size, args.batches_per_transaction, + args.retry_timeout, addrs.join(",") + ), + "qwpudp" => println!( + "Ingestion started. Protocol: qwpudp (QWP/UDP datagrams, ingest-only, unauthenticated, \ + no store-and-forward, no failover; query client disabled, batch-size={}) | addr: {}", + args.batch_size, addrs[0] + ), + _ => println!( + "Ingestion started. Protocol: ilp (HTTP, retry-timeout-ms={}) | addrs: {}", + args.retry_timeout, addrs.join(",") + ), + } + + let total_sent = AtomicU64::new(0); + let stop = AtomicBool::new(false); + let start = Instant::now(); + + let base = args.total_events / args.num_senders as u64; + let rem = args.total_events % args.num_senders as u64; + let mut worker_errors: Vec = Vec::new(); + + thread::scope(|scope| { + let args_ref = &args; + let rows_ref = &rows; + let sent_ref = &total_sent; + let stop_ref = &stop; + + // Progress reporter: rows/s (client-side, sent), once per second. + scope.spawn(move || { + let mut last = 0u64; + while !stop_ref.load(Ordering::Relaxed) { + thread::sleep(Duration::from_millis(1000)); + let now = sent_ref.load(Ordering::Relaxed); + println!("[progress] sent={now} rate={} rows/s", now - last); + last = now; + } + }); + + // Probe (QWP/WebSocket only): ingest-only transports (qwpudp, ilp) have no query path. + if args_ref.protocol == "qwp" && args_ref.probe_interval_ms > 0 { + scope.spawn(move || run_probe(args_ref, stop_ref)); + } + + let mut handles = Vec::with_capacity(args.num_senders); + for id in 0..args.num_senders { + let events = base + if (id as u64) < rem { 1 } else { 0 }; + handles.push(scope.spawn(move || run_worker(id, events, args_ref, rows_ref, sent_ref))); + } + + for (id, h) in handles.into_iter().enumerate() { + match h.join() { + Ok(Ok(())) => {} + Ok(Err(e)) => worker_errors.push(format!("Sender {id}: {e}")), + Err(_) => worker_errors.push(format!("Sender {id}: panicked")), + } + } + stop_ref.store(true, Ordering::Relaxed); + }); + + if !worker_errors.is_empty() { + for e in &worker_errors { + eprintln!("Worker failed: {e}"); + } + std::process::exit(1); + } + + let elapsed = start.elapsed().as_secs_f64(); + let rate = if elapsed > 0.0 { args.total_events as f64 / elapsed } else { 0.0 }; + println!( + "All workers completed. protocol={} events={} elapsed={:.3} s throughput={:.0} rows/s", + args.protocol, args.total_events, elapsed, rate + ); +} + +fn run_worker( + worker_id: usize, + total_events: u64, + args: &Args, + rows: &[TradeRow], + total_sent: &AtomicU64, +) -> questdb::Result<()> { + println!("Sender {worker_id} will send {total_events} events"); + + let is_qwp = args.protocol == "qwp"; + let is_udp = args.protocol == "qwpudp"; + // Single worker on a QWP transport: stamp each row with the current time (nanos) client-side. + // A single thread is monotonic (no O3) and this avoids QWP's per-batch atNow() stamping. + // ILP, or more than one worker, use atNow() (server-side, O3-safe). + let per_row_ts = (is_qwp || is_udp) && args.num_senders == 1; + // Flush cadence: qwp commits every batch-size*batches-per-transaction rows; qwpudp flushes + // every batch-size rows (bounds the datagram burst); ilp flushes every batch-size rows too + // (the Rust HTTP client has no auto-flush, so the buffer must be drained explicitly). + let commit_every: u64 = if is_qwp { + args.batch_size * args.batches_per_transaction + } else { + args.batch_size + }; + + // Rate limiting (when --rate > 0): pace this worker to its share of the aggregate target, + // rate / num-senders rows/second. interval_nanos is the ideal spacing between this worker's + // rows; we track a deadline from loop entry and sleep only when running ahead, so rows go + // out back-to-back until we get ahead. Reaches high rates a fixed per-row sleep cannot. + let interval_nanos: f64 = if args.rate > 0 { + 1_000_000_000.0 * args.num_senders as f64 / args.rate as f64 + } else { + 0.0 + }; + let rate_limited = interval_nanos > 0.0; + + let mut sender = Sender::from_conf(build_ingest_conf(args, worker_id))?; + let mut buffer = sender.new_buffer(); + let n = rows.len(); + let mut sent: u64 = 0; + let pace_start = Instant::now(); + + for i in 0..total_events { + let r = &rows[(i as usize) % n]; + buffer + .table("trades")? + .symbol("symbol", r.symbol.as_str())? + .symbol("side", r.side.as_str())? + .column_f64("price", r.price)? + .column_f64("amount", r.amount)? + .column_str("trade_id", format!("{worker_id}-{}", i + 1))?; + + if args.timestamp_from_file { + let ts = r.ts_nanos + args.seconds_offset * 1_000_000_000; + buffer.at(TimestampNanos::new(ts))?; + } else if args.seconds_offset != 0 { + buffer.at(TimestampNanos::new(now_nanos() + args.seconds_offset * 1_000_000_000))?; + } else if per_row_ts { + buffer.at(TimestampNanos::new(now_nanos()))?; + } else { + buffer.at_now()?; + } + + sent += 1; + total_sent.fetch_add(1, Ordering::Relaxed); + + if sent.is_multiple_of(commit_every) { + sender.flush(&mut buffer)?; + } + if rate_limited { + // Deadline for the row just sent (sent is 1-based). Sleep only while ahead of + // schedule; the 1ms floor coalesces many rows into one sleep at high rates. + let target_nanos = (sent as f64 * interval_nanos) as u64; + let elapsed_nanos = pace_start.elapsed().as_nanos() as u64; + if target_nanos > elapsed_nanos { + let sleep_nanos = target_nanos - elapsed_nanos; + if sleep_nanos > 1_000_000 { + thread::sleep(Duration::from_nanos(sleep_nanos)); + } + } + } else if args.delay_ms > 0 { + thread::sleep(Duration::from_millis(args.delay_ms)); + } + } + + sender.flush(&mut buffer)?; + // QWP/WebSocket: drain any un-acked store-and-forward frames before closing. + if is_qwp { + sender.close_drain()?; + } + println!("Sender {worker_id} finished sending {sent} events"); + Ok(()) +} + +/// Build the ingestion connect string for the configured transport. +fn build_ingest_conf(args: &Args, worker_id: usize) -> String { + let addrs = args.addr_list(); + match args.protocol.as_str() { + "qwpudp" => format!("qwpudp::addr={};", addrs[0]), + "ilp" => { + let scheme = if args.tls() { "https" } else { "http" }; + let mut c = format!("{scheme}::addr={};", addrs.join(",")); + append_auth(&mut c, args); + if args.tls() { + c.push_str("tls_verify=unsafe_off;"); + } + c.push_str(&format!("retry_timeout={};", args.retry_timeout)); + c + } + _ => { + // qwp (WebSocket) + let scheme = if args.tls() { "qwpwss" } else { "qwpws" }; + let who = format!("{}-{worker_id}", args.sender_id); + let sf = format!("{}/{who}", args.store_forward_dir); + let _ = std::fs::create_dir_all(&sf); + let mut c = format!("{scheme}::addr={};", addrs.join(",")); + append_auth(&mut c, args); + if args.tls() { + c.push_str("tls_verify=unsafe_off;"); + } + c.push_str(&format!("sender_id={who};")); + c.push_str(&format!("sf_dir={sf};")); + c.push_str(&format!("reconnect_max_duration_millis={};", args.retry_timeout)); + c.push_str("reconnect_initial_backoff_millis=100;"); + c.push_str("reconnect_max_backoff_millis=5000;"); + if args.enterprise { + c.push_str("request_durable_ack=true;"); + } + c + } + } +} + +fn append_auth(c: &mut String, args: &Args) { + if args.has_token() { + c.push_str(&format!("token={};", args.token.as_ref().unwrap())); + } else if args.has_basic() { + c.push_str(&format!( + "username={};password={};", + args.username.as_ref().unwrap(), + args.password.as_ref().unwrap() + )); + } +} + +/// Connect string for the QWP query client (probe): same hosts/auth as the senders, +/// target=any (replica-fallback reads), failover on, zone bias. +fn build_reader_conf(args: &Args) -> String { + let addrs = args.addr_list(); + let scheme = if args.tls() { "wss" } else { "ws" }; + let mut c = format!("{scheme}::addr={};target=any;failover=true;", addrs.join(",")); + append_auth(&mut c, args); + if args.tls() { + c.push_str("tls_verify=unsafe_off;"); + } + if !args.zone.is_empty() { + c.push_str(&format!("zone={};", args.zone)); + } + c.push_str("compression=raw;"); + c +} + +/// Probe thread: every interval, poll the latest ingested timestamp and the serving node's +/// live role over a QWP query client. Independent of the senders; fails over automatically. +fn run_probe(args: &Args, stop: &AtomicBool) { + let mut reader = match Reader::from_conf(build_reader_conf(args)) { + Ok(r) => r, + Err(e) => { + eprintln!("[query client] connect failed: {e}"); + return; + } + }; + if let Some(si) = reader.server_info() { + println!( + "[query client] connected, serving node={} role={} zone={} cluster={}", + or_none(&si.node_id), + si.role.as_str(), + or_none(si.zone_id.as_deref().unwrap_or("")), + or_none(&si.cluster_id) + ); + } else { + println!("[query client] connected"); + } + + let mut was_down = false; + while !stop.load(Ordering::Relaxed) { + match read_latest_ts(&mut reader) { + Ok(latest) => { + if was_down { + println!("[query client] connection restored"); + was_down = false; + } + if let Some(micros) = latest { + // trades designated timestamp is microseconds by default. + let ts = chrono::DateTime::from_timestamp_micros(micros) + .map(|dt| dt.to_rfc3339_opts(chrono::SecondsFormat::Micros, true)) + .unwrap_or_else(|| micros.to_string()); + + // Ask the serving node for its live role. A status-query failure must not + // look like a connection loss, so it is swallowed (current stays None). + let (current, target) = read_switch_status(&mut reader).unwrap_or((None, None)); + + let (node, zone) = match reader.server_info() { + Some(si) => ( + or_none(&si.node_id), + or_none(si.zone_id.as_deref().unwrap_or("")), + ), + None => ("(none)".to_string(), "(none)".to_string()), + }; + + let served = match current { + Some(role) => { + let switching = target + .as_deref() + .is_some_and(|t| !t.eq_ignore_ascii_case(&role)); + let sw = match (&target, switching) { + (Some(t), true) => format!(" (switching -> {t})"), + _ => String::new(), + }; + format!(" served by role={role}{sw} node={node} zone={zone}") + } + None => { + let handshake = reader + .server_info() + .map(|si| si.role.as_str()) + .unwrap_or_else(|| "unknown".to_string()); + format!( + " served by role={handshake} node={node} zone={zone} \ + (handshake role; live 'switch status' unavailable, may be stale)" + ) + } + }; + println!("[probe] latest trades timestamp = {ts} (raw={micros}){served}"); + } + } + Err(e) => { + if !was_down { + println!("[query client] connection lost ({e}), will retry"); + was_down = true; + } + } + } + thread::sleep(Duration::from_millis(args.probe_interval_ms)); + } +} + +/// Run PROBE_QUERY and return the last non-null timestamp (micros), if any. +fn read_latest_ts(reader: &mut Reader) -> Result, questdb::egress::Error> { + let mut latest: Option = None; + let mut cursor = reader + .prepare(PROBE_QUERY) + .on_failover_reset(|ev: &FailoverEvent| { + println!( + "[query client] failed over {} -> {} (attempt {})", + ev.failed_addr, ev.new_addr, ev.attempts + ); + }) + .execute()?; + while let Some(view) = cursor.next_batch()? { + if let Ok(ColumnView::Timestamp(col)) = view.column(0) { + for r in 0..view.row_count() { + if !col.is_null(r) { + latest = Some(col.value(r)); + } + } + } + } + Ok(latest) +} + +/// Run STATUS_QUERY and pull current_role / target_role out of the result by column name. +fn read_switch_status( + reader: &mut Reader, +) -> Result<(Option, Option), questdb::egress::Error> { + let mut current: Option = None; + let mut target: Option = None; + let mut cursor = reader.prepare(STATUS_QUERY).execute()?; + while let Some(view) = cursor.next_batch()? { + let names: Vec = view + .schema() + .columns() + .iter() + .map(|c| c.name.to_lowercase()) + .collect(); + for r in 0..view.row_count() { + for (i, name) in names.iter().enumerate() { + if !name.contains("role") { + continue; + } + let value = col_string(&view, i, r); + if name.contains("current") { + current = value; + } else if name.contains("target") { + target = value; + } else if current.is_none() { + // A single unqualified "role" column (older servers) is the current one. + current = value; + } + } + } + } + Ok((current, target)) +} + +/// Read a string value from a Symbol or Varchar column; None for null or non-string types. +fn col_string(view: &BatchView, col: usize, row: usize) -> Option { + match view.column(col) { + Ok(ColumnView::Symbol(c)) => c.resolve(row).map(|s| s.to_string()), + Ok(ColumnView::Varchar(c)) => c.value(row).map(|s| s.to_string()), + _ => None, + } +} + +fn or_none(s: &str) -> String { + if s.is_empty() { + "(none)".to_string() + } else { + s.to_string() + } +} + +/// Load the CSV (optionally gzipped) into memory. Requires symbol,side,price,amount and, +/// when `need_timestamp`, a timestamp column parsed to microseconds since epoch. +fn load_csv(path: &str, need_timestamp: bool) -> Result, Box> { + let file = std::fs::File::open(path)?; + let reader: Box = if path.ends_with(".gz") { + Box::new(flate2::read::GzDecoder::new(file)) + } else { + Box::new(file) + }; + let mut rdr = csv::ReaderBuilder::new().has_headers(true).from_reader(reader); + + let headers = rdr.headers()?.clone(); + let idx = |name: &str| -> Result> { + headers + .iter() + .position(|h| h.trim() == name) + .ok_or_else(|| format!("CSV missing required column: {name}").into()) + }; + let i_symbol = idx("symbol")?; + let i_side = idx("side")?; + let i_price = idx("price")?; + let i_amount = idx("amount")?; + let i_ts = if need_timestamp { Some(idx("timestamp")?) } else { None }; + + let mut out = Vec::new(); + for rec in rdr.records() { + let rec = rec?; + if rec.is_empty() { + continue; + } + let ts_nanos = match i_ts { + Some(i) => { + let raw = rec.get(i).unwrap_or("").trim(); + // Parse to nanoseconds so a TIMESTAMP_NS source keeps full precision; a micros + // target column truncates server-side. timestamp_nanos_opt is None only for + // dates outside ~1677-2262, which trade data never is. + chrono::DateTime::parse_from_rfc3339(raw) + .map_err(|e| format!("bad timestamp {raw:?}: {e}"))? + .timestamp_nanos_opt() + .ok_or_else(|| format!("timestamp out of nanosecond range: {raw:?}"))? + } + None => 0, + }; + out.push(TradeRow { + symbol: rec.get(i_symbol).unwrap_or("").trim().to_string(), + side: rec.get(i_side).unwrap_or("").trim().to_string(), + price: rec.get(i_price).unwrap_or("").trim().parse()?, + amount: rec.get(i_amount).unwrap_or("").trim().parse()?, + ts_nanos, + }); + } + Ok(out) +} diff --git a/src/main/java/com/example/sender/CsvParallelSender.java b/src/main/java/com/example/sender/CsvParallelSender.java index 05da68d..14375bc 100644 --- a/src/main/java/com/example/sender/CsvParallelSender.java +++ b/src/main/java/com/example/sender/CsvParallelSender.java @@ -36,6 +36,13 @@ public class CsvParallelSender { private static final String DEFAULT_ADDRS = "questdb:9000"; private static final long DEFAULT_TOTAL_EVENTS = 1_000_000L; private static final int DEFAULT_DELAY_MS = 50; + // Target aggregate generation rate in rows/second across ALL workers. 0 disables rate + // limiting and falls back to --delay-ms. When > 0 it takes precedence over --delay-ms: + // each worker paces itself to its share (rate / num-senders) against a deadline schedule, + // so the process approximates the target regardless of worker count. Unlike a fixed + // per-row sleep, it sends rows back-to-back and only sleeps when ahead of schedule, so it + // can sustain high rates (e.g. 300000) that a --delay-ms of >=1 could never reach. + private static final long DEFAULT_RATE = 0L; private static final int DEFAULT_NUM_SENDERS = 10; private static final int DEFAULT_RETRY_TIMEOUT = 360000; private static final String DEFAULT_CSV = "./trades20250728.csv.gz"; @@ -69,6 +76,12 @@ public class CsvParallelSender { // Rows sent (client-side) across all workers, for the once-per-second progress reporter. private static final AtomicLong TOTAL_SENT = new AtomicLong(); + // Rows the server has acknowledged, per worker (summed by the reporter). Fed from the QWP + // ack watermark (getAckedFsn) - the real committed count, with no extra query round-trips. + private static AtomicLong[] ACKED_ROWS = new AtomicLong[0]; + // Set once all rows are submitted: silences the per-second progress + probe lines so the + // commit-drain tail is quiet and only the final summary line is printed. + private static volatile boolean SUBMIT_COMPLETE = false; public static void main(String[] args) throws Exception { // Parse CLI flags @@ -80,6 +93,7 @@ public static void main(String[] args) throws Exception { final String password = a.get("--password"); // optional final long totalEvents = Long.parseLong(a.getOrDefault("--total-events", String.valueOf(DEFAULT_TOTAL_EVENTS))); final int delayMs = Integer.parseInt(a.getOrDefault("--delay-ms", String.valueOf(DEFAULT_DELAY_MS))); + final long rate = Long.parseLong(a.getOrDefault("--rate", String.valueOf(DEFAULT_RATE))); final int numSenders = Integer.parseInt(a.getOrDefault("--num-senders", String.valueOf(DEFAULT_NUM_SENDERS))); final int retryTimeout = Integer.parseInt(a.getOrDefault("--retry-timeout", String.valueOf(DEFAULT_RETRY_TIMEOUT))); final String csvPath = a.getOrDefault("--csv", DEFAULT_CSV); @@ -148,12 +162,25 @@ public static void main(String[] args) throws Exception { System.err.println("--total-events must be > 0"); System.exit(2); } + if (rate < 0) { + System.err.println("--rate must be >= 0 (0 disables rate limiting, falling back to --delay-ms)"); + System.exit(2); + } + // --rate takes precedence: when set it drives the pacing and --delay-ms is ignored. + if (rate > 0 && delayMs > 0 && a.containsKey("--delay-ms")) { + System.err.println("[warn] --rate " + rate + " overrides --delay-ms " + delayMs + + " (rate limiting drives pacing; the per-row delay is ignored)"); + } final SenderCfg cfg = new SenderCfg(protocol, addrsCsv, token, username, password, retryTimeout, senderIdBase, storeForwardDir, batchSize, batchesPerTransaction, numSenders, enterprise, zone, - connectTimeoutMs); + connectTimeoutMs, rate); + final String pacing = rate > 0 + ? "rate=" + rate + " rows/s (aggregate across " + numSenders + " workers)" + : "delay-ms=" + delayMs; final String conf = buildConf(addrsCsv, token, username, password, retryTimeout); + System.out.println("Pacing: " + pacing); System.out.println("Ingestion started. Protocol: " + protocol + (protocol.equals("qwp") ? " (WebSocket, sender-id=" + senderIdBase + ", store-and-forward=" + storeForwardDir @@ -180,18 +207,52 @@ public static void main(String[] args) throws Exception { // Time only the ingestion: start right before the workers begin sending. final long startNanos = System.nanoTime(); - // Progress reporter: prints rows/s (client-side, sent) once per second. + // Per-worker acknowledged-row counters, summed by the reporter. Fed from the QWP ack + // watermark (getAckedFsn) - the real committed progress, with no extra query round-trips. + ACKED_ROWS = new AtomicLong[numSenders]; + for (int i = 0; i < numSenders; i++) { + ACKED_ROWS[i] = new AtomicLong(); + } + + // Records (to ~1s resolution) when all rows were submitted and when all were acknowledged, + // so the summary can separate the submit phase from the commit-drain tail. + final AtomicLong appendDoneNanos = new AtomicLong(0); + final AtomicLong commitDoneNanos = new AtomicLong(0); + + // Progress reporter: once per second, prints BOTH counters - submitted (client-side, can + // run ahead of the server) and acknowledged (rows the server has actually committed). The + // gap between them is the buffered backlog; during the drain, submitted is flat while + // acknowledged keeps climbing (real work), so there is no misleading "0 rows/s". final Thread reporter = new Thread(() -> { - long last = 0; + long lastSub = 0; + long lastAck = 0; while (true) { try { Thread.sleep(1000); } catch (InterruptedException ie) { return; } - long now = TOTAL_SENT.get(); - System.out.printf("[progress] sent=%,d rate=%,d rows/s%n", now, now - last); - last = now; + final long sub = TOTAL_SENT.get(); + long ack = 0; + for (AtomicLong al : ACKED_ROWS) { + ack += al.get(); + } + if (sub >= totalEvents) { + appendDoneNanos.compareAndSet(0, System.nanoTime()); + SUBMIT_COMPLETE = true; + } + if (ack >= totalEvents) { + commitDoneNanos.compareAndSet(0, System.nanoTime()); + } + // Once everything is submitted, go quiet - no per-second spam while the client + // drains. The final "All workers completed ..." summary is the single last line. + if (SUBMIT_COMPLETE) { + continue; + } + System.out.printf("[progress] submitted=%,d (+%,d/s) | acknowledged=%,d (+%,d/s)%n", + sub, sub - lastSub, ack, ack - lastAck); + lastSub = sub; + lastAck = ack; } }); reporter.setDaemon(true); @@ -228,8 +289,17 @@ public static void main(String[] args) throws Exception { } final double elapsedSec = (System.nanoTime() - startNanos) / 1_000_000_000.0; final double rowsPerSec = elapsedSec > 0 ? totalEvents / elapsedSec : 0; - System.out.printf("All workers completed. protocol=%s events=%d elapsed=%.3f s throughput=%,.0f rows/s%n", + System.out.printf("All workers completed. protocol=%s events=%d elapsed=%.3f s throughput=%,.0f rows/s (acknowledged, end-to-end)%n", protocol, totalEvents, elapsedSec, rowsPerSec); + // Split the wall time into submit vs commit-drain, so the fast "submitted" rate is not + // mistaken for durable throughput. appendDoneNanos/commitDoneNanos are ~1s-resolution. + final long submitNanos = appendDoneNanos.get(); + if (submitNanos > 0) { + final double submitSec = (submitNanos - startNanos) / 1_000_000_000.0; + final double submitRate = submitSec > 0 ? totalEvents / submitSec : 0; + System.out.printf(" submit phase %.3f s (%,.0f rows/s submitted) + commit drain %.3f s%n", + submitSec, submitRate, Math.max(0.0, elapsedSec - submitSec)); + } } private static void runWorker( @@ -259,6 +329,19 @@ private static void runWorker( : isUdp ? cfg.batchSize : 0L; + // Rate limiting (when --rate > 0): pace this worker to its share of the aggregate + // target, rate / num-senders rows/second. intervalNanos is the ideal spacing between + // this worker's rows; we track a deadline schedule from loop entry and only sleep when + // we are running ahead of it, so rows go out back-to-back until we get ahead. This + // reaches high rates that a fixed per-row Thread.sleep cannot. 0 => rate disabled. + final double intervalNanos = cfg.rate > 0 + ? 1_000_000_000.0 * cfg.numSenders / cfg.rate + : 0.0; + final boolean rateLimited = intervalNanos > 0.0; + final long paceStartNanos = System.nanoTime(); + // QWP ack tracking: map each flush sequence -> cumulative rows, so getAckedFsn() can be + // resolved to acknowledged rows. Only QWP carries frame sequence numbers and acks. + final java.util.TreeMap fsnToRows = isQwp ? new java.util.TreeMap<>() : null; try ( Sender sender = buildSender(cfg, senderId)) { //( Sender sender = Sender.fromConfig(conf)) { final int n = rows.size(); for (long i = 0; i < totalEvents; i++) { @@ -279,11 +362,11 @@ private static void runWorker( if (secondsOffset != 0) { ts = ts.plusSeconds(secondsOffset); } - sender.at(ts); + atNanos(sender, ts); } else if (secondsOffset != 0) { - sender.at(Instant.now().plusSeconds(secondsOffset)); + atNanos(sender, Instant.now().plusSeconds(secondsOffset)); } else if (perRowMicros) { - sender.at(Instant.now()); + atNanos(sender, Instant.now()); } else { sender.atNow(); } @@ -291,12 +374,34 @@ private static void runWorker( sent++; TOTAL_SENT.incrementAndGet(); - // QWP-only: commit a transaction every batchSize * batchesPerTransaction rows. + // Commit a transaction every batchSize * batchesPerTransaction rows (QWP), or flush + // every batchSize rows (UDP). For QWP, record the flush sequence and refresh this + // worker's acknowledged-row count from the ack watermark (no extra round-trip). if (commitEveryRows > 0 && sent % commitEveryRows == 0) { - sender.flush(); + if (isQwp) { + fsnToRows.put(sender.flushAndGetSequence(), sent); + ACKED_ROWS[senderId].set(ackedRows(sender, fsnToRows)); + } else { + sender.flush(); + } } - if (delayMs > 0) { + if (rateLimited) { + // Deadline for the row we have just sent (sent is 1-based here). Sleep only + // while ahead of schedule; if behind, fall through and keep sending. The 1ms + // floor coalesces many rows into one sleep so we do not burn CPU sleeping for + // sub-millisecond slivers at high rates. + final long targetNanos = paceStartNanos + Math.round(sent * intervalNanos); + final long sleepNanos = targetNanos - System.nanoTime(); + if (sleepNanos > 1_000_000L) { + try { + Thread.sleep(sleepNanos / 1_000_000L, (int) (sleepNanos % 1_000_000L)); + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Interrupted", ie); + } + } + } else if (delayMs > 0) { try { Thread.sleep(delayMs); } catch (InterruptedException ie) { @@ -305,8 +410,20 @@ private static void runWorker( } } } - // Explicit flush at the end of this connection's work - sender.flush(); + // Final flush. For QWP, wait for the server to acknowledge everything so the acked + // counter climbs to the full count during the drain (the real committed progress). The + // await loop drives the connection and refreshes the counter; capped to avoid a hang. + if (isQwp) { + final long finalSeq = sender.flushAndGetSequence(); + fsnToRows.put(finalSeq, sent); + final long drainDeadline = System.nanoTime() + 600_000_000_000L; // 10 min cap + while (!sender.awaitAckedFsn(finalSeq, 200L) && System.nanoTime() < drainDeadline) { + ACKED_ROWS[senderId].set(ackedRows(sender, fsnToRows)); + } + } else { + sender.flush(); + } + ACKED_ROWS[senderId].set(sent); System.out.printf("Sender %d finished sending %d events%n", senderId, sent); } catch (Exception e) { System.err.printf("Sender %d got error: %s%s%n", senderId, e.toString(), upgradeHint(e)); @@ -314,6 +431,25 @@ private static void runWorker( } } + // Rows the server has acknowledged for this sender: map the ack watermark (getAckedFsn) back to + // the cumulative row count recorded at the latest fully-acknowledged flush. Zero round-trips - + // the ack watermark rides the ingest connection. + private static long ackedRows(Sender sender, java.util.TreeMap fsnToRows) { + final java.util.Map.Entry e = fsnToRows.floorEntry(sender.getAckedFsn()); + return e != null ? e.getValue() : 0L; + } + + // Send the designated timestamp at NANOSECOND resolution. The client's at(Instant) path + // delivers only microseconds (sub-microsecond digits are dropped before the wire), so we + // convert to epoch-nanos and use at(long, NANOS) to carry full nanosecond precision. + // QuestDB stores at the target column's resolution: a micros TIMESTAMP column silently + // truncates the extra digits, a TIMESTAMP_NS column keeps them. The multiply stays within + // long range for any realistic date (epochSecond * 1e9 overflows only past year ~2262). + private static void atNanos(Sender sender, Instant ts) { + final long epochNanos = ts.getEpochSecond() * 1_000_000_000L + ts.getNano(); + sender.at(epochNanos, ChronoUnit.NANOS); + } + private static List loadCsv(String path, boolean needTimestamp) throws Exception { List out = new ArrayList<>(1024); try (InputStream in0 = Files.newInputStream(Path.of(path)); @@ -725,8 +861,10 @@ public void onFailoverReset(QwpServerInfo info) { served = " served by role=" + handshake + " node=" + node + " zone=" + zone + " (handshake role; live 'switch status' unavailable, may be stale)"; } - System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", - ts, latest[0], served); + if (!SUBMIT_COMPLETE) { + System.out.printf("[probe] latest trades timestamp = %s (raw=%d)%s%n", + ts, latest[0], served); + } // Explain a missing live role at most once per 30s so it does not spam. if (currentRole[0] == null && statusDiag[0] != null) { final long now = System.currentTimeMillis(); @@ -834,6 +972,7 @@ private static Map parseArgs(String[] args) { case "--password": case "--total-events": case "--delay-ms": + case "--rate": case "--num-senders": case "--csv": case "--timestamp-from-file": @@ -885,11 +1024,13 @@ private static final class SenderCfg { final boolean enterprise; final String zone; final int connectTimeoutMs; + // Target aggregate rows/second across all workers; 0 = disabled (use delayMs). + final long rate; SenderCfg(String protocol, String addrsCsv, String token, String username, String password, int retryTimeout, String senderIdBase, String storeForwardDir, int batchSize, int batchesPerTransaction, int numSenders, boolean enterprise, String zone, - int connectTimeoutMs) { + int connectTimeoutMs, long rate) { this.protocol = protocol; this.addrsCsv = addrsCsv; this.token = token; @@ -904,6 +1045,7 @@ private static final class SenderCfg { this.enterprise = enterprise; this.zone = zone; this.connectTimeoutMs = connectTimeoutMs; + this.rate = rate; } } }