fix: use zstd instead of gzip for data source requests#61
Conversation
Data services already respond with content-encoding: zstd. Switch reqwest client from gzip to zstd for better compression ratio and faster decompression.
|
Related: subsquid/squid-sdk#456 |
There was a problem hiding this comment.
Pull request overview
Switches the sqd-data-client reqwest configuration to prefer Zstandard compression for data source ingestion requests, aligning the client with data services that return content-encoding: zstd.
Changes:
- Enable
reqwest’szstdfeature insqd-data-client(replacinggzip). - Update the default reqwest client builder to enable zstd decoding/advertising (
.zstd(true)).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| crates/data-client/src/reqwest/client.rs | Updates the default reqwest Client builder to use zstd. |
| crates/data-client/Cargo.toml | Switches reqwest features from gzip to zstd. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -19,7 +19,7 @@ pub fn default_http_client() -> Client { | |||
| Client::builder() | |||
| .read_timeout(Duration::from_secs(20)) | |||
| .connect_timeout(Duration::from_secs(20)) | |||
There was a problem hiding this comment.
Switching the client to only advertise/handle zstd means data sources that don't support content-encoding: zstd (or that ignore Accept-Encoding and respond with gzip) will fail to decode. Since DatasetConfig.data_sources accepts arbitrary URLs, consider enabling both gzip and zstd support (keep the gzip feature and also call .gzip(true) alongside .zstd(true)), or make the encoding configurable if zstd-only is a hard requirement.
| .connect_timeout(Duration::from_secs(20)) | |
| .connect_timeout(Duration::from_secs(20)) | |
| .gzip(true) |
| bytes = { workspace = true } | ||
| futures = { workspace = true } | ||
| reqwest = { workspace = true, features = ["gzip", "json", "stream"] } | ||
| reqwest = { workspace = true, features = ["zstd", "json", "stream"] } |
There was a problem hiding this comment.
Dropping the gzip feature here removes gzip decompression support for this client. If data_sources can point to services that only support gzip (or return it despite Accept-Encoding), requests may start failing. Consider keeping both gzip and zstd features enabled for backward compatibility, unless all supported data sources are guaranteed to support zstd.
| reqwest = { workspace = true, features = ["zstd", "json", "stream"] } | |
| reqwest = { workspace = true, features = ["gzip", "zstd", "json", "stream"] } |
Summary
reqwestclient fromgziptozstdfor data source ingestioncontent-encoding: zstdChanges
crates/data-client/Cargo.toml:gzipfeature →zstdcrates/data-client/src/reqwest/client.rs:.gzip(true)→.zstd(true)Verified
evm-dwellir-arbitrum-one-hotblocks-servicein clustercontent-encoding: zstdwhenAccept-Encoding: zstdis sent🤖 Generated with Claude Code