Skip to content

Batch INSERT statements in executemany to reduce round trips#605

Open
srchilukoori wants to merge 1 commit into
trinodb:masterfrom
srchilukoori:batch-executemany-insert
Open

Batch INSERT statements in executemany to reduce round trips#605
srchilukoori wants to merge 1 commit into
trinodb:masterfrom
srchilukoori:batch-executemany-insert

Conversation

@srchilukoori
Copy link
Copy Markdown

Description

Fixes #602.

Cursor.executemany() executes INSERT...VALUES one row at a time.
For 1,000 rows that means 1,000 HTTP round trips to Trino.

This adds a regex check for INSERT...VALUES patterns and batches
rows into multi-row VALUES clauses, 100 per request. Non-INSERT
statements still use the existing row-by-row path. Parameters are
formatted with the existing _format_prepared_param() method.

Scenario Row-by-row Batched Speedup
500 rows x 2 cols 8.9s 0.23s ~39x
1,000 rows x 25 cols (mixed types) timeout 3.1s (322 rows/s)

Non-technical explanation

Bulk inserts via SQLAlchemy text() or cursor.executemany() now
send fewer queries to Trino instead of one per row.

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

* Batch INSERT...VALUES in `executemany()` into multi-row statements to reduce HTTP round trips. ({issue}`602`)

@cla-bot
Copy link
Copy Markdown

cla-bot Bot commented May 25, 2026

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Sadha Chilukoori.
This is most likely caused by a git client misconfiguration; please make sure to:

  1. check if your git client is configured with an email to sign commits git config --list | grep email
  2. If not, set it up using git config --global user.email email@example.com
  3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

SQLAlchemy bulk insert is executed per row

1 participant