⚡ Optimize database table existence checks to avoid N+1 schema queries - #16
⚡ Optimize database table existence checks to avoid N+1 schema queries#16zrt219 wants to merge 1 commit into
Conversation
…hema queries This change refactors `validate_omop_database` and `_upsert_sql_tables` to query the database schema for table names exactly once for supported databases (SQLite and DuckDB) using `sqlite_master`. It then uses this pre-fetched set to quickly check for table existence, bypassing the slower and repetitive `SELECT 1 FROM table LIMIT 1` exception-driven check. This significantly improves the performance of validation and upsert operations by avoiding N+1 schema queries when repeatedly checking for tables, while still falling back to the original safe check when necessary (e.g. for non-supported database dialects). Co-authored-by: zrt219 <199104500+zrt219@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
1 participant
💡 What: Refactored
validate_omop_databaseand_upsert_sql_tablesto query the existing database schema for tables once upfront (usingsqlite_masterfor sqlite/duckdb where it is safe), and passed this cached set into the iteration loops to prevent repetitive_table_existsslow checks.🎯 Why: Previously, the code relied entirely on
SELECT 1 FROM table LIMIT 1catching an exception to determine if a table existed. This occurred in a hot path for every table check loop (e.g., invalidate_omop_databaseand_upsert_sql_tables), resulting in slow N+1 schema queries. The new approach fetches the schema list safely where supported and does quickinchecks on the set.📊 Measured Improvement: In a benchmark testing 2000 sequential
_table_existsoperations against a DuckDB memory database containing 10 tables, the execution time improved from ~0.75 seconds to ~0.01 seconds. In a SQLite memory database, it dropped from ~0.01 seconds to ~0.001 seconds. The overall validation loop performance speeds up proportionally.PR created automatically by Jules for task 8412359730842490020 started by @zrt219