FIX: Cache SQLDescribeParam results for NULL parameters to reduce server round-trips by jahnvi480 · Pull Request #614 · microsoft/mssql-python

jahnvi480 · 2026-06-02T03:16:41Z

Work Item / Issue Reference

AB#45399

GitHub Issue: #610

Summary

This pull request implements a robust caching mechanism for SQL parameter type resolution when binding None (NULL) parameters in the MSSQL Python driver, addressing issues with repeated round-trips to SQLDescribeParam and previous incorrect type fallbacks. The changes affect both the Python and C++ layers, improving performance and correctness, especially for all-NULL columns and VARBINARY types. Comprehensive tests are added to ensure correctness and coverage of the new cache logic.

Enhancements to NULL parameter type resolution and caching:

Introduced a thread-safe cache in ddbc_bindings.cpp to store SQLDescribeParam results per statement handle and parameter index, reducing redundant round-trips and ensuring correct type inference for NULL parameters. This replaces the previous fallback to SQL_VARCHAR, which caused issues with certain column types.
Updated both single and array parameter binding (BindParameters and BindParameterArray) to use the new cache for resolving types of NULL parameters, ensuring consistency and correctness for all-NULL columns.

Cache invalidation and integration:

Added cache invalidation logic when a new SQL statement is prepared, ensuring that parameter type caches are not reused across different statements. This is handled in both SQLExecute_wrap and SQLExecuteMany_wrap.

Python interface and type mapping:

Modified _map_sql_type in cursor.py to always return SQL_UNKNOWN_TYPE for None parameters, delegating type resolution to the C++ cache. Removed the previous Python-side fallback to SQL_VARCHAR for all-NULL columns in executemany.

Testing improvements:

Added extensive tests in test_004_cursor.py to cover cache hit/miss scenarios, cache invalidation, all-NULL columns, and bypassing the cache with explicit type hints, ensuring the new logic is robust and correct.

Codebase maintenance:

Included necessary C++ headers for thread-safety and updated function pointer loading logs for clarity.
Minor loop index and logging improvements for clarity and correctness in parameter binding.

These changes collectively resolve previous issues with NULL parameter handling, improve driver efficiency, and provide comprehensive test coverage for the new behavior.

…lared_parameters round-trips (GH-610) When a parameterized query includes None values, the driver previously sent SQL_UNKNOWN_TYPE to the C++ layer, which triggered a SQLDescribeParam call (internally sp_describe_undeclared_parameters) for every NULL param on every execute(). This caused thousands of unnecessary server round-trips per minute for workloads with frequent NULLs (e.g. sp_set_session_context). The describe call fails most of the time — especially for stored procedure calls — and the C++ layer already falls back to SQL_VARCHAR, so the round-trip adds latency with no benefit. Fix: Return SQL_VARCHAR directly from _map_sql_type() for None params, matching the existing executemany() all-NULL column optimisation. Closes #610

Copilot

Pull request overview

Optimizes NULL (None) parameter type mapping in mssql_python to avoid triggering SQLDescribeParam (and the associated sp_describe_undeclared_parameters server round-trips) on every execution involving NULL parameters, improving performance for heavily parameterized workloads.

Changes:

Update Cursor._map_sql_type to map None parameters to SQL_VARCHAR instead of SQL_UNKNOWN_TYPE.
Add a unit test asserting _map_sql_type returns the expected tuple for None parameters (GH-610).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
`mssql_python/cursor.py`	Changes NULL parameter SQL type mapping to `SQL_VARCHAR` to avoid `SQLDescribeParam` round-trips.
`tests/test_001_globals.py`	Adds a regression test verifying `_map_sql_type(None, ...)` returns `SQL_VARCHAR` and related defaults.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-06-02T03:22:20Z

📊 Code Coverage Report

🔥 Diff Coverage 69%	🎯 Overall Coverage 80%	📈 Total Lines Covered: `6689` out of `8288` 📁 Project: `mssql-python`

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

mssql_python/pybind/ddbc_bindings.cpp (68.8%): Missing lines 495-496,677-678,1485,1539-1541,1616,1728-1730,1812,1833,2573-2576,2578-2583,2594,2881-2884,3378
mssql_python/pybind/ddbc_bindings.h (100%)

Summary

Total: 97 lines
Missing: 30 lines
Coverage: 69%

mssql_python/pybind/ddbc_bindings.cpp

Lines 491-500

  491         "SQLDescribeParam", (void*)hStmt, paramIndex);
  492     RETCODE rc = SQLDescribeParam_ptr(
  493         hStmt, static_cast<SQLUSMALLINT>(paramIndex + 1),
  494         &type, &size, &digits, &nullable);
! 495 
! 496     DescribedParamInfo info;
  497     if (SQL_SUCCEEDED(rc)) {
  498         info = {type, size, digits, true};
  499         LOG("ResolveNullParamType: SQLDescribeParam succeeded for param[%d] "
  500             "-> sqlType=%d, columnSize=%lu, decimalDigits=%d",

Lines 673-682

  673                 SQLSMALLINT sqlType = paramInfo.paramSQLType;
  674                 SQLULEN columnSize = paramInfo.columnSize;
  675                 SQLSMALLINT decimalDigits = paramInfo.decimalDigits;
  676                 if (sqlType == SQL_UNKNOWN_TYPE) {
! 677                     auto resolved = ResolveNullParamType(handle, hStmt, paramIndex);
! 678                     sqlType = resolved.sqlType;
  679                     columnSize = resolved.columnSize;
  680                     decimalDigits = resolved.decimalDigits;
  681                 }
  682                 dataPtr = nullptr;

Lines 1481-1489

  1481 
  1482     // Release the GIL during the blocking ODBC catalog call
  1483     py::gil_scoped_release release;
  1484     return SQLProcedures_ptr(
! 1485         StatementHandle->get(), catalog.empty() ? nullptr : reinterpretU16stringAsSqlWChar(catalog),
  1486         catalog.empty() ? 0 : SQL_NTS,
  1487         schema.empty() ? nullptr : reinterpretU16stringAsSqlWChar(schema),
  1488         schema.empty() ? 0 : SQL_NTS,
  1489         procedure.empty() ? nullptr : reinterpretU16stringAsSqlWChar(procedure),

Lines 1535-1545

  1535     // Release the GIL during the blocking ODBC catalog call
  1536     py::gil_scoped_release release;
  1537     return SQLPrimaryKeys_ptr(StatementHandle->get(),
  1538                               catalog.empty() ? nullptr : reinterpretU16stringAsSqlWChar(catalog),
! 1539                               catalog.empty() ? 0 : SQL_NTS,
! 1540                               schema.empty() ? nullptr : reinterpretU16stringAsSqlWChar(schema),
! 1541                               schema.empty() ? 0 : SQL_NTS,
  1542                               table.empty() ? nullptr : reinterpretU16stringAsSqlWChar(table),
  1543                               table.empty() ? 0 : SQL_NTS);
  1544 }

Lines 1612-1620

  1612         SQLSMALLINT messageLen;
  1613 
  1614         SQLRETURN diagReturn =
  1615             SQLGetDiagRec_ptr(handleType, rawHandle, 1, sqlState, &nativeError, message,
! 1616                               SQL_MAX_MESSAGE_LENGTH_SQLSERVER, &messageLen);
  1617 
  1618         if (SQL_SUCCEEDED(diagReturn)) {
  1619             std::u16string sqlStateUtf16 = dupeSqlWCharAsUtf16Le(sqlState, 5);
  1620             std::u16string messageUtf16 = dupeSqlWCharAsUtf16Le(

Lines 1724-1734

  1724                             catalog.empty() ? nullptr : reinterpretU16stringAsSqlWChar(catalog),
  1725                             catalog.empty() ? 0 : SQL_NTS,
  1726                             schema.empty() ? nullptr : reinterpretU16stringAsSqlWChar(schema),
  1727                             schema.empty() ? 0 : SQL_NTS,
! 1728                             table.empty() ? nullptr : reinterpretU16stringAsSqlWChar(table),
! 1729                             table.empty() ? 0 : SQL_NTS,
! 1730                             tableType.empty() ? nullptr : reinterpretU16stringAsSqlWChar(tableType),
  1731                             tableType.empty() ? 0 : SQL_NTS);
  1732     }
  1733 
  1734     LOG("SQLTables: Catalog metadata query %s - SQLRETURN=%d",

Lines 1808-1816

  1808                     rc, (void*)hStmt);
  1809                 return rc;
  1810             }
  1811             // GH-610: Clear per-handle describe cache (new prepare = new param types)
! 1812             statementHandle->clearDescribeCache();
  1813             isStmtPrepared[0] = py::cast(true);
  1814         } else {
  1815             // Make sure the statement has been prepared earlier if we're not
  1816             // preparing now

Lines 1829-1837

  1829             charEncoding = encodingSettings["encoding"].cast<std::string>();
  1830         }
  1831 
  1832         std::vector<std::shared_ptr<void>> paramBuffers;
! 1833         rc = BindParameters(*statementHandle, hStmt, params, paramInfos, paramBuffers, charEncoding);
  1834         if (!SQL_SUCCEEDED(rc)) {
  1835             return rc;
  1836         }

Lines 2569-2587

  2569                     LOG("BindParameterArray: Binding SQL_C_DEFAULT (NULL) array - param_index=%d, "
  2570                         "count=%zu",
  2571                         paramIndex, paramSetSize);
  2572 
! 2573                     // GH-610: Resolve SQL type for all-NULL columns via per-handle cache.
! 2574                     SQLSMALLINT resolvedSqlType = info.paramSQLType;
! 2575                     SQLULEN resolvedColSize = info.columnSize;
! 2576                     SQLSMALLINT resolvedDecDigits = info.decimalDigits;
  2577                     if (resolvedSqlType == SQL_UNKNOWN_TYPE) {
! 2578                         auto resolved = ResolveNullParamType(handle, hStmt, paramIndex);
! 2579                         resolvedSqlType = resolved.sqlType;
! 2580                         resolvedColSize = resolved.columnSize;
! 2581                         resolvedDecDigits = resolved.decimalDigits;
! 2582                     }
! 2583 
  2584                     char* nullBuffer = AllocateParamBufferArray<char>(tempBuffers, paramSetSize);
  2585                     strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
  2586 
  2587                     for (size_t i = 0; i < paramSetSize; ++i) {

Lines 2590-2598

  2590                     }
  2591 
  2592                     dataPtr = nullBuffer;
  2593                     bufferLength = 1;
! 2594 
  2595                     // Override info fields so SQLBindParameter below uses resolved type
  2596                     const_cast<ParamInfo&>(info).paramSQLType = resolvedSqlType;
  2597                     const_cast<ParamInfo&>(info).columnSize = resolvedColSize;
  2598                     const_cast<ParamInfo&>(info).decimalDigits = resolvedDecDigits;

Lines 2877-2888

  2877     py::gil_scoped_release release;
  2878     return SQLSpecialColumns_ptr(StatementHandle->get(), identifierType,
  2879                                  catalog.empty() ? nullptr
  2880                                                  : reinterpretU16stringAsSqlWChar(catalog),
! 2881                                  catalog.empty() ? 0 : SQL_NTS,
! 2882                                  schema.empty() ? nullptr : reinterpretU16stringAsSqlWChar(schema),
! 2883                                  schema.empty() ? 0 : SQL_NTS,
! 2884                                  table.empty() ? nullptr : reinterpretU16stringAsSqlWChar(table),
  2885                                  table.empty() ? 0 : SQL_NTS, scope, nullable);
  2886 }
  2887 
  2888 // Wrap SQLFetch to retrieve rows

Lines 3374-3382

  3374                                 // exact number of characters via dataLen, so do not rely on
  3375                                 // null termination. This preserves embedded NULs and avoids
  3376                                 // any risk of reading past the valid range if the driver
  3377                                 // omits the terminator.
! 3378                                 row.append(py::cast(
  3379                                     dupeSqlWCharAsUtf16Le(dataBuffer.data(), numCharsInData)));
  3380                                 LOG("SQLGetData: Appended NVARCHAR string "
  3381                                     "length=%lu for column %d",
  3382                                     (unsigned long)numCharsInData, i);

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 76.3%
mssql_python.row.py: 76.9%
mssql_python.__init__.py: 77.3%
mssql_python.pybind.connection.connection.cpp: 77.3%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.logging.py: 85.5%
mssql_python.connection.py: 85.6%

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

… jahnvi/fix-null-param-describe-610

…o longer reachable since _map_sql_type(None) now returns SQL_VARCHAR directly.

… jahnvi/fix-null-param-describe-610

…declared_parameters round-trips (GH-610) When cursor.execute() includes None (NULL) params, the C++ BindParameters layer calls SQLDescribeParam for each NULL param on every execution. This triggers sp_describe_undeclared_parameters on SQL Server — a full network round-trip per NULL param per call. Fix: Add a statement-level cache in C++ that stores SQLDescribeParam results per (hStmt, paramIndex). First execution describes and caches; all subsequent executions of the same prepared statement get cache hits with zero round-trips. Cache is cleared on SQLPrepare (new SQL = new param types). Changes: - ddbc_bindings.cpp: Add DescribedParamInfo cache struct, ResolveNullParamType helper, cache invalidation on SQLPrepare in both SQLExecute_wrap and SQLExecuteMany_wrap. Both BindParameters and BindParameterArray use the cache for SQL_UNKNOWN_TYPE NULL params. - ddbc_bindings.h: Restore SQLDescribeParamFunc typedef and extern. - cursor.py: Revert _map_sql_type to return SQL_UNKNOWN_TYPE for None (let C++ cache handle resolution). Remove executemany SQL_VARCHAR hardcoded fallback (C++ BindParameterArray now resolves via cache). - test_004_cursor.py: Update test to verify SQL_UNKNOWN_TYPE for None. Closes #610

Add 7 integration tests covering all cache code paths: - Cache miss (first execute with NULL param) - Cache hit (repeated execute with same SQL + NULL) - Cache invalidation (different SQL triggers re-prepare) - executemany all-NULL column (BindParameterArray path) - executemany multiple all-NULL columns - All-NULL params in execute - setinputsizes bypass (explicit type skips cache)

… jahnvi/fix-null-param-describe-610

…om/microsoft/mssql-python into jahnvi/fix-null-param-describe-610

…H-610) Address reviewer feedback (sumitmsft): 1. Memory leak fix: Cache entries are now freed when SqlHandle::free() is called (all exit paths), preventing orphaned entries from freed HSTMT handles, pooled connections, and implicit frees. 2. Contention fix: Cache moved from global std::shared_mutex + std::unordered_map to a per-SqlHandle member. No mutex needed since each handle is used by one thread. Eliminates the global serialization point for multi-threaded workloads. Changes: - ddbc_bindings.h: Add DescribedParamInfo struct and describeCache member + clearDescribeCache() to SqlHandle class. - ddbc_bindings.cpp: Add ResolveNullParamType() helper that uses per-handle cache. Update BindParameters and BindParameterArray signatures to take SqlHandle&. Clear cache on SQLPrepare in both SQLExecute_wrap and SQLExecuteMany_wrap. Clear cache in SqlHandle::free(). - Remove global g_describeCache and g_describeCacheMutex. 522 tests pass, 0 regressions.

Copilot AI review requested due to automatic review settings June 2, 2026 03:16

Copilot started reviewing on behalf of jahnvi480 June 2, 2026 03:16 View session

github-actions Bot added the pr-size: small Minimal code update label Jun 2, 2026

Copilot AI reviewed Jun 2, 2026

View reviewed changes

jahnvi480 added 2 commits June 2, 2026 14:01

Merge branch 'main' of https://github.com/microsoft/mssql-python into…

217c601

… jahnvi/fix-null-param-describe-610

Moving test from global and putting it in cursor

b6f1478

jahnvi480 force-pushed the jahnvi/fix-null-param-describe-610 branch from 34b05de to b6f1478 Compare June 2, 2026 08:33

subrata-ms previously approved these changes Jun 2, 2026

View reviewed changes

Removed the dead executemany() SQL_UNKNOWN_TYPE check. That path is n…

af90409

…o longer reachable since _map_sql_type(None) now returns SQL_VARCHAR directly.

jahnvi480 dismissed subrata-ms’s stale review via af90409 June 2, 2026 12:45

jahnvi480 requested a review from subrata-ms June 2, 2026 12:48

Removing the dead code

00bb017

sumitmsft reviewed Jun 3, 2026

View reviewed changes

Comment thread mssql_python/cursor.py Outdated

Merge branch 'main' of https://github.com/microsoft/mssql-python into…

d48b07e

… jahnvi/fix-null-param-describe-610

jahnvi480 marked this pull request as draft June 3, 2026 10:29

jahnvi480 added 2 commits June 5, 2026 09:13

Merge branch 'main' of https://github.com/microsoft/mssql-python into…

ddd2893

… jahnvi/fix-null-param-describe-610

github-actions Bot added pr-size: medium Moderate update size and removed pr-size: small Minimal code update labels Jun 5, 2026

jahnvi480 marked this pull request as ready for review June 5, 2026 08:05

jahnvi480 changed the title ~~FIX: Skip SQLDescribeParam for NULL params to avoid sp_describe_undeclared_parameters round-trips~~ FIX: Cache SQLDescribeParam results for NULL parameters to reduce server round-trips Jun 5, 2026

jahnvi480 and others added 2 commits June 5, 2026 13:43

Resolving issues

61af48f

Merge branch 'main' into jahnvi/fix-null-param-describe-610

304a1b8

sumitmsft reviewed Jun 5, 2026

View reviewed changes

Comment thread mssql_python/pybind/ddbc_bindings.cpp Outdated

sumitmsft reviewed Jun 5, 2026

View reviewed changes

Comment thread mssql_python/pybind/ddbc_bindings.cpp Outdated

jahnvi480 added 2 commits June 8, 2026 09:28

Merge branch 'main' of https://github.com/microsoft/mssql-python into…

aa362e6

… jahnvi/fix-null-param-describe-610

Merge branch 'jahnvi/fix-null-param-describe-610' of https://github.c…

779cb31

…om/microsoft/mssql-python into jahnvi/fix-null-param-describe-610

jahnvi480 requested a review from sumitmsft June 8, 2026 04:47

Merge branch 'main' into jahnvi/fix-null-param-describe-610

17a8184

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FIX: Cache SQLDescribeParam results for NULL parameters to reduce server round-trips#614

FIX: Cache SQLDescribeParam results for NULL parameters to reduce server round-trips#614
jahnvi480 wants to merge 15 commits into
mainfrom
jahnvi/fix-null-param-describe-610

jahnvi480 commented Jun 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Jun 2, 2026 •

edited

Loading

🔥 Diff Coverage

69%

🎯 Overall Coverage

80%

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jahnvi480 commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Work Item / Issue Reference

Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Code Coverage Report

🔥 Diff Coverage

69%

🎯 Overall Coverage

80%

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

Summary

mssql_python/pybind/ddbc_bindings.cpp

📋 Files Needing Attention

🔗 Quick Links

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jahnvi480 commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading