feat: split raw data storage into per-dataset SQLite files

[astafan8]

Summary

This PR implements split raw data storage for QCoDeS: an opt-in feature that writes raw measurement data (results table rows) into individual per-dataset SQLite files while keeping all metadata in the main database. The goal is to prevent the main DB file from growing excessively large as datasets accumulate, making metadata browsing and experiment management faster.

It also includes dataset management helpers for maintaining the main database and raw data files over time.

Motivation

The main QCoDeS SQLite database stores both metadata (experiments, runs, parameter layouts, dependencies) and raw measurement data (results tables) in a single file. Over time, this file can grow to many gigabytes, slowing down operations that only need metadata. By splitting the raw data into per-dataset files, the main DB stays lightweight while data integrity is preserved.

Design Decisions

Architecture: transparent routing via _data_conn property

• A single _data_conn property on DataSet is the routing point for all data read/write operations
• Returns the per-dataset raw data connection when split is enabled, otherwise falls back to self.conn (main DB)
• All write paths (add_results, _BackgroundWriter) and read paths (get_parameter_data, DataSetCacheWithDBBackend, number_of_results, __len__) go through this property
• Zero changes to public DataSet API — all existing methods work identically

Config: follows existing export path pattern

• Two new config options in dataset section of qcodesrc.json:
  • raw_data_to_separate_db (bool, default false)
  • raw_data_path (string, default "{db_location}")
• Reuses _expand_export_path() from export_config.py for path expansion (e.g., ~/experiments.db -> ~/experiments_db/)
• Pattern mirrors the existing export_path / export_type config approach

Per-dataset files: lightweight, GUID-named

• Each file is named <guid>.db and contains only the results table + numpy type adapters
• No QCoDeS metadata schema in per-dataset files — they are minimal
• Path to per-dataset file is persisted in run metadata (raw_data_db_path dynamic column) for automatic reconnection on load_by_id()

Empty results table kept in main DB

• We considered removing the results table from the main DB entirely, but this would break:
  • _Subscriber trigger creation (SQLite triggers require the table to exist)
  • __len__ / number_of_results before dataset is started (when raw data DB doesn't yet exist)
  • Low-level query functions that inspect table structure via PRAGMA TABLE_INFO
  • The _check_if_table_found logic used in _get_datasetprotocol_from_guid to distinguish DataSet vs DataSetInMem
• Decision: keep the empty table schema (column definitions, no rows) — negligible overhead, full backward compatibility

get_parameter_data bypass for raw data DB

• The standard get_parameter_data() in queries.py calls get_rundescriber_from_result_table_name() which queries the runs table — this table doesn't exist in the raw data DB
• Solution: when _raw_data_conn is set, bypass the top-level function and call get_shaped_parameter_data_for_one_paramtree() directly with the already-held rundescriber

Subscriber triggers on data connection

• _Subscriber.__init__ creates SQL triggers for real-time data callbacks
• Changed to use _data_conn instead of self.conn so triggers fire on the correct DB where data is actually inserted

BackgroundWriter support

• _BackgroundWriter maintains a _raw_data_conns dict keyed by file path
• Queue items include optional raw_data_path key for routing
• Connections are lazily created and reused across datasets sharing the same raw data DB path

Dataset Management Helpers

Three management functions are exposed via qcodes.dataset:

update_raw_data_paths(db_path, new_raw_data_folder)

Updates stored paths in the main database after raw data files have been moved to a new folder. Scans all datasets with a raw_data_db_path metadata entry, checks if the corresponding .db file exists in the new folder, and updates the path.

purge_orphaned_datasets(db_path, *, dry_run=True)

Finds and removes dataset records from the main DB whose raw data files no longer exist on disk. Use case: after users archive or delete selected raw data files, this cleans up stale references. Defaults to dry_run=True — first see what would be removed, then set dry_run=False to execute.

cleanup_datasets(db_path, *, older_than_days=None, sample_name=None, larger_than_mb=None, dry_run=True)

Removes datasets (both DB records AND raw data files on disk) matching specified criteria. Criteria use AND logic. Use cases:

• Remove datasets older than N days
• Remove datasets for a specific sample name (exact match)
• Remove datasets with raw data files larger than N MB
• Combine criteria (e.g., older than 30 days AND sample "test-sample")

Defaults to dry_run=True. Both purge and cleanup return dataclass results with full details.

Compatibility with existing features (extract/export)

Audited all functions that read raw measurement data to ensure they work correctly when data is in separate per-dataset SQLite files:

• extract_runs_into_db: Fixed to use _data_conn instead of conn so raw data is read from the per-dataset file (not the empty results table in the main DB)
• export_datasets_and_create_metadata_db: NetCDF export path works correctly (goes through get_parameter_data which already routes via _data_conn); the _copy_dataset_as_is fallback path is fixed by the extract fix above
• unsubscribe / unsubscribe_all: Fixed to use _data_conn to remove triggers that were created on the raw data connection
• get_parameter_data, to_pandas_*, to_xarray_*, export, cache, plotting: All work correctly — they go through _data_conn or get_parameter_data() which already routes properly
• Added integration tests for extract_runs_into_db and export_datasets_and_create_metadata_db with split storage enabled

Files Changed

File	Type	Description
src/qcodes/dataset/_raw_data_storage.py	New	Helper module: storage config helpers, create_raw_data_db(), connect_to_raw_data_db(), update_raw_data_paths(), purge_orphaned_datasets(), cleanup_datasets()
src/qcodes/dataset/sqlite/queries.py	Modified	Added get_datasets_with_raw_data_path(), remove_dataset_from_db()
tests/dataset/test_raw_data_storage.py	New	35 tests (unit, integration, management, extract/export compatibility)
src/qcodes/dataset/data_set.py	Modified	_data_conn property, _raw_data_conn attribute, routing in __init__, _perform_start_actions, add_results, get_parameter_data, number_of_results, __len__, _BackgroundWriter, unsubscribe/unsubscribe_all
src/qcodes/dataset/database_extract_runs.py	Modified	_extract_single_dataset_into_db uses _data_conn
src/qcodes/dataset/data_set_cache.py	Modified	load_data_from_db() uses _data_conn
src/qcodes/dataset/subscriber.py	Modified	Trigger creation uses _data_conn
src/qcodes/dataset/__init__.py	Modified	Exports new public functions
src/qcodes/configuration/qcodesrc.json	Modified	Added config defaults
src/qcodes/configuration/qcodesrc_schema.json	Modified	Added config schema
docs/dataset/introduction.rst	Modified	New sections for split storage and management
docs/dataset/dataset_design.rst	Modified	Design notes on split storage
docs/examples/DataSet/Database.ipynb	Modified	Config, usage, and management documentation
docs/changes/newsfragments/8219.new	New	Towncrier news fragment

Verification

Own tests: 35/35 pass

• Unit tests for all helper functions (config reading, path generation, DB creation)
• Integration tests: write/read round-trip, cache, load_by_id, multiple datasets, background writer
• Non-interference: feature disabled -> data goes to main DB as before
• Management: purge dry run/actual, cleanup by age/sample/size, error cases
• Extract/export compatibility: extract_runs_into_db and export_datasets_and_create_metadata_db with split data

Full test suite with feature globally enabled: 1031 passed, 5 failed, 35 skipped

The 5 failures are all expected/explained:

Test	Reason
test_raw_data_conn_is_none	Own test for disabled mode — overridden by global enable
test_data_in_main_db	Own test for disabled mode — overridden by global enable
test_get_parameter_data	Low-level query test calls queries.get_parameter_data(ds.conn, ...) directly, bypassing DataSet
test_get_parameter_data_independent_parameters	Same as above
test_get_run_attributes	Metadata assertion expects exact {'foo': 'bar'} but split adds raw_data_db_path

Full test suite with feature disabled (default): all pass unchanged

Code quality

• Ruff lint: all checks passed
• Ruff format: all checks passed
• Pyright type check: 0 errors, 0 warnings

[codecov]

Codecov Report

❌ Patch coverage is 93.54839% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.29%. Comparing base (8e5ea8a) to head (2aeb373).

Files with missing lines	Patch %	Lines
src/qcodes/dataset/_raw_data_storage.py	94.73%	11 Missing ⚠️
src/qcodes/dataset/data_set.py	88.46%	6 Missing ⚠️
src/qcodes/dataset/sqlite/queries.py	93.75%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8219      +/-   ##
==========================================
+ Coverage   71.11%   71.29%   +0.18%     
==========================================
  Files         305      306       +1     
  Lines       32004    32271     +267     
==========================================
+ Hits        22760    23009     +249     
- Misses       9244     9262      +18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

[astafan8]

Added: update_raw_data_paths helper

Added a utility function for users who move their per-dataset raw data files to a new location. This mirrors the pattern used for exported netCDF files.

Usage:

`python
from qcodes.dataset import update_raw_data_paths

update_raw_data_paths(
db_path="/path/to/main_database.db",
new_raw_data_folder="/new/location/of/raw_files/"
)
`

The function:

• Scans all datasets in the main DB that have raw_data_db_path metadata
• For each, checks if the corresponding .db file exists in the new folder
• Updates the stored path to point to the new location
• Logs warnings for any files not found in the new folder

4 tests added, documentation updated in introduction.rst and Database.ipynb.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 8 comments.

[astafan8]

Suggested change

	loading metadata can become slower due to the file size. To address this,
	managing the database file can become inconvenient due to the file size. To address this,

[astafan8]

ok, so instead of copying this over here from connect funcition, can't you extract that part into a function and re-use it in here and in connect? the same is true for the rest of this function