From efca7d808e06c798c977aa341b5254806a472b32 Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Fri, 12 Jun 2026 13:38:28 +0200 Subject: [PATCH 01/12] feat: split raw data storage into per-dataset SQLite files Add optional configuration to write results-table data into individual per-dataset SQLite files (.db) while keeping all metadata in the main database. This keeps the main DB lightweight as it grows. Config options (dataset section of qcodesrc.json): - raw_data_to_separate_db (bool, default false) - raw_data_path (string, default '{db_location}') Implementation: - New module: qcodes.dataset.raw_data_storage (helper functions) - DataSet._data_conn property routes reads/writes to correct DB - BackgroundWriter supports per-dataset raw data connections - Subscriber triggers created on data connection for compatibility - Per-dataset DB path persisted in run metadata for auto-reconnect - Empty results table schema kept in main DB for compatibility - 19 new tests, all existing tests pass unchanged - Documentation added to dataset intro, design docs, and Database notebook Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/dataset/dataset_design.rst | 30 ++ docs/dataset/introduction.rst | 30 ++ docs/examples/DataSet/Database.ipynb | 65 ++++ src/qcodes/configuration/qcodesrc.json | 4 +- src/qcodes/configuration/qcodesrc_schema.json | 10 + src/qcodes/dataset/data_set.py | 113 ++++++- src/qcodes/dataset/data_set_cache.py | 5 +- src/qcodes/dataset/raw_data_storage.py | 174 ++++++++++ src/qcodes/dataset/subscriber.py | 2 +- tests/dataset/test_raw_data_storage.py | 315 ++++++++++++++++++ 10 files changed, 736 insertions(+), 12 deletions(-) create mode 100644 src/qcodes/dataset/raw_data_storage.py create mode 100644 tests/dataset/test_raw_data_storage.py diff --git a/docs/dataset/dataset_design.rst b/docs/dataset/dataset_design.rst index f44795586083..53a223292a52 100644 --- a/docs/dataset/dataset_design.rst +++ b/docs/dataset/dataset_design.rst @@ -75,3 +75,33 @@ We note that the dataset currently exclusively supports storing data in an SQLite database. This is not an intrinsic limitation of the dataset and measurement layer. It is possible that at a future state support for writing to a different backend will be added. + +.. _sec:design_split_storage: + +Split Raw Data Storage +====================== + +As the main SQLite database grows with many datasets, browsing experiments and +loading metadata can become slower due to the file size. To address this, +QCoDeS supports an optional **split raw data storage** mode (see +:ref:`sec:intro_split_raw_data` for user-facing details). + +From a design perspective, this feature adds a thin routing layer inside the +``DataSet`` class without changing any public interfaces: + +- A ``_data_conn`` property transparently returns either the main database + connection or a per-dataset raw data connection, depending on the + configuration. +- Write paths (``add_results``, ``_BackgroundWriter``) and read paths + (``get_parameter_data``, ``DataSetCacheWithDBBackend``, ``number_of_results``, + ``__len__``) all go through this single routing point. +- The per-dataset SQLite file is a lightweight database containing only the + results table and numpy type adapters -- no QCoDeS metadata schema. +- Subscriber triggers (used for real-time data callbacks) are created on the + data connection so that they fire regardless of which database holds the + results table. + +The implementation is contained in ``qcodes.dataset.raw_data_storage`` (helper +functions) and a handful of additions to ``qcodes.dataset.data_set`` (routing +logic). The ``Measurement`` context manager, ``DataSaver``, and all export +functions work without modification. diff --git a/docs/dataset/introduction.rst b/docs/dataset/introduction.rst index b5de7a35e4a3..b0dda5c1eb12 100644 --- a/docs/dataset/introduction.rst +++ b/docs/dataset/introduction.rst @@ -75,3 +75,33 @@ For dataset operations, QCoDeS provides functions for: - **Exporting datasets**: :doc:`Exporting data to other file formats <../examples/DataSet/Exporting-data-to-other-file-formats>` - **Extracting runs between databases**: :doc:`Extracting runs from one DB file to another <../examples/DataSet/Extracting-runs-from-one-DB-file-to-another>` and :func:`qcodes.dataset.extract_runs_into_db` - **Bulk export and metadata-only databases**: :func:`qcodes.dataset.export_datasets_and_create_metadata_db` for creating lightweight metadata-only databases while exporting all data to NetCDF files + +.. _sec:intro_split_raw_data: + +Split Raw Data Storage +====================== + +By default, all measurement data (the results table rows) is stored in the same SQLite database alongside metadata such as experiments, runs, parameter layouts, and dependencies. Over time, the main database file can grow very large, which can slow down operations like browsing experiments and loading metadata. + +QCoDeS supports an optional **split raw data storage** mode in which the actual measurement data for each ``DataSet`` is written to an individual, per-dataset SQLite file while all metadata remains in the main database. Each per-dataset file is named after the dataset's GUID (e.g. ``.db``) and is stored in a configurable folder. + +This feature is controlled by two configuration options in ``qcodesrc.json``: + +- ``dataset.raw_data_to_separate_db`` (bool, default ``false``): enables or disables split storage. +- ``dataset.raw_data_path`` (string, default ``"{db_location}"``): the folder where per-dataset files are created. The ``{db_location}`` placeholder is expanded to a folder derived from the main database path (e.g. ``~/experiments.db`` becomes ``~/experiments_db/``). + +When enabled: + +- The main database retains the full results table schema (column definitions) but no data rows are written to it, keeping it lightweight. +- All ``INSERT`` and ``SELECT`` operations on results data are transparently routed to the per-dataset file. +- The path to the per-dataset file is persisted in the run's metadata (``raw_data_db_path``), so ``load_by_id`` and related loading functions automatically reconnect to the correct file. +- All public ``DataSet`` APIs (``get_parameter_data``, ``to_pandas_dataframe``, ``to_xarray_dataset``, ``cache``, ``export``, etc.) work identically whether split storage is enabled or not. + +Example runtime configuration:: + + import qcodes as qc + + qc.config.dataset.raw_data_to_separate_db = True + qc.config.dataset.raw_data_path = "/data/raw_measurements/" + +For more details on database management, see the :doc:`Database notebook <../examples/DataSet/Database>`. diff --git a/docs/examples/DataSet/Database.ipynb b/docs/examples/DataSet/Database.ipynb index 9aeaa6bd818d..a294ec187062 100644 --- a/docs/examples/DataSet/Database.ipynb +++ b/docs/examples/DataSet/Database.ipynb @@ -167,6 +167,71 @@ "\n", "Moreover, we have also written an [example notebook](Extracting-runs-from-one-DB-file-to-another.ipynb) of transferring `DataSets` between database flies that may serve as a template for more complex data organization protocols." ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Split Raw Data Storage\n", + "\n", + "As the main database grows with many datasets, browsing experiments and loading metadata can become slower. QCoDeS supports an optional **split raw data storage** mode that writes the raw measurement data for each dataset into its own individual SQLite file, while keeping all metadata (experiments, runs, parameters, dependencies) in the main database.\n", + "\n", + "This keeps the main database lightweight and makes it faster to work with, while still allowing all existing `DataSet` APIs to function identically.\n", + "\n", + "### Configuration\n", + "\n", + "Split raw data storage is controlled by two configuration options:\n", + "\n", + "- `dataset.raw_data_to_separate_db` (bool, default `False`): enables or disables split storage.\n", + "- `dataset.raw_data_path` (string, default `\"{db_location}\"`): the folder where per-dataset SQLite files are created. The `{db_location}` placeholder expands to a folder derived from the main database path (e.g. `~/experiments.db` becomes `~/experiments_db/`).\n", + "\n", + "You can enable it at runtime:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Enable split raw data storage\n", + "qc.config.dataset.raw_data_to_separate_db = True\n", + "\n", + "# Optionally set a custom path for per-dataset files\n", + "qc.config.dataset.raw_data_path = \"/data/raw_measurements/\"\n", + "\n", + "# Or use the default which derives from the main DB location:\n", + "# qc.config.dataset.raw_data_path = \"{db_location}\"\n", + "# e.g. ~/experiments.db -> ~/experiments_db/" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Or permanently in your `qcodesrc.json`:\n", + "\n", + "```json\n", + "{\n", + " \"dataset\": {\n", + " \"raw_data_to_separate_db\": true,\n", + " \"raw_data_path\": \"{db_location}\"\n", + " }\n", + "}\n", + "```\n", + "\n", + "### How It Works\n", + "\n", + "When split storage is enabled:\n", + "\n", + "1. When a measurement starts (`mark_started()`), a per-dataset SQLite file named `.db` is created in the configured folder.\n", + "2. All measurement data (results table rows) is written to this per-dataset file instead of the main database.\n", + "3. The main database retains the results table schema (column definitions) but contains no data rows, keeping it small.\n", + "4. The path to the per-dataset file is saved in the run metadata, so `load_by_id()` and related functions automatically find and reconnect to the correct file.\n", + "5. All `DataSet` methods (`get_parameter_data`, `to_pandas_dataframe`, `to_xarray_dataset`, `cache`, `export`, etc.) work transparently with split storage.\n", + "\n", + "> **Note:** Datasets created with split storage enabled can always be loaded later, even if the configuration is changed back to the default, as long as the per-dataset files remain at their original paths." + ] } ], "metadata": { diff --git a/src/qcodes/configuration/qcodesrc.json b/src/qcodes/configuration/qcodesrc.json index 583f36bf8326..977cc3571966 100644 --- a/src/qcodes/configuration/qcodesrc.json +++ b/src/qcodes/configuration/qcodesrc.json @@ -79,7 +79,9 @@ "export_chunked_export_of_large_files_enabled": false, "export_chunked_threshold": 1000, "in_memory_cache": true, - "load_from_exported_file": false + "load_from_exported_file": false, + "raw_data_to_separate_db": false, + "raw_data_path": "{db_location}" }, "telemetry": { diff --git a/src/qcodes/configuration/qcodesrc_schema.json b/src/qcodes/configuration/qcodesrc_schema.json index ddd4929be22c..e7e49931d4f5 100644 --- a/src/qcodes/configuration/qcodesrc_schema.json +++ b/src/qcodes/configuration/qcodesrc_schema.json @@ -382,6 +382,16 @@ "type": "boolean", "default": true, "description": "Should the data be cached in memory as it is measured. Useful to disable for large datasets to save on memory consumption." + }, + "raw_data_to_separate_db": { + "type": "boolean", + "default": false, + "description": "If true, raw measurement data (results tables) will be written to individual per-dataset SQLite files instead of the main database. Metadata remains in the main database." + }, + "raw_data_path": { + "type": "string", + "default": "{db_location}", + "description": "Path to the folder where per-dataset raw data SQLite files are stored. {db_location} is a directory in the same folder as the .db file with a matching name, e.g. for ~/experiments.db raw data files will be stored in ~/experiments_db/" } }, "description": "Settings related to the DataSet and Measurement Context manager", diff --git a/src/qcodes/dataset/data_set.py b/src/qcodes/dataset/data_set.py index f36ad008772f..5d0c7f17e65b 100644 --- a/src/qcodes/dataset/data_set.py +++ b/src/qcodes/dataset/data_set.py @@ -66,6 +66,7 @@ get_run_timestamp_from_run_id, get_runid_from_guid, get_sample_name_from_experiment_id, + get_shaped_parameter_data_for_one_paramtree, mark_run_complete, remove_trigger, run_exists, @@ -99,6 +100,12 @@ load_to_xarray_dataset_dict, xarray_to_h5netcdf_with_complex_numbers, ) +from .raw_data_storage import ( + connect_to_raw_data_db, + create_raw_data_db, + get_raw_data_db_path, + is_raw_data_storage_enabled, +) from .subscriber import _Subscriber if TYPE_CHECKING: @@ -141,22 +148,40 @@ def __init__(self, queue: Queue[Any], conn: AtomicConnection): def run(self) -> None: self.conn = connect(self.path) + self._raw_data_conns: dict[str, AtomicConnection] = {} while self.keep_writing: item = self.queue.get() if item["keys"] == "stop": self.keep_writing = False self.conn.close() + for raw_conn in self._raw_data_conns.values(): + raw_conn.close() elif item["keys"] == "finalize": _WRITERS[self.path].active_datasets.remove(item["values"]) else: - self.write_results(item["keys"], item["values"], item["table_name"]) + conn = self._get_conn_for_item(item) + self.write_results( + conn, item["keys"], item["values"], item["table_name"] + ) self.queue.task_done() + def _get_conn_for_item(self, item: dict[str, Any]) -> AtomicConnection: + raw_data_path = item.get("raw_data_path") + if raw_data_path is None: + return self.conn + if raw_data_path not in self._raw_data_conns: + self._raw_data_conns[raw_data_path] = connect_to_raw_data_db(raw_data_path) + return self._raw_data_conns[raw_data_path] + def write_results( - self, keys: Sequence[str], values: Sequence[list[Any]], table_name: str + self, + conn: AtomicConnection, + keys: Sequence[str], + values: Sequence[list[Any]], + table_name: str, ) -> None: - insert_many_values(self.conn, table_name, keys, values) + insert_many_values(conn, table_name, keys, values) def shutdown(self) -> None: """ @@ -272,6 +297,7 @@ def __init__( self._cache: DataSetCacheWithDBBackend = DataSetCacheWithDBBackend(self) self._results: list[dict[str, VALUE]] = [] self._in_memory_cache = in_memory_cache + self._raw_data_conn: AtomicConnection | None = None if run_id is not None: if not run_exists(self.conn, run_id): @@ -290,6 +316,13 @@ def __init__( self._export_info = ExportInfo.from_str( self.metadata.get("export_info", "") ) + # If this dataset was saved with raw data in a separate db, + # re-open that connection for reads. + raw_db_path = self._metadata.get("raw_data_db_path") + if raw_db_path is not None and Path(raw_db_path).is_file(): + self._raw_data_conn = connect_to_raw_data_db( + raw_db_path, read_only=read_only + ) else: # Actually perform all the side effects needed for the creation # of a new dataset. Note that a dataset is created (in the DB) @@ -358,6 +391,17 @@ def prepare( def cache(self) -> DataSetCacheWithDBBackend: return self._cache + @property + def _data_conn(self) -> AtomicConnection: + """Connection to use for results-table data operations. + + Returns the separate raw-data connection when split storage is + active, otherwise falls back to the main database connection. + """ + if self._raw_data_conn is not None: + return self._raw_data_conn + return self.conn + @property def run_id(self) -> int: return self._run_id @@ -420,7 +464,7 @@ def snapshot_raw(self) -> str | None: @property def number_of_results(self) -> int: sql = f'SELECT COUNT(*) FROM "{self.table_name}"' - cursor = atomic_transaction(self.conn, sql) + cursor = atomic_transaction(self._data_conn, sql) return one(cursor, "COUNT(*)") @property @@ -682,12 +726,34 @@ def _perform_start_actions(self, start_bg_writer: bool) -> None: Perform the actions that must take place once the run has been started """ paramspecs = new_to_old(self._rundescriber.interdeps).paramspecs + raw_data_enabled = is_raw_data_storage_enabled() for spec in paramspecs: add_parameter( - spec, conn=self.conn, run_id=self.run_id, insert_into_results_table=True + spec, + conn=self.conn, + run_id=self.run_id, + insert_into_results_table=True, ) + # When raw data split is enabled, create a per-dataset SQLite file + # for results data with the full results table. + if raw_data_enabled: + raw_db_path = get_raw_data_db_path(self.guid) + self._raw_data_conn = create_raw_data_db( + raw_db_path, + self.table_name, + self._rundescriber.interdeps.paramspecs, + ) + # Persist the raw data path in metadata so we can find it when + # loading the dataset later. + raw_path_str = str(raw_db_path) + self._metadata["raw_data_db_path"] = raw_path_str + with atomic(self.conn) as aconn: + add_data_to_dynamic_columns( + aconn, self.run_id, {"raw_data_db_path": raw_path_str} + ) + desc_str = serial.to_json_for_storage(self.description) update_run_description(self.conn, self.run_id, desc_str) @@ -770,14 +836,18 @@ def add_results(self, results: Sequence[Mapping[str, VALUE]]) -> None: writer_status = self._writer_status if writer_status.write_in_background: - item = { + item: dict[str, Any] = { "keys": list(expected_keys), "values": values, "table_name": self.table_name, } + if self._raw_data_conn is not None: + item["raw_data_path"] = self._raw_data_conn.path_to_dbfile writer_status.data_write_queue.put(item) else: - insert_many_values(self.conn, self.table_name, list(expected_keys), values) + insert_many_values( + self._data_conn, self.table_name, list(expected_keys), values + ) def _raise_if_not_writable(self) -> None: if self.pristine: @@ -869,6 +939,25 @@ def get_parameter_data( else: valid_param_names = self._validate_parameters(*params) + + if self._raw_data_conn is not None: + # When raw data lives in a separate DB, we bypass + # get_parameter_data (which looks up the rundescriber + # from the main DB) and call the lower-level function + # directly with the rundescriber we already hold. + output: ParameterData = {} + for param_name in valid_param_names: + output[param_name] = get_shaped_parameter_data_for_one_paramtree( + self._raw_data_conn, + self.table_name, + self._rundescriber, + param_name, + start, + end, + callback, + ) + return output + return get_parameter_data( self.conn, self.table_name, valid_param_names, start, end, callback ) @@ -1225,7 +1314,7 @@ def get_metadata(self, tag: str) -> VALUE | None: return get_data_by_tag_and_table_name(self.conn, tag, self.table_name) def __len__(self) -> int: - return length(self.conn, self.table_name) + return length(self._data_conn, self.table_name) def __repr__(self) -> str: out = [] @@ -1878,7 +1967,13 @@ def _get_datasetprotocol_from_guid( if _check_if_table_found(conn, result_table_name): d = DataSet(conn=conn, run_id=run_id) else: - d = DataSetInMem._load_from_db(conn=conn, guid=guid) + # The results table may be absent from the main DB when raw data + # is stored in a separate per-dataset SQLite file. + metadata = get_metadata_from_run_id(conn, run_id) + if metadata.get("raw_data_db_path") is not None: + d = DataSet(conn=conn, run_id=run_id) + else: + d = DataSetInMem._load_from_db(conn=conn, guid=guid) return d diff --git a/src/qcodes/dataset/data_set_cache.py b/src/qcodes/dataset/data_set_cache.py index d3d361d2ce0d..2338b4143684 100644 --- a/src/qcodes/dataset/data_set_cache.py +++ b/src/qcodes/dataset/data_set_cache.py @@ -553,12 +553,15 @@ def load_data_from_db(self) -> None: self._loaded_from_completed_ds = True if self._data == {}: self.prepare() + # Use the raw-data connection when the dataset stores results + # in a separate per-dataset SQLite file. + data_conn = self._dataset._data_conn ( self._write_status, self._read_status, self._data, ) = load_new_data_from_db_and_append( - self._dataset.conn, + data_conn, self._dataset.table_name, self.rundescriber, self._write_status, diff --git a/src/qcodes/dataset/raw_data_storage.py b/src/qcodes/dataset/raw_data_storage.py new file mode 100644 index 000000000000..c581a97c631f --- /dev/null +++ b/src/qcodes/dataset/raw_data_storage.py @@ -0,0 +1,174 @@ +""" +Module for managing per-dataset raw data SQLite files. + +When the ``dataset.raw_data_to_separate_db`` config option is enabled, +measurement data (results tables) are written to individual SQLite files +- one per dataset - instead of the main QCoDeS database file. All metadata +(runs, experiments, layouts, dependencies) remains in the main database. + +The per-dataset files are stored in the folder given by +``dataset.raw_data_path`` and are named ``.db``. +""" + +from __future__ import annotations + +import logging +import sqlite3 +from pathlib import Path +from typing import TYPE_CHECKING + +import numpy as np + +import qcodes +from qcodes.dataset.export_config import _expand_export_path +from qcodes.dataset.sqlite.connection import AtomicConnection +from qcodes.dataset.sqlite.database import ( + _adapt_array, + _adapt_complex, + _adapt_float, + _convert_array, + _convert_complex, + _convert_numeric, +) +from qcodes.utils.types import complex_types, numpy_floats, numpy_ints + +if TYPE_CHECKING: + from collections.abc import Sequence + + from qcodes.parameters import ParamSpecBase + +log = logging.getLogger(__name__) + +_RAW_DATA_CONFIG_SECTION = "dataset" +_RAW_DATA_ENABLED_KEY = "raw_data_to_separate_db" +_RAW_DATA_PATH_KEY = "raw_data_path" + + +def is_raw_data_storage_enabled() -> bool: + """Return True if per-dataset raw data storage is enabled in config.""" + return bool( + qcodes.config[_RAW_DATA_CONFIG_SECTION].get(_RAW_DATA_ENABLED_KEY, False) + ) + + +def get_raw_data_folder() -> Path: + """Return the resolved folder path for raw data SQLite files. + + The path template from config is expanded the same way as the + export path (``{db_location}`` is replaced with a folder derived + from the main database path). + """ + raw_path_template: str = qcodes.config[_RAW_DATA_CONFIG_SECTION].get( + _RAW_DATA_PATH_KEY, "{db_location}" + ) + return Path(_expand_export_path(raw_path_template)).expanduser().absolute() + + +def get_raw_data_db_path(guid: str, folder: Path | None = None) -> Path: + """Return the full path for a dataset's raw data SQLite file. + + Args: + guid: The GUID of the dataset. + folder: Override folder. If *None*, uses :func:`get_raw_data_folder`. + + """ + if folder is None: + folder = get_raw_data_folder() + return folder / f"{guid}.db" + + +def connect_to_raw_data_db( + path: str | Path, + *, + read_only: bool = False, +) -> AtomicConnection: + """Open (or create) a lightweight SQLite connection for raw data. + + Unlike the main QCoDeS :func:`~qcodes.dataset.sqlite.database.connect`, + this does **not** create the full metadata schema (experiments, runs, ...). + It only registers the numpy/sqlite type adapters that QCoDeS needs to + round-trip array and numeric data. + + Args: + path: Path to the raw-data SQLite file. + read_only: Open the database in read-only mode. + + Returns: + An :class:`AtomicConnection` to the raw-data database. + + """ + # Register adapters/converters (idempotent calls) + sqlite3.register_adapter(np.ndarray, _adapt_array) + sqlite3.register_converter("array", _convert_array) + for numpy_int in numpy_ints: + sqlite3.register_adapter(numpy_int, int) + sqlite3.register_converter("numeric", _convert_numeric) + for numpy_float in (float, *numpy_floats): + sqlite3.register_adapter(numpy_float, _adapt_float) + for complex_type in complex_types: + sqlite3.register_adapter(complex_type, _adapt_complex) # type: ignore[arg-type] + sqlite3.register_converter("complex", _convert_complex) + + uri = f"file:{path!s}" + if read_only: + uri += "?mode=ro" + + conn = sqlite3.connect( + uri, + detect_types=sqlite3.PARSE_DECLTYPES, + check_same_thread=True, + uri=True, + factory=AtomicConnection, + ) + return conn + + +def create_raw_data_db( + path: str | Path, + table_name: str, + paramspecs: Sequence[ParamSpecBase], +) -> AtomicConnection: + """Create a per-dataset raw-data SQLite file with a results table. + + The file is created if it does not exist. The parent directory is + created if needed. + + Args: + path: Full path for the new SQLite file. + table_name: Name of the results table to create (matches the name + in the main database). + paramspecs: Parameter specifications describing the columns. + + Returns: + An :class:`AtomicConnection` to the newly created database. + + """ + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + + conn = connect_to_raw_data_db(path) + + if paramspecs: + columns = ",".join(p.sql_repr() for p in paramspecs) + sql = f""" + CREATE TABLE IF NOT EXISTS "{table_name}" ( + id INTEGER PRIMARY KEY, + {columns} + ); + """ + else: + sql = f""" + CREATE TABLE IF NOT EXISTS "{table_name}" ( + id INTEGER PRIMARY KEY + ); + """ + + conn.execute(sql) + conn.commit() + + log.info( + "Created raw data database at %s with table %s", + path, + table_name, + ) + return conn diff --git a/src/qcodes/dataset/subscriber.py b/src/qcodes/dataset/subscriber.py index 6b540599396c..5229878d5190 100644 --- a/src/qcodes/dataset/subscriber.py +++ b/src/qcodes/dataset/subscriber.py @@ -64,7 +64,7 @@ def __init__( self.callback_id = f"callback{self._id}" self.trigger_id = f"sub{self._id}" - conn = dataSet.conn + conn = dataSet._data_conn conn.create_function(self.callback_id, -1, self._cache_data_to_queue) diff --git a/tests/dataset/test_raw_data_storage.py b/tests/dataset/test_raw_data_storage.py new file mode 100644 index 000000000000..96d0c0055faf --- /dev/null +++ b/tests/dataset/test_raw_data_storage.py @@ -0,0 +1,315 @@ +""" +Tests for per-dataset raw data SQLite storage. + +When ``dataset.raw_data_to_separate_db`` is enabled, measurement data +(results tables) are written to individual SQLite files while metadata +remains in the main database. +""" + +from __future__ import annotations + +import gc +import sqlite3 +from pathlib import Path +from typing import TYPE_CHECKING + +import numpy as np +import pytest + +import qcodes as qc +from qcodes.dataset import new_data_set, new_experiment +from qcodes.dataset.data_set import DataSet, load_by_id +from qcodes.dataset.descriptions.dependencies import InterDependencies_ +from qcodes.dataset.raw_data_storage import ( + connect_to_raw_data_db, + create_raw_data_db, + get_raw_data_db_path, + get_raw_data_folder, + is_raw_data_storage_enabled, +) +from qcodes.dataset.sqlite.database import initialise_database +from qcodes.parameters import ParamSpecBase + +if TYPE_CHECKING: + from collections.abc import Generator + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + + +@pytest.fixture() +def _raw_data_db(tmp_path: Path) -> Generator[None, None, None]: + """Set up a temp DB with raw_data_to_separate_db enabled.""" + db_path = str(tmp_path / "test.db") + qc.config["core"]["db_location"] = db_path + qc.config["core"]["db_debug"] = False + qc.config["dataset"]["raw_data_to_separate_db"] = True + qc.config["dataset"]["raw_data_path"] = str(tmp_path / "raw_data") + initialise_database() + try: + yield + finally: + qc.config["dataset"]["raw_data_to_separate_db"] = False + qc.config["dataset"]["raw_data_path"] = "{db_location}" + gc.collect() + + +@pytest.fixture() +def _raw_data_experiment(_raw_data_db: None) -> Generator[None, None, None]: + """Create a test experiment inside the raw data DB.""" + e = new_experiment("test-experiment", sample_name="test-sample") + try: + yield + finally: + e.conn.close() + + +# --------------------------------------------------------------------------- +# Unit tests - raw_data_storage module +# --------------------------------------------------------------------------- + + +class TestRawDataStorageHelpers: + def test_is_raw_data_storage_enabled_default(self, tmp_path: Path) -> None: + assert not is_raw_data_storage_enabled() + + def test_is_raw_data_storage_enabled_on(self, _raw_data_db: None) -> None: + assert is_raw_data_storage_enabled() + + def test_get_raw_data_folder(self, _raw_data_db: None, tmp_path: Path) -> None: + folder = get_raw_data_folder() + assert folder == (tmp_path / "raw_data") + + def test_get_raw_data_db_path(self, tmp_path: Path) -> None: + folder = tmp_path / "raw_data" + path = get_raw_data_db_path("abc-123", folder) + assert path == folder / "abc-123.db" + + def test_create_raw_data_db(self, tmp_path: Path) -> None: + db_path = tmp_path / "raw" / "test.db" + params = [ + ParamSpecBase("x", "numeric"), + ParamSpecBase("y", "numeric"), + ] + conn = create_raw_data_db(db_path, "results_table", params) + try: + # Verify table exists and has the right columns + cursor = conn.execute("PRAGMA table_info('results_table')") + columns = {row[1] for row in cursor.fetchall()} + assert "id" in columns + assert "x" in columns + assert "y" in columns + finally: + conn.close() + + def test_connect_to_raw_data_db(self, tmp_path: Path) -> None: + db_path = tmp_path / "test.db" + conn = connect_to_raw_data_db(db_path) + try: + # Should be a valid connection with numpy adapters + conn.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, val REAL)") + conn.execute("INSERT INTO t (val) VALUES (?)", (3.14,)) + conn.commit() + cursor = conn.execute("SELECT val FROM t") + assert cursor.fetchone()[0] == pytest.approx(3.14) + finally: + conn.close() + + def test_connect_to_raw_data_db_read_only(self, tmp_path: Path) -> None: + db_path = tmp_path / "test.db" + # Create first + conn = connect_to_raw_data_db(db_path) + conn.execute("CREATE TABLE t (id INTEGER PRIMARY KEY)") + conn.commit() + conn.close() + # Now open read-only + conn = connect_to_raw_data_db(db_path, read_only=True) + try: + with pytest.raises(sqlite3.OperationalError): + conn.execute("INSERT INTO t (id) VALUES (1)") + finally: + conn.close() + + +# --------------------------------------------------------------------------- +# Integration tests - DataSet with split raw data +# --------------------------------------------------------------------------- + + +@pytest.mark.usefixtures("_raw_data_experiment") +class TestDataSetWithSplitRawData: + def _make_dataset_with_data( + self, n_rows: int = 10 + ) -> tuple[DataSet, list[dict[str, float]]]: + """Create a dataset, add data, mark completed, return ds and data.""" + ds = new_data_set("test-split") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + + results = [{"x": float(i), "y": float(i**2)} for i in range(n_rows)] + ds.add_results(results) + ds.mark_completed() + return ds, results + + def test_raw_data_conn_is_set(self) -> None: + """When split is enabled, DataSet should have a raw data connection.""" + ds = new_data_set("test-split") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + assert ds._raw_data_conn is not None + ds.conn.close() + + def test_raw_data_file_created(self, tmp_path: Path) -> None: + """A per-dataset SQLite file should be created.""" + ds, _ = self._make_dataset_with_data() + raw_folder = tmp_path / "raw_data" + raw_file = raw_folder / f"{ds.guid}.db" + assert raw_file.is_file() + ds.conn.close() + + def test_raw_data_db_path_in_metadata(self) -> None: + """The path to the raw data file should be stored in metadata.""" + ds, _ = self._make_dataset_with_data() + assert "raw_data_db_path" in ds.metadata + assert Path(ds.metadata["raw_data_db_path"]).is_file() + ds.conn.close() + + def test_data_is_in_raw_db_not_main(self) -> None: + """Data should be in the raw data file, not the main DB.""" + ds, _ = self._make_dataset_with_data(n_rows=5) + table_name = ds.table_name + main_conn = ds.conn + + # Main DB should have the results table (schema) but no data rows + cursor = main_conn.execute( + "SELECT name FROM sqlite_master WHERE type='table' AND name=?", + (table_name,), + ) + assert cursor.fetchone() is not None + cursor = main_conn.execute(f'SELECT COUNT(*) FROM "{table_name}"') + assert cursor.fetchone()[0] == 0 + + # Raw data DB should have the actual data + raw_conn = ds._raw_data_conn + assert raw_conn is not None + cursor = raw_conn.execute(f'SELECT COUNT(*) FROM "{table_name}"') + raw_count = cursor.fetchone()[0] + assert raw_count == 5 + ds.conn.close() + + def test_number_of_results(self) -> None: + """number_of_results should read from the raw data file.""" + ds, _ = self._make_dataset_with_data(n_rows=7) + assert ds.number_of_results == 7 + ds.conn.close() + + def test_get_parameter_data(self) -> None: + """get_parameter_data should read from the raw data file.""" + ds, results = self._make_dataset_with_data(n_rows=5) + data = ds.get_parameter_data() + assert "y" in data + np.testing.assert_array_almost_equal( + data["y"]["x"], np.array([r["x"] for r in results]) + ) + np.testing.assert_array_almost_equal( + data["y"]["y"], np.array([r["y"] for r in results]) + ) + ds.conn.close() + + def test_cache_loads_from_raw_data(self) -> None: + """The cache should also read from the raw data file.""" + ds, results = self._make_dataset_with_data(n_rows=5) + cache = ds.cache + cache.load_data_from_db() + cache_data = cache.data() + assert "y" in cache_data + np.testing.assert_array_almost_equal( + cache_data["y"]["x"], np.array([r["x"] for r in results]) + ) + ds.conn.close() + + def test_load_by_id_with_split_data(self) -> None: + """Loading by ID should automatically use the raw data connection.""" + ds, results = self._make_dataset_with_data(n_rows=3) + run_id = ds.run_id + ds.conn.close() + + # Re-load from the database + loaded = load_by_id(run_id) + assert isinstance(loaded, DataSet) + assert loaded._raw_data_conn is not None + data = loaded.get_parameter_data() + np.testing.assert_array_almost_equal( + data["y"]["y"], np.array([r["y"] for r in results]) + ) + loaded.conn.close() + + def test_multiple_datasets_split(self) -> None: + """Multiple datasets should each get their own raw data file.""" + ds1, _ = self._make_dataset_with_data(n_rows=3) + ds2, _ = self._make_dataset_with_data(n_rows=5) + + assert ds1._raw_data_conn is not None + assert ds2._raw_data_conn is not None + assert ds1._raw_data_conn.path_to_dbfile != ds2._raw_data_conn.path_to_dbfile + assert ds1.number_of_results == 3 + assert ds2.number_of_results == 5 + ds1.conn.close() + ds2.conn.close() + + def test_metadata_remains_in_main_db(self) -> None: + """Run metadata should be in the main DB, not the raw data DB.""" + ds, _ = self._make_dataset_with_data() + main_conn = ds.conn + # runs table should be populated in main DB + cursor = main_conn.execute( + "SELECT name, is_completed FROM runs WHERE run_id=?", (ds.run_id,) + ) + row = cursor.fetchone() + assert row is not None + assert row[0] == "test-split" + assert row[1] == 1 # completed + ds.conn.close() + + +# --------------------------------------------------------------------------- +# Tests that the feature doesn't interfere when disabled +# --------------------------------------------------------------------------- + + +@pytest.mark.usefixtures("experiment") +class TestDataSetWithoutSplitRawData: + def test_raw_data_conn_is_none(self) -> None: + """When split is disabled, _raw_data_conn should be None.""" + ds = new_data_set("test-no-split") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + assert ds._raw_data_conn is None + ds.conn.close() + + def test_data_in_main_db(self) -> None: + """When split is disabled, data should be in the main DB as usual.""" + ds = new_data_set("test-no-split") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + ds.add_results([{"x": 1.0, "y": 2.0}]) + ds.mark_completed() + + cursor = ds.conn.execute(f'SELECT COUNT(*) FROM "{ds.table_name}"') + assert cursor.fetchone()[0] == 1 + assert ds.number_of_results == 1 + ds.conn.close() From f9a3ea4b6001e79329be471a790e8a4a9feb94e2 Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Tue, 30 Jun 2026 10:04:43 +0200 Subject: [PATCH 02/12] address PR review feedback from Jens and Copilot - Rename raw_data_storage.py to _raw_data_storage.py (private module) - Raise FileNotFoundError when per-dataset raw data file is missing on load instead of silently falling back to empty main DB table - Quote column names in create_raw_data_db to handle SQL keyword names - Close both _raw_data_conn and conn in all tests to prevent leaked file handles - Add test_missing_raw_data_file_raises to verify the error behavior - Fix import ordering (ruff I001) and raw regex pattern (ruff RUF043) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...w_data_storage.py => _raw_data_storage.py} | 2 +- src/qcodes/dataset/data_set.py | 27 ++++++---- tests/dataset/test_raw_data_storage.py | 52 +++++++++++++------ 3 files changed, 55 insertions(+), 26 deletions(-) rename src/qcodes/dataset/{raw_data_storage.py => _raw_data_storage.py} (98%) diff --git a/src/qcodes/dataset/raw_data_storage.py b/src/qcodes/dataset/_raw_data_storage.py similarity index 98% rename from src/qcodes/dataset/raw_data_storage.py rename to src/qcodes/dataset/_raw_data_storage.py index c581a97c631f..d092cfc23615 100644 --- a/src/qcodes/dataset/raw_data_storage.py +++ b/src/qcodes/dataset/_raw_data_storage.py @@ -149,7 +149,7 @@ def create_raw_data_db( conn = connect_to_raw_data_db(path) if paramspecs: - columns = ",".join(p.sql_repr() for p in paramspecs) + columns = ",".join(f'"{p.name}" {p.type}' for p in paramspecs) sql = f""" CREATE TABLE IF NOT EXISTS "{table_name}" ( id INTEGER PRIMARY KEY, diff --git a/src/qcodes/dataset/data_set.py b/src/qcodes/dataset/data_set.py index 5d0c7f17e65b..106b1fa17bc7 100644 --- a/src/qcodes/dataset/data_set.py +++ b/src/qcodes/dataset/data_set.py @@ -86,6 +86,12 @@ NumpyJSONEncoder, ) +from ._raw_data_storage import ( + connect_to_raw_data_db, + create_raw_data_db, + get_raw_data_db_path, + is_raw_data_storage_enabled, +) from .data_set_cache import DataSetCacheWithDBBackend from .data_set_in_memory import DataSetInMem, load_from_file from .descriptions.versioning import serialization as serial @@ -100,12 +106,6 @@ load_to_xarray_dataset_dict, xarray_to_h5netcdf_with_complex_numbers, ) -from .raw_data_storage import ( - connect_to_raw_data_db, - create_raw_data_db, - get_raw_data_db_path, - is_raw_data_storage_enabled, -) from .subscriber import _Subscriber if TYPE_CHECKING: @@ -319,10 +319,17 @@ def __init__( # If this dataset was saved with raw data in a separate db, # re-open that connection for reads. raw_db_path = self._metadata.get("raw_data_db_path") - if raw_db_path is not None and Path(raw_db_path).is_file(): - self._raw_data_conn = connect_to_raw_data_db( - raw_db_path, read_only=read_only - ) + if raw_db_path is not None: + if Path(raw_db_path).is_file(): + self._raw_data_conn = connect_to_raw_data_db( + raw_db_path, read_only=read_only + ) + else: + raise FileNotFoundError( + f"Raw data file for dataset {self.guid} not found at " + f"'{raw_db_path}'. The per-dataset SQLite file may " + f"have been moved or deleted." + ) else: # Actually perform all the side effects needed for the creation # of a new dataset. Note that a dataset is created (in the DB) diff --git a/tests/dataset/test_raw_data_storage.py b/tests/dataset/test_raw_data_storage.py index 96d0c0055faf..384c8be71543 100644 --- a/tests/dataset/test_raw_data_storage.py +++ b/tests/dataset/test_raw_data_storage.py @@ -18,15 +18,15 @@ import qcodes as qc from qcodes.dataset import new_data_set, new_experiment -from qcodes.dataset.data_set import DataSet, load_by_id -from qcodes.dataset.descriptions.dependencies import InterDependencies_ -from qcodes.dataset.raw_data_storage import ( +from qcodes.dataset._raw_data_storage import ( connect_to_raw_data_db, create_raw_data_db, get_raw_data_db_path, get_raw_data_folder, is_raw_data_storage_enabled, ) +from qcodes.dataset.data_set import DataSet, load_by_id +from qcodes.dataset.descriptions.dependencies import InterDependencies_ from qcodes.dataset.sqlite.database import initialise_database from qcodes.parameters import ParamSpecBase @@ -156,6 +156,13 @@ def _make_dataset_with_data( ds.mark_completed() return ds, results + @staticmethod + def _close_ds(ds: DataSet) -> None: + """Close both main and raw data connections.""" + if ds._raw_data_conn is not None: + ds._raw_data_conn.close() + ds.conn.close() + def test_raw_data_conn_is_set(self) -> None: """When split is enabled, DataSet should have a raw data connection.""" ds = new_data_set("test-split") @@ -165,7 +172,7 @@ def test_raw_data_conn_is_set(self) -> None: ds.set_interdependencies(idps) ds.mark_started() assert ds._raw_data_conn is not None - ds.conn.close() + self._close_ds(ds) def test_raw_data_file_created(self, tmp_path: Path) -> None: """A per-dataset SQLite file should be created.""" @@ -173,14 +180,14 @@ def test_raw_data_file_created(self, tmp_path: Path) -> None: raw_folder = tmp_path / "raw_data" raw_file = raw_folder / f"{ds.guid}.db" assert raw_file.is_file() - ds.conn.close() + self._close_ds(ds) def test_raw_data_db_path_in_metadata(self) -> None: """The path to the raw data file should be stored in metadata.""" ds, _ = self._make_dataset_with_data() assert "raw_data_db_path" in ds.metadata assert Path(ds.metadata["raw_data_db_path"]).is_file() - ds.conn.close() + self._close_ds(ds) def test_data_is_in_raw_db_not_main(self) -> None: """Data should be in the raw data file, not the main DB.""" @@ -203,13 +210,13 @@ def test_data_is_in_raw_db_not_main(self) -> None: cursor = raw_conn.execute(f'SELECT COUNT(*) FROM "{table_name}"') raw_count = cursor.fetchone()[0] assert raw_count == 5 - ds.conn.close() + self._close_ds(ds) def test_number_of_results(self) -> None: """number_of_results should read from the raw data file.""" ds, _ = self._make_dataset_with_data(n_rows=7) assert ds.number_of_results == 7 - ds.conn.close() + self._close_ds(ds) def test_get_parameter_data(self) -> None: """get_parameter_data should read from the raw data file.""" @@ -222,7 +229,7 @@ def test_get_parameter_data(self) -> None: np.testing.assert_array_almost_equal( data["y"]["y"], np.array([r["y"] for r in results]) ) - ds.conn.close() + self._close_ds(ds) def test_cache_loads_from_raw_data(self) -> None: """The cache should also read from the raw data file.""" @@ -234,13 +241,13 @@ def test_cache_loads_from_raw_data(self) -> None: np.testing.assert_array_almost_equal( cache_data["y"]["x"], np.array([r["x"] for r in results]) ) - ds.conn.close() + self._close_ds(ds) def test_load_by_id_with_split_data(self) -> None: """Loading by ID should automatically use the raw data connection.""" ds, results = self._make_dataset_with_data(n_rows=3) run_id = ds.run_id - ds.conn.close() + self._close_ds(ds) # Re-load from the database loaded = load_by_id(run_id) @@ -250,7 +257,7 @@ def test_load_by_id_with_split_data(self) -> None: np.testing.assert_array_almost_equal( data["y"]["y"], np.array([r["y"] for r in results]) ) - loaded.conn.close() + self._close_ds(loaded) def test_multiple_datasets_split(self) -> None: """Multiple datasets should each get their own raw data file.""" @@ -262,8 +269,8 @@ def test_multiple_datasets_split(self) -> None: assert ds1._raw_data_conn.path_to_dbfile != ds2._raw_data_conn.path_to_dbfile assert ds1.number_of_results == 3 assert ds2.number_of_results == 5 - ds1.conn.close() - ds2.conn.close() + self._close_ds(ds1) + self._close_ds(ds2) def test_metadata_remains_in_main_db(self) -> None: """Run metadata should be in the main DB, not the raw data DB.""" @@ -277,7 +284,22 @@ def test_metadata_remains_in_main_db(self) -> None: assert row is not None assert row[0] == "test-split" assert row[1] == 1 # completed - ds.conn.close() + self._close_ds(ds) + + def test_missing_raw_data_file_raises(self, tmp_path: Path) -> None: + """Loading a dataset whose raw data file is missing should raise.""" + ds, _ = self._make_dataset_with_data(n_rows=3) + run_id = ds.run_id + raw_path = Path(ds.metadata["raw_data_db_path"]) + self._close_ds(ds) + + # Delete the raw data file + raw_path.unlink() + assert not raw_path.exists() + + # Attempting to load should raise FileNotFoundError + with pytest.raises(FileNotFoundError, match=r"Raw data file.*not found"): + load_by_id(run_id) # --------------------------------------------------------------------------- From 32b1c966023456c67072b027bfa28696c3bc8b6b Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Tue, 30 Jun 2026 18:34:58 +0200 Subject: [PATCH 03/12] Add update_raw_data_paths helper for relocated raw data files Implement a helper function that updates the raw_data_db_path metadata in the main database when individual raw data SQLite files have been moved to a new location. This mirrors the existing pattern used for exported netCDF files. The function: - Scans all datasets in the main DB that have raw_data_db_path metadata - For each, checks if the corresponding .db file exists in the new folder - Updates the stored path in the metadata to point to the new location - Reports which datasets were updated and which were skipped (file missing) Exposed as qcodes.dataset.update_raw_data_paths() for user convenience. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/qcodes/dataset/__init__.py | 2 + src/qcodes/dataset/_raw_data_storage.py | 86 ++++++++++++++++++++++++- tests/dataset/test_raw_data_storage.py | 86 +++++++++++++++++++++++++ 3 files changed, 173 insertions(+), 1 deletion(-) diff --git a/src/qcodes/dataset/__init__.py b/src/qcodes/dataset/__init__.py index 88e5b43d7f24..37359d4b51de 100644 --- a/src/qcodes/dataset/__init__.py +++ b/src/qcodes/dataset/__init__.py @@ -3,6 +3,7 @@ and from disk """ +from ._raw_data_storage import update_raw_data_paths from .data_set import ( get_guids_by_run_spec, load_by_counter, @@ -120,4 +121,5 @@ "plot_dataset", "reset_default_experiment_id", "rundescriber_from_json", + "update_raw_data_paths", ] diff --git a/src/qcodes/dataset/_raw_data_storage.py b/src/qcodes/dataset/_raw_data_storage.py index d092cfc23615..1aa6ec285200 100644 --- a/src/qcodes/dataset/_raw_data_storage.py +++ b/src/qcodes/dataset/_raw_data_storage.py @@ -21,7 +21,7 @@ import qcodes from qcodes.dataset.export_config import _expand_export_path -from qcodes.dataset.sqlite.connection import AtomicConnection +from qcodes.dataset.sqlite.connection import AtomicConnection, atomic from qcodes.dataset.sqlite.database import ( _adapt_array, _adapt_complex, @@ -29,7 +29,9 @@ _convert_array, _convert_complex, _convert_numeric, + connect, ) +from qcodes.dataset.sqlite.query_helpers import is_column_in_table from qcodes.utils.types import complex_types, numpy_floats, numpy_ints if TYPE_CHECKING: @@ -172,3 +174,85 @@ def create_raw_data_db( table_name, ) return conn + + +def update_raw_data_paths( + db_path: str | Path, + new_raw_data_folder: str | Path, +) -> list[tuple[int, str, str]]: + """Update raw data file paths in the main database after files have moved. + + Use this when per-dataset raw data files have been relocated to a new + folder but the main database still references the old paths3. + + The function scans all runs that have a ``raw_data_db_path`` metadata + entry, verifies that a file with the expected GUID-based name exists in + *new_raw_data_folder*, and updates the stored path in the database. + + Args: + db_path: Path to the main QCoDeS database file. + new_raw_data_folder: The new folder where the per-dataset SQLite + files now reside. + + Returns: + A list of ``(run_id, old_path, new_path)`` tuples for every run + whose path was updated. + + Raises: + FileNotFoundError: If the main database file does not exist. + FileNotFoundError: If *new_raw_data_folder* does not exist. + + """ + db_path = Path(db_path) + new_raw_data_folder = Path(new_raw_data_folder) + + if not db_path.is_file(): + raise FileNotFoundError(f"Database file not found: {db_path}") + if not new_raw_data_folder.is_dir(): + raise FileNotFoundError( + f"New raw data folder not found: {new_raw_data_folder}" + ) + + conn = connect(str(db_path)) + + if not is_column_in_table(conn, "runs", "raw_data_db_path"): + log.info("No raw_data_db_path column found in %s; nothing to update.", db_path) + conn.close() + return [] + + cursor = conn.execute( + "SELECT run_id, raw_data_db_path FROM runs " + "WHERE raw_data_db_path IS NOT NULL" + ) + rows = cursor.fetchall() + + updated: list[tuple[int, str, str]] = [] + + for run_id, old_path_str in rows: + old_path = Path(old_path_str) + # The per-dataset file name is always .db — preserved on move + new_path = new_raw_data_folder / old_path.name + + if not new_path.is_file(): + log.warning( + "Run %d: expected raw data file %s not found in new folder; skipping.", + run_id, + new_path, + ) + continue + + if str(new_path) == old_path_str: + continue # already correct + + new_path_str = str(new_path) + with atomic(conn) as aconn: + aconn.execute( + "UPDATE runs SET raw_data_db_path = ? WHERE run_id = ?", + (new_path_str, run_id), + ) + updated.append((run_id, old_path_str, new_path_str)) + log.info("Run %d: updated raw_data_db_path from %s to %s", run_id, old_path_str, new_path_str) + + conn.close() + log.info("Updated %d raw data paths in %s", len(updated), db_path) + return updated diff --git a/tests/dataset/test_raw_data_storage.py b/tests/dataset/test_raw_data_storage.py index 384c8be71543..2b4358b2e4f1 100644 --- a/tests/dataset/test_raw_data_storage.py +++ b/tests/dataset/test_raw_data_storage.py @@ -9,6 +9,7 @@ from __future__ import annotations import gc +import shutil import sqlite3 from pathlib import Path from typing import TYPE_CHECKING @@ -24,6 +25,7 @@ get_raw_data_db_path, get_raw_data_folder, is_raw_data_storage_enabled, + update_raw_data_paths, ) from qcodes.dataset.data_set import DataSet, load_by_id from qcodes.dataset.descriptions.dependencies import InterDependencies_ @@ -335,3 +337,87 @@ def test_data_in_main_db(self) -> None: assert cursor.fetchone()[0] == 1 assert ds.number_of_results == 1 ds.conn.close() + + +# --------------------------------------------------------------------------- +# Tests for update_raw_data_paths helper +# --------------------------------------------------------------------------- + + +@pytest.mark.usefixtures("_raw_data_experiment") +class TestUpdateRawDataPaths: + @staticmethod + def _close_ds(ds: DataSet) -> None: + if ds._raw_data_conn is not None: + ds._raw_data_conn.close() + ds.conn.close() + + def test_update_after_move(self, tmp_path: Path) -> None: + """Paths should be updated after raw data files are moved.""" + ds = new_data_set("test-move") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + ds.add_results([{"x": 1.0, "y": 2.0}]) + ds.mark_completed() + + raw_path = Path(ds.metadata["raw_data_db_path"]) + db_path = ds.path_to_db + self._close_ds(ds) + + # Move the raw data file to a new folder + new_folder = tmp_path / "moved_raw" + new_folder.mkdir() + new_file = new_folder / raw_path.name + + shutil.move(str(raw_path), str(new_file)) + + # Run the update + updated = update_raw_data_paths(db_path, new_folder) + assert len(updated) == 1 + run_id, old_path, new_path = updated[0] + assert old_path == str(raw_path) + assert new_path == str(new_file) + + # Verify the dataset can now be loaded + loaded = load_by_id(run_id) + assert isinstance(loaded, DataSet) + assert loaded.number_of_results == 1 + data = loaded.get_parameter_data() + assert data["y"]["x"][0] == 1.0 + self._close_ds(loaded) + + def test_update_skips_missing_files(self, tmp_path: Path) -> None: + """Runs whose files are not in the new folder should be skipped.""" + ds = new_data_set("test-skip") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + ds.add_results([{"x": 1.0, "y": 2.0}]) + ds.mark_completed() + + db_path = ds.path_to_db + self._close_ds(ds) + + # Point to an empty folder — file not there + empty_folder = tmp_path / "empty" + empty_folder.mkdir() + + updated = update_raw_data_paths(db_path, empty_folder) + assert len(updated) == 0 + + def test_update_nonexistent_db_raises(self, tmp_path: Path) -> None: + """Should raise if the main DB file doesn't exist.""" + with pytest.raises(FileNotFoundError, match="Database file not found"): + update_raw_data_paths(tmp_path / "nonexistent.db", tmp_path) + + def test_update_nonexistent_folder_raises(self, tmp_path: Path) -> None: + """Should raise if the new folder doesn't exist.""" + db_path = tmp_path / "test.db" + db_path.touch() + with pytest.raises(FileNotFoundError, match="New raw data folder not found"): + update_raw_data_paths(db_path, tmp_path / "no_such_folder") From d897d95a0d2f58fc277acafb08fe3246b10824e3 Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Tue, 30 Jun 2026 18:35:53 +0200 Subject: [PATCH 04/12] Add update_raw_data_paths documentation to intro.rst and Database.ipynb Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/dataset/introduction.rst | 11 +++++++++++ docs/examples/DataSet/Database.ipynb | 28 +++++++++++++++++++++++++++- 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/docs/dataset/introduction.rst b/docs/dataset/introduction.rst index b0dda5c1eb12..782fa2c9a70a 100644 --- a/docs/dataset/introduction.rst +++ b/docs/dataset/introduction.rst @@ -104,4 +104,15 @@ Example runtime configuration:: qc.config.dataset.raw_data_to_separate_db = True qc.config.dataset.raw_data_path = "/data/raw_measurements/" +If the per-dataset raw data files are moved to a different folder (e.g. during data migration or archival), the stored paths in the main database will become stale. Use the :func:`~qcodes.dataset.update_raw_data_paths` helper to update them:: + + from qcodes.dataset import update_raw_data_paths + + update_raw_data_paths( + db_path="/path/to/main_database.db", + new_raw_data_folder="/new/location/of/raw_files/" + ) + +This scans all datasets with a ``raw_data_db_path`` metadata entry, checks whether the corresponding ``.db`` file exists in the new folder, and updates the stored path accordingly. + For more details on database management, see the :doc:`Database notebook <../examples/DataSet/Database>`. diff --git a/docs/examples/DataSet/Database.ipynb b/docs/examples/DataSet/Database.ipynb index a294ec187062..b1cf905223c2 100644 --- a/docs/examples/DataSet/Database.ipynb +++ b/docs/examples/DataSet/Database.ipynb @@ -230,7 +230,33 @@ "4. The path to the per-dataset file is saved in the run metadata, so `load_by_id()` and related functions automatically find and reconnect to the correct file.\n", "5. All `DataSet` methods (`get_parameter_data`, `to_pandas_dataframe`, `to_xarray_dataset`, `cache`, `export`, etc.) work transparently with split storage.\n", "\n", - "> **Note:** Datasets created with split storage enabled can always be loaded later, even if the configuration is changed back to the default, as long as the per-dataset files remain at their original paths." + "> **Note:** Datasets created with split storage enabled can always be loaded later, even if the configuration is changed back to the default, as long as the per-dataset files remain at their original paths.\n", + "\n", + "### Updating Paths After Moving Raw Data Files\n", + "\n", + "If you move the per-dataset raw data files to a different folder, the paths stored in the main database become stale. Use `update_raw_data_paths` to fix them:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from qcodes.dataset import update_raw_data_paths\n", + "\n", + "# After moving raw data files to a new folder:\n", + "# update_raw_data_paths(\n", + "# db_path=\"/path/to/main_database.db\",\n", + "# new_raw_data_folder=\"/new/location/of/raw_files/\"\n", + "# )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The function scans all datasets that have a `raw_data_db_path` metadata entry, checks whether the corresponding `.db` file exists in the new folder, and updates the stored path. Datasets whose files are not found in the new folder are skipped with a warning." ] } ], From ce732d64c2167cd5b0a2b088a1959b5a568ada13 Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Thu, 2 Jul 2026 16:59:41 +0200 Subject: [PATCH 05/12] Fix ruff format and unused import lint issues - Apply ruff format to _raw_data_storage.py (line length adjustments) - Comment out unused import in Database.ipynb example cell Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/examples/DataSet/Database.ipynb | 2 +- src/qcodes/dataset/_raw_data_storage.py | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/examples/DataSet/Database.ipynb b/docs/examples/DataSet/Database.ipynb index b1cf905223c2..68249464532e 100644 --- a/docs/examples/DataSet/Database.ipynb +++ b/docs/examples/DataSet/Database.ipynb @@ -243,7 +243,7 @@ "metadata": {}, "outputs": [], "source": [ - "from qcodes.dataset import update_raw_data_paths\n", + "# from qcodes.dataset import update_raw_data_paths\n", "\n", "# After moving raw data files to a new folder:\n", "# update_raw_data_paths(\n", diff --git a/src/qcodes/dataset/_raw_data_storage.py b/src/qcodes/dataset/_raw_data_storage.py index 1aa6ec285200..58492ebb60db 100644 --- a/src/qcodes/dataset/_raw_data_storage.py +++ b/src/qcodes/dataset/_raw_data_storage.py @@ -209,9 +209,7 @@ def update_raw_data_paths( if not db_path.is_file(): raise FileNotFoundError(f"Database file not found: {db_path}") if not new_raw_data_folder.is_dir(): - raise FileNotFoundError( - f"New raw data folder not found: {new_raw_data_folder}" - ) + raise FileNotFoundError(f"New raw data folder not found: {new_raw_data_folder}") conn = connect(str(db_path)) @@ -221,8 +219,7 @@ def update_raw_data_paths( return [] cursor = conn.execute( - "SELECT run_id, raw_data_db_path FROM runs " - "WHERE raw_data_db_path IS NOT NULL" + "SELECT run_id, raw_data_db_path FROM runs WHERE raw_data_db_path IS NOT NULL" ) rows = cursor.fetchall() @@ -251,7 +248,12 @@ def update_raw_data_paths( (new_path_str, run_id), ) updated.append((run_id, old_path_str, new_path_str)) - log.info("Run %d: updated raw_data_db_path from %s to %s", run_id, old_path_str, new_path_str) + log.info( + "Run %d: updated raw_data_db_path from %s to %s", + run_id, + old_path_str, + new_path_str, + ) conn.close() log.info("Updated %d raw data paths in %s", len(updated), db_path) From a195b320ded335f7f14d63897d80a9cc2060e50d Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Thu, 2 Jul 2026 17:26:43 +0200 Subject: [PATCH 06/12] Fix pyright type errors in test_raw_data_storage Add assert statements to narrow path_to_db from str | None to str before passing to update_raw_data_paths(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/dataset/test_raw_data_storage.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/dataset/test_raw_data_storage.py b/tests/dataset/test_raw_data_storage.py index 2b4358b2e4f1..03ca6ad047a6 100644 --- a/tests/dataset/test_raw_data_storage.py +++ b/tests/dataset/test_raw_data_storage.py @@ -365,6 +365,7 @@ def test_update_after_move(self, tmp_path: Path) -> None: raw_path = Path(ds.metadata["raw_data_db_path"]) db_path = ds.path_to_db + assert db_path is not None self._close_ds(ds) # Move the raw data file to a new folder @@ -401,6 +402,7 @@ def test_update_skips_missing_files(self, tmp_path: Path) -> None: ds.mark_completed() db_path = ds.path_to_db + assert db_path is not None self._close_ds(ds) # Point to an empty folder — file not there From 07266e6ceab2263c2837ad72f03b78a60b75b124 Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Thu, 16 Jul 2026 15:56:57 -0700 Subject: [PATCH 07/12] Add purge_orphaned_datasets and cleanup_datasets management functions Two new functions for managing per-dataset raw data storage: - purge_orphaned_datasets(): Finds and removes dataset records from the main DB whose raw data files no longer exist on disk. Useful after archiving/deleting selected raw data files. Defaults to dry_run=True. - cleanup_datasets(): Removes datasets (both DB records AND raw data files) matching criteria: older_than_days, sample_name (exact match), or larger_than_mb. Criteria use AND logic. Defaults to dry_run=True. Both functions return dataclass results with full details of what was found/removed. The _remove_dataset_from_db helper correctly deletes from runs, layouts, dependencies tables and drops the results table. Documentation added to introduction.rst and Database.ipynb. 9 new tests covering both functions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/dataset/introduction.rst | 33 +++ docs/examples/DataSet/Database.ipynb | 53 ++++ src/qcodes/dataset/__init__.py | 8 +- src/qcodes/dataset/_raw_data_storage.py | 357 ++++++++++++++++++++++++ tests/dataset/test_raw_data_storage.py | 256 ++++++++++++++++- 5 files changed, 705 insertions(+), 2 deletions(-) diff --git a/docs/dataset/introduction.rst b/docs/dataset/introduction.rst index 782fa2c9a70a..4e6fb0b6d042 100644 --- a/docs/dataset/introduction.rst +++ b/docs/dataset/introduction.rst @@ -115,4 +115,37 @@ If the per-dataset raw data files are moved to a different folder (e.g. during d This scans all datasets with a ``raw_data_db_path`` metadata entry, checks whether the corresponding ``.db`` file exists in the new folder, and updates the stored path accordingly. +Purging Orphaned Datasets +------------------------- + +After archiving or deleting selected raw data files, some dataset records in the main database may reference files that no longer exist on disk. Use :func:`~qcodes.dataset.purge_orphaned_datasets` to identify and remove those orphaned records:: + + from qcodes.dataset import purge_orphaned_datasets + + # First, see what would be removed (dry run — the default) + result = purge_orphaned_datasets("/path/to/main_database.db") + print(f"{len(result.orphaned_datasets)} orphaned datasets found") + + # Then actually remove them + result = purge_orphaned_datasets("/path/to/main_database.db", dry_run=False) + +Cleaning Up Datasets by Criteria +--------------------------------- + +To free disk space by removing datasets (both the raw data files **and** their records in the main database) based on age, sample name, or file size, use :func:`~qcodes.dataset.cleanup_datasets`:: + + from qcodes.dataset import cleanup_datasets + + # Remove datasets older than 30 days (dry run first) + result = cleanup_datasets("/path/to/db.db", older_than_days=30) + print(f"{len(result.matching_datasets)} datasets would be removed") + + # Remove datasets for a specific sample + result = cleanup_datasets("/path/to/db.db", sample_name="old-sample", dry_run=False) + + # Remove datasets with raw data larger than 500 MB + result = cleanup_datasets("/path/to/db.db", larger_than_mb=500, dry_run=False) + +Criteria are combined with AND logic — a dataset must match **all** specified criteria to be selected for removal. + For more details on database management, see the :doc:`Database notebook <../examples/DataSet/Database>`. diff --git a/docs/examples/DataSet/Database.ipynb b/docs/examples/DataSet/Database.ipynb index 68249464532e..3d2f5e7e44fa 100644 --- a/docs/examples/DataSet/Database.ipynb +++ b/docs/examples/DataSet/Database.ipynb @@ -258,6 +258,59 @@ "source": [ "The function scans all datasets that have a `raw_data_db_path` metadata entry, checks whether the corresponding `.db` file exists in the new folder, and updates the stored path. Datasets whose files are not found in the new folder are skipped with a warning." ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Purging Orphaned Datasets\n", + "\n", + "After archiving or deleting selected raw data files, some dataset records in the main database may reference files that no longer exist. Use `purge_orphaned_datasets` to clean up those orphaned records:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# from qcodes.dataset import purge_orphaned_datasets\n", + "\n", + "# Dry run (default) — see what would be removed:\n", + "# result = purge_orphaned_datasets(\"/path/to/main_database.db\")\n", + "# print(f\"{len(result.orphaned_datasets)} orphaned datasets found\")\n", + "\n", + "# Actually remove orphaned records:\n", + "# result = purge_orphaned_datasets(\"/path/to/main_database.db\", dry_run=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Cleaning Up Datasets by Criteria\n", + "\n", + "To free disk space by removing datasets (both raw data files and DB records) based on age, sample name, or file size, use `cleanup_datasets`. Criteria are combined with AND logic — a dataset must match **all** specified criteria to be selected." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# from qcodes.dataset import cleanup_datasets\n", + "\n", + "# Remove datasets older than 30 days (dry run first):\n", + "# result = cleanup_datasets(\"/path/to/db.db\", older_than_days=30)\n", + "# print(f\"{len(result.matching_datasets)} datasets would be removed\")\n", + "\n", + "# Remove datasets for a specific sample:\n", + "# result = cleanup_datasets(\"/path/to/db.db\", sample_name=\"old-sample\", dry_run=False)\n", + "\n", + "# Remove datasets with raw data files larger than 500 MB:\n", + "# result = cleanup_datasets(\"/path/to/db.db\", larger_than_mb=500, dry_run=False)" + ] } ], "metadata": { diff --git a/src/qcodes/dataset/__init__.py b/src/qcodes/dataset/__init__.py index 37359d4b51de..2658afee94a7 100644 --- a/src/qcodes/dataset/__init__.py +++ b/src/qcodes/dataset/__init__.py @@ -3,7 +3,11 @@ and from disk """ -from ._raw_data_storage import update_raw_data_paths +from ._raw_data_storage import ( + cleanup_datasets, + purge_orphaned_datasets, + update_raw_data_paths, +) from .data_set import ( get_guids_by_run_spec, load_by_counter, @@ -85,6 +89,7 @@ "ThreadPoolParamsCaller", "TogetherSweep", "call_params_threaded", + "cleanup_datasets", "connect", "datasaver_builder", "do0d", @@ -119,6 +124,7 @@ "new_experiment", "plot_by_id", "plot_dataset", + "purge_orphaned_datasets", "reset_default_experiment_id", "rundescriber_from_json", "update_raw_data_paths", diff --git a/src/qcodes/dataset/_raw_data_storage.py b/src/qcodes/dataset/_raw_data_storage.py index 58492ebb60db..feaa88cd472d 100644 --- a/src/qcodes/dataset/_raw_data_storage.py +++ b/src/qcodes/dataset/_raw_data_storage.py @@ -13,7 +13,10 @@ from __future__ import annotations import logging +import os import sqlite3 +from dataclasses import dataclass, field +from datetime import UTC, datetime, timedelta from pathlib import Path from typing import TYPE_CHECKING @@ -258,3 +261,357 @@ def update_raw_data_paths( conn.close() log.info("Updated %d raw data paths in %s", len(updated), db_path) return updated + + +# --------------------------------------------------------------------------- +# Dataset management helpers +# --------------------------------------------------------------------------- + + +@dataclass +class DatasetInfo: + """Summary information about a dataset in the main database.""" + + run_id: int + guid: str + experiment_name: str + sample_name: str + run_timestamp: float | None + completed_timestamp: float | None + result_table_name: str + raw_data_db_path: str | None + raw_data_size_bytes: int | None + + +@dataclass +class PurgeResult: + """Result of a purge_orphaned_datasets operation.""" + + total_datasets_with_raw_data: int + orphaned_datasets: list[DatasetInfo] + removed_datasets: list[DatasetInfo] + dry_run: bool + errors: list[tuple[int, Exception]] = field(default_factory=list) + + +@dataclass +class CleanupResult: + """Result of a cleanup_datasets operation.""" + + total_datasets_scanned: int + matching_datasets: list[DatasetInfo] + removed_datasets: list[DatasetInfo] + total_size_freed_bytes: int + dry_run: bool + errors: list[tuple[int, Exception]] = field(default_factory=list) + + +def _get_datasets_with_raw_data( + conn: AtomicConnection, +) -> list[DatasetInfo]: + """Query all datasets that have a raw_data_db_path metadata entry.""" + if not is_column_in_table(conn, "runs", "raw_data_db_path"): + return [] + + sql = """ + SELECT r.run_id, r.guid, e.name, e.sample_name, + r.run_timestamp, r.completed_timestamp, + r.result_table_name, r.raw_data_db_path + FROM runs r + JOIN experiments e ON r.exp_id = e.exp_id + WHERE r.raw_data_db_path IS NOT NULL + """ + cursor = conn.execute(sql) + rows = cursor.fetchall() + + datasets: list[DatasetInfo] = [] + for row in rows: + ( + run_id, + guid, + exp_name, + sample_name, + run_ts, + completed_ts, + table_name, + raw_path, + ) = row + + raw_size: int | None = None + if raw_path and Path(raw_path).is_file(): + raw_size = Path(raw_path).stat().st_size + + datasets.append( + DatasetInfo( + run_id=run_id, + guid=guid, + experiment_name=exp_name, + sample_name=sample_name, + run_timestamp=run_ts, + completed_timestamp=completed_ts, + result_table_name=table_name, + raw_data_db_path=raw_path, + raw_data_size_bytes=raw_size, + ) + ) + return datasets + + +def _remove_dataset_from_db(conn: AtomicConnection, ds_info: DatasetInfo) -> None: + """Remove a single dataset's records from the main database. + + Deletes the run row, associated layouts, dependencies, and drops + the results table (if it exists in the main DB). + """ + run_id = ds_info.run_id + table_name = ds_info.result_table_name + + with atomic(conn) as aconn: + # Get layout_ids for this run (needed for dependencies) + cursor = aconn.execute( + "SELECT layout_id FROM layouts WHERE run_id = ?", (run_id,) + ) + layout_ids = [row[0] for row in cursor.fetchall()] + + # Delete dependencies referencing these layouts + if layout_ids: + placeholders = ",".join("?" * len(layout_ids)) + aconn.execute( + f"DELETE FROM dependencies WHERE dependent IN ({placeholders})" + f" OR independent IN ({placeholders})", + (*layout_ids, *layout_ids), + ) + + # Delete layouts + aconn.execute("DELETE FROM layouts WHERE run_id = ?", (run_id,)) + + # Drop the results table in the main DB (if it exists) + aconn.execute(f'DROP TABLE IF EXISTS "{table_name}"') + + # Delete the run row + aconn.execute("DELETE FROM runs WHERE run_id = ?", (run_id,)) + + +def purge_orphaned_datasets( + db_path: str | Path, + *, + dry_run: bool = True, +) -> PurgeResult: + """Find and optionally remove dataset records whose raw data files are missing. + + When using split raw data storage, users may archive and delete + individual per-dataset SQLite files. This function identifies + datasets in the main database that reference raw data files which + no longer exist on disk, and optionally removes those dataset + records from the main database. + + Args: + db_path: Path to the main QCoDeS database file. + dry_run: If *True* (default), only report which datasets would + be removed without making any changes. Set to *False* to + actually delete the orphaned dataset records. + + Returns: + A :class:`PurgeResult` with the list of orphaned datasets and, + if *dry_run* is False, the list of datasets that were removed. + + Raises: + FileNotFoundError: If the main database file does not exist. + + """ + db_path = Path(db_path) + if not db_path.is_file(): + raise FileNotFoundError(f"Database file not found: {db_path}") + + conn = connect(str(db_path)) + all_datasets = _get_datasets_with_raw_data(conn) + + # Find orphaned datasets: raw_data_db_path is set but the file is missing + orphaned = [ + ds + for ds in all_datasets + if ds.raw_data_db_path and not Path(ds.raw_data_db_path).is_file() + ] + + log.info( + "Found %d datasets with raw data references, %d orphaned (file missing).", + len(all_datasets), + len(orphaned), + ) + + removed: list[DatasetInfo] = [] + errors: list[tuple[int, Exception]] = [] + + if not dry_run and orphaned: + for ds_info in orphaned: + try: + _remove_dataset_from_db(conn, ds_info) + removed.append(ds_info) + log.info( + "Removed orphaned dataset run_id=%d (guid=%s) from database.", + ds_info.run_id, + ds_info.guid, + ) + except Exception as exc: + log.error("Failed to remove run_id=%d: %s", ds_info.run_id, exc) + errors.append((ds_info.run_id, exc)) + + conn.close() + + result = PurgeResult( + total_datasets_with_raw_data=len(all_datasets), + orphaned_datasets=orphaned, + removed_datasets=removed, + dry_run=dry_run, + errors=errors, + ) + + if dry_run: + log.info("Dry run: %d orphaned datasets would be removed.", len(orphaned)) + else: + log.info("Removed %d orphaned datasets.", len(removed)) + + return result + + +def cleanup_datasets( + db_path: str | Path, + *, + older_than_days: int | None = None, + sample_name: str | None = None, + larger_than_mb: float | None = None, + dry_run: bool = True, +) -> CleanupResult: + """Remove datasets (DB records and raw data files) matching given criteria. + + This function helps manage disk space by removing datasets that match + one or more of the specified criteria. It removes both the raw data + SQLite file on disk and the corresponding records in the main database + (run metadata, layouts, dependencies, results table schema). + + Criteria are combined with AND logic: a dataset must match **all** + specified criteria to be selected for removal. Specify at least one + criterion. + + Args: + db_path: Path to the main QCoDeS database file. + older_than_days: Remove datasets whose *completed_timestamp* + (or *run_timestamp* if not completed) is older than this + many days ago. + sample_name: Remove datasets belonging to experiments with this + exact sample name. + larger_than_mb: Remove datasets whose raw data file is larger + than this many megabytes. + dry_run: If *True* (default), only report which datasets would + be removed without making any changes. Set to *False* to + actually delete datasets and their raw data files. + + Returns: + A :class:`CleanupResult` with details of the operation. + + Raises: + FileNotFoundError: If the main database file does not exist. + ValueError: If no criteria are specified. + + """ + db_path = Path(db_path) + if not db_path.is_file(): + raise FileNotFoundError(f"Database file not found: {db_path}") + + if older_than_days is None and sample_name is None and larger_than_mb is None: + raise ValueError("At least one cleanup criterion must be specified.") + + conn = connect(str(db_path)) + all_datasets = _get_datasets_with_raw_data(conn) + + # Apply filters (AND logic) + matching: list[DatasetInfo] = [] + cutoff_ts: float | None = None + if older_than_days is not None: + cutoff_dt = datetime.now(tz=UTC) - timedelta(days=older_than_days) + cutoff_ts = cutoff_dt.timestamp() + + size_threshold_bytes: int | None = None + if larger_than_mb is not None: + size_threshold_bytes = int(larger_than_mb * 1024 * 1024) + + for ds in all_datasets: + # Age filter + if cutoff_ts is not None: + ts = ds.completed_timestamp or ds.run_timestamp + if ts is None or ts >= cutoff_ts: + continue + + # Sample name filter + if sample_name is not None: + if ds.sample_name != sample_name: + continue + + # Size filter + if size_threshold_bytes is not None: + if ( + ds.raw_data_size_bytes is None + or ds.raw_data_size_bytes <= size_threshold_bytes + ): + continue + + matching.append(ds) + + log.info( + "Found %d datasets with raw data, %d match cleanup criteria.", + len(all_datasets), + len(matching), + ) + + removed: list[DatasetInfo] = [] + errors: list[tuple[int, Exception]] = [] + total_freed: int = 0 + + if not dry_run and matching: + for ds_info in matching: + try: + # Delete the raw data file from disk + if ds_info.raw_data_db_path: + raw_path = Path(ds_info.raw_data_db_path) + if raw_path.is_file(): + file_size = raw_path.stat().st_size + os.remove(raw_path) + total_freed += file_size + log.info("Deleted raw data file: %s", raw_path) + + # Remove dataset records from the main DB + _remove_dataset_from_db(conn, ds_info) + removed.append(ds_info) + log.info( + "Removed dataset run_id=%d (guid=%s) from database.", + ds_info.run_id, + ds_info.guid, + ) + except Exception as exc: + log.error("Failed to remove run_id=%d: %s", ds_info.run_id, exc) + errors.append((ds_info.run_id, exc)) + + conn.close() + + result = CleanupResult( + total_datasets_scanned=len(all_datasets), + matching_datasets=matching, + removed_datasets=removed, + total_size_freed_bytes=total_freed, + dry_run=dry_run, + errors=errors, + ) + + if dry_run: + total_size = sum( + ds.raw_data_size_bytes for ds in matching if ds.raw_data_size_bytes + ) + log.info( + "Dry run: %d datasets (%d bytes) would be removed.", + len(matching), + total_size, + ) + else: + log.info("Removed %d datasets, freed %d bytes.", len(removed), total_freed) + + return result diff --git a/tests/dataset/test_raw_data_storage.py b/tests/dataset/test_raw_data_storage.py index 03ca6ad047a6..2536b3154718 100644 --- a/tests/dataset/test_raw_data_storage.py +++ b/tests/dataset/test_raw_data_storage.py @@ -11,6 +11,7 @@ import gc import shutil import sqlite3 +import time from pathlib import Path from typing import TYPE_CHECKING @@ -20,16 +21,18 @@ import qcodes as qc from qcodes.dataset import new_data_set, new_experiment from qcodes.dataset._raw_data_storage import ( + cleanup_datasets, connect_to_raw_data_db, create_raw_data_db, get_raw_data_db_path, get_raw_data_folder, is_raw_data_storage_enabled, + purge_orphaned_datasets, update_raw_data_paths, ) from qcodes.dataset.data_set import DataSet, load_by_id from qcodes.dataset.descriptions.dependencies import InterDependencies_ -from qcodes.dataset.sqlite.database import initialise_database +from qcodes.dataset.sqlite.database import connect, initialise_database from qcodes.parameters import ParamSpecBase if TYPE_CHECKING: @@ -423,3 +426,254 @@ def test_update_nonexistent_folder_raises(self, tmp_path: Path) -> None: db_path.touch() with pytest.raises(FileNotFoundError, match="New raw data folder not found"): update_raw_data_paths(db_path, tmp_path / "no_such_folder") + + +# --------------------------------------------------------------------------- +# purge_orphaned_datasets and cleanup_datasets tests +# --------------------------------------------------------------------------- + + +class TestPurgeOrphanedDatasets: + """Tests for purge_orphaned_datasets.""" + + @staticmethod + def _close_ds(ds: DataSet) -> None: + if ds._raw_data_conn is not None: + ds._raw_data_conn.close() + ds.conn.close() + + @pytest.mark.usefixtures("_raw_data_db") + def test_dry_run_reports_orphans(self, tmp_path: Path) -> None: + """Dry run should report orphaned datasets without removing them.""" + + new_experiment("test-exp", sample_name="test-sample") + ds = new_data_set("test-purge") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + ds.add_results([{"x": 1.0, "y": 2.0}]) + ds.mark_completed() + + raw_path = Path(ds.metadata["raw_data_db_path"]) + db_path = ds.path_to_db + assert db_path is not None + self._close_ds(ds) + + # Delete the raw data file to create an orphan + raw_path.unlink() + + result = purge_orphaned_datasets(db_path, dry_run=True) + assert result.total_datasets_with_raw_data == 1 + assert len(result.orphaned_datasets) == 1 + assert len(result.removed_datasets) == 0 + assert result.dry_run is True + + # Verify the dataset is still in the DB (direct query since load_by_id + # will raise FileNotFoundError due to missing raw data file) + + conn = connect(db_path) + cursor = conn.execute("SELECT COUNT(*) FROM runs WHERE run_id = 1") + assert cursor.fetchone()[0] == 1 + conn.close() + + @pytest.mark.usefixtures("_raw_data_db") + def test_removes_orphans_when_not_dry_run(self, tmp_path: Path) -> None: + """With dry_run=False, orphaned datasets should be removed.""" + + new_experiment("test-exp", sample_name="test-sample") + ds = new_data_set("test-purge-remove") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + ds.add_results([{"x": 1.0, "y": 2.0}]) + ds.mark_completed() + + raw_path = Path(ds.metadata["raw_data_db_path"]) + db_path = ds.path_to_db + assert db_path is not None + run_id = ds.run_id + self._close_ds(ds) + + # Delete the raw data file + raw_path.unlink() + + result = purge_orphaned_datasets(db_path, dry_run=False) + assert len(result.orphaned_datasets) == 1 + assert len(result.removed_datasets) == 1 + assert result.removed_datasets[0].run_id == run_id + assert result.dry_run is False + + # Verify the dataset no longer exists in the DB + + conn = connect(db_path) + cursor = conn.execute("SELECT COUNT(*) FROM runs WHERE run_id = ?", (run_id,)) + assert cursor.fetchone()[0] == 0 + conn.close() + + @pytest.mark.usefixtures("_raw_data_db") + def test_skips_datasets_with_existing_files(self, tmp_path: Path) -> None: + """Datasets whose raw data files exist should not be purged.""" + + new_experiment("test-exp", sample_name="test-sample") + ds = new_data_set("test-no-purge") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + ds.add_results([{"x": 1.0, "y": 2.0}]) + ds.mark_completed() + + db_path = ds.path_to_db + assert db_path is not None + self._close_ds(ds) + + # Don't delete the file — should not be orphaned + result = purge_orphaned_datasets(db_path, dry_run=False) + assert len(result.orphaned_datasets) == 0 + assert len(result.removed_datasets) == 0 + + def test_nonexistent_db_raises(self, tmp_path: Path) -> None: + """Should raise if the DB file doesn't exist.""" + + with pytest.raises(FileNotFoundError, match="Database file not found"): + purge_orphaned_datasets(tmp_path / "nonexistent.db") + + +class TestCleanupDatasets: + """Tests for cleanup_datasets.""" + + @staticmethod + def _close_ds(ds: DataSet) -> None: + if ds._raw_data_conn is not None: + ds._raw_data_conn.close() + ds.conn.close() + + @pytest.mark.usefixtures("_raw_data_db") + def test_cleanup_by_sample_name(self, tmp_path: Path) -> None: + """Should remove datasets matching exact sample name.""" + + new_experiment("exp1", sample_name="sample-A") + ds1 = new_data_set("ds1") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds1.set_interdependencies(idps) + ds1.mark_started() + ds1.add_results([{"x": 1.0, "y": 2.0}]) + ds1.mark_completed() + raw_path1 = Path(ds1.metadata["raw_data_db_path"]) + db_path = ds1.path_to_db + assert db_path is not None + self._close_ds(ds1) + + # Create another dataset with a different sample + new_experiment("exp2", sample_name="sample-B") + ds2 = new_data_set("ds2") + ds2.set_interdependencies(idps) + ds2.mark_started() + ds2.add_results([{"x": 3.0, "y": 4.0}]) + ds2.mark_completed() + raw_path2 = Path(ds2.metadata["raw_data_db_path"]) + self._close_ds(ds2) + + # Dry run first + result = cleanup_datasets(db_path, sample_name="sample-A", dry_run=True) + assert len(result.matching_datasets) == 1 + assert len(result.removed_datasets) == 0 + assert raw_path1.is_file() + + # Actual removal + result = cleanup_datasets(db_path, sample_name="sample-A", dry_run=False) + assert len(result.matching_datasets) == 1 + assert len(result.removed_datasets) == 1 + assert not raw_path1.is_file() + assert raw_path2.is_file() # other sample untouched + + @pytest.mark.usefixtures("_raw_data_db") + def test_cleanup_by_size(self, tmp_path: Path) -> None: + """Should remove datasets larger than the specified threshold.""" + + new_experiment("exp1", sample_name="test-sample") + ds = new_data_set("ds-large") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + # Add enough data so the file has some size + for i in range(100): + ds.add_results([{"x": float(i), "y": float(i * 2)}]) + ds.mark_completed() + raw_path = Path(ds.metadata["raw_data_db_path"]) + db_path = ds.path_to_db + assert db_path is not None + self._close_ds(ds) + + file_size_mb = raw_path.stat().st_size / (1024 * 1024) + + # Set threshold below actual size — should match + result = cleanup_datasets( + db_path, larger_than_mb=file_size_mb * 0.5, dry_run=True + ) + assert len(result.matching_datasets) == 1 + + # Set threshold above actual size — should not match + result = cleanup_datasets( + db_path, larger_than_mb=file_size_mb * 2, dry_run=True + ) + assert len(result.matching_datasets) == 0 + + @pytest.mark.usefixtures("_raw_data_db") + def test_cleanup_by_age(self, tmp_path: Path) -> None: + """Should remove datasets older than the specified number of days.""" + + new_experiment("exp1", sample_name="test-sample") + ds = new_data_set("ds-old") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + ds.add_results([{"x": 1.0, "y": 2.0}]) + ds.mark_completed() + db_path = ds.path_to_db + assert db_path is not None + run_id = ds.run_id + self._close_ds(ds) + + # Set the completed_timestamp to 10 days ago + old_ts = time.time() - (10 * 86400) + conn = connect(db_path) + conn.execute( + "UPDATE runs SET completed_timestamp = ? WHERE run_id = ?", + (old_ts, run_id), + ) + conn.commit() + conn.close() + + # older_than_days=5 should match (ds is 10 days old) + result = cleanup_datasets(db_path, older_than_days=5, dry_run=True) + assert len(result.matching_datasets) == 1 + + # older_than_days=15 should NOT match (ds is only 10 days old) + result = cleanup_datasets(db_path, older_than_days=15, dry_run=True) + assert len(result.matching_datasets) == 0 + + def test_no_criteria_raises(self, tmp_path: Path) -> None: + """Should raise if no criteria are specified.""" + + db_path = tmp_path / "test.db" + db_path.touch() + with pytest.raises(ValueError, match="At least one cleanup criterion"): + cleanup_datasets(db_path) + + def test_nonexistent_db_raises(self, tmp_path: Path) -> None: + """Should raise if the DB file doesn't exist.""" + + with pytest.raises(FileNotFoundError, match="Database file not found"): + cleanup_datasets(tmp_path / "nonexistent.db", older_than_days=1) From 0f21ee934c3419b7f2f2784f6d5161a34800c75c Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Thu, 16 Jul 2026 16:09:02 -0700 Subject: [PATCH 08/12] Add towncrier news fragment for split raw data storage Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/changes/newsfragments/8219.new | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 docs/changes/newsfragments/8219.new diff --git a/docs/changes/newsfragments/8219.new b/docs/changes/newsfragments/8219.new new file mode 100644 index 000000000000..c4c8dbea84b0 --- /dev/null +++ b/docs/changes/newsfragments/8219.new @@ -0,0 +1,14 @@ +Added **split raw data storage** for the DataSet: an opt-in feature +(``dataset.raw_data_to_separate_db``) that writes raw measurement data +into individual per-dataset SQLite files while keeping all metadata in +the main database. This prevents the main DB file from growing +excessively large. + +Management helpers included: + +- ``update_raw_data_paths()``: update stored paths after raw data files + have been moved to a new folder. +- ``purge_orphaned_datasets()``: find and remove dataset records whose + raw data files no longer exist on disk (``dry_run=True`` by default). +- ``cleanup_datasets()``: remove datasets (DB records and raw data files) + by age, sample name, or file size (``dry_run=True`` by default). From c76baadf757e771582a17126c2d61048badda875 Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Thu, 16 Jul 2026 16:31:10 -0700 Subject: [PATCH 09/12] Address PR review comments: move queries to queries.py, add prints, simplify - Move _get_datasets_with_raw_data and _remove_dataset_from_db to src/qcodes/dataset/sqlite/queries.py as public functions - Add print() statements alongside logging for user visibility - Use 'with closing(conn):' pattern for connection management - Simplify orphan detection (rely on raw_data_size_bytes is None) - Simplify cleanup_datasets docstring - Simplify introduction.rst to reference Database notebook - Update news fragment with config option name and DB context Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/changes/newsfragments/8219.new | 10 +- docs/dataset/introduction.rst | 35 +-- src/qcodes/dataset/_raw_data_storage.py | 307 ++++++++++-------------- src/qcodes/dataset/sqlite/queries.py | 69 ++++++ 4 files changed, 212 insertions(+), 209 deletions(-) diff --git a/docs/changes/newsfragments/8219.new b/docs/changes/newsfragments/8219.new index c4c8dbea84b0..1743811bb763 100644 --- a/docs/changes/newsfragments/8219.new +++ b/docs/changes/newsfragments/8219.new @@ -1,8 +1,10 @@ Added **split raw data storage** for the DataSet: an opt-in feature -(``dataset.raw_data_to_separate_db``) that writes raw measurement data -into individual per-dataset SQLite files while keeping all metadata in -the main database. This prevents the main DB file from growing -excessively large. +enabled via the ``dataset.raw_data_to_separate_db`` config option that +writes raw measurement data into individual per-dataset SQLite files +while keeping all metadata in the main database. This prevents the +main DB file from growing excessively large and allows users to have a +single main SQLite QCoDeS database on their PC as opposed to many that +grow infinitely. Management helpers included: diff --git a/docs/dataset/introduction.rst b/docs/dataset/introduction.rst index 4e6fb0b6d042..d3769431a33b 100644 --- a/docs/dataset/introduction.rst +++ b/docs/dataset/introduction.rst @@ -115,37 +115,14 @@ If the per-dataset raw data files are moved to a different folder (e.g. during d This scans all datasets with a ``raw_data_db_path`` metadata entry, checks whether the corresponding ``.db`` file exists in the new folder, and updates the stored path accordingly. -Purging Orphaned Datasets -------------------------- +Managing Datasets +----------------- -After archiving or deleting selected raw data files, some dataset records in the main database may reference files that no longer exist on disk. Use :func:`~qcodes.dataset.purge_orphaned_datasets` to identify and remove those orphaned records:: +QCoDeS provides helpers for managing per-dataset raw data files over time: - from qcodes.dataset import purge_orphaned_datasets +- :func:`~qcodes.dataset.purge_orphaned_datasets` — remove dataset records whose raw data files no longer exist on disk. +- :func:`~qcodes.dataset.cleanup_datasets` — remove datasets (DB records and raw data files) by age, sample name, or file size. - # First, see what would be removed (dry run — the default) - result = purge_orphaned_datasets("/path/to/main_database.db") - print(f"{len(result.orphaned_datasets)} orphaned datasets found") - - # Then actually remove them - result = purge_orphaned_datasets("/path/to/main_database.db", dry_run=False) - -Cleaning Up Datasets by Criteria ---------------------------------- - -To free disk space by removing datasets (both the raw data files **and** their records in the main database) based on age, sample name, or file size, use :func:`~qcodes.dataset.cleanup_datasets`:: - - from qcodes.dataset import cleanup_datasets - - # Remove datasets older than 30 days (dry run first) - result = cleanup_datasets("/path/to/db.db", older_than_days=30) - print(f"{len(result.matching_datasets)} datasets would be removed") - - # Remove datasets for a specific sample - result = cleanup_datasets("/path/to/db.db", sample_name="old-sample", dry_run=False) - - # Remove datasets with raw data larger than 500 MB - result = cleanup_datasets("/path/to/db.db", larger_than_mb=500, dry_run=False) - -Criteria are combined with AND logic — a dataset must match **all** specified criteria to be selected for removal. +For usage examples, see the :doc:`Database notebook <../examples/DataSet/Database>`. For more details on database management, see the :doc:`Database notebook <../examples/DataSet/Database>`. diff --git a/src/qcodes/dataset/_raw_data_storage.py b/src/qcodes/dataset/_raw_data_storage.py index feaa88cd472d..4a9f6c880972 100644 --- a/src/qcodes/dataset/_raw_data_storage.py +++ b/src/qcodes/dataset/_raw_data_storage.py @@ -15,6 +15,7 @@ import logging import os import sqlite3 +from contextlib import closing from dataclasses import dataclass, field from datetime import UTC, datetime, timedelta from pathlib import Path @@ -34,6 +35,10 @@ _convert_numeric, connect, ) +from qcodes.dataset.sqlite.queries import ( + get_datasets_with_raw_data_path, + remove_dataset_from_db, +) from qcodes.dataset.sqlite.query_helpers import is_column_in_table from qcodes.utils.types import complex_types, numpy_floats, numpy_ints @@ -306,37 +311,23 @@ class CleanupResult: errors: list[tuple[int, Exception]] = field(default_factory=list) -def _get_datasets_with_raw_data( +def _build_dataset_info_list( conn: AtomicConnection, ) -> list[DatasetInfo]: - """Query all datasets that have a raw_data_db_path metadata entry.""" - if not is_column_in_table(conn, "runs", "raw_data_db_path"): - return [] - - sql = """ - SELECT r.run_id, r.guid, e.name, e.sample_name, - r.run_timestamp, r.completed_timestamp, - r.result_table_name, r.raw_data_db_path - FROM runs r - JOIN experiments e ON r.exp_id = e.exp_id - WHERE r.raw_data_db_path IS NOT NULL - """ - cursor = conn.execute(sql) - rows = cursor.fetchall() + """Query datasets with raw data paths and enrich with file size info.""" + rows = get_datasets_with_raw_data_path(conn) datasets: list[DatasetInfo] = [] - for row in rows: - ( - run_id, - guid, - exp_name, - sample_name, - run_ts, - completed_ts, - table_name, - raw_path, - ) = row - + for ( + run_id, + guid, + exp_name, + sample_name, + run_ts, + completed_ts, + table_name, + raw_path, + ) in rows: raw_size: int | None = None if raw_path and Path(raw_path).is_file(): raw_size = Path(raw_path).stat().st_size @@ -357,41 +348,6 @@ def _get_datasets_with_raw_data( return datasets -def _remove_dataset_from_db(conn: AtomicConnection, ds_info: DatasetInfo) -> None: - """Remove a single dataset's records from the main database. - - Deletes the run row, associated layouts, dependencies, and drops - the results table (if it exists in the main DB). - """ - run_id = ds_info.run_id - table_name = ds_info.result_table_name - - with atomic(conn) as aconn: - # Get layout_ids for this run (needed for dependencies) - cursor = aconn.execute( - "SELECT layout_id FROM layouts WHERE run_id = ?", (run_id,) - ) - layout_ids = [row[0] for row in cursor.fetchall()] - - # Delete dependencies referencing these layouts - if layout_ids: - placeholders = ",".join("?" * len(layout_ids)) - aconn.execute( - f"DELETE FROM dependencies WHERE dependent IN ({placeholders})" - f" OR independent IN ({placeholders})", - (*layout_ids, *layout_ids), - ) - - # Delete layouts - aconn.execute("DELETE FROM layouts WHERE run_id = ?", (run_id,)) - - # Drop the results table in the main DB (if it exists) - aconn.execute(f'DROP TABLE IF EXISTS "{table_name}"') - - # Delete the run row - aconn.execute("DELETE FROM runs WHERE run_id = ?", (run_id,)) - - def purge_orphaned_datasets( db_path: str | Path, *, @@ -423,40 +379,38 @@ def purge_orphaned_datasets( if not db_path.is_file(): raise FileNotFoundError(f"Database file not found: {db_path}") - conn = connect(str(db_path)) - all_datasets = _get_datasets_with_raw_data(conn) - - # Find orphaned datasets: raw_data_db_path is set but the file is missing - orphaned = [ - ds - for ds in all_datasets - if ds.raw_data_db_path and not Path(ds.raw_data_db_path).is_file() - ] - - log.info( - "Found %d datasets with raw data references, %d orphaned (file missing).", - len(all_datasets), - len(orphaned), - ) + with closing(connect(str(db_path))) as conn: + all_datasets = _build_dataset_info_list(conn) - removed: list[DatasetInfo] = [] - errors: list[tuple[int, Exception]] = [] - - if not dry_run and orphaned: - for ds_info in orphaned: - try: - _remove_dataset_from_db(conn, ds_info) - removed.append(ds_info) - log.info( - "Removed orphaned dataset run_id=%d (guid=%s) from database.", - ds_info.run_id, - ds_info.guid, - ) - except Exception as exc: - log.error("Failed to remove run_id=%d: %s", ds_info.run_id, exc) - errors.append((ds_info.run_id, exc)) + # Orphaned = raw_data_size_bytes is None (file not found on disk) + orphaned = [ds for ds in all_datasets if ds.raw_data_size_bytes is None] - conn.close() + msg = ( + f"Found {len(all_datasets)} datasets with raw data references in {db_path}, " + f"{len(orphaned)} orphaned (file missing)." + ) + log.info(msg) + print(msg) + + removed: list[DatasetInfo] = [] + errors: list[tuple[int, Exception]] = [] + + if not dry_run and orphaned: + for ds_info in orphaned: + try: + remove_dataset_from_db( + conn, ds_info.run_id, ds_info.result_table_name + ) + removed.append(ds_info) + log.info( + "Removed orphaned dataset run_id=%d (guid=%s) from %s.", + ds_info.run_id, + ds_info.guid, + db_path, + ) + except Exception as exc: + log.error("Failed to remove run_id=%d: %s", ds_info.run_id, exc) + errors.append((ds_info.run_id, exc)) result = PurgeResult( total_datasets_with_raw_data=len(all_datasets), @@ -467,9 +421,11 @@ def purge_orphaned_datasets( ) if dry_run: - log.info("Dry run: %d orphaned datasets would be removed.", len(orphaned)) + msg = f"Dry run: {len(orphaned)} orphaned datasets would be removed from {db_path}." else: - log.info("Removed %d orphaned datasets.", len(removed)) + msg = f"Removed {len(removed)} orphaned datasets from {db_path}." + log.info(msg) + print(msg) return result @@ -482,12 +438,11 @@ def cleanup_datasets( larger_than_mb: float | None = None, dry_run: bool = True, ) -> CleanupResult: - """Remove datasets (DB records and raw data files) matching given criteria. + """Remove datasets and their raw data files matching given criteria. This function helps manage disk space by removing datasets that match one or more of the specified criteria. It removes both the raw data - SQLite file on disk and the corresponding records in the main database - (run metadata, layouts, dependencies, results table schema). + SQLite file on disk and the corresponding records in the main database. Criteria are combined with AND logic: a dataset must match **all** specified criteria to be selected for removal. Specify at least one @@ -521,77 +476,79 @@ def cleanup_datasets( if older_than_days is None and sample_name is None and larger_than_mb is None: raise ValueError("At least one cleanup criterion must be specified.") - conn = connect(str(db_path)) - all_datasets = _get_datasets_with_raw_data(conn) - - # Apply filters (AND logic) - matching: list[DatasetInfo] = [] - cutoff_ts: float | None = None - if older_than_days is not None: - cutoff_dt = datetime.now(tz=UTC) - timedelta(days=older_than_days) - cutoff_ts = cutoff_dt.timestamp() - - size_threshold_bytes: int | None = None - if larger_than_mb is not None: - size_threshold_bytes = int(larger_than_mb * 1024 * 1024) - - for ds in all_datasets: - # Age filter - if cutoff_ts is not None: - ts = ds.completed_timestamp or ds.run_timestamp - if ts is None or ts >= cutoff_ts: - continue - - # Sample name filter - if sample_name is not None: - if ds.sample_name != sample_name: - continue - - # Size filter - if size_threshold_bytes is not None: - if ( - ds.raw_data_size_bytes is None - or ds.raw_data_size_bytes <= size_threshold_bytes - ): - continue - - matching.append(ds) - - log.info( - "Found %d datasets with raw data, %d match cleanup criteria.", - len(all_datasets), - len(matching), - ) - - removed: list[DatasetInfo] = [] - errors: list[tuple[int, Exception]] = [] - total_freed: int = 0 - - if not dry_run and matching: - for ds_info in matching: - try: - # Delete the raw data file from disk - if ds_info.raw_data_db_path: - raw_path = Path(ds_info.raw_data_db_path) - if raw_path.is_file(): - file_size = raw_path.stat().st_size - os.remove(raw_path) - total_freed += file_size - log.info("Deleted raw data file: %s", raw_path) - - # Remove dataset records from the main DB - _remove_dataset_from_db(conn, ds_info) - removed.append(ds_info) - log.info( - "Removed dataset run_id=%d (guid=%s) from database.", - ds_info.run_id, - ds_info.guid, - ) - except Exception as exc: - log.error("Failed to remove run_id=%d: %s", ds_info.run_id, exc) - errors.append((ds_info.run_id, exc)) - - conn.close() + with closing(connect(str(db_path))) as conn: + all_datasets = _build_dataset_info_list(conn) + + # Apply filters (AND logic) + matching: list[DatasetInfo] = [] + cutoff_ts: float | None = None + if older_than_days is not None: + cutoff_dt = datetime.now(tz=UTC) - timedelta(days=older_than_days) + cutoff_ts = cutoff_dt.timestamp() + + size_threshold_bytes: int | None = None + if larger_than_mb is not None: + size_threshold_bytes = int(larger_than_mb * 1024 * 1024) + + for ds in all_datasets: + # Age filter + if cutoff_ts is not None: + ts = ds.completed_timestamp or ds.run_timestamp + if ts is None or ts >= cutoff_ts: + continue + + # Sample name filter + if sample_name is not None: + if ds.sample_name != sample_name: + continue + + # Size filter + if size_threshold_bytes is not None: + if ( + ds.raw_data_size_bytes is None + or ds.raw_data_size_bytes <= size_threshold_bytes + ): + continue + + matching.append(ds) + + msg = ( + f"Found {len(all_datasets)} datasets with raw data in {db_path}, " + f"{len(matching)} match cleanup criteria." + ) + log.info(msg) + print(msg) + + removed: list[DatasetInfo] = [] + errors: list[tuple[int, Exception]] = [] + total_freed: int = 0 + + if not dry_run and matching: + for ds_info in matching: + try: + # Delete the raw data file from disk + if ds_info.raw_data_db_path: + raw_path = Path(ds_info.raw_data_db_path) + if raw_path.is_file(): + file_size = raw_path.stat().st_size + os.remove(raw_path) + total_freed += file_size + log.info("Deleted raw data file: %s", raw_path) + + # Remove dataset records from the main DB + remove_dataset_from_db( + conn, ds_info.run_id, ds_info.result_table_name + ) + removed.append(ds_info) + log.info( + "Removed dataset run_id=%d (guid=%s) from %s.", + ds_info.run_id, + ds_info.guid, + db_path, + ) + except Exception as exc: + log.error("Failed to remove run_id=%d: %s", ds_info.run_id, exc) + errors.append((ds_info.run_id, exc)) result = CleanupResult( total_datasets_scanned=len(all_datasets), @@ -606,12 +563,10 @@ def cleanup_datasets( total_size = sum( ds.raw_data_size_bytes for ds in matching if ds.raw_data_size_bytes ) - log.info( - "Dry run: %d datasets (%d bytes) would be removed.", - len(matching), - total_size, - ) + msg = f"Dry run: {len(matching)} datasets ({total_size} bytes) would be removed from {db_path}." else: - log.info("Removed %d datasets, freed %d bytes.", len(removed), total_freed) + msg = f"Removed {len(removed)} datasets from {db_path}, freed {total_freed} bytes." + log.info(msg) + print(msg) return result diff --git a/src/qcodes/dataset/sqlite/queries.py b/src/qcodes/dataset/sqlite/queries.py index 4825f99f8bcd..a2f85f464cc2 100644 --- a/src/qcodes/dataset/sqlite/queries.py +++ b/src/qcodes/dataset/sqlite/queries.py @@ -2307,3 +2307,72 @@ def _get_result_table_name_by_guid(conn: AtomicConnection, guid: str) -> str: sql = "SELECT result_table_name FROM runs WHERE guid=?" formatted_name = one(transaction(conn, sql, guid), "result_table_name") return formatted_name + + +def get_datasets_with_raw_data_path( + conn: AtomicConnection, +) -> list[tuple[int, str, str, str, float | None, float | None, str, str]]: + """Get all datasets that have a raw_data_db_path metadata column set. + + Returns: + A list of tuples: + ``(run_id, guid, experiment_name, sample_name, run_timestamp, + completed_timestamp, result_table_name, raw_data_db_path)``. + Returns an empty list if the column does not exist. + + """ + if not is_column_in_table(conn, "runs", "raw_data_db_path"): + return [] + + sql = """ + SELECT r.run_id, r.guid, e.name, e.sample_name, + r.run_timestamp, r.completed_timestamp, + r.result_table_name, r.raw_data_db_path + FROM runs r + JOIN experiments e ON r.exp_id = e.exp_id + WHERE r.raw_data_db_path IS NOT NULL + """ + cursor = atomic_transaction(conn, sql) + return cursor.fetchall() + + +def remove_dataset_from_db( + conn: AtomicConnection, run_id: int, result_table_name: str +) -> None: + """Remove a single dataset's records from the database. + + Deletes the run row, associated layouts and dependencies, and drops + the results table (if it exists). + + Args: + conn: Connection to the database. + run_id: The run_id of the dataset to remove. + result_table_name: Name of the dataset's results table. + + """ + with atomic(conn) as aconn: + # Get layout_ids for this run (needed for dependencies) + cursor = transaction( + aconn, "SELECT layout_id FROM layouts WHERE run_id = ?", run_id + ) + layout_ids = [row[0] for row in cursor.fetchall()] + + # Delete dependencies referencing these layouts + if layout_ids: + placeholders = ",".join("?" * len(layout_ids)) + transaction( + aconn, + f"DELETE FROM dependencies WHERE dependent IN ({placeholders})" + f" OR independent IN ({placeholders})", + *layout_ids, + *layout_ids, + ) + + # Delete layouts + transaction(aconn, "DELETE FROM layouts WHERE run_id = ?", run_id) + + # Drop the results table in the DB (if it exists) + transaction(aconn, f'DROP TABLE IF EXISTS "{result_table_name}"') + + # Delete the run row + transaction(aconn, "DELETE FROM runs WHERE run_id = ?", run_id) From 84e119d4eb6ac5130114799fb56b5c39d1961884 Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Thu, 16 Jul 2026 16:41:32 -0700 Subject: [PATCH 10/12] Use log.debug for per-dataset reports, Path.unlink() for file removal - Change per-dataset log.info to log.debug in purge/cleanup/update_paths - Replace os.remove with Path.unlink() for robustness - Remove unused os import Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/qcodes/dataset/_raw_data_storage.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/src/qcodes/dataset/_raw_data_storage.py b/src/qcodes/dataset/_raw_data_storage.py index 4a9f6c880972..915b4372f57d 100644 --- a/src/qcodes/dataset/_raw_data_storage.py +++ b/src/qcodes/dataset/_raw_data_storage.py @@ -13,7 +13,6 @@ from __future__ import annotations import logging -import os import sqlite3 from contextlib import closing from dataclasses import dataclass, field @@ -256,7 +255,7 @@ def update_raw_data_paths( (new_path_str, run_id), ) updated.append((run_id, old_path_str, new_path_str)) - log.info( + log.debug( "Run %d: updated raw_data_db_path from %s to %s", run_id, old_path_str, @@ -402,7 +401,7 @@ def purge_orphaned_datasets( conn, ds_info.run_id, ds_info.result_table_name ) removed.append(ds_info) - log.info( + log.debug( "Removed orphaned dataset run_id=%d (guid=%s) from %s.", ds_info.run_id, ds_info.guid, @@ -531,16 +530,16 @@ def cleanup_datasets( raw_path = Path(ds_info.raw_data_db_path) if raw_path.is_file(): file_size = raw_path.stat().st_size - os.remove(raw_path) + raw_path.unlink() total_freed += file_size - log.info("Deleted raw data file: %s", raw_path) + log.debug("Deleted raw data file: %s", raw_path) # Remove dataset records from the main DB remove_dataset_from_db( conn, ds_info.run_id, ds_info.result_table_name ) removed.append(ds_info) - log.info( + log.debug( "Removed dataset run_id=%d (guid=%s) from %s.", ds_info.run_id, ds_info.guid, From 91dd3165923a9f50190a651ff633f1e171cfacef Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Thu, 16 Jul 2026 17:00:24 -0700 Subject: [PATCH 11/12] Fix extract/export and unsubscribe to use _data_conn for split storage - _extract_single_dataset_into_db: use dataset._data_conn instead of dataset.conn so raw data is read from the per-dataset file - unsubscribe/unsubscribe_all: use _data_conn to remove triggers that were created on the raw data connection - Add tests for extract_runs_into_db and export_datasets_and_create_metadata_db with split raw data Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/qcodes/dataset/data_set.py | 7 +- src/qcodes/dataset/database_extract_runs.py | 6 +- tests/dataset/test_raw_data_storage.py | 77 +++++++++++++++++++++ 3 files changed, 84 insertions(+), 6 deletions(-) diff --git a/src/qcodes/dataset/data_set.py b/src/qcodes/dataset/data_set.py index 106b1fa17bc7..ef9d475111b4 100644 --- a/src/qcodes/dataset/data_set.py +++ b/src/qcodes/dataset/data_set.py @@ -1292,7 +1292,7 @@ def unsubscribe(self, uuid: str) -> None: """ Remove subscriber with the provided uuid """ - with atomic(self.conn) as conn: + with atomic(self._data_conn) as conn: sub = self.subscribers[uuid] remove_trigger(conn, sub.trigger_id) sub.schedule_stop() @@ -1307,8 +1307,9 @@ def unsubscribe_all(self) -> None: SELECT name FROM sqlite_master WHERE type = 'trigger' """ - triggers = atomic_transaction(self.conn, sql).fetchall() - with atomic(self.conn) as conn: + data_conn = self._data_conn + triggers = atomic_transaction(data_conn, sql).fetchall() + with atomic(data_conn) as conn: for (trigger,) in triggers: remove_trigger(conn, trigger) for sub in self.subscribers.values(): diff --git a/src/qcodes/dataset/database_extract_runs.py b/src/qcodes/dataset/database_extract_runs.py index bfbccc36092c..deb74982a5bc 100644 --- a/src/qcodes/dataset/database_extract_runs.py +++ b/src/qcodes/dataset/database_extract_runs.py @@ -166,8 +166,6 @@ def _extract_single_dataset_into_db( f"GUID: {dataset.guid} and run_id: {dataset.run_id}" ) - source_conn = dataset.conn - run_id = get_runid_from_guid(target_conn, dataset.guid) if run_id is not None: @@ -177,8 +175,10 @@ def _extract_single_dataset_into_db( dataset, target_conn, target_exp_id ) assert target_table_name is not None + # Use _data_conn to read from the raw data connection, which may + # be a separate per-dataset SQLite file when split storage is enabled. _populate_results_table( - source_conn, target_conn, dataset.table_name, target_table_name + dataset._data_conn, target_conn, dataset.table_name, target_table_name ) diff --git a/tests/dataset/test_raw_data_storage.py b/tests/dataset/test_raw_data_storage.py index 2536b3154718..010b6de66cd6 100644 --- a/tests/dataset/test_raw_data_storage.py +++ b/tests/dataset/test_raw_data_storage.py @@ -31,6 +31,10 @@ update_raw_data_paths, ) from qcodes.dataset.data_set import DataSet, load_by_id +from qcodes.dataset.database_extract_runs import ( + export_datasets_and_create_metadata_db, + extract_runs_into_db, +) from qcodes.dataset.descriptions.dependencies import InterDependencies_ from qcodes.dataset.sqlite.database import connect, initialise_database from qcodes.parameters import ParamSpecBase @@ -677,3 +681,76 @@ def test_nonexistent_db_raises(self, tmp_path: Path) -> None: with pytest.raises(FileNotFoundError, match="Database file not found"): cleanup_datasets(tmp_path / "nonexistent.db", older_than_days=1) + + +# --------------------------------------------------------------------------- +# Integration tests - extract/export with split raw data +# --------------------------------------------------------------------------- + + +@pytest.mark.usefixtures("_raw_data_experiment") +class TestExtractExportWithSplitRawData: + """Verify that extract_runs_into_db and export_datasets_and_create_metadata_db + work correctly when raw data is stored in separate per-dataset SQLite files.""" + + @staticmethod + def _make_dataset_with_data( + n_rows: int = 10, + ) -> tuple[DataSet, list[dict[str, float]]]: + ds = new_data_set("test-split") + x = ParamSpecBase("x", "numeric") + y = ParamSpecBase("y", "numeric") + idps = InterDependencies_(dependencies={y: (x,)}) + ds.set_interdependencies(idps) + ds.mark_started() + results = [{"x": float(i), "y": float(i**2)} for i in range(n_rows)] + ds.add_results(results) + ds.mark_completed() + return ds, results + + @staticmethod + def _close_ds(ds: DataSet) -> None: + if ds._raw_data_conn is not None: + ds._raw_data_conn.close() + ds.conn.close() + + def test_extract_runs_into_db_with_split_data(self, tmp_path: Path) -> None: + """extract_runs_into_db should copy data from external raw data file.""" + ds, results = self._make_dataset_with_data(n_rows=5) + run_id = ds.run_id + source_db = ds.path_to_db + assert source_db is not None + self._close_ds(ds) + + target_db = str(tmp_path / "target.db") + extract_runs_into_db(source_db, target_db, run_id) + + # Verify data was copied to target DB + target_conn = connect(target_db) + target_ds = load_by_id(1, conn=target_conn) + data = target_ds.get_parameter_data() + np.testing.assert_array_almost_equal( + data["y"]["y"], np.array([r["y"] for r in results]) + ) + assert target_ds.number_of_results == 5 + target_conn.close() + + def test_export_and_create_metadata_db_with_split_data( + self, tmp_path: Path + ) -> None: + """export_datasets_and_create_metadata_db should work with split data.""" + ds, _results = self._make_dataset_with_data(n_rows=5) + source_db = ds.path_to_db + assert source_db is not None + self._close_ds(ds) + + target_db = tmp_path / "metadata_only.db" + export_path = tmp_path / "netcdf_exports" + + statuses = export_datasets_and_create_metadata_db( + source_db, target_db, export_path=export_path + ) + + assert len(statuses) == 1 + # Should either be exported to NetCDF or copied as-is + assert next(iter(statuses.values())) in ("exported", "copied_as_is") From 2aeb373f5a34d99d0794a1495f76fd63a0ce2c26 Mon Sep 17 00:00:00 2001 From: Mikhail Astafev Date: Sat, 18 Jul 2026 10:31:43 +0200 Subject: [PATCH 12/12] Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Mikhail Astafev --- docs/dataset/introduction.rst | 6 ++---- src/qcodes/dataset/_raw_data_storage.py | 2 +- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/dataset/introduction.rst b/docs/dataset/introduction.rst index d3769431a33b..aeaa7cb1f705 100644 --- a/docs/dataset/introduction.rst +++ b/docs/dataset/introduction.rst @@ -81,7 +81,7 @@ For dataset operations, QCoDeS provides functions for: Split Raw Data Storage ====================== -By default, all measurement data (the results table rows) is stored in the same SQLite database alongside metadata such as experiments, runs, parameter layouts, and dependencies. Over time, the main database file can grow very large, which can slow down operations like browsing experiments and loading metadata. +By default, all measurement data (the results table rows) is stored in the same SQLite database alongside metadata such as experiments, runs, parameters and their dependencies. Over time, the main database file can grow very large, which can slow down operations like browsing experiments and make managing a large database file inconvenient. QCoDeS supports an optional **split raw data storage** mode in which the actual measurement data for each ``DataSet`` is written to an individual, per-dataset SQLite file while all metadata remains in the main database. Each per-dataset file is named after the dataset's GUID (e.g. ``.db``) and is stored in a configurable folder. @@ -123,6 +123,4 @@ QCoDeS provides helpers for managing per-dataset raw data files over time: - :func:`~qcodes.dataset.purge_orphaned_datasets` — remove dataset records whose raw data files no longer exist on disk. - :func:`~qcodes.dataset.cleanup_datasets` — remove datasets (DB records and raw data files) by age, sample name, or file size. -For usage examples, see the :doc:`Database notebook <../examples/DataSet/Database>`. - -For more details on database management, see the :doc:`Database notebook <../examples/DataSet/Database>`. +For usage examples and more details on database management, see the :doc:`Database notebook <../examples/DataSet/Database>`. diff --git a/src/qcodes/dataset/_raw_data_storage.py b/src/qcodes/dataset/_raw_data_storage.py index 915b4372f57d..5aea66be95d2 100644 --- a/src/qcodes/dataset/_raw_data_storage.py +++ b/src/qcodes/dataset/_raw_data_storage.py @@ -4,7 +4,7 @@ When the ``dataset.raw_data_to_separate_db`` config option is enabled, measurement data (results tables) are written to individual SQLite files - one per dataset - instead of the main QCoDeS database file. All metadata -(runs, experiments, layouts, dependencies) remains in the main database. +(runs, experiments, parameters) remains in the main database. The per-dataset files are stored in the folder given by ``dataset.raw_data_path`` and are named ``.db``.