feat: split raw data storage into per-dataset SQLite files#8219

Draft

astafan8 wants to merge 16 commits into

microsoft:mainfrom

astafan8:feature/split-raw-data-sqlite

Commits on Jun 12, 2026

feat: split raw data storage into per-dataset SQLite files

Mikhail Astafev
and
Copilot
committed

Commits on Jun 30, 2026

address PR review feedback from Jens and Copilot

Mikhail Astafev
and
Copilot
committed
Add update_raw_data_paths helper for relocated raw data files

Mikhail Astafev
and
Copilot
committed
Add update_raw_data_paths documentation to intro.rst and Database.ipynb

Mikhail Astafev
and
Copilot
committed

Commits on Jul 2, 2026

Merge branch 'main' into feature/split-raw-data-sqlite
astafan8
authored
Fix ruff format and unused import lint issues

Mikhail Astafev
and
Copilot
committed
Fix pyright type errors in test_raw_data_storage

Mikhail Astafev
and
Copilot
committed

Commits on Jul 3, 2026

Merge branch 'main' into feature/split-raw-data-sqlite
astafan8
authored

Commits on Jul 16, 2026

Add purge_orphaned_datasets and cleanup_datasets management functions

Mikhail Astafev
and
Copilot
committed
Add towncrier news fragment for split raw data storage

Mikhail Astafev
and
Copilot
committed
Address PR review comments: move queries to queries.py, add prints, simplify

Mikhail Astafev
and
Copilot
committed
Use log.debug for per-dataset reports, Path.unlink() for file removal

Mikhail Astafev
and
Copilot
committed
Merge branch 'main' into feature/split-raw-data-sqlite
astafan8
authored

Commits on Jul 17, 2026

Fix extract/export and unsubscribe to use _data_conn for split storage

Mikhail Astafev
and
Copilot
committed
Merge branch 'main' into feature/split-raw-data-sqlite
astafan8
authored

Commits on Jul 18, 2026

Apply suggestions from code review

astafan8
and
Copilot
authored