We need to enable creating a SQL database to search the tracked datasets that have been tracked in the local store.
This SQL database should be created if it does not exist (after asking the user), and it should be placed in the .cache/ folder in the local store. If the database exists when creating a new snapshot, then it should also be updated automatically, it should only be updated once the local store snapshot has finished being created.
This enables the user to search the tracked datasets using dstrack search.
User should also be able to re-index the database whenever they like to. Deleting the SQL database should never lead to a broken pipeline.
We need to enable creating a SQL database to search the tracked datasets that have been tracked in the local store.
This SQL database should be created if it does not exist (after asking the user), and it should be placed in the
.cache/folder in the local store. If the database exists when creating a new snapshot, then it should also be updated automatically, it should only be updated once the local store snapshot has finished being created.This enables the user to search the tracked datasets using
dstrack search.User should also be able to re-index the database whenever they like to. Deleting the SQL database should never lead to a broken pipeline.