feat/3525-update-dataset-schema-api

[munish7771]

Description

This PR adds a sync_schema method to dlt.Dataset, allowing us to refresh a dataset’s schema when it becomes stale.
It supports:

1. Fetching the latest schema by dataset name (default behavior)
2. Updating from an explicit dlt.Schema or schema name
3. Resetting to a local-only schema state

Related Issues

• Fixes feat: API to update dlt.Dataset.schema #3525

Additional Context

• Added a new function sync_schema to dlt\dataset\dataset.py
• Added unit tests to tests\dataset\test_dataset.py

[munish7771]

@rudolfix The refresh breaks when called via pipeline.dataset().refresh()
Looks like the pipeline.dataset() resolves the default schema to a full schema object before passing it to dataset. In refresh() it can't tell if the schema came from the user or the pipeline default. What do you suggest?

[rudolfix]

@munish7771 feel free to store origin schema argument to be able to pass to new dataset instance in refresh(). ie. self._schema_origin = schema in __init__

[munish7771]

@rudolfix what i meant was if we create the dataset using the pipeline api using dataset(), the function always provides the dataset constructor with a schema even if we explicitly didn't add any. in this case it will always be a caught exception.
for-eg, with the current code changes if i run the following -

import dlt

pipeline = dlt.pipeline(
    pipeline_name="test_pipeline",
    destination="duckdb",
    dataset_name="test_dataset"
)
pipeline.run([{"id": 1, "name": "Sharma"}], table_name="users")
dataset = pipeline.dataset()
dataset.refresh()

it gives error - TypeError: refresh() is not supported when the Dataset was created with a Schema instance.

is this intended? One way I can think of is to change the behaviour of the pipeline.dataset() function, but I am not sure if its that way for a reason.

[zilto]

@munish7771 Can you rebase your branch on devel?

Some tests are failing on CI, but are unrelated to your changes. They were recently fixed and merged to devel

[munish7771]

Replaced by a clean PR due to rebase issues → #3802
FYI: @zilto @rudolfix