Skip to content

Add GCS and ADLS import - #1439

Merged
andredlng merged 2 commits into
mainfrom
import-gcs-azure
Jul 28, 2026
Merged

Add GCS and ADLS import#1439
andredlng merged 2 commits into
mainfrom
import-gcs-azure

Conversation

@andredlng

@andredlng andredlng commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds datacontract import gcs and datacontract import adls, closing the last two file sources that still told users to download an object by hand:

datacontract import gcs  --source s3://my-bucket/orders/*.json       --output datacontract.yaml
datacontract import adls --source abfss://my-container/orders/*.json --output datacontract.yaml

Both guides previously said: download one object with gsutil/az, run import json or import parquet on the local copy, then replace the generated type: local server with the real one, filling in location, format and delimiter.

Almost no new machinery. The S3 importer already read objects through the same duckdb connection the test path uses, and that connection already knew how to authenticate against GCS (setup_gcs_connection) and Azure (setup_azure_connection). One importer now serves all three, so s3_importer.py became object_storage_importer.py. import s3 is unchanged.

The import format and the ODCS server type are deliberately not the same string. ODCS has no adls server type — only azure, which is also the only one the engine dispatches on — so a contract written by import adls still says type: azure and connects as before:

SERVER_TYPES = {"s3": "s3", "gcs": "gcs", "adls": "azure"}

The command is named for what the docs already call this source, "Azure Blob / ADLS", rather than for the server type.

Verification

S3 is verified three ways on the refactored code:

  • the MinIO-backed suite, including a round trip that imports a contract and tests it unedited
  • a real AWS import against a private bucket with only an SSO session, then datacontract test on the unedited file: 4/4 passed
  • the produced contract is byte-identical to the one the pre-refactor importer produced, so the generalisation is provably a no-op for S3

GCS and ADLS reads are not verified against real storage. The repo's existing test_test_gcs_* and test_test_azure_* suites are gated on credentials being set and skip everywhere, including CI, so there was no seam to reuse. The new tests cover what can be checked without a cloud account:

  • each storage writes the right server type into the contract, including adlsazure
  • each reaches duckdb through its own credential setup, not S3's
  • each rejects a missing location

That is weaker than the end-to-end proof the other importers have, and worth weighing before merging. One real import against each would close it.

31 tests pass.

Docs

New imports/gcs.md and imports/adls.md; both testing guides get the one-command step 3. The GCS pages note that duckdb reads Google Cloud Storage through its S3-compatible endpoint, so the location uses s3:// rather than gs:// — a long-standing trap that already had its own troubleshooting entry.

@andredlng andredlng self-assigned this Jul 28, 2026
The S3 importer already read objects through the same duckdb connection the
test path uses, and that connection already knew how to authenticate against
GCS and Azure. It now serves all three, taking the server type from the format
it is registered under, so the two remaining file sources stop telling users to
download an object and hand-replace a type: local server.

The GCS and Azure reads need real credentials, which is how the existing gcs
and azure suites are gated, so the tests here cover the dispatch: each storage
writes its own server type and reaches duckdb through its own setup.
The docs already call this source Azure Blob / ADLS, so the command follows.

The import format and the ODCS server type are no longer the same string: ODCS
has no adls server type, only azure, so a contract written by this import still
says type: azure and the engine connects as before.
@andredlng andredlng changed the title Add GCS and Azure import Add GCS and ADLS import Jul 28, 2026
@github-actions

Copy link
Copy Markdown

📖 Docs preview

Preview site: https://brave-water-0cee36e03-1439.westeurope.7.azurestaticapps.net

Changed pages (most-changed first):

@andredlng
andredlng merged commit 3ac0d91 into main Jul 28, 2026
18 checks passed
@andredlng
andredlng deleted the import-gcs-azure branch July 28, 2026 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant