Add GCS and ADLS import - #1439
Merged
Merged
Conversation
The S3 importer already read objects through the same duckdb connection the test path uses, and that connection already knew how to authenticate against GCS and Azure. It now serves all three, taking the server type from the format it is registered under, so the two remaining file sources stop telling users to download an object and hand-replace a type: local server. The GCS and Azure reads need real credentials, which is how the existing gcs and azure suites are gated, so the tests here cover the dispatch: each storage writes its own server type and reaches duckdb through its own setup.
The docs already call this source Azure Blob / ADLS, so the command follows. The import format and the ODCS server type are no longer the same string: ODCS has no adls server type, only azure, so a contract written by this import still says type: azure and the engine connects as before.
andredlng
force-pushed
the
import-gcs-azure
branch
from
July 28, 2026 11:35
47cb292 to
d9d9b2d
Compare
📖 Docs previewPreview site: https://brave-water-0cee36e03-1439.westeurope.7.azurestaticapps.net Changed pages (most-changed first):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
datacontract import gcsanddatacontract import adls, closing the last two file sources that still told users to download an object by hand:Both guides previously said: download one object with
gsutil/az, runimport jsonorimport parqueton the local copy, then replace the generatedtype: localserver with the real one, filling in location, format and delimiter.Almost no new machinery. The S3 importer already read objects through the same duckdb connection the test path uses, and that connection already knew how to authenticate against GCS (
setup_gcs_connection) and Azure (setup_azure_connection). One importer now serves all three, sos3_importer.pybecameobject_storage_importer.py.import s3is unchanged.The import format and the ODCS server type are deliberately not the same string. ODCS has no
adlsserver type — onlyazure, which is also the only one the engine dispatches on — so a contract written byimport adlsstill saystype: azureand connects as before:The command is named for what the docs already call this source, "Azure Blob / ADLS", rather than for the server type.
Verification
S3 is verified three ways on the refactored code:
datacontract teston the unedited file: 4/4 passedGCS and ADLS reads are not verified against real storage. The repo's existing
test_test_gcs_*andtest_test_azure_*suites are gated on credentials being set and skip everywhere, including CI, so there was no seam to reuse. The new tests cover what can be checked without a cloud account:adls→azureThat is weaker than the end-to-end proof the other importers have, and worth weighing before merging. One real import against each would close it.
31 tests pass.
Docs
New
imports/gcs.mdandimports/adls.md; both testing guides get the one-command step 3. The GCS pages note that duckdb reads Google Cloud Storage through its S3-compatible endpoint, so the location usess3://rather thangs://— a long-standing trap that already had its own troubleshooting entry.