Current state:
Currently, gcsfs does not natively support generating signed URLs using IAM-based credentials provided by GCP Workload Identity or other non-private key credentials. This is a limitation when running on environments such as Google Kubernetes Engine (GKE) with Workload Identity, where only google.auth.compute_engine.credentials.Credentials are available (which is what IAM-based credentials which are provided via Workload Identity Federation and the recommended approach: GCP Best Practices for Workload Identity). These credentials do not include a private key, which is required for the current signing functionality in gcsfs.
Use case:
In environments like GKE with Workload Identity:
- Using service account key files is discouraged for security reasons.
- Credentials provided via Workload Identity rely on token-based authentication and require the use of the
signBlob API to generate signed URLs.
Adding support for IAM-based signing would enable secure and native integration with GCP for generating signed URLs in such environments.
Proposed Solution:
Implement an alternative signing mechanism in gcsfs that leverages the IAM Service Account Credentials API to generate signed URLs. This could be achieved by:
1. Detecting the type of credentials in use (e.g., google.auth.compute_engine.credentials.Credentials).
2. Using the signBlob API to sign requests when private key-based credentials are unavailable.
Example of Desired Behavior:
The sign() method in gcsfs could use IAM credentials to generate signed URLs transparently, even in environments where private keys are unavailable. For example:
import gcsfs
fs = gcsfs.GCSFileSystem()
signed_url = fs.sign("bucket-name/object-name", expiration=3600)
print(signed_url) # Uses IAM-based signing if private key is not presentNotes:
- Proper configuration is needed to ensure the correct IAM permissions (e.g., roles/iam.serviceAccountTokenCreator) are in place.
Finally: I would be happy to open a PR if it is relevant.
Current state:
Currently,
gcsfsdoes not natively support generating signed URLs using IAM-based credentials provided by GCP Workload Identity or other non-private key credentials. This is a limitation when running on environments such as Google Kubernetes Engine (GKE) with Workload Identity, where onlygoogle.auth.compute_engine.credentials.Credentialsare available (which is what IAM-based credentials which are provided via Workload Identity Federation and the recommended approach: GCP Best Practices for Workload Identity). These credentials do not include a private key, which is required for the current signing functionality in gcsfs.Use case:
In environments like GKE with Workload Identity:
signBlobAPI to generate signed URLs.Adding support for IAM-based signing would enable secure and native integration with GCP for generating signed URLs in such environments.
Proposed Solution:
Implement an alternative signing mechanism in gcsfs that leverages the IAM Service Account Credentials API to generate signed URLs. This could be achieved by:
1. Detecting the type of credentials in use (e.g.,
google.auth.compute_engine.credentials.Credentials).2. Using the signBlob API to sign requests when private key-based credentials are unavailable.
Example of Desired Behavior:
The
sign()method ingcsfscould use IAM credentials to generate signed URLs transparently, even in environments where private keys are unavailable. For example:Notes:
Finally: I would be happy to open a PR if it is relevant.