Use requester-pays for mirroring in AnVIL deployments (#7794)#7806
Draft
nadove-ucsc wants to merge 1 commit intodevelopfrom
Draft
Use requester-pays for mirroring in AnVIL deployments (#7794)#7806nadove-ucsc wants to merge 1 commit intodevelopfrom
nadove-ucsc wants to merge 1 commit intodevelopfrom
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #7806 +/- ##
===========================================
- Coverage 85.03% 85.02% -0.02%
===========================================
Files 162 162
Lines 23306 23313 +7
===========================================
+ Hits 19819 19821 +2
- Misses 3487 3492 +5 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7794-use-requester-pays-for-mirroring
branch
4 times, most recently
from
f3d77dd to
40d25df
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7794-use-requester-pays-for-mirroring
branch
from
40d25df to
1ad5cc7
Compare
|
Coverage is 85.102% — issues/nadove-ucsc/7794-use-requester-pays-for-mirroring into develop. No base build found for |
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7794-use-requester-pays-for-mirroring
branch
2 times, most recently
from
94ae6f0 to
dcfedd3
Compare
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7794-use-requester-pays-for-mirroring
branch
2 times, most recently
from
f05d424 to
22cff68
Compare
hannes-ucsc
requested changes
Apr 18, 2026
| def get(self, access_method: AccessMethod = AccessMethod.https) -> Access: | ||
| def get(self, | ||
| access_method: AccessMethod = AccessMethod.https, | ||
| access_headers: Mapping[str, str] | None = None |
Member
There was a problem hiding this comment.
Suggested change
| access_headers: Mapping[str, str] | None = None | |
| headers: Mapping[str, str] | None = None |
The method in "access method" has nothing to do with what the HTTP standard refers to as method. Access method is purely a DRS concept. IOW, there are just standard, optional HTTP headers for the request, there is no correspondence to "access".
| which must be done manually. | ||
|
|
||
| To set up a workspace ofr requester-pays for a given deployment: | ||
| 1. If the deployment's service accounts have not already been |
Member
There was a problem hiding this comment.
Empty lines between paragraphs. Multi-line list items constitute paragraphs.
|
|
||
| ### 3.2.3 Terra workspace for egress charges | ||
|
|
||
| Terra has a feature called "requester pays" that allows for data egress charges |
Member
There was a problem hiding this comment.
Technically speaking, "requester pays" is a GCS feature. AWS S3 has it too, and uses the same name. IOW, TDR (not Terra) just exposes that feature to us.
|
|
||
| 9. `_refresh` | ||
|
|
||
| ### 3.2.3 Terra workspace for egress charges |
Member
There was a problem hiding this comment.
Suggested change
| ### 3.2.3 Terra workspace for egress charges | |
| ### 3.2.3 Terra workspace for requester-pays with TDR |
Comment on lines
+760
to
+762
| # The Google Project ID associated with the Terra workspace to charge | ||
| # for file downloads while mirroring. If left unset, Terra pays the | ||
| # egress cost for the downloads. |
Member
There was a problem hiding this comment.
Suggested change
| # The Google Project ID associated with the Terra workspace to charge | |
| # for file downloads while mirroring. If left unset, Terra pays the | |
| # egress cost for the downloads. | |
| # The GCP project ID of the Terra workspace to charge | |
| # for file downloads from TDR while mirroring. If left unset, egress charges are incurred to the owner of the GCS bucket the files are stored in. Otherwise, the egress will be charged to the GCP billing account associated with the workspace. |
And add reference to the README section that explains how to set this up.
| # for file downloads while mirroring. If left unset, Terra pays the | ||
| # egress cost for the downloads. | ||
| # | ||
| 'AZUL_TERRA_BILLING_PROJECT': None, |
Member
There was a problem hiding this comment.
I don't think this is a billing project.
Suggested change
| 'AZUL_TERRA_BILLING_PROJECT': None, | |
| 'AZUL_TDR_REQUESTER_PAYS_PROJECT': None, |
nadove-ucsc
force-pushed
the
issues/nadove-ucsc/7794-use-requester-pays-for-mirroring
branch
from
22cff68 to
e953780
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked issues: #7794
Checklist
Author
developissues/<GitHub handle of author>/<issue#>-<slug>1 when the issue title describes a problem, the corresponding PR
title is
Fix:followed by the issue titleAuthor (partiality)
ptag to titles of partial commitspartialor completely resolves all linked issuespartiallabelAuthor (reindex)
rtag to commit title or the changes introduced by this PR will not require reindexing of any deploymentreindex:devor the changes introduced by it will not require reindexing ofdevreindex:anvildevor the changes introduced by it will not require reindexing ofanvildevreindex:anvilprodor the changes introduced by it will not require reindexing ofanvilprodreindex:prodor the changes introduced by it will not require reindexing ofprodreindex:partialand its description documents the specific reindexing procedure fordev,anvildev,anvilprodandprodor requires a full reindex or carries none of the labelsreindex:dev,reindex:anvildev,reindex:anvilprodandreindex:prodAuthor (API changes)
APIor this PR does not modify a REST APIa(A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST APIapp.pyor this PR does not modify a REST APIAuthor (upgrading deployments)
make docker_images.jsonand committed the resulting changes or this PR does not modifyazul_docker_images, or any other variables referenced in the definition of that variableutag to commit title or this PR does not require upgrading deploymentsupgradeor does not require upgrading deploymentsdeploy:sharedor does not modifydocker_images.json, and does not require deploying thesharedcomponent for any other reasondeploy:gitlabor does not require deploying thegitlabcomponentdeploy:runneror does not require deploying therunnerimageAuthor (hotfixes)
Ftag to main commit title or this PR does not include permanent fix for a temporary hotfixanvilprodandprod) have temporary hotfixes for any of the issues linked to this PRAuthor (before every review)
develop, squashed fixups from prior reviewsmake requirements_updateor this PR does not modifyDockerfile,environment,requirements*.txt,common.mk,Makefileorenvironment.bootRtag to commit title or this PR does not modifyrequirements*.txtreqsor does not modifyrequirements*.txtmake integration_testpasses in personal deployment or this PR does not modify functionality that could affect the IT outcomePeer reviewer (after approval)
Note that after requesting changes, the PR must be assigned to only the author.
System administrator (after approval)
demoorno demono demono sandboxN reviewslabel is accurateOperator
reindex:…labels andrcommit title tagno demodevelopOperator (deploy
.sharedand.gitlabcomponents)_select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlab_select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlabdeploy:gitlabdeploy:gitlabSystem administrator (post-deploy of
.gitlabcomponent)dev.gitlabare complete or this PR is not labeleddeploy:gitlabanvildev.gitlabare complete or this PR is not labeleddeploy:gitlabOperator (deploy runner image)
_select dev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runner_select anvildev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runnerOperator (sandbox build)
sandboxlabel or PR is labeledno sandboxdevor PR is labeledno sandboxanvildevor PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxor this PR does not remove catalogs or otherwise causes unreferenced indices insandboxanvilboxor this PR does not remove catalogs or otherwise causes unreferenced indices inanvilboxsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevOperator (merge the branch)
pif the PR is also labeledpartialOperator (main build)
devanvildevdevdevanvildevanvildev_select dev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shared_select anvildev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shareddevanvildevOperator (reindex)
devor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevdevor this PR does not require reindexingdevdeploy_browserjob in the GitLab pipeline for this PR indevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdeploy_browserjob in the GitLab pipeline for this PR inanvildevor this PR does not require reindexinganvildevOperator (mirroring)
devor this PR does not require mirroringdevanvildevor this PR does not require mirroringanvildevdevor this PR does not require mirroringdevanvildevor this PR does not require mirroringanvildevdevor this PR does not require mirroringdevanvildevor this PR does not require mirroringanvildevOperator
deploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels to the next promotion PRs or this PR carries none of these labelsdeploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labelsShorthand for review comments
Lline is too longWline wrapping is wrongQbad quotesFother formatting problem