Fetch all source files before processing easystack file - #283
Conversation
|
This one has tarballs from the GNU mirror, which was causing issues in EESSI/software-layer#1557 bot: build repo:eessi.io-2026.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2 |
|
New job on instance
|
|
bot: build repo:eessi.io-2026.06-software instance:eessi-bot-deucalion for:arch=aarch64/a64fx |
|
New job on instance
|
|
The a64fx build already showed how this improves things: |
|
Trying it again with bot: build repo:eessi.io-2026.06-software instance:eessi-bot-deucalion for:arch=aarch64/a64fx |
|
New job on instance
|
|
Also trying one without the last item ( bot: build repo:eessi.io-2026.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2 |
|
New job on instance
|
|
a64fx build (still with 3 items in the easystack): Then it exits, as expected. For the zen 2 build (only 2 items): So, also as expected. Will do one more without download issues to ensure that this still works as well. |
|
bot: build repo:eessi.io-2026.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2 |
|
New job on instance
|
|
That still works, so everything seems to work as expected. Removed the easystack file again. Note that this doesn't require a build/ingest. |
|
|
||
| if [ -f ${easystack_file} ]; then | ||
| echo_green "Downloading sources for easystack file ${easystack_file}..." | ||
| ${EB} --fetch-all --easystack ${easystack_file} --robot |
There was a problem hiding this comment.
--fetch-all is only supported from EasyBuild 5.3.0 onwards, do we need a version check here?
There was a problem hiding this comment.
Good point. I guess it's not very likely that we'll still be using older versions, but I've added a check by running eb --help and grepping for --fetch-all first. Otherwise it falls back to --fetch.
|
Does this mean that if the sources for already existing software in the easystack file cannot be fetched, the build process will fail? so we need to fix the existing easyconfigs first? |
I guess that could happen, theoretically, e.g. if we would clean up the sources for some already installed applications. But I don't think we've ever done that. Also, we usually stick to the latest EB release, so the easyconfigs should be quite up-to-date, and it should not really happen that we are trying to use outdated source URLs from an old easyconfig. |
As discussed in the support meeting, I've introduced an environment variable ( |
3 participants
This should make it fail early in case source tarballs cannot be downloaded. Note that if you're processing multiple easystack files, it will not fetch the sources for all of them. Doing that would require more changes to the code, and we're not doing that very often anyway: almost all PRs change only one easystack file, so this should already be a big improvement.
I'll give it a try with an actual easystack file.