Feature/872 make secret array configurable

doc/changes/unreleased.md

@@ -2,8 +2,36 @@
 
 ## Summary
 
-Updated the nox DB-version default to come from `BaseConfig` instead of the hardcoded `7.1.9`,
-so ITDE-related test flows use the configured Exasol baseline and unit-test help no longer advertises `--db-version`.
+In this major release, several modifications were made to the PTB's workflow templates and actions:
+
+* the default DB-version was updated to come from `BaseConfig` instead of the
+  hardcoded `7.1.9`, so ITDE-related test flows use the configured Exasol baseline
+  and unit-test help no longer advertises `--db-version`.
+* the `github_template_dict.custom_workflows` entry now auto-detects secret names
+  from custom workflow files and passes them into PTB-controlled workflow templates.
+  For example:
+
+  ```yaml
+  on:
+    workflow_call:
+      secrets:
+        PYPI_TOKEN:
+          required: true
+        SONAR_TOKEN:
+          required: true
+  ```
+* the Python environment GitHub action now accepts `extras` as a comma-separated
+  list, which makes it easier to pass multiple optional dependency groups in one
+  value. Additionally, it supports `all-extras`, so that all extras are installed
+  without further specification needed.
+* the new `workflow:audit` Nox session runs `zizmor` against GitHub Actions and
+  reusable workflows, so security checks are part of the normal `checks.yml`
+  pipeline instead of being a separate manual step. It also keeps the audit
+  configuration in the project root via `.zizmor.yml`; see the
+  [zizmor configuration guide](https://exasol.github.io/python-toolbox/main/user_guide/features/managing_dependencies/zizmor_configuration.html)
+  and the
+  [troubleshooting guide for findings](https://exasol.github.io/python-toolbox/main/user_guide/troubleshooting/handle_zizmor_findings.html)
+  for details on tuning or suppressing findings locally.
 
 ## Feature
 
@@ -18,10 +46,12 @@ so ITDE-related test flows use the configured Exasol baseline and unit-test help
 ## Feature
 
 * #878: Added Nox session `workflow:audit` which uses `zizmor` and added it in `checks.yml`
+* #872: Added `custom_workflows` to `github_template_dict` for automatic custom workflow secret extraction
 
 ## Refactoring
 
 * #744: Extracted shared minimum-version selection logic into `minimum_declared_version()`
+* #699: Switched `extras` in the Python environment GitHub action to comma-separation
 
 ## Documentation
 

doc/user_guide/features/github_workflows/index.rst

@@ -36,10 +36,15 @@ compare the rendered workflow templates against the files in ``.github/workflows
 Workflows
 ---------
 
-The PTB has three categories of workflows:
+The PTB allows for two categories of workflows:
   #. those maintained by the PTB, which can be modified using the :ref:`workflow_patcher`.
-  #. those which are seeded by the PTB but owned and maintained by the project after initial creation.
-  #. those which extend the PTB-provided workflows and are maintained by the project (not the PTB).
+  #. custom workflows, which are project-owned.
+
+Custom workflows can optionally be 
+* seeded by the PTB, i.e. PTB generates an initial version but ignores future changes.
+* extend PTB-provided workflows, i.e. ending in `-extension.yml`
+
+Besides that, you can also create individual workflow files which are ignored by the PTB.
 
 Maintained by the PTB
 ^^^^^^^^^^^^^^^^^^^^^
@@ -108,8 +113,26 @@ Maintained by the PTB
        the action, and uploads the results to Sonar.
 
 
-Not Maintained by the PTB
-^^^^^^^^^^^^^^^^^^^^^^^^^
+Custom Workflows
+^^^^^^^^^^^^^^^^
+
+Custom workflows are project-owned. The two custom workflow forms are:
+
+* workflows seeded once by the PTB and then owned by the project
+* workflow extensions that are enabled when the project adds the corresponding file
+
+
+The PTB detects whether the relevant workflow file is present and activates a block
+where is it called in corresponding calling workflow. If the file is absent, the PTB
+does not add that block to the PTB-controlled workflows.
+
+For these custom workflows to work correctly, the reusable workflow must follow
+the secret declaration format described in :ref:`custom_workflow_secrets` so the
+calling PTB workflow can pass the required secrets through.
+
+
+Workflows Initially Seeded by the PTB
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The PTB seeds these workflows for new projects, but after that the project owns
 them and PTB regeneration does not overwrite them.
@@ -126,7 +149,7 @@ them and PTB regeneration does not overwrite them.
      - Runs long-running checks, which typically involve an Exasol database instance.
 
 Workflow Extensions
-^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~
 
 To use a workflow extension, a user must simply add the file to their project's
 ``.github/workflows`` directory. The PTB checks that this file exists, and if it does,

doc/user_guide/features/github_workflows/workflow_variables.rst

@@ -19,6 +19,36 @@ standardized baseline that can be overridden in individual projects.
   :start-at: github_template_dict
   :end-before: @computed_field
 
+.. _custom_workflow_secrets:
+
+Custom Workflow Secrets
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The PTB extracts secret names from reusable custom workflow files and exposes them
+through :py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict` under the
+``custom_workflows`` entry. PTB-controlled workflow templates use those extracted
+names when they call reusable workflows and forward secrets via ``secrets:``.
+
+To make a custom workflow compatible with this extraction, declare its secrets at the
+top of the reusable workflow file under ``on.workflow_call`` and before ``jobs``.
+The extractor reads that section and collects the secret names automatically.
+
+For example, ``slow-checks.yml`` can define its reusable workflow header like this:
+
+.. code-block:: yaml
+
+   name: Slow-Checks
+
+   on:
+     workflow_call:
+       secrets:
+         PYPI_TOKEN:
+           required: true
+         SONAR_TOKEN:
+           required: true
+
+Those extracted secret names are then made available to the PTB templates that
+reference the custom workflow.
 
 .. _workflow_matrix:
 

doc/user_guide/troubleshooting/handle_zizmor_findings.rst

@@ -32,7 +32,11 @@ A typical line-level ignore looks like this:
 
 .. code-block:: yaml
 
-   secrets: inherit # zizmor: ignore[secrets-inherit] - PTB cannot customize inherited secrets here yet.
+    - name: Set up Poetry (${{ inputs.poetry-version }})
+      shell: bash
+      run: | # zizmor: ignore[github-env] - This shared action is used by many workflows, and downstream steps need `poetry` on PATH; we do not have a safer replacement yet.
+        POETRY_VERSION="${INPUTS_POETRY_VERSION}" "$PYTHON_BINARY" "${{ github.action_path }}/ext/get_poetry.py"
+        echo "$HOME/.local/bin" >> $GITHUB_PATH
 
 Use configuration rules in ``.zizmor.yml`` only when the finding is genuinely
 project-wide. If you add a temporary rule while working through a batch of

exasol/toolbox/config.py

@@ -26,6 +26,9 @@
     PLUGIN_ATTR_NAME,
 )
 from exasol.toolbox.util.version import Version
+from exasol.toolbox.util.workflows.custom_workflow_extractor import (
+    CustomWorkflowExtractor,
+)
 
 WORKFLOW_HEADER_PREFIX = (
     "# Generated and maintained by the exasol-toolbox.\n"
@@ -316,26 +319,19 @@ def github_template_dict(self) -> dict[str, Any]:
         Dictionary of variables to dynamically render Jinja2 templates into valid YAML
         configurations.
         """
-        cd_extension = self.github_workflow_directory / "cd-extension.yml"
-        fast_tests_extension = (
-            self.github_workflow_directory / "fast-tests-extension.yml"
-        )
-        merge_gate_extension = (
-            self.github_workflow_directory / "merge-gate-extension.yml"
+        custom_workflow_extractor = CustomWorkflowExtractor(
+            github_workflow_directory=self.github_workflow_directory,
+            sonar_token_name=self.sonar_token_name,
         )
 
         return {
+            "custom_workflows": custom_workflow_extractor.build_custom_workflow_dict(),
             "dependency_manager_version": self.dependency_manager.version,
             "minimum_python_version": self.minimum_python_version,
             "os_version": self.os_version,
             "python_versions": self.python_versions,
             "sonar_token_name": self.sonar_token_name,
             "workflow_header": f"{WORKFLOW_HEADER_PREFIX}{__version__}.",
-            "workflow_extension": {
-                "cd": cd_extension.is_file(),
-                "fast_tests": fast_tests_extension.is_file(),
-                "merge_gate": merge_gate_extension.is_file(),
-            },
         }
 
     @computed_field  # type: ignore[misc]

exasol/toolbox/templates/github/workflows/cd.yml

@@ -24,13 +24,18 @@ jobs:
     secrets:
       PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
 
-  (% if workflow_extension.cd %)
+  (% if custom_workflows["cd-extension"].exists %)
   cd-extension:
     name: Extension
     uses: ./.github/workflows/cd-extension.yml
     needs:
       - check-release-tag
-    secrets: inherit # zizmor: ignore[secrets-inherit] - PTB cannot customize inherited secrets here yet; tracked in https://github.com/exasol/python-toolbox/issues/872.
+    (% if custom_workflows["cd-extension"].secrets %)
+    secrets:
+      (% for secret_name in custom_workflows["cd-extension"].secrets %)
+      (( secret_name )): ${{ secrets.(( secret_name )) }}
+      (% endfor %)
+    (% endif %)
     permissions:
       contents: write
   (% endif %)

exasol/toolbox/templates/github/workflows/ci.yml

@@ -9,7 +9,12 @@ jobs:
   merge-gate:
     name: Merge Gate
     uses: ./.github/workflows/merge-gate.yml
-    secrets: inherit # zizmor: ignore[secrets-inherit] - PTB cannot customize inherited secrets here yet; tracked in https://github.com/exasol/python-toolbox/issues/872.
+    (% if custom_workflows["merge-gate"].secrets %)
+    secrets:
+      (% for secret_name in custom_workflows["merge-gate"].secrets %)
+      (( secret_name )): ${{ secrets.(( secret_name )) }}
+      (% endfor %)
+    (% endif %)
     permissions:
       contents: read
 

exasol/toolbox/templates/github/workflows/fast-tests.yml

@@ -42,10 +42,16 @@ jobs:
           include-hidden-files: true
           overwrite: false
 
-(% if workflow_extension.fast_tests %)
+(% if custom_workflows["fast-tests-extension"].exists %)
   fast-tests-extension:
     name: Extension
     uses: ./.github/workflows/fast-tests-extension.yml
+    (% if custom_workflows["fast-tests-extension"].secrets %)
+    secrets:
+      (% for secret_name in custom_workflows["fast-tests-extension"].secrets %)
+      (( secret_name )): ${{ secrets.(( secret_name )) }}
+      (% endfor %)
+    (% endif %)
     permissions:
       contents: read
   (% endif %)

exasol/toolbox/templates/github/workflows/merge-gate.yml

@@ -3,6 +3,13 @@ name: Merge-Gate
 
 on:
   workflow_call:
+    (% if custom_workflows["merge-gate"].secrets %)
+    secrets:
+      (% for secret_name in custom_workflows["merge-gate"].secrets %)
+        (( secret_name )):
+          required: true
+      (% endfor %)
+    (% endif %)
 
 jobs:
   run-fast-checks:
@@ -46,15 +53,25 @@ jobs:
     needs:
       - approve-run-slow-tests
     uses: ./.github/workflows/slow-checks.yml
-    secrets: inherit # zizmor: ignore[secrets-inherit] - PTB cannot customize inherited secrets here yet; tracked in https://github.com/exasol/python-toolbox/issues/872.
+    (% if custom_workflows["slow-checks"].secrets %)
+    secrets:
+      (% for secret_name in custom_workflows["slow-checks"].secrets %)
+      (( secret_name )): ${{ secrets.(( secret_name )) }}
+      (% endfor %)
+    (% endif %)
     permissions:
       contents: read
 
-  (% if workflow_extension.merge_gate %)
+  (% if custom_workflows["merge-gate-extension"].exists %)
   merge-gate-extension:
     name: Extension
     uses: ./.github/workflows/merge-gate-extension.yml
-    secrets: inherit # zizmor: ignore[secrets-inherit] - PTB cannot customize inherited secrets here yet; tracked in https://github.com/exasol/python-toolbox/issues/872.
+    (% if custom_workflows["merge-gate-extension"].secrets %)
+    secrets:
+      (% for secret_name in custom_workflows["merge-gate-extension"].secrets %)
+      (( secret_name )): ${{ secrets.(( secret_name )) }}
+      (% endfor %)
+    (% endif %)
     permissions:
       contents: read
   (% endif %)
@@ -71,9 +88,9 @@ jobs:
       - run-fast-checks
       - run-fast-tests
       - run-slow-checks
-     (% if workflow_extension.merge_gate %)
+    (% if custom_workflows["merge-gate-extension"].exists %)
       - merge-gate-extension
-     (% endif %)
+    (% endif %)
     # To prevent accidentally merges, this step is required. For more details
     # see: https://github.com/exasol/python-toolbox/issues/563
     steps:

exasol/toolbox/templates/github/workflows/periodic-validation.yml

@@ -45,7 +45,12 @@ jobs:
     uses: ./.github/workflows/slow-checks.yml
     needs:
       - restrict-to-default-branch
-    secrets: inherit # zizmor: ignore[secrets-inherit] - PTB cannot customize inherited secrets here yet; tracked in https://github.com/exasol/python-toolbox/issues/872.
+    (% if custom_workflows["slow-checks"].secrets %)
+    secrets:
+      (% for secret_name in custom_workflows["slow-checks"].secrets %)
+      (( secret_name )): ${{ secrets.(( secret_name )) }}
+      (% endfor %)
+    (% endif %)
     permissions:
       contents: read
 

exasol/toolbox/templates/github/workflows/slow-checks.yml

@@ -21,8 +21,6 @@ jobs:
     runs-on: "(( os_version ))"
     permissions:
       contents: read
-    env:
-      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     strategy:
       fail-fast: false
       matrix: ${{ fromJson(needs.build-matrix.outputs.matrix) }}