Add option to run tests in parallel (as default)

Author: Daniel Gerber

.gitignore

@@ -9,3 +9,5 @@ Tests/kaas/kaas-sonobuoy-tests/results/
 *.tar.gz
 .direnv
 .vscode
+Tests/sono-*
+report.yaml

Tests/README.md

@@ -49,10 +49,15 @@ Install KaaS-specific requirements:
 pip install -r kaas/requirements.txt
 ```
 
-Given a kubeconfig file `path/to/kubeconfig.yaml`, run
+Given a kubeconfig file `path/to/kubeconfig.yaml`, run all tests in parallel with:
 
 ```shell
-./scs-compliance-check.py -v -a kubeconfig=path/to/kubeconfig.yaml -a subject_root=. -s SUBJECT -o report.yaml scs-compatible-kaas.yaml
+./scs-compliance-check.py -v -a kubeconfig=path/to/kubeconfig.yaml \
+   -a subject_root=. \
+   -a e2e-parallel="--e2e-parallel=true" \
+   -s SUBJECT \
+   -o report.yaml \
+   scs-compatible-kaas.yaml
 ```
 
 Replace `SUBJECT` with an arbitrary, but meaningful subject name. Also, please note that the check
@@ -64,6 +69,10 @@ Additionally, the directory `sono-results` will be generated. It contains a JUni
 `plugins/e2e/results/global/junit_01.xml`. You can render it to HTML with a tool like junit2html.
 This might give you hints as to why a test failed.
 
+If you need to run tests without parallelization simply remove `-a e2e-parallel="--e2e-parallel=true"`
+or set it to `false`.
+Some additional information can be found [here](https://github.com/vmware-tanzu/sonobuoy/issues/1435).
+
 ## Usage information (help output)
 
 ```text
@@ -183,7 +192,7 @@ We use a **layered approach** to allow for selective installation:
 - at the very top we have `test-requirements.in`.
 
 Whenever you change or recompile one of these layers,
-*all layers above that layer have to be recompiled as well*.
+_all layers above that layer have to be recompiled as well_.
 
 Note: The Python version used for running `pip-compile` should be consistent. The currently
 used version is documented in the header of the `requirements.txt`. It should match the

Tests/scs-compatible-kaas.yaml

@@ -11,7 +11,7 @@ variables:
   - kubeconfig
 scripts:
 - executable: ./kaas/sonobuoy_handler/run_sonobuoy.py
-  args: run -k {kubeconfig} --scs-sonobuoy-config kaas/scs-sonobuoy-config.yaml -r {subject_root}/sono-results-e2e -c 'cncf-k8s-conformance'  -a '--mode=certified-conformance'
+  args: run -k {kubeconfig} --scs-sonobuoy-config kaas/scs-sonobuoy-config.yaml -r {subject_root}/sono-results-e2e -c 'cncf-k8s-conformance'  -a '--mode=certified-conformance {e2e-parallel}'
   #~ args: -k {kubeconfig} -r {subject_root}/sono-results -c 'cncf-k8s-conformance'  -a '--plugin-env e2e.E2E_DRYRUN=true'
   testcases:
   - id: cncf-k8s-conformance
@@ -31,7 +31,7 @@ scripts:
     description: Must fulfill all requirements of scs-0214-v2.
     url: https://docs.scs.community/standards/scs-0214-v2-k8s-node-distribution#decision
 - executable: ./kaas/sonobuoy_handler/run_sonobuoy.py
-  args: run -k {kubeconfig} --scs-sonobuoy-config kaas/scs-sonobuoy-config.yaml -r {subject_root}/sono-results-0219 -c 'kaas-networking-check' -a '--e2e-focus "NetworkPolicy"'
+  args: run -k {kubeconfig} --scs-sonobuoy-config kaas/scs-sonobuoy-config.yaml -r {subject_root}/sono-results-0219 -c 'kaas-networking-check' -a '--e2e-focus "NetworkPolicy" {e2e-parallel}'
   testcases:
   - id: kaas-networking-check
     description: Must fulfill all requirements of scs-0219-v1.