You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
=Templates for Common Files in Operator Repositories
1
+
#Templates for Common Files in Operator Repositories
2
2
3
3
Stackable develops and maintains a growing number of operators for open source software.
4
4
5
5
The high level structure of the repository is consistent across all repositories, which in the past has led to repetitive maintenance tasks in many repositories even for small changes.
6
6
7
7
This repository is intended to help with these changes by keeping common files as templates and offering tooling to roll these out to all repositories that are under management by this tool.
8
8
9
-
==Structure of this repository
9
+
##Structure of this repository
10
10
11
-
=== GitHub Actions
11
+
###GitHub Actions
12
12
13
13
The definition files for GitHub actions that in turn execute the playbooks doing the actual work are kept in `.github`
14
14
15
-
=== Playbook
15
+
###Playbook
16
16
17
17
`playbook` contains the ansible playbook which is executed to perform the needed changes.
18
18
19
-
=== Templates
19
+
###Templates
20
20
21
21
Everything under the top level folder `template` is replicated to the target repositories.
22
22
The folder and file structure under `template` is considered as source structure which will be synchronized across all repositories in the following manner:
23
23
24
-
* Regular files and folders are simply copied
25
-
* Files with an extension of `.j2` will be processed as jinja2 templates and copied to the target repositories with the `.j2` extension removed. The default jinja2 variable delimiter has been replaced with `{[ }]` since a lot of the template files contain `{{ }}` and caused issues.
26
-
* File and directory names which contain `\[[product]]` will have this substituted for the value of the `product_string` variable. For example: "stackable-\[[product]]-operator.service" would become `stackable-kafka-operator.service`.
24
+
- Regular files and folders are simply copied
25
+
- Files with an extension of `.j2` will be processed as jinja2 templates and copied to the target repositories with the `.j2` extension removed. The default jinja2 variable delimiter has been replaced with `{[ }]` since a lot of the template files contain `{{ }}` and caused issues.
26
+
- File and directory names which contain `\[[product]]` will have this substituted for the value of the `product_string` variable. For example: "stackable-\[[product]]-operator.service" would become `stackable-kafka-operator.service`.
27
27
28
28
To remove files or directories that already exist in the target repositories these need to be configured in `repositories.yaml` under the `retired_files` key.
29
29
30
30
Anything that is listed here will be deleted from the target repositories.
31
31
32
32
NOTE: Deletion is the last step that is performed, so if there is an overlap between files existing in the template folder and this setting, the files would not be rolled out, since they'd get deleted before creating the pull request.
33
33
34
-
=== Configuration
34
+
###Configuration
35
35
36
36
All user-facing configuration is kept in `repositories.yaml`.
37
37
@@ -41,62 +41,51 @@ So in principle it is as simple as pushing any commit to `main` which will trigg
41
41
42
42
Target repositories are configured in `repositories.yaml` in the following form:
43
43
44
-
----
44
+
```yaml
45
45
- name: kafka-operator
46
46
url: stackabletech/kafka-operator.git
47
47
product_string: kafka
48
48
pretty_string: Kafka
49
-
----
50
-
51
-
|===
52
-
|Field |Description
53
-
54
-
|name
55
-
|This is only used internally to name working directories and the like.
56
-
57
-
|url
58
-
|The github repository for this operator. Need to be in the form of `<org>/<repo>.git`
59
-
60
-
|product_string
61
-
| A lower case string to use in config files, file names and the like. Should not contain whitespaces. This can sometimes be a shortened version of the full name, for example for Open Policy Agent this would be "opa"
62
-
63
-
|pretty_string
64
-
| The actual name of the product, including whitespaces and proper capitalization. This is intended to be used in doc or man files or similar things.
65
-
66
-
|include_productconfig
67
-
| Whether to include files from the `deploy/config-spec` folder into the os package.
68
-
69
-
*Default*: true
70
-
|===
71
-
49
+
```
50
+
51
+
| Field | Description |
52
+
| ----- | ----------- |
53
+
| `name` | This is only used internally to name working directories and the like. |
54
+
| `url` | The github repository for this operator. Need to be in the form of `<org>/<repo>.git`. |
55
+
| `product_string` | A lower case string to use in config files, file names and the like. Should not contain whitespaces. This can sometimes be a shortened version of the full name, for example for Open Policy Agent this would be "opa" |
56
+
| `pretty_string` | The actual name of the product, including whitespaces and proper capitalization. This is intended to be used in doc or man files or similar things. |
57
+
| `config.include_productconfig` | Whether to include files from the `deploy/config-spec` folder into the os package. *Default*: true |
58
+
| `config.has_product` | Indicates that the operator manages a product. This is particularly useful to differentiate between core and product operators. *Default*: true |
59
+
| `config.run_as` | Whether to run the operator as a default Deployment, or something custom. |
72
60
73
61
These are the only variables currently being used on the playbooks, but can be extended easily as more are needed.
74
62
75
-
NOTE: If a new variable is introduced, it needs to be added to all repository objects!
63
+
> [!NOTE]
64
+
> If a new variable is introduced, it needs to be added to all repository objects!
76
65
77
66
Additional settings can be found in `playbook/group_vars/all`, but these are not intended to be freely changed and should be treated with care.
78
67
79
-
80
-
== Making changes to the template
68
+
## Making changes to the template
81
69
82
70
If you want to make a change that should be rolled out to all operators, make the change in the `template` directory.
83
71
Consult the section above to learn more about the structure of the template.
84
72
85
-
=== Test changes locally
73
+
### Test changes locally
86
74
87
-
1. Run the `test.sh` script. To limit the run to a single operator, add `--extra-vars "shard_repositories=['nifi-operator']"` (or your choice of operator).
75
+
1. Run the `test.sh` script. To limit the run to a single operator, add `--extra-vars "shard_repositories#['nifi-operator']"` (or your choice of operator).
88
76
It will automatically delete and recreate a `work` directory.
89
77
2. The changes can be examined with `git status`.
90
78
When the pull request is later merged into the `main` branch then pull requests with these changes will be created automatically.
91
79
3. Depending on the change, it makes sense to run the integration tests for all changed operators.
92
80
If the tests are not run in this stage and if there is even just one integration test failing in the subsequential generated pull requests then the operator-templating must be adapted which creates again pull requests for all operators.
93
81
Changes in the GitHub workflow actions cannot be tested until finally merged.
94
82
95
-
== Deploying changes
83
+
## Deploying changes
96
84
97
85
Changes are rolled out via GitHub actions.
98
86
99
-
=== Authentication
87
+
### Authentication
88
+
100
89
Since this tool needs to authenticate itself in order to push commits it needs credentials of a user.
101
90
This is currently solved via a personal access token that needs to be provided as a repository (or org) secret with the name `GH_ACCESS_TOKEN`.
102
91
@@ -106,9 +95,11 @@ The personal access token needs to have the following permissions for this to wo
106
95
- org:read
107
96
- workflow
108
97
109
-
== Limitations
98
+
## Limitations
99
+
110
100
There is currently no synchronization with existing PRs on the target repositories whatsoever. A new pull request will be created for every commit made to this repository.
111
101
112
102
To update a PR that was created via this tool, it will have to be closed and necessary changes pushed here, which will result in a new PR.
113
103
114
-
WARNING: The Helm Chart files that are rolled out by the templates in their current form do not include a ClusterRole object which may be needed for this to work with RBAC.
104
+
> [!WARNING]
105
+
> The Helm Chart files that are rolled out by the templates in their current form do not include a ClusterRole object which may be needed for this to work with RBAC.
0 commit comments