Skip to content

Commit ce0af05

Browse files
joshuamkite-nfbjoshuamkite-nfb
committed
feat: Add JSON schema generation and validation for digger.yml (#2541)
- Fixed schema-gen tool for jsonschema v0.13.0 API changes - Added comprehensive developer documentation in schema-gen/ - Integrated schema validation into CI/CD pipeline - Updated user documentation with IDE support guidance - Validated schema against 3 production digger.yml configs - Created Makefile with schema-generate, schema-validate, and schema-check targets New dependencies (scoped to schema-gen tool): - github.com/invopop/jsonschema v0.13.0 - github.com/wk8/go-ordered-map/v2 v2.1.8 AI Assistance: This PR was developed with AI assistance (Claude via Roo Code) under human direction. The human directed the overall approach, validation strategy, and ensured adherence to repository conventions. The AI helped with code fixes, documentation, and validation testing.
1 parent f25a582 commit ce0af05

9 files changed

Lines changed: 4009 additions & 2 deletions

File tree

‎.github/workflows/libs_test.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,11 +57,15 @@ jobs:
5757
with:
5858
tofu_version: 1.8.5
5959

60+
- name: Validate digger.yml Schema
61+
run: make schema-check
62+
working-directory: libs/digger_config
63+
6064
- name: Test
6165
run: |
6266
if [ "${{ steps.pr_check.outputs.is_external }}" = "true" ]; then
6367
export IS_EXTERNAL_PR="true"
64-
go test ./...
68+
go test ./...
6569
else
6670
go test ./...
6771
fi

‎docs/ce/reference/digger.yml.mdx‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,14 @@ description: "Reference for configuring Digger via the digger.yml configuration
77
Fields marked as **Required** must be set. Optional fields have sensible defaults shown below.
88
</Note>
99

10+
<Tip>
11+
**IDE Support**: Add this to the top of your `digger.yml` for autocomplete and validation:
12+
```yaml
13+
# yaml-language-server: $schema=https://docs.opentaco.dev/schemas/digger.yml.json
14+
```
15+
The [JSON Schema](https://docs.opentaco.dev/schemas/digger.yml.json) is automatically generated from the codebase and is the authoritative source for validation.
16+
</Tip>
17+
1018
You can configure Digger by dropping a `digger.yml` file at the root level of your repo.
1119

1220
```yml
@@ -202,7 +210,7 @@ workflows:
202210
</ParamField>
203211

204212
<ParamField path="comment_render_mode" type="string" default="basic">
205-
How to render plan output in comments. Options: `basic`, `detailed`.
213+
How to render plan output in comments. Options: `basic`, `group_by_module`.
206214
</ParamField>
207215

208216
<ParamField path="mention_drifted_projects_in_pr" type="boolean" default="false">

‎libs/digger_config/Makefile‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
.PHONY: schema-generate schema-validate test-schema
2+
3+
# Generate JSON schema from Go structs
4+
schema-generate:
5+
@echo "Generating JSON schema from Go structs..."
6+
@cd cmd/schema-gen && GOWORK=off go mod download && GOWORK=off go run main.go -output ../../digger-schema.json
7+
@echo "✓ Schema generated successfully"
8+
9+
# Validate existing schema matches Go structs
10+
schema-validate:
11+
@echo "Validating JSON schema against Go structs..."
12+
@cd cmd/schema-gen && GOWORK=off go mod download && GOWORK=off go run main.go -validate -output ../../digger-schema.json
13+
@echo "✓ Schema validation passed"
14+
15+
# Run schema tests
16+
test-schema:
17+
@echo "Running schema tests..."
18+
@go test -v -run "^Test.*" schema_test.go config.go converters.go digger_config.go utils.go validators.go yaml.go
19+
@echo "✓ All schema tests passed"
20+
21+
# Run all schema checks (for CI)
22+
schema-check: schema-validate test-schema
23+
@echo "✓ All schema checks passed"

‎libs/digger_config/README.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Digger Config Library
2+
3+
Go library for parsing and validating Digger configuration files (`digger.yml`).
4+
5+
## Components
6+
7+
### Configuration Parsing
8+
- **[`digger_config.go`](digger_config.go)** - Main configuration loading and parsing
9+
- **[`yaml.go`](yaml.go)** - YAML struct definitions
10+
- **[`converters.go`](converters.go)** - Convert YAML structs to internal types
11+
- **[`validators.go`](validators.go)** - Configuration validation logic
12+
13+
### JSON Schema
14+
- **[`digger-schema.json`](digger-schema.json)** - JSON Schema for `digger.yml`
15+
- **[`cmd/schema-gen/`](cmd/schema-gen/)** - Schema generation tool
16+
- **[`schema_test.go`](schema_test.go)** - Schema validation tests
17+
18+
See [`cmd/schema-gen/README.md`](cmd/schema-gen/README.md) for schema generation documentation.
19+
20+
### Terragrunt Support
21+
- **[`terragrunt/`](terragrunt/)** - Terragrunt configuration parsing
22+
23+
## Usage
24+
25+
```go
26+
import "github.com/diggerhq/digger/libs/digger_config"
27+
28+
// Load configuration
29+
config, _, _, _, err := digger_config.LoadDiggerConfig(".", true, nil, nil)
30+
if err != nil {
31+
log.Fatal(err)
32+
}
33+
34+
// Validate
35+
if err := digger_config.ValidateDiggerConfig(config); err != nil {
36+
log.Fatal(err)
37+
}
38+
```
39+
40+
## Development
41+
42+
### Running Tests
43+
44+
```bash
45+
# All tests
46+
go test ./...
47+
48+
# Schema tests only
49+
make test-schema
50+
```
51+
52+
### Maintaining the Schema
53+
54+
When modifying configuration structs:
55+
56+
```bash
57+
# Regenerate schema
58+
make schema-generate
59+
60+
# Validate it matches
61+
make schema-validate
62+
63+
# Run all checks
64+
make schema-check
65+
```
66+
67+
See [`cmd/schema-gen/README.md`](cmd/schema-gen/README.md) for details.
68+
69+
## Configuration Reference
70+
71+
See the [digger.yml reference documentation](../../docs/ce/reference/digger.yml.mdx) for user-facing configuration documentation.

‎libs/digger_config/cmd/schema-gen/README.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Schema Generator
2+
3+
Automatically generates and validates the JSON Schema for `digger.yml` configuration files.
4+
5+
## Overview
6+
7+
This tool generates [`digger-schema.json`](../../digger-schema.json) from the Go struct definitions in [`yaml.go`](../../yaml.go), ensuring the schema stays in sync with the actual configuration code.
8+
9+
## Usage
10+
11+
### Generate Schema
12+
13+
```bash
14+
cd libs/digger_config
15+
make schema-generate
16+
```
17+
18+
This creates/updates `digger-schema.json` with:
19+
- All fields from `DiggerConfigYaml` and nested structs
20+
- Type information (string, boolean, array, object)
21+
- Enum constraints from code constants
22+
- Field descriptions and default values
23+
- YAML field name mapping (using `yaml:` struct tags)
24+
25+
### Validate Schema
26+
27+
```bash
28+
make schema-validate
29+
```
30+
31+
Validates that the existing schema matches the current Go struct definitions. Fails if:
32+
- Schema is missing properties that exist in Go structs
33+
- Critical enum values don't match code constants
34+
35+
### Run All Checks
36+
37+
```bash
38+
make schema-check
39+
```
40+
41+
Runs both validation and all schema tests. Use this in CI/CD.
42+
43+
## How It Works
44+
45+
1. **Reflection**: Uses [`github.com/invopop/jsonschema`](https://github.com/invopop/jsonschema) to reflect on Go structs
46+
2. **YAML Tag Mapping**: Configured with `FieldNameTag: "yaml"` to use YAML field names
47+
3. **Enhancement**: [`enhanceSchema()`](main.go:75) adds:
48+
- Enum values from code constants (e.g., `CommentRenderModeBasic`)
49+
- Field descriptions
50+
- Default values
51+
4. **Validation**: Compares generated schema against existing file
52+
53+
## Schema ID
54+
55+
The schema is published at:
56+
```
57+
https://docs.opentaco.dev/schemas/digger.yml.json
58+
```
59+
60+
## Development
61+
62+
### Adding New Fields
63+
64+
When you add fields to `DiggerConfigYaml`:
65+
66+
1. Add the field with proper `yaml:` tag:
67+
```go
68+
NewField string `yaml:"new_field"`
69+
```
70+
71+
2. Regenerate the schema:
72+
```bash
73+
make schema-generate
74+
```
75+
76+
3. If the field has enum values, add them in [`enhanceSchema()`](main.go:75):
77+
```go
78+
if prop, ok := props.Get("new_field"); ok && prop != nil {
79+
prop.Enum = []interface{}{"value1", "value2"}
80+
prop.Description = "Description of new field"
81+
}
82+
```
83+
84+
### Testing
85+
86+
Schema tests are in [`schema_test.go`](../../schema_test.go). Add tests for:
87+
- New enum fields
88+
- Required field validation
89+
- Default value behavior
90+
91+
## Dependencies
92+
93+
- `github.com/invopop/jsonschema` v0.13.0 - Schema generation
94+
- `github.com/wk8/go-ordered-map/v2` v2.1.8 - Ordered property maps (transitive)
95+
96+
## Related Files
97+
98+
- [`../../digger-schema.json`](../../digger-schema.json) - Generated schema
99+
- [`../../yaml.go`](../../yaml.go) - Source structs
100+
- [`../../schema_test.go`](../../schema_test.go) - Schema validation tests
101+
- [`../../Makefile`](../../Makefile) - Build targets

0 commit comments

Comments
 (0)