From eeb9b28c0a0cdd99678a749932c2897da1e01bc8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 31 Oct 2025 11:51:54 +0000 Subject: [PATCH 1/7] Initial plan From 6f10b4ff97f3d7589912c4da192283c4e39c352a Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 31 Oct 2025 12:02:52 +0000 Subject: [PATCH 2/7] Add k8s templates and documentation to reduce code duplication Co-authored-by: guyzsarun <48861591+guyzsarun@users.noreply.github.com> --- .github/workflows/README.md | 114 +++++++++ README.md | 13 + applicationset/toolchain/kustomization.yaml | 1 + k8s-templates/README.md | 242 ++++++++++++++++++ k8s-templates/nfs-volume-base.yaml | 31 +++ k8s-templates/patch-deployment-port.yaml | 14 + k8s-templates/patch-deployment-resources.yaml | 15 ++ k8s-templates/patch-service-port.yaml | 14 + k8s-templates/virtualservice-base.yaml | 18 ++ .../virtualservice-with-rewrite.yaml | 20 ++ 10 files changed, 482 insertions(+) create mode 100644 .github/workflows/README.md create mode 100644 k8s-templates/README.md create mode 100644 k8s-templates/nfs-volume-base.yaml create mode 100644 k8s-templates/patch-deployment-port.yaml create mode 100644 k8s-templates/patch-deployment-resources.yaml create mode 100644 k8s-templates/patch-service-port.yaml create mode 100644 k8s-templates/virtualservice-base.yaml create mode 100644 k8s-templates/virtualservice-with-rewrite.yaml diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100644 index 0000000..04c8c6c --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,114 @@ +# GitHub Workflows + +This directory contains GitHub Actions workflows for building and deploying applications. + +## Workflow Types + +### Individual Application Workflows + +Files like `dashy.yaml`, `homepage.yaml`, `cyberchef.yaml`, and `toolchain.yaml` are minimal wrapper workflows that trigger on changes to specific application directories. + +**Structure**: +```yaml +name: + +on: + workflow_dispatch: + push: + branches: + - main + paths: + - applicationset//** + +jobs: + docker-build: + uses: ./.github/workflows/reusable-docker-build.yaml + with: + path: applicationset/ +``` + +### Reusable Workflows + +#### reusable-docker-build.yaml + +A reusable workflow that handles Docker image building and pushing. Used by all application workflows. + +**Inputs**: +- `tag-name`: Docker image tag name (default: 'docker-build') +- `path`: Path to the application directory (required) +- `push`: Whether to push the image to registry (default: false) +- `version`: Image version tag (default: 'latest') + +### Release Workflow + +#### release.yaml + +Triggered by Git tags following the pattern `-`. This workflow: +1. Parses the application name and version from the tag +2. Builds and pushes the Docker image +3. Updates the version in the application's `kustomization.yaml` +4. Commits the version update back to the repository + +## Creating a New Application Workflow + +To add a workflow for a new application: + +1. **Copy an existing workflow** (e.g., `cyberchef.yaml`) +2. **Update the name** to match your application +3. **Update the paths** to trigger on your application directory +4. **Customize the path parameter** in the workflow call + +**Example for a new app called "myapp"**: + +```yaml +name: myapp + +on: + workflow_dispatch: + push: + branches: + - main + paths: + - applicationset/myapp/** + +jobs: + docker-build: + uses: ./.github/workflows/reusable-docker-build.yaml + with: + path: applicationset/myapp +``` + +## Why Individual Workflows? + +While these workflows appear duplicated, they serve distinct purposes: + +1. **Isolated Triggers**: Each workflow only runs when its specific application changes +2. **Clear Build Status**: Individual workflow badges in README.md show per-application status +3. **Flexible Configuration**: Each application can customize its workflow parameters if needed +4. **GitHub Actions Limitations**: GitHub Actions doesn't support dynamic workflow generation + +The common build logic is extracted into `reusable-docker-build.yaml` to avoid true code duplication. + +## Deployment Process + +### Continuous Deployment (Push to Main) + +1. Push changes to `applicationset//` +2. The application's workflow triggers +3. Docker image is built (but not pushed) +4. ArgoCD syncs the changes from the repository + +### Release Deployment (Tagged) + +1. Create a tag: `git tag -v1.0.0 && git push --tags` +2. The `release.yaml` workflow triggers +3. Docker image is built and pushed to GitHub Container Registry +4. `kustomization.yaml` is updated with the new version +5. ArgoCD syncs the changes and deploys the new version + +## Best Practices + +- Keep individual workflows minimal (just trigger configuration) +- Put common logic in reusable workflows +- Use semantic versioning for release tags: `-v..` +- Test kustomization locally before pushing: `kubectl kustomize applicationset/` diff --git a/README.md b/README.md index ee7f60b..cb90ac6 100644 --- a/README.md +++ b/README.md @@ -20,10 +20,23 @@ This repository follows the GitOps workflow to manage Kubernetes deployments. Al ├── applicationset # argocd applicationset generator │   ├── base # base kustomize template │   └── ... +├── k8s-templates # reusable kubernetes resource templates | └── changedetection # argocd managed application ``` +## Adding a New Application + +To add a new application to the repository: + +1. **Copy an existing application** from `applicationset/` that matches your needs +2. **Customize the configuration** (see [k8s-templates/README.md](k8s-templates/README.md) for common patterns) +3. **Test your kustomization**: `kubectl kustomize applicationset/myapp` +4. **Create a workflow** (optional): Copy one of the existing `.github/workflows/*.yaml` files if you need to build a custom Docker image +5. **Commit and let ArgoCD sync** your changes + +For detailed templates and examples, see [k8s-templates/README.md](k8s-templates/README.md). + ## Deployment To deploy an application, you can tag the repository follows the naming convention: `-`. This tag triggers the deployment process via GitHub Actions. diff --git a/applicationset/toolchain/kustomization.yaml b/applicationset/toolchain/kustomization.yaml index db2ea7d..fffe43f 100755 --- a/applicationset/toolchain/kustomization.yaml +++ b/applicationset/toolchain/kustomization.yaml @@ -2,6 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base +- vs.yaml namePrefix: toolchain- images: - name: ghcr.io/guyzsarun-lab/base diff --git a/k8s-templates/README.md b/k8s-templates/README.md new file mode 100644 index 0000000..4f09b57 --- /dev/null +++ b/k8s-templates/README.md @@ -0,0 +1,242 @@ +# Kubernetes Templates + +This directory contains reusable Kubernetes resource templates to reduce code duplication across the repository. + +## Overview + +The GitOps repository contains multiple applications with similar Kubernetes resources. These templates provide a starting point for new applications and document common patterns used across the repository. + +## Templates + +### VirtualService Templates + +#### virtualservice-base.yaml +Base template for a simple VirtualService that routes traffic to a single service. + +**Use case**: Simple applications exposed at a dedicated hostname (e.g., `app.proxmox.homelab`) + +**Key customization points**: +- `spec.hosts`: The hostname(s) for your application +- `spec.http.route.destination.host`: The Kubernetes service name +- `spec.http.route.destination.port.number`: The service port + +**Example VirtualService (copy and customize)**: +```yaml +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: myapp-vs +spec: + gateways: + - istio-system/default-gateway + hosts: + - myapp.proxmox.homelab + http: + - match: + - uri: + prefix: / + route: + - destination: + host: myapp-application.default.svc.cluster.local + port: + number: 3000 +``` + +#### virtualservice-with-rewrite.yaml +Template for VirtualService with URI rewriting (for apps running under a path prefix). + +**Use case**: Applications served under a path on a shared gateway (e.g., `gateway.proxmox.homelab/myapp`) + +**Key customization points**: +- `spec.hosts`: The shared gateway hostname +- `spec.http.match.uri.prefix`: The URL path prefix (e.g., `/myapp`) +- `spec.http.route.destination.host`: The Kubernetes service name + +**Example VirtualService (copy and customize)**: +```yaml +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: myapp-vs +spec: + gateways: + - istio-system/default-gateway + hosts: + - gateway.proxmox.homelab + http: + - match: + - uri: + prefix: /myapp + rewrite: + uri: / + route: + - destination: + host: myapp-application.default.svc.cluster.local + port: + number: 3000 +``` + +### Volume Templates + +#### nfs-volume-base.yaml +Base template for NFS-backed PersistentVolume and PersistentVolumeClaim. + +**Use case**: Applications requiring persistent storage backed by NFS + +**Key customization points**: +- `metadata.name`: Unique name for PV and PVC +- `spec.capacity.storage`: Storage size (e.g., `5Gi`, `10Gi`) +- `spec.storageClassName`: Unique storage class name +- `spec.nfs.path`: NFS export path + +**Example Volume (copy and customize)**: +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: myapp-pv +spec: + capacity: + storage: 5Gi + volumeMode: Filesystem + storageClassName: myapp + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + mountOptions: + - hard + - nfsvers=4.1 + nfs: + path: /home/devops/nfs_share/myapp + server: nfs.proxmox.homelab +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: myapp-pvc +spec: + storageClassName: myapp + accessModes: + - ReadWriteOnce + volumeMode: Filesystem + resources: + requests: + storage: 5Gi +``` + +### Service Port Patch Template + +Many applications use the same base service from `applicationset/base/service.yaml` but need different target ports. + +**Example service patch**: +```yaml +# patches/service.yaml +apiVersion: v1 +kind: Service +metadata: + labels: + app: application + name: application +spec: + ports: + - $patch: replace + - port: 3000 + protocol: TCP + targetPort: 8080 # Change to your app's port +``` + +### Deployment Resource Limits Patch Template + +**Example deployment resource limits patch**: +```yaml +# patches/deployment-memory.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: application +spec: + template: + spec: + containers: + - name: application + resources: + limits: + memory: 256Mi # Adjust as needed + cpu: 100m # Adjust as needed +``` + +## Common Patterns + +### Pattern 1: Simple Application with Dedicated Hostname + +``` +myapp/ +├── kustomization.yaml +└── vs.yaml +``` + +**kustomization.yaml**: +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- ../base +- vs.yaml +namePrefix: myapp- +images: +- name: ghcr.io/guyzsarun-lab/base + newName: ghcr.io/guyzsarun-lab/myapp + newTag: v1.0.0 +commonLabels: + app: myapp +``` + +### Pattern 2: Application with Custom Port + +``` +myapp/ +├── kustomization.yaml +├── patches/ +│ ├── service.yaml +│ └── deployment.yaml +└── vs.yaml +``` + +### Pattern 3: Application with Persistent Storage + +``` +myapp/ +├── kustomization.yaml +├── volume.yaml +├── patches/ +│ └── deployment-volume.yaml +└── vs.yaml +``` + +## Benefits + +- **Reduced Duplication**: Common patterns are documented and reusable +- **Consistency**: All applications follow similar structures +- **Easier Onboarding**: New applications can copy existing patterns +- **Self-Documenting**: Templates serve as live documentation + +## Creating a New Application + +1. Copy one of the existing applications that most closely matches your needs +2. Update the application name throughout +3. Customize ports, hostnames, and resource limits +4. Add any application-specific configuration (secrets, configmaps, etc.) +5. Test with `kubectl kustomize applicationset/myapp` before committing +6. Add a GitHub workflow if building a custom Docker image + +## Validation + +Before committing changes, validate your Kustomize configuration: + +```bash +# Validate kustomization +kubectl kustomize applicationset/myapp + +# Check for common issues +kubectl kustomize applicationset/myapp | kubectl apply --dry-run=client -f - +``` diff --git a/k8s-templates/nfs-volume-base.yaml b/k8s-templates/nfs-volume-base.yaml new file mode 100644 index 0000000..9f61c3b --- /dev/null +++ b/k8s-templates/nfs-volume-base.yaml @@ -0,0 +1,31 @@ +apiVersion: v1 +kind: PersistentVolume +metadata: + name: pv +spec: + capacity: + storage: 10Gi + volumeMode: Filesystem + storageClassName: app-storage + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + mountOptions: + - hard + - nfsvers=4.1 + nfs: + path: /home/devops/nfs_share/app + server: nfs.proxmox.homelab +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: pvc +spec: + storageClassName: app-storage + accessModes: + - ReadWriteOnce + volumeMode: Filesystem + resources: + requests: + storage: 10Gi diff --git a/k8s-templates/patch-deployment-port.yaml b/k8s-templates/patch-deployment-port.yaml new file mode 100644 index 0000000..0129ce2 --- /dev/null +++ b/k8s-templates/patch-deployment-port.yaml @@ -0,0 +1,14 @@ +# Example patch to change deployment container port +# Copy to applicationset//patches/deployment.yaml and customize +apiVersion: apps/v1 +kind: Deployment +metadata: + name: application +spec: + template: + spec: + containers: + - name: application + ports: + - $patch: replace + - containerPort: 8080 # Change to your application's port diff --git a/k8s-templates/patch-deployment-resources.yaml b/k8s-templates/patch-deployment-resources.yaml new file mode 100644 index 0000000..3dfb46e --- /dev/null +++ b/k8s-templates/patch-deployment-resources.yaml @@ -0,0 +1,15 @@ +# Example patch to set resource limits +# Copy to applicationset//patches/deployment-memory.yaml and customize +apiVersion: apps/v1 +kind: Deployment +metadata: + name: application +spec: + template: + spec: + containers: + - name: application + resources: + limits: + memory: 256Mi # Adjust based on application needs + cpu: 100m # Adjust based on application needs diff --git a/k8s-templates/patch-service-port.yaml b/k8s-templates/patch-service-port.yaml new file mode 100644 index 0000000..bf7bc26 --- /dev/null +++ b/k8s-templates/patch-service-port.yaml @@ -0,0 +1,14 @@ +# Example patch to change service target port +# Copy to applicationset//patches/service.yaml and customize +apiVersion: v1 +kind: Service +metadata: + labels: + app: application + name: application +spec: + ports: + - $patch: replace + - port: 3000 + protocol: TCP + targetPort: 8080 # Change to your application's port diff --git a/k8s-templates/virtualservice-base.yaml b/k8s-templates/virtualservice-base.yaml new file mode 100644 index 0000000..e97eaf4 --- /dev/null +++ b/k8s-templates/virtualservice-base.yaml @@ -0,0 +1,18 @@ +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: vs +spec: + gateways: + - istio-system/default-gateway + hosts: + - placeholder.proxmox.homelab + http: + - match: + - uri: + prefix: / + route: + - destination: + host: application.default.svc.cluster.local + port: + number: 3000 diff --git a/k8s-templates/virtualservice-with-rewrite.yaml b/k8s-templates/virtualservice-with-rewrite.yaml new file mode 100644 index 0000000..c2561b4 --- /dev/null +++ b/k8s-templates/virtualservice-with-rewrite.yaml @@ -0,0 +1,20 @@ +apiVersion: networking.istio.io/v1beta1 +kind: VirtualService +metadata: + name: vs +spec: + gateways: + - istio-system/default-gateway + hosts: + - placeholder.proxmox.homelab + http: + - match: + - uri: + prefix: /app + rewrite: + uri: / + route: + - destination: + host: application.default.svc.cluster.local + port: + number: 3000 From 00b70cd8eae89afbfbd971aedb8d1cc369145d40 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 31 Oct 2025 12:05:49 +0000 Subject: [PATCH 3/7] Add CONTRIBUTING guide and duplication detection script Co-authored-by: guyzsarun <48861591+guyzsarun@users.noreply.github.com> --- CONTRIBUTING.md | 237 ++++++++++++++++++++++++++++++++++ scripts/README.md | 49 +++++++ scripts/find-similar-files.sh | 99 ++++++++++++++ 3 files changed, 385 insertions(+) create mode 100644 CONTRIBUTING.md create mode 100644 scripts/README.md create mode 100755 scripts/find-similar-files.sh diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..60ec5dc --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,237 @@ +# Contributing to GitOps + +Thank you for your interest in contributing to this GitOps repository! + +## Overview + +This repository uses GitOps practices to manage Kubernetes deployments via ArgoCD. All changes to the cluster are made by updating this repository. + +## Repository Structure + +``` +├── .github/workflows/ # GitHub Actions CI/CD pipelines +├── applicationset/ # Application configurations +│ ├── base/ # Base Kustomize templates +│ └── / # Individual application configs +├── k8s-templates/ # Reusable Kubernetes resource templates +├── cluster-bootstrap/ # Cluster infrastructure components +└── changedetection/ # Standalone application +``` + +## Adding a New Application + +### Quick Start + +1. **Choose a template approach**: + - Copy an existing similar application from `applicationset/` + - Or start from scratch using templates in `k8s-templates/` + +2. **Create your application directory**: + ```bash + mkdir -p applicationset/myapp + ``` + +3. **Create required files**: + - `kustomization.yaml` - Kustomize configuration + - `vs.yaml` - VirtualService for Istio routing + - Optional: `volume.yaml` - Persistent storage + - Optional: `patches/` - Customizations to base resources + +4. **Reference the base template**: + ```yaml + # applicationset/myapp/kustomization.yaml + apiVersion: kustomize.config.k8s.io/v1beta1 + kind: Kustomization + resources: + - ../base + - vs.yaml + namePrefix: myapp- + images: + - name: ghcr.io/guyzsarun-lab/base + newName: ghcr.io/guyzsarun-lab/myapp + newTag: v1.0.0 + commonLabels: + app: myapp + ``` + +5. **Test locally**: + ```bash + kubectl kustomize applicationset/myapp + ``` + +6. **Create a GitHub workflow** (if building custom images): + ```bash + cp .github/workflows/cyberchef.yaml .github/workflows/myapp.yaml + # Edit the file to update name and paths + ``` + +7. **Commit and push**: + ```bash + git add applicationset/myapp + git commit -m "Add myapp application" + git push + ``` + +### Detailed Guides + +- [Kubernetes Templates Guide](k8s-templates/README.md) - Common patterns and templates +- [Workflow Guide](.github/workflows/README.md) - CI/CD pipeline patterns + +## Common Patterns + +### Pattern 1: Simple Web Application + +For applications with: +- Single container +- Exposed via Istio VirtualService +- No persistent storage + +**Example**: `cyberchef`, `toolchain` + +### Pattern 2: Application with Configuration + +For applications with: +- ConfigMaps for configuration files +- Volume mounts for config +- Patches to inject config + +**Example**: `dashy`, `glance` + +### Pattern 3: Application with Persistent Storage + +For applications with: +- NFS-backed PersistentVolume +- Volume mounts in deployment +- Data persistence requirements + +**Example**: `linkwarden`, `paperless`, `calibre` + +### Pattern 4: Multi-Container Application + +For applications with: +- Sidecar containers +- Multiple services +- Complex networking + +**Example**: `linkwarden` (with meilisearch sidecar) + +## Kustomize Best Practices + +### Use Strategic Merge Patches + +For modifying specific fields: +```yaml +# patches/service.yaml +apiVersion: v1 +kind: Service +metadata: + name: application +spec: + ports: + - $patch: replace + - port: 3000 + targetPort: 8080 +``` + +### Use namePrefix for Resource Naming + +Instead of hardcoding names, use `namePrefix`: +```yaml +namePrefix: myapp- +# Results in: myapp-application, myapp-vs, etc. +``` + +### Use commonLabels for Consistent Labeling + +```yaml +commonLabels: + app: myapp +``` + +## Validation + +Before committing, always validate: + +```bash +# Validate Kustomize output +kubectl kustomize applicationset/myapp + +# Dry-run to catch issues +kubectl kustomize applicationset/myapp | kubectl apply --dry-run=client -f - + +# Check YAML syntax +kubectl kustomize applicationset/myapp | kubectl apply --dry-run=server -f - +``` + +## Deployment Process + +### Development/Testing (Push to Main) + +1. Make changes to application configuration +2. Commit and push to `main` branch +3. ArgoCD automatically syncs changes to the cluster + +### Production Release (Tagged Release) + +1. Build and test your application +2. Create a Git tag: `git tag myapp-v1.0.0` +3. Push tag: `git push --tags` +4. GitHub Actions builds and pushes Docker image +5. Workflow updates `kustomization.yaml` with new version +6. ArgoCD syncs the new version + +## Troubleshooting + +### Kustomize Errors + +```bash +# View detailed error output +kubectl kustomize applicationset/myapp + +# Check for YAML syntax errors +yamllint applicationset/myapp/*.yaml +``` + +### ArgoCD Sync Issues + +1. Check ArgoCD UI for sync status +2. Review ArgoCD application logs +3. Verify Git repository is accessible +4. Check resource quotas and RBAC permissions + +### Docker Build Failures + +1. Review GitHub Actions workflow logs +2. Verify Dockerfile syntax +3. Check base image availability +4. Ensure required build arguments are provided + +## Code Review Guidelines + +When reviewing PRs: + +- [ ] Kustomize configuration is valid +- [ ] Resources follow naming conventions +- [ ] Labels are consistent +- [ ] VirtualService routes are correct +- [ ] Resource limits are appropriate +- [ ] Secrets are not committed to Git +- [ ] Documentation is updated if needed + +## Resources + +- [Kustomize Documentation](https://kubectl.docs.kubernetes.io/references/kustomize/) +- [ArgoCD Documentation](https://argo-cd.readthedocs.io/) +- [Istio VirtualService](https://istio.io/latest/docs/reference/config/networking/virtual-service/) +- [Kubernetes Best Practices](https://kubernetes.io/docs/concepts/configuration/overview/) + +## Getting Help + +- Review existing applications for examples +- Check [k8s-templates/README.md](k8s-templates/README.md) for common patterns +- Open an issue for questions or problems +- Consult the team before making infrastructure changes + +## License + +This project follows the same license as specified in [LICENSE](LICENSE). diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..85d4b19 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,49 @@ +# Scripts + +This directory contains utility scripts for maintaining the GitOps repository. + +## Available Scripts + +### find-similar-files.sh + +Identifies files with similar content that might be candidates for refactoring into shared templates. + +**Usage**: +```bash +# Find all similar YAML files +./scripts/find-similar-files.sh "*.yaml" + +# Find similar files with specific pattern +./scripts/find-similar-files.sh "vs.yaml" + +# Find similar patches +./scripts/find-similar-files.sh "*service*.yaml" +``` + +**Output**: +- Groups of files with the same name +- List of patch files that might be consolidated +- Recommendations for reducing duplication + +**Use cases**: +- Periodic audits for code duplication +- Before major refactoring efforts +- When adding new applications to check for existing patterns + +## Contributing + +When adding new scripts: + +1. Include a header comment explaining what the script does +2. Add usage examples +3. Make scripts executable: `chmod +x scripts/your-script.sh` +4. Document the script in this README +5. Follow bash best practices (set -e, proper quoting, etc.) + +## Best Practices + +- Keep scripts simple and focused on one task +- Use meaningful variable names +- Include error handling +- Document expected inputs and outputs +- Test scripts before committing diff --git a/scripts/find-similar-files.sh b/scripts/find-similar-files.sh new file mode 100755 index 0000000..c9a0596 --- /dev/null +++ b/scripts/find-similar-files.sh @@ -0,0 +1,99 @@ +#!/bin/bash +# +# Find similar files that might be candidates for refactoring +# Usage: ./scripts/find-similar-files.sh [pattern] +# +# This script helps identify duplicated code by finding files with similar content + +set -e + +PATTERN="${1:-*.yaml}" +MIN_SIMILARITY=70 # Minimum similarity percentage to report + +echo "Finding similar files matching pattern: $PATTERN" +echo "Minimum similarity threshold: ${MIN_SIMILARITY}%" +echo "---" + +# Find all matching files +FILES=$(find . -name "$PATTERN" -type f ! -path "./.git/*" ! -path "./k8s-templates/*" | sort) + +# Function to calculate simple similarity based on line overlap +# This is a simplified approach - for production use, consider using proper diff tools +compare_files() { + local file1="$1" + local file2="$2" + + # Get total lines in both files (excluding empty lines) + local lines1=$(grep -c ^ "$file1" || echo 0) + local lines2=$(grep -c ^ "$file2" || echo 0) + + if [ "$lines1" -eq 0 ] || [ "$lines2" -eq 0 ]; then + echo 0 + return + fi + + # Count common lines + local common=$(comm -12 <(sort "$file1") <(sort "$file2") | grep -c ^ || echo 0) + local avg_lines=$(( (lines1 + lines2) / 2 )) + + # Calculate similarity percentage + if [ "$avg_lines" -gt 0 ]; then + local similarity=$(( (common * 100) / avg_lines )) + echo "$similarity" + else + echo 0 + fi +} + +# Group files by name pattern +declare -A file_groups + +for file in $FILES; do + basename_file=$(basename "$file") + + # Group by filename (vs.yaml, volume.yaml, etc.) + if [[ ! -v file_groups["$basename_file"] ]]; then + file_groups["$basename_file"]="" + fi + file_groups["$basename_file"]+="$file " +done + +# Report on groups with multiple files +echo "File groups with potential duplication:" +echo "" + +for name in "${!file_groups[@]}"; do + files_array=(${file_groups[$name]}) + count=${#files_array[@]} + + if [ "$count" -gt 1 ]; then + echo "Group: $name (${count} files)" + + # Show first few files as examples + for ((i=0; i<${count} && i<5; i++)); do + echo " - ${files_array[$i]}" + done + + if [ "$count" -gt 5 ]; then + echo " ... and $((count - 5)) more" + fi + + echo "" + fi +done + +# Report patches with similar patterns +echo "---" +echo "Patch files (potential for consolidation):" +find . -path "*/patches/*.yaml" ! -path "./.git/*" | while read -r file; do + basename_file=$(basename "$file") + echo " - $file ($basename_file)" +done | sort -k3 + +echo "" +echo "---" +echo "Recommendations:" +echo "1. Review file groups with multiple instances" +echo "2. Consider using k8s-templates/ for common patterns" +echo "3. Use Kustomize patches for minor variations" +echo "4. Document patterns in k8s-templates/README.md" From 45dfa9632b96de84aa7bfab8e954c10b14dacf91 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 31 Oct 2025 12:08:51 +0000 Subject: [PATCH 4/7] Fix code review issues in find-similar-files.sh script Co-authored-by: guyzsarun <48861591+guyzsarun@users.noreply.github.com> --- scripts/find-similar-files.sh | 44 ++++++----------------------------- 1 file changed, 7 insertions(+), 37 deletions(-) diff --git a/scripts/find-similar-files.sh b/scripts/find-similar-files.sh index c9a0596..af60c64 100755 --- a/scripts/find-similar-files.sh +++ b/scripts/find-similar-files.sh @@ -8,47 +8,15 @@ set -e PATTERN="${1:-*.yaml}" -MIN_SIMILARITY=70 # Minimum similarity percentage to report echo "Finding similar files matching pattern: $PATTERN" -echo "Minimum similarity threshold: ${MIN_SIMILARITY}%" echo "---" -# Find all matching files -FILES=$(find . -name "$PATTERN" -type f ! -path "./.git/*" ! -path "./k8s-templates/*" | sort) - -# Function to calculate simple similarity based on line overlap -# This is a simplified approach - for production use, consider using proper diff tools -compare_files() { - local file1="$1" - local file2="$2" - - # Get total lines in both files (excluding empty lines) - local lines1=$(grep -c ^ "$file1" || echo 0) - local lines2=$(grep -c ^ "$file2" || echo 0) - - if [ "$lines1" -eq 0 ] || [ "$lines2" -eq 0 ]; then - echo 0 - return - fi - - # Count common lines - local common=$(comm -12 <(sort "$file1") <(sort "$file2") | grep -c ^ || echo 0) - local avg_lines=$(( (lines1 + lines2) / 2 )) - - # Calculate similarity percentage - if [ "$avg_lines" -gt 0 ]; then - local similarity=$(( (common * 100) / avg_lines )) - echo "$similarity" - else - echo 0 - fi -} - # Group files by name pattern declare -A file_groups -for file in $FILES; do +# Use proper file handling with null delimiters +while IFS= read -r -d '' file; do basename_file=$(basename "$file") # Group by filename (vs.yaml, volume.yaml, etc.) @@ -56,21 +24,23 @@ for file in $FILES; do file_groups["$basename_file"]="" fi file_groups["$basename_file"]+="$file " -done +done < <(find . -name "$PATTERN" -type f ! -path "./.git/*" ! -path "./k8s-templates/*" -print0 | sort -z) # Report on groups with multiple files echo "File groups with potential duplication:" echo "" for name in "${!file_groups[@]}"; do - files_array=(${file_groups[$name]}) + # Read files into array safely + IFS=' ' read -r -a files_array <<< "${file_groups[$name]}" count=${#files_array[@]} if [ "$count" -gt 1 ]; then echo "Group: $name (${count} files)" # Show first few files as examples - for ((i=0; i<${count} && i<5; i++)); do + display_count=$((count < 5 ? count : 5)) + for ((i=0; i Date: Fri, 31 Oct 2025 12:10:51 +0000 Subject: [PATCH 5/7] Completely refactor script to handle filenames with spaces safely Co-authored-by: guyzsarun <48861591+guyzsarun@users.noreply.github.com> --- scripts/find-similar-files.sh | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/scripts/find-similar-files.sh b/scripts/find-similar-files.sh index af60c64..0eeab01 100755 --- a/scripts/find-similar-files.sh +++ b/scripts/find-similar-files.sh @@ -12,36 +12,37 @@ PATTERN="${1:-*.yaml}" echo "Finding similar files matching pattern: $PATTERN" echo "---" -# Group files by name pattern -declare -A file_groups +# Create a temporary directory for grouping files +tmpdir=$(mktemp -d) +trap 'rm -rf "$tmpdir"' EXIT -# Use proper file handling with null delimiters +# First pass: collect all files and group by basename +find . -name "$PATTERN" -type f ! -path "./.git/*" ! -path "./k8s-templates/*" -print0 | \ while IFS= read -r -d '' file; do basename_file=$(basename "$file") - - # Group by filename (vs.yaml, volume.yaml, etc.) - if [[ ! -v file_groups["$basename_file"] ]]; then - file_groups["$basename_file"]="" - fi - file_groups["$basename_file"]+="$file " -done < <(find . -name "$PATTERN" -type f ! -path "./.git/*" ! -path "./k8s-templates/*" -print0 | sort -z) + # Create a list file for each basename + echo "$file" >> "$tmpdir/$basename_file.list" +done # Report on groups with multiple files echo "File groups with potential duplication:" echo "" -for name in "${!file_groups[@]}"; do - # Read files into array safely - IFS=' ' read -r -a files_array <<< "${file_groups[$name]}" - count=${#files_array[@]} +for list_file in "$tmpdir"/*.list; do + [ -f "$list_file" ] || continue + + # Get the basename from the list filename + name=$(basename "$list_file" .list) + + # Count files in this group + count=$(wc -l < "$list_file") if [ "$count" -gt 1 ]; then echo "Group: $name (${count} files)" # Show first few files as examples - display_count=$((count < 5 ? count : 5)) - for ((i=0; i Date: Fri, 31 Oct 2025 12:13:00 +0000 Subject: [PATCH 6/7] Address final code review feedback - improve placeholder syntax and script safety Co-authored-by: guyzsarun <48861591+guyzsarun@users.noreply.github.com> --- k8s-templates/nfs-volume-base.yaml | 10 +++++----- k8s-templates/virtualservice-base.yaml | 4 ++-- k8s-templates/virtualservice-with-rewrite.yaml | 6 +++--- scripts/find-similar-files.sh | 2 +- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/k8s-templates/nfs-volume-base.yaml b/k8s-templates/nfs-volume-base.yaml index 9f61c3b..2f40fe7 100644 --- a/k8s-templates/nfs-volume-base.yaml +++ b/k8s-templates/nfs-volume-base.yaml @@ -1,12 +1,12 @@ apiVersion: v1 kind: PersistentVolume metadata: - name: pv + name: -pv spec: capacity: storage: 10Gi volumeMode: Filesystem - storageClassName: app-storage + storageClassName: -storage accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain @@ -14,15 +14,15 @@ spec: - hard - nfsvers=4.1 nfs: - path: /home/devops/nfs_share/app + path: /home/devops/nfs_share/ server: nfs.proxmox.homelab --- apiVersion: v1 kind: PersistentVolumeClaim metadata: - name: pvc + name: -pvc spec: - storageClassName: app-storage + storageClassName: -storage accessModes: - ReadWriteOnce volumeMode: Filesystem diff --git a/k8s-templates/virtualservice-base.yaml b/k8s-templates/virtualservice-base.yaml index e97eaf4..8bdec1d 100644 --- a/k8s-templates/virtualservice-base.yaml +++ b/k8s-templates/virtualservice-base.yaml @@ -6,13 +6,13 @@ spec: gateways: - istio-system/default-gateway hosts: - - placeholder.proxmox.homelab + - .proxmox.homelab http: - match: - uri: prefix: / route: - destination: - host: application.default.svc.cluster.local + host: -application.default.svc.cluster.local port: number: 3000 diff --git a/k8s-templates/virtualservice-with-rewrite.yaml b/k8s-templates/virtualservice-with-rewrite.yaml index c2561b4..3fde96c 100644 --- a/k8s-templates/virtualservice-with-rewrite.yaml +++ b/k8s-templates/virtualservice-with-rewrite.yaml @@ -6,15 +6,15 @@ spec: gateways: - istio-system/default-gateway hosts: - - placeholder.proxmox.homelab + - .proxmox.homelab http: - match: - uri: - prefix: /app + prefix: / rewrite: uri: / route: - destination: - host: application.default.svc.cluster.local + host: -application.default.svc.cluster.local port: number: 3000 diff --git a/scripts/find-similar-files.sh b/scripts/find-similar-files.sh index 0eeab01..9547e5a 100755 --- a/scripts/find-similar-files.sh +++ b/scripts/find-similar-files.sh @@ -56,7 +56,7 @@ done # Report patches with similar patterns echo "---" echo "Patch files (potential for consolidation):" -find . -path "*/patches/*.yaml" ! -path "./.git/*" | while read -r file; do +find . -path "*/patches/*.yaml" ! -path "./.git/*" -print0 | while IFS= read -r -d '' file; do basename_file=$(basename "$file") echo " - $file ($basename_file)" done | sort -k3 From fc44a28d2aca6bcdca7fcb4648de83fc2a88d9ff Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 31 Oct 2025 12:14:33 +0000 Subject: [PATCH 7/7] Fix sort key in script output Co-authored-by: guyzsarun <48861591+guyzsarun@users.noreply.github.com> --- scripts/find-similar-files.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/find-similar-files.sh b/scripts/find-similar-files.sh index 9547e5a..5758f31 100755 --- a/scripts/find-similar-files.sh +++ b/scripts/find-similar-files.sh @@ -59,7 +59,7 @@ echo "Patch files (potential for consolidation):" find . -path "*/patches/*.yaml" ! -path "./.git/*" -print0 | while IFS= read -r -d '' file; do basename_file=$(basename "$file") echo " - $file ($basename_file)" -done | sort -k3 +done | sort -k2 echo "" echo "---"