ci: verify workflow execution for wave 4p (throwaway) - #2
Closed
DoDiODev wants to merge 98 commits into
Closed
Conversation
…sions - upgrade backend dependencies (Go modules, gorm/mysql driver) and pin python/e2e deps - wrap MySQL dialector migrator to prevent duplicate PRIMARY KEY errors on AddColumn - refactor backend Docker setup (new streamlined Dockerfile, remove Dockerfile.local) and align build workflow dockerfile paths - upgrade config-ui stack (Node 24, Yarn 4, React 19, AntD 6, Vite 8, TypeScript 6) and regenerate lockfile - migrate to ESLint flat config and add Vitest setup/tests - patch miller-columns-select for React 19 compatibility and add regression coverage - switch app bootstrap to createRoot and apply related type-safety/runtime fixes - make Grafana path/proxy and compose Grafana DB settings configurable for local/devcontainer environments
…sions - upgrade backend dependencies (Go modules, gorm/mysql driver) and pin python/e2e deps - wrap MySQL dialector migrator to prevent duplicate PRIMARY KEY errors on AddColumn - refactor backend Docker setup (new streamlined Dockerfile, remove Dockerfile.local) and align build workflow dockerfile paths - upgrade config-ui stack (Node 24, Yarn 4, React 19, AntD 6, Vite 8, TypeScript 6) and regenerate lockfile - migrate to ESLint flat config and add Vitest setup/tests - patch miller-columns-select for React 19 compatibility and add regression coverage - switch app bootstrap to createRoot and apply related type-safety/runtime fixes - make Grafana path/proxy and compose Grafana DB settings configurable for local/devcontainer environments
…date Wave 1) Pin the previously floating `mysql:8` tag to the reproducible `mysql:8.4.10` (8.4 LTS line) across all developer/datasource compose files so local and devcontainer environments resolve to a deterministic MySQL version: - docker-compose-dev-mysql.yml - docker-compose.datasources.yml - .devcontainer/docker-compose.yml Align the swag CLI install in backend/Dockerfile.server from v1.16.1 to v1.16.6, matching backend/Makefile, backend/Dockerfile, backend/Dockerfile.local and the github.com/swaggo/swag v1.16.6 library in go.mod.
- grafana: bump base image 11.6.2 -> 12.4.5 (latest 12.x stable)
- grafana: drop grafana-piechart-panel install; dashboards use core "piechart"
- grafana: align dashboard pluginVersion metadata 11.6.2 -> 12.4.5 (144 refs, 13 files)
- config-ui: bump nginx-unprivileged 1.30.3 -> 1.31.2
- node.js stays on 24.x LTS (no change); node 26.x deferred
Smoke-tested: grafana image builds and starts, /api/health returns
{"database":"ok","version":"12.4.5"} with 0 error-level logs; dashboard JSON re-validated.
Move the Python runtime from EOL 3.9 (end-of-life 2025-10-31) to the active, security-supported 3.11 line. Runtime-only change: no Python library version bumps — Pydantic v1, SQLModel 0.0.8 and dbt 1.7 all run unchanged on 3.11 (library modernization in next step). Changes: - scripts-local/setup.sh: PYTHON_VERSION_REQUIRED 3.9 → 3.11, add PYTHON_PATCH_VERSION 3.11.9, rename ensure_python_39 → ensure_python_311 (pyenv 3.11.9) and update the call site. - backend/python/uv.sh: default DEVLAKE_PYTHON_VERSION 3.9 → 3.11. - pyproject.toml (pydevlake, azuredevops, fakeplugin): python "~3.9" → ">=3.11,<3.12"; re-locked all three poetry.lock files on 3.11. - backend/Dockerfile.server: python:3.9-slim-bookworm → python:3.11-slim-bookworm. - .devcontainer/devcontainer.json: python feature 3.9 → 3.11; node feature 18 → 24 (aligns with Node 24 LTS runtime, stale-pin fix). - Docs: update dependency_overview.csv (Python runtime + Node rows), restructure DEPENDENCY_UPDATE_PLAN.md into Wave 3a/3b and seed the Wave 4 "drop dbt-mysql → dbt-postgres 1.10" epic, and refresh backend/python/DevelopmentSetup.md. Validation: make build-pydevlake on 3.11.15; pytest 18 passed / 1 skipped (pydevlake + azuredevops + fakeplugin); shell + JSON syntax checks pass.
The Grafana 11→12 bump surfaced runtime regressions in the dashboard image.
Harden the entrypoint and remove broken duplicate dashboards.
entrypoint.sh:
- Resolve the home dashboard path for both layouts (baked-in image uses
/etc/grafana/dashboards/<mode>/Homepage.json, dev compose bind-mounts the
variant folder directly to /etc/grafana/dashboards). A wrong path made
Grafana 12 return HTTP 500 "Failed to load home dashboard".
- Self-heal read-only legacy volumes (uid 472, 0640) for the gid-0 runtime
user, else print an actionable remediation instead of a cryptic SQLite
"attempt to write a readonly database" crash.
- Remove the deprecated Angular grafana-piechart-panel from the data volume
(Grafana 12 dropped Angular support; dashboards use the core piechart).
- Default GF_SECURITY_ADMIN_PASSWORD to admin so the datasource bootstrap API
stops returning HTTP 401, and treat "already exists" as success (no 10x
retry loop on provisioned datasources).
dashboards:
- Remove grafana/dashboards/{mysql,postgresql}/ai-model-roi.json: mislabeled
duplicates of the PostgreSQL "AI Cost-Efficiency" dashboard. In the MySQL
deployment they referenced a non-existent "postgresql" datasource, which
fails the whole dashboard under Grafana 12's Scenes engine; the PostgreSQL
folder also had a duplicate uid collision.
Verified on dev mysql + postgresql compose: Grafana 12.4.5 healthy, home
dashboard 200, all 58 provisioned dashboards load, 0 level=error.
The convertIssues subtask failed with "Error 1406: Data too long for column component" because the domain issues.component column stayed varchar(255). The previous migration (20240813) targeted the wrong column name "components" (plural) and was never registered. - Add migration 20260629 to widen issues.component to text (MySQL + Postgres) - Register the migration in All() and fix the buggy 20240813 column name - Fix unhandled-error lint warnings in mysql_migrator.go - Document pending q_dev aws-sdk-go v1 -> v2 migration in TODO.md
Migrate the Python plugin framework from Pydantic v1 to v2 and bump supporting libraries. This is Wave 3b of the dependency modernization plan, split into two independent changes: — sqlglot bump (Grafana only): - grafana/scripts/requirements.txt: sqlglot 25.29.0 → 30.11.0 — Pydantic v2 / SQLModel migration: - pydantic ^1.10.2 → ^2.13, sqlmodel ^0.0.8 → ^0.0.38 - inflect ^6.0.2 → ^7.5, fire ^0.4.0 → ^0.7 - pytest ^7.2.2 → ^9.1, jsonpointer ^2.3 → ^3.1 Key migration changes: - model.py: @validator → @field_validator(mode='before'), class Config → model_config = ConfigDict(...), sa_column=Column(...) → sa_type=DateTime()/Text(), explicit default=None on all Optional fields - message.py: const=True → Literal, __root__ → RootModel - model_info.py: .schema() → .model_json_schema(), definitions → $defs, flatten Pydantic v2 anyOf for Optional fields to plain "type" string (Go backend's conversion.go requires type as a string, not an array) - extractor.py: __fields__ → model_fields, field.field_info.extra → field_info.json_schema_extra, added _is_optional() annotation check - __init__.py: Field wrapper sets json_schema_extra manually (SQLModel 0.0.38 drops schema_extra from FieldInfo) - Serialization: .json() → .model_dump_json(), .dict() → .model_dump() All 10 pydevlake tests pass. azuredevops and fakeplugin imports verified. dbt-postgres/dbt-mysql pins intentionally untouched
Bump PostgreSQL images to 18.1: - docker-compose-dev-postgresql.yml: postgres:17.2 → postgres:18.1 - backend/test/e2e/remote/docker-compose.test.yml: postgres:14.2-alpine → postgres:18.1-alpine
Image & entrypoint:
- Bump base image to grafana/grafana:13.0.2
- Change PG datasource type from "postgres" to "grafana-postgresql-datasource"
(plugin renamed in Grafana 13)
- Add "database" to jsonData (required by new PG plugin)
- Delete ALL existing datasources by ID on startup (handles read-only
legacy-provisioned datasources from old volumes)
- postgresVersion 1400 → 1800
- Comments "Grafana 12" → "Grafana 12+"
Dashboard JSON files:
- Align pluginVersion to "13.0.2" (1267 occurrences)
- Replace deprecated name-string datasource references with UID-based objects:
- postgresql: {"type": "grafana-postgresql-datasource", "uid": "devlake-postgres-api"}
- mysql: {"type": "mysql", "uid": "devlake-mysql-api"}
Validated on both MySQL (:3001) and PostgreSQL (:3002) compose variants:
- /api/health → version 13.0.2
- Datasource health → "Database Connection OK"
- 58 dashboards load, 0 level=error logs
Updated Dockerfile.local to use build caches for improved build performance.
Refactor Dockerfile to support multi-platform builds and optimize dependencies.
The convertIssueCodeBlocks subtask failed with: Error 1406 (22001): Data too long for column 'component' at row 91 Root cause: The domain layer table cq_issue_code_blocks had component as varchar(256) (GORM default), while SonarQube component paths can exceed this limit. The tool layer had varchar(500) which was also insufficient. Changes: - Change CqIssueCodeBlock.Component to gorm:"type:text" (domain layer) - Change SonarqubeIssueCodeBlock.Component to gorm:"type:text" (tool layer) - Add migration to alter cq_issue_code_blocks.component to TEXT - Add migration to drop index and alter _tool_sonarqube_issue_code_blocks.component to TEXT - Add unit tests verifying long component handling and GORM tag correctness - Add E2E test reproducing the original error with >256 and >500 char paths
- Migration scripts: safely drop indexes before changing component columns to TEXT, ignoring errors if index does not exist (supports both MySQL and PostgreSQL) - CI build workflow: use Dockerfile.server instead of Dockerfile
# Conflicts: # .gitignore # config-ui/src/plugins/components/connection-form/index.tsx
The fix_versions column in _tool_jira_issues was originally created as varchar(255) by migration 20250619, but the JiraIssue model was later changed to type:text without a corresponding migration to alter the existing column. On existing databases the column stayed varchar(255), causing the extractIssues subtask to fail when an issue's joined fix-version names exceed 255 characters: Error 1406 (22001): Data too long for column 'fix_versions' Add migration 20260707_change_fix_versions_to_text to alter the column to text, matching the model definition and the domain-layer issues table, and register it in migrationscripts/register.go.
… main Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # backend/plugins/jira/models/migrationscripts/register.go
Signed-off-by: DoDiODev <DoDiDev@proton.me> # Conflicts: # backend/core/models/migrationscripts/register.go # backend/plugins/sonarqube/models/migrationscripts/register.go # docker-compose-dev-postgresql.yml
Update the pinned config-ui dependencies to their current releases and regenerate yarn.lock: - antd 6.4.5 -> 6.6.0 - @ant-design/icons 6.2.5 -> 6.3.2 - cron-parser 5.6.0 -> 5.8.1 - react-router-dom 7.18.0 -> 7.18.2 - styled-components 6.4.2 -> 6.5.1 - eslint 10.5.0 -> 10.8.1 - prettier 3.8.4 -> 3.9.6 - vite 8.1.0 -> 8.2.1 - vitest 4.1.9 -> 4.1.10 All target versions stay within their current major line and keep their declared engines/peerDependencies compatible with the pinned Node 24 and React 19 versions. TypeScript is deliberately left at 6.0.3, since 7.x is a major upgrade that deserves its own change. Validated locally with yarn install, yarn build, yarn test and yarn lint. Signed-off-by: DoDiODev <DoDiDev@proton.me>
Container-Image- und Frontend-Refresh (PR-Branch pr/wave4i-a, Basis upstream/main @ 7df8ba0): - Container-Images: Grafana 13.0.2 -> 13.1.3, PostgreSQL 18.1 -> 18.4 (2 Stellen), MySQL 8.4.10 -> 8.4.11 (5 Stellen), nginx-unprivileged 1.31.2 -> 1.31.3 - Node-Pin (volta): 24.17.0 -> 24.19.0 - Frontend-Deps: antd 6.6.0, @ant-design/icons 6.3.2, cron-parser 5.8.1, react-router-dom 7.18.2, styled-components 6.5.1, eslint 10.8.1, prettier 3.9.6, vite 8.2.1, vitest 4.1.10 + regeneriertes yarn.lock Bewusst nicht enthalten: TypeScript 7 (eigene Welle), Yarn 4.18.0 (4 Stellen + 3-MB-Release-Bundle), @playwright/test, Dashboard-JSONs, go.mod/go.sum. Validiert: yarn install/build/test/lint, config-ui-Image-Build, beide Compose-Stacks (PG 18.4 / MySQL 8.4.11) mit Grafana 13.1.3 inkl. Datasource-Health und Dashboard-Provisionierung; Fork-CI 8/8 gruen. Merge ohne Konflikte (versionUpgrade hatte keine eigenen Aenderungen an den betroffenen Dateien).
The named volumes of the `devlake-mysql` / `devlake-postgresql` compose projects hold the entire local configuration state (connections, blueprints, projects, scope configs, collected data, Grafana state). There is no automatic backup and no way to recover a removed Docker volume. A verification run that cleaned up with `docker compose down -v` on the *dev* project name wiped that state. Document the rule (use a dedicated `-p` project name for throwaway verification stacks) and point at the backup helper. Signed-off-by: DoDiODev <DoDiDev@proton.me>
Signed-off-by: DoDiODev <DoDiDev@proton.me> (cherry picked from commit a6bd93d)
Signed-off-by: DoDiODev <DoDiDev@proton.me> # Conflicts: # .github/workflows/golangci-lint.yml # .github/workflows/test.yml
Signed-off-by: DoDiODev <DoDiDev@proton.me> (cherry picked from commit 5c509c7)
Signed-off-by: DoDiODev <DoDiDev@proton.me>
# Conflicts: # .github/workflows/golangci-lint.yml # .github/workflows/test.yml # backend/.mockery.core.yml # backend/.mockery.helpers.yml # backend/Makefile # devops/docker/lake-builder/Dockerfile
The go-git based collector (used when UseGoGit is set) had no test coverage at all, which makes any go-git upgrade a leap of faith. Adds a test that builds a repository on disk with a known shape -- three commits, one tag, a second branch checked out so HEAD is not the default branch -- and asserts CountCommits, CountTags and CountBranches, that the counting helpers stop on a cancelled context, and that opening a directory which is not a repository fails. The test passes both on go-git v5.12.0 and on the version this branch upgrades to, so it documents unchanged behaviour across the bump. Signed-off-by: DoDiODev <DoDiDev@proton.me>
github.com/go-git/go-git/v5 v5.12.0 -> v5.19.2 github.com/go-git/go-billy/v5 v5.5.0 -> v5.9.0 (pulled in) github.com/cloudflare/circl v1.3.7 -> v1.6.3 (pulled in) govulncheck reports 12 of these findings as reachable from this code base before the bump (8 in go-git, 2 in go-billy, 2 in circl) and none after. The go-git ones include GO-2025-3367 and GO-2025-3368 (argument injection through crafted URLs) and GO-2026-4909/4910. go-git is used in exactly one place: plugins/gitextractor/parser/repo_gogit.go opens an already cloned repository with PlainOpen and walks commits, tags, branches and trees. Cloning is done by the git CLI (parser.NewGitcliCloner), so the go-git transports are not on the path -- this bump does not change how DevLake talks to remotes. No source change was required. Verified with the collector test added in the previous commit, which passes on both the old and the new version. Minimal version selection also pulls up golang.org/x/crypto, x/net, x/sys, x/text, x/tools, ProtonMail/go-crypto, pjbgf/sha1cd, skeema/knownhosts, cyphar/filepath-securejoin, Microsoft/go-winio, klauspost/cpuid and golang/protobuf. Signed-off-by: DoDiODev <DoDiDev@proton.me>
golang.org/x/crypto v0.41.0 -> v0.55.0 (15 CVEs, 7 CRITICAL) golang.org/x/net v0.43.0 -> v0.57.0 (CVE-2026-25680) golang.org/x/oauth2 v0.13.0 -> v0.36.0 (CVE-2025-22868) 17 known CVEs in total. govulncheck confirms 7 of them are reachable from this code base: 6 in x/crypto and 1 in x/text, which minimal version selection pulls up along with the rest of the golang.org/x set (x/sys, x/term, x/sync, x/mod, x/tools). After the bump govulncheck reports none of them. The reachable x/crypto findings are in the SSH stack and are reached through core/utils/io.go, where archives are copied with github.com/viant/afs -- that library registers an SSH/SCP backend, so ssh.Dial, ssh.ParsePrivateKey and the agent signer end up in the build graph. They are not reached through gitextractor: cloning is done by the git CLI, not by a Go SSH client. x/oauth2 is used directly by server/api/auth and helpers/oidchelper; its finding is a malformed token that allocates without bound during parsing. No source change was required -- none of the bumped packages changed an API surface this code base touches. Signed-off-by: DoDiODev <DoDiDev@proton.me>
… and protobuf github.com/golang-jwt/jwt/v5 v5.0.0-rc.1 -> v5.3.1 github.com/sirupsen/logrus v1.9.0 -> v1.9.4 google.golang.org/protobuf v1.31.0 -> v1.36.12 jwt/v5 was pinned to a release candidate published in March 2023. Besides being a pre-release in a security-sensitive position, v5.0.0-rc.1 is affected by CVE-2025-30204 (HIGH), which govulncheck reports as reachable from this code base. v5.3.1 is the current stable release. jwt/v5 is used by the GitHub App token flow (plugins/github/token, plugins/github/models/connection.go), the Teambition connection and the OIDC session helper (helpers/oidchelper/session.go). Moving from the RC to stable needed no source change; the signing, parsing and claims APIs used here are unchanged. logrus v1.9.0 (CVE-2025-65637, HIGH) and protobuf v1.31.0 (CVE-2024-24786) are bumped to current releases in the same pass. Verified with the unit tests of the affected packages, including server/api/auth (JWT session handling) and helpers/oidchelper. Signed-off-by: DoDiODev <DoDiDev@proton.me>
# Conflicts: # backend/go.mod # backend/go.sum
# Conflicts: # backend/go.mod # backend/go.sum
…tal filter apache#8959 changed `GetMergeRequestsIterator` to filter merge requests by `GREATEST(gmr.gitlab_updated_at, COALESCE(gmr.commit_updated_at, gmr.gitlab_updated_at))`, but `commit_updated_at` does not exist on `_tool_gitlab_merge_requests`: there is neither a model field nor a migration script, and nothing ever writes the value. As a result every incremental "Collect MR Notes" run (the only caller that passes a stateful collector; the MR commit collector passes nil) aborts with: Error 1054 (42S22): Unknown column 'gmr.commit_updated_at' in 'where clause' This makes the intended behaviour actually work instead of reverting it: * add `CommitUpdatedAt` to `GitlabMergeRequest` plus a migration script * maintain the column in the MR commit extractor, setting it to the latest authored date of the MR's commits, so MRs that received new commits without their own `updated_at` being bumped (e.g. force-pushes) are picked up again * the collector filter is unchanged and now resolves against a real column Verified on MySQL 8 and PostgreSQL 14 (extractor UPDATE and the GREATEST filter produce identical results on both).
The `generate mock` step of the `golangci-lint` workflow fails with
ERR encountered error when loading package
error="-: error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping."
make: *** [Makefile:80: mock] Error 1
The job runs inside `mericodev/lake-builder:latest` as root, while the
checkout is owned by the runner user, so git refuses to operate on the
repository ("dubious ownership") and Go's VCS stamping aborts while
mockery loads the packages.
`test.yml` and `test-e2e.yml` already run
`git config --global --add safe.directory $(pwd)` right after the
checkout for exactly this reason, which is why the unit-test and e2e
jobs pass while lint does not. Add the same step to the lint job.
…e#9054 Re-checked before opening the upstream PR: x/net moved on to v0.58.0 and logrus to v1.10.0 since the wave was prepared on 2026-08-12. Signed-off-by: DoDiODev <DoDiDev@proton.me>
# Conflicts: # backend/go.mod # backend/go.sum
Signed-off-by: DoDiODev <DoDiDev@proton.me>
…ps/fakeplugin
certifi and pytest are bumped in the two direct pyproject.toml ranges that
still pinned pre-CVE-fix versions (certifi ^2023.7.22, pytest ^7.2.x); all
three Poetry lockfiles are regenerated with the project-pinned Poetry 2.4.1
to stay consistent with their pyproject.toml (poetry export/lock already
required `poetry lock` on upstream/main).
- backend/python/pydevlake/pyproject.toml: certifi ^2023.7.22 -> ^2026.7.22,
pytest (main + dev group) ^7.2.x -> ^9.1.1
- backend/python/plugins/azuredevops/pyproject.toml: dev pytest ^7.2.2 ->
^9.1.1
- backend/python/{pydevlake,plugins/azuredevops,test/fakeplugin}/poetry.lock:
regenerated (`poetry lock --regenerate`)
No Pydantic v2/SQLModel/runtime/dbt changes; this is a lockfile-security-only
refresh, independent of apache#8970 and the later Python 3.14 wave.
Validation (Python 3.11.15, Poetry 2.4.1):
- `poetry check --lock` clean in all three projects
- `pip-audit` against the exported lock graphs: 0 known findings in all
three (only the expected skip for the local, non-PyPI package `pydevlake`)
- `make build-pydevlake` and `make unit-test-python`: pydevlake 10 passed,
azuredevops 8 passed / 1 skipped
- azuredevops remote plugin starts (`run.sh --help` lists collect/convert/
extract/make_pipeline/plugin_info/remote_scopes/test_connection)
…curity) Not a blind merge: versionUpgrade already carries the fork-only Pydantic v2 / SQLModel migration (pydantic ^2.13, sqlmodel ^0.0.38, etc.), which upstream's topic branch does not have yet. Conflicts in all 5 touched files were resolved by keeping the higher/integrated stand and re-applying only the 4n range bumps on top of it: - certifi ^2023.7.22 -> ^2026.7.22 (pydevlake) - pytest ^9.1 -> ^9.1.1 (pydevlake main + dev group, azuredevops dev group; functionally a no-op tightening since ^9.1 already resolved to 9.1.1, but keeps the explicit range consistent with the topic branch's intent) - all three poetry.lock files regenerated from the resolved pyproject.toml on the integration stand (`poetry lock --regenerate`, Poetry 2.4.1) Validation on versionUpgrade (Python 3.11.15, Poetry 2.4.1): - `poetry check --lock` clean in all three projects - `pip-audit` against the exported lock graphs: 0 known findings in all three (only the expected skip for the local, non-PyPI package `pydevlake`) - `make build-pydevlake` and `make unit-test-python`: pydevlake 10 passed, azuredevops 8 passed / 1 skipped (identical to the upstream topic branch) - azuredevops remote plugin starts (`run.sh --help`) Topic commit: f123c16 (pr/wave4n-python-lock-security, origin/DoDiODev).
# Conflicts: # backend/go.mod # backend/go.sum
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Throwaway validation only; no production change intended.
Verifies fork-ci and pull-request workflow triggering for the Wave 4p fixture commit. This PR will be deleted after observing the checks.