diff --git a/.claude-plugin/DEPRECATED.md b/.claude-plugin/DEPRECATED.md new file mode 100644 index 00000000..f4e3b663 --- /dev/null +++ b/.claude-plugin/DEPRECATED.md @@ -0,0 +1,57 @@ +# DEPRECATED — the `databricks-ai-dev-kit` Claude Code plugin + +The **`databricks-ai-dev-kit`** Claude Code plugin published by *this repo* is +**deprecated and no longer published**. It has been removed from the marketplace +(`marketplace.json` lists no plugins) and `plugin.json` is marked +`"deprecated": true`. These files are kept only for historical reference. + +> **This is not a move away from plugins.** Skills are still delivered as a plugin +> for the agents that support one — just the *official* `databricks` plugin managed +> by the Databricks CLI, not this repo's marketplace plugin. See "How skills install +> now" below. + +## Why + +- Skills are now delivered through the Databricks CLI: `databricks aitools install` + (Databricks CLI v1.0.0+), backed by + [github.com/databricks/databricks-agent-skills](https://github.com/databricks/databricks-agent-skills). + The main installer (`install.sh` / `install.ps1`) already delegates to it. +- The CLI owns the install lifecycle (`install` / `update` / `uninstall` / `list`), + so skills no longer need a repo-published marketplace plugin to be bundled and + auto-loaded. + +## How skills install now + +`databricks aitools install` decides delivery per agent: + +- **Claude Code, Codex, GitHub Copilot** — installs the official **`databricks` + plugin** through each agent's own plugin CLI (a *different* plugin from the retired + `databricks-ai-dev-kit` one). +- **Cursor, OpenCode, Antigravity** — writes **raw skill files** (no headless plugin + install available). +- `--skills-only` forces raw skill files for every agent. + +So on Claude Code you still get a plugin — the CLI-managed `databricks` plugin — +rather than the deprecated `databricks-ai-dev-kit` plugin. + +## Migration for existing plugin users + +Use either of the following to get the up-to-date skills: + +- Run the installer: + + ```bash + bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) + ``` + +- Or install directly via the Databricks CLI (v1.0.0+): + + ```bash + databricks aitools install + ``` + +## Notes + +- The frozen historical copies of the bundled skills live at git tag `v0.1.13`. +- Files under `.claude-plugin/` remain in the repo for reference only; they are + no longer installed or published. diff --git a/.claude-plugin/check_update.sh b/.claude-plugin/check_update.sh index e7a744c1..74fbf4b2 100755 --- a/.claude-plugin/check_update.sh +++ b/.claude-plugin/check_update.sh @@ -1,4 +1,9 @@ #!/bin/bash +# DEPRECATED: The databricks-ai-dev-kit Claude Code plugin is no longer published +# (see .claude-plugin/DEPRECATED.md). Install skills via install.sh or +# `databricks aitools install`. This script is kept for reference and still runs +# if invoked directly. +# # Version update check for Databricks AI Dev Kit. # Stdout from this script is injected as context Claude can see at session start. # Silent on success (up to date) or failure (network error, missing files). @@ -47,32 +52,18 @@ if [ -z "$remote_ver" ]; then fi fi -# If versions differ, output a message for Claude to relay to the user +# If versions differ, emit an informational update notice for Claude to relay. +# Deliberately avoids urgent/imperative phrasing so it does not alarm users. +# This is a routine, first-party update notice. if [ -n "$remote_ver" ] && [ "$remote_ver" != "$local_ver" ]; then cat < set[str]: """Get actual skill directories on the filesystem.""" return { - d.name for d in SKILLS_DIR.iterdir() if d.is_dir() and d.name not in SKIP_DIRS and not d.name.startswith(".") + d.name for d in SKILL_DIRS_ROOT.iterdir() if d.is_dir() and d.name not in SKIP_DIRS and not d.name.startswith(".") } @@ -120,7 +125,7 @@ def main() -> int: actual_skills = get_local_skill_dirs() # --- Validate each skill directory's SKILL.md and frontmatter --- - for skill_dir in sorted(SKILLS_DIR.iterdir()): + for skill_dir in sorted(SKILL_DIRS_ROOT.iterdir()): if not skill_dir.is_dir(): continue if skill_dir.name in SKIP_DIRS or skill_dir.name.startswith("."): diff --git a/DEPRECATED-databricks-skills/README.md b/DEPRECATED-databricks-skills/README.md new file mode 100644 index 00000000..947c6f4b --- /dev/null +++ b/DEPRECATED-databricks-skills/README.md @@ -0,0 +1,46 @@ +# Deprecated: Bundled Databricks Skills (frozen) + +> **These skill copies are deprecated and no longer maintained.** + +The skill folders under `DEPRECATED-databricks-skills/` are frozen, historical copies +of the Databricks skills that used to ship in this repository. They are kept only so +that existing tooling (and anyone who copied them directly) does not break. They will +**not** receive updates. + +## Use the supported path instead + +Skills are now installed and maintained through the Databricks CLI: + +```bash +databricks aitools install +``` + +This requires Databricks CLI **v1.0.0+** and pulls the current, maintained skills from +[github.com/databricks/databricks-agent-skills](https://github.com/databricks/databricks-agent-skills). +The AI Dev Kit installer (`install.sh` / `install.ps1`) already delegates to it, so a normal +install gets you the up-to-date skills with no manual copying. + +## Getting the exact historical content + +If you specifically need the files exactly as they were bundled, check out the last release +that shipped them: + +```bash +git checkout v0.1.13 -- databricks-skills +``` + +## Migration notes + +- Some skills were **renamed** when they moved to `databricks-agent-skills`. The mapping and + other breaking changes are documented in the root [`README.md`](../README.md) "breaking change" note. +- APX skills has been removed in favor of other app skills (see the root README). + +## Genie Code (workspace upload) +Genie Code already has coverage for most skills available publicly and we will continue to sync improvements +whenever possible. The one flow `databricks aitools install` does not fully cover is uploading skills to a +Databricks workspace for **Genie Code**. Keep that simple — do not clone this tree to copy files: + +- **Recommended:** run the notebook uploader + [`../install_genie_code_skills.py`](../install_genie_code_skills.py) + in your workspace. It downloads skills from GitHub (pinned to release `v0.1.13`) and uploads them + via the SDK — no local clone needed. diff --git a/databricks-skills/databricks-agent-bricks/1-knowledge-assistants.md b/DEPRECATED-databricks-skills/databricks-agent-bricks/1-knowledge-assistants.md similarity index 100% rename from databricks-skills/databricks-agent-bricks/1-knowledge-assistants.md rename to DEPRECATED-databricks-skills/databricks-agent-bricks/1-knowledge-assistants.md diff --git a/databricks-skills/databricks-agent-bricks/2-supervisor-agents.md b/DEPRECATED-databricks-skills/databricks-agent-bricks/2-supervisor-agents.md similarity index 100% rename from databricks-skills/databricks-agent-bricks/2-supervisor-agents.md rename to DEPRECATED-databricks-skills/databricks-agent-bricks/2-supervisor-agents.md diff --git a/databricks-skills/databricks-agent-bricks/SKILL.md b/DEPRECATED-databricks-skills/databricks-agent-bricks/SKILL.md similarity index 100% rename from databricks-skills/databricks-agent-bricks/SKILL.md rename to DEPRECATED-databricks-skills/databricks-agent-bricks/SKILL.md diff --git a/databricks-skills/databricks-ai-functions/1-task-functions.md b/DEPRECATED-databricks-skills/databricks-ai-functions/1-task-functions.md similarity index 100% rename from databricks-skills/databricks-ai-functions/1-task-functions.md rename to DEPRECATED-databricks-skills/databricks-ai-functions/1-task-functions.md diff --git a/databricks-skills/databricks-ai-functions/2-ai-query.md b/DEPRECATED-databricks-skills/databricks-ai-functions/2-ai-query.md similarity index 100% rename from databricks-skills/databricks-ai-functions/2-ai-query.md rename to DEPRECATED-databricks-skills/databricks-ai-functions/2-ai-query.md diff --git a/databricks-skills/databricks-ai-functions/3-ai-forecast.md b/DEPRECATED-databricks-skills/databricks-ai-functions/3-ai-forecast.md similarity index 100% rename from databricks-skills/databricks-ai-functions/3-ai-forecast.md rename to DEPRECATED-databricks-skills/databricks-ai-functions/3-ai-forecast.md diff --git a/databricks-skills/databricks-ai-functions/SKILL.md b/DEPRECATED-databricks-skills/databricks-ai-functions/SKILL.md similarity index 100% rename from databricks-skills/databricks-ai-functions/SKILL.md rename to DEPRECATED-databricks-skills/databricks-ai-functions/SKILL.md diff --git a/databricks-skills/databricks-aibi-dashboards/1-widget-specifications.md b/DEPRECATED-databricks-skills/databricks-aibi-dashboards/1-widget-specifications.md similarity index 100% rename from databricks-skills/databricks-aibi-dashboards/1-widget-specifications.md rename to DEPRECATED-databricks-skills/databricks-aibi-dashboards/1-widget-specifications.md diff --git a/databricks-skills/databricks-aibi-dashboards/2-advanced-widget-specifications.md b/DEPRECATED-databricks-skills/databricks-aibi-dashboards/2-advanced-widget-specifications.md similarity index 100% rename from databricks-skills/databricks-aibi-dashboards/2-advanced-widget-specifications.md rename to DEPRECATED-databricks-skills/databricks-aibi-dashboards/2-advanced-widget-specifications.md diff --git a/databricks-skills/databricks-aibi-dashboards/3-examples.md b/DEPRECATED-databricks-skills/databricks-aibi-dashboards/3-examples.md similarity index 100% rename from databricks-skills/databricks-aibi-dashboards/3-examples.md rename to DEPRECATED-databricks-skills/databricks-aibi-dashboards/3-examples.md diff --git a/databricks-skills/databricks-aibi-dashboards/3-filters.md b/DEPRECATED-databricks-skills/databricks-aibi-dashboards/3-filters.md similarity index 100% rename from databricks-skills/databricks-aibi-dashboards/3-filters.md rename to DEPRECATED-databricks-skills/databricks-aibi-dashboards/3-filters.md diff --git a/databricks-skills/databricks-aibi-dashboards/4-examples.md b/DEPRECATED-databricks-skills/databricks-aibi-dashboards/4-examples.md similarity index 100% rename from databricks-skills/databricks-aibi-dashboards/4-examples.md rename to DEPRECATED-databricks-skills/databricks-aibi-dashboards/4-examples.md diff --git a/databricks-skills/databricks-aibi-dashboards/5-troubleshooting.md b/DEPRECATED-databricks-skills/databricks-aibi-dashboards/5-troubleshooting.md similarity index 100% rename from databricks-skills/databricks-aibi-dashboards/5-troubleshooting.md rename to DEPRECATED-databricks-skills/databricks-aibi-dashboards/5-troubleshooting.md diff --git a/databricks-skills/databricks-aibi-dashboards/SKILL.md b/DEPRECATED-databricks-skills/databricks-aibi-dashboards/SKILL.md similarity index 100% rename from databricks-skills/databricks-aibi-dashboards/SKILL.md rename to DEPRECATED-databricks-skills/databricks-aibi-dashboards/SKILL.md diff --git a/databricks-skills/databricks-apps-python/1-authorization.md b/DEPRECATED-databricks-skills/databricks-apps-python/1-authorization.md similarity index 100% rename from databricks-skills/databricks-apps-python/1-authorization.md rename to DEPRECATED-databricks-skills/databricks-apps-python/1-authorization.md diff --git a/databricks-skills/databricks-apps-python/2-app-resources.md b/DEPRECATED-databricks-skills/databricks-apps-python/2-app-resources.md similarity index 100% rename from databricks-skills/databricks-apps-python/2-app-resources.md rename to DEPRECATED-databricks-skills/databricks-apps-python/2-app-resources.md diff --git a/databricks-skills/databricks-apps-python/3-frameworks.md b/DEPRECATED-databricks-skills/databricks-apps-python/3-frameworks.md similarity index 100% rename from databricks-skills/databricks-apps-python/3-frameworks.md rename to DEPRECATED-databricks-skills/databricks-apps-python/3-frameworks.md diff --git a/databricks-skills/databricks-apps-python/4-deployment.md b/DEPRECATED-databricks-skills/databricks-apps-python/4-deployment.md similarity index 100% rename from databricks-skills/databricks-apps-python/4-deployment.md rename to DEPRECATED-databricks-skills/databricks-apps-python/4-deployment.md diff --git a/databricks-skills/databricks-apps-python/5-lakebase.md b/DEPRECATED-databricks-skills/databricks-apps-python/5-lakebase.md similarity index 100% rename from databricks-skills/databricks-apps-python/5-lakebase.md rename to DEPRECATED-databricks-skills/databricks-apps-python/5-lakebase.md diff --git a/databricks-skills/databricks-apps-python/6-mcp-approach.md b/DEPRECATED-databricks-skills/databricks-apps-python/6-mcp-approach.md similarity index 100% rename from databricks-skills/databricks-apps-python/6-mcp-approach.md rename to DEPRECATED-databricks-skills/databricks-apps-python/6-mcp-approach.md diff --git a/databricks-skills/databricks-apps-python/SKILL.md b/DEPRECATED-databricks-skills/databricks-apps-python/SKILL.md similarity index 100% rename from databricks-skills/databricks-apps-python/SKILL.md rename to DEPRECATED-databricks-skills/databricks-apps-python/SKILL.md diff --git a/databricks-skills/databricks-apps-python/examples/fm-minimal-chat.py b/DEPRECATED-databricks-skills/databricks-apps-python/examples/fm-minimal-chat.py similarity index 100% rename from databricks-skills/databricks-apps-python/examples/fm-minimal-chat.py rename to DEPRECATED-databricks-skills/databricks-apps-python/examples/fm-minimal-chat.py diff --git a/databricks-skills/databricks-apps-python/examples/fm-parallel-calls.py b/DEPRECATED-databricks-skills/databricks-apps-python/examples/fm-parallel-calls.py similarity index 100% rename from databricks-skills/databricks-apps-python/examples/fm-parallel-calls.py rename to DEPRECATED-databricks-skills/databricks-apps-python/examples/fm-parallel-calls.py diff --git a/databricks-skills/databricks-apps-python/examples/fm-structured-outputs.py b/DEPRECATED-databricks-skills/databricks-apps-python/examples/fm-structured-outputs.py similarity index 100% rename from databricks-skills/databricks-apps-python/examples/fm-structured-outputs.py rename to DEPRECATED-databricks-skills/databricks-apps-python/examples/fm-structured-outputs.py diff --git a/databricks-skills/databricks-apps-python/examples/llm_config.py b/DEPRECATED-databricks-skills/databricks-apps-python/examples/llm_config.py similarity index 100% rename from databricks-skills/databricks-apps-python/examples/llm_config.py rename to DEPRECATED-databricks-skills/databricks-apps-python/examples/llm_config.py diff --git a/databricks-skills/databricks-bundles/SDP_guidance.md b/DEPRECATED-databricks-skills/databricks-bundles/SDP_guidance.md similarity index 100% rename from databricks-skills/databricks-bundles/SDP_guidance.md rename to DEPRECATED-databricks-skills/databricks-bundles/SDP_guidance.md diff --git a/databricks-skills/databricks-bundles/SKILL.md b/DEPRECATED-databricks-skills/databricks-bundles/SKILL.md similarity index 100% rename from databricks-skills/databricks-bundles/SKILL.md rename to DEPRECATED-databricks-skills/databricks-bundles/SKILL.md diff --git a/databricks-skills/databricks-bundles/alerts_guidance.md b/DEPRECATED-databricks-skills/databricks-bundles/alerts_guidance.md similarity index 100% rename from databricks-skills/databricks-bundles/alerts_guidance.md rename to DEPRECATED-databricks-skills/databricks-bundles/alerts_guidance.md diff --git a/databricks-skills/databricks-config/SKILL.md b/DEPRECATED-databricks-skills/databricks-config/SKILL.md similarity index 100% rename from databricks-skills/databricks-config/SKILL.md rename to DEPRECATED-databricks-skills/databricks-config/SKILL.md diff --git a/databricks-skills/databricks-dbsql/SKILL.md b/DEPRECATED-databricks-skills/databricks-dbsql/SKILL.md similarity index 100% rename from databricks-skills/databricks-dbsql/SKILL.md rename to DEPRECATED-databricks-skills/databricks-dbsql/SKILL.md diff --git a/databricks-skills/databricks-dbsql/ai-functions.md b/DEPRECATED-databricks-skills/databricks-dbsql/ai-functions.md similarity index 100% rename from databricks-skills/databricks-dbsql/ai-functions.md rename to DEPRECATED-databricks-skills/databricks-dbsql/ai-functions.md diff --git a/databricks-skills/databricks-dbsql/best-practices.md b/DEPRECATED-databricks-skills/databricks-dbsql/best-practices.md similarity index 100% rename from databricks-skills/databricks-dbsql/best-practices.md rename to DEPRECATED-databricks-skills/databricks-dbsql/best-practices.md diff --git a/databricks-skills/databricks-dbsql/geospatial-collations.md b/DEPRECATED-databricks-skills/databricks-dbsql/geospatial-collations.md similarity index 100% rename from databricks-skills/databricks-dbsql/geospatial-collations.md rename to DEPRECATED-databricks-skills/databricks-dbsql/geospatial-collations.md diff --git a/databricks-skills/databricks-dbsql/materialized-views-pipes.md b/DEPRECATED-databricks-skills/databricks-dbsql/materialized-views-pipes.md similarity index 100% rename from databricks-skills/databricks-dbsql/materialized-views-pipes.md rename to DEPRECATED-databricks-skills/databricks-dbsql/materialized-views-pipes.md diff --git a/databricks-skills/databricks-dbsql/sql-scripting.md b/DEPRECATED-databricks-skills/databricks-dbsql/sql-scripting.md similarity index 100% rename from databricks-skills/databricks-dbsql/sql-scripting.md rename to DEPRECATED-databricks-skills/databricks-dbsql/sql-scripting.md diff --git a/databricks-skills/databricks-docs/SKILL.md b/DEPRECATED-databricks-skills/databricks-docs/SKILL.md similarity index 100% rename from databricks-skills/databricks-docs/SKILL.md rename to DEPRECATED-databricks-skills/databricks-docs/SKILL.md diff --git a/databricks-skills/databricks-execution-compute/SKILL.md b/DEPRECATED-databricks-skills/databricks-execution-compute/SKILL.md similarity index 100% rename from databricks-skills/databricks-execution-compute/SKILL.md rename to DEPRECATED-databricks-skills/databricks-execution-compute/SKILL.md diff --git a/databricks-skills/databricks-execution-compute/references/1-databricks-connect.md b/DEPRECATED-databricks-skills/databricks-execution-compute/references/1-databricks-connect.md similarity index 100% rename from databricks-skills/databricks-execution-compute/references/1-databricks-connect.md rename to DEPRECATED-databricks-skills/databricks-execution-compute/references/1-databricks-connect.md diff --git a/databricks-skills/databricks-execution-compute/references/2-serverless-job.md b/DEPRECATED-databricks-skills/databricks-execution-compute/references/2-serverless-job.md similarity index 100% rename from databricks-skills/databricks-execution-compute/references/2-serverless-job.md rename to DEPRECATED-databricks-skills/databricks-execution-compute/references/2-serverless-job.md diff --git a/databricks-skills/databricks-execution-compute/references/3-interactive-cluster.md b/DEPRECATED-databricks-skills/databricks-execution-compute/references/3-interactive-cluster.md similarity index 100% rename from databricks-skills/databricks-execution-compute/references/3-interactive-cluster.md rename to DEPRECATED-databricks-skills/databricks-execution-compute/references/3-interactive-cluster.md diff --git a/databricks-skills/databricks-genie/SKILL.md b/DEPRECATED-databricks-skills/databricks-genie/SKILL.md similarity index 100% rename from databricks-skills/databricks-genie/SKILL.md rename to DEPRECATED-databricks-skills/databricks-genie/SKILL.md diff --git a/databricks-skills/databricks-genie/conversation.md b/DEPRECATED-databricks-skills/databricks-genie/conversation.md similarity index 100% rename from databricks-skills/databricks-genie/conversation.md rename to DEPRECATED-databricks-skills/databricks-genie/conversation.md diff --git a/databricks-skills/databricks-genie/spaces.md b/DEPRECATED-databricks-skills/databricks-genie/spaces.md similarity index 100% rename from databricks-skills/databricks-genie/spaces.md rename to DEPRECATED-databricks-skills/databricks-genie/spaces.md diff --git a/databricks-skills/databricks-iceberg/1-managed-iceberg-tables.md b/DEPRECATED-databricks-skills/databricks-iceberg/1-managed-iceberg-tables.md similarity index 100% rename from databricks-skills/databricks-iceberg/1-managed-iceberg-tables.md rename to DEPRECATED-databricks-skills/databricks-iceberg/1-managed-iceberg-tables.md diff --git a/databricks-skills/databricks-iceberg/2-uniform-and-compatibility.md b/DEPRECATED-databricks-skills/databricks-iceberg/2-uniform-and-compatibility.md similarity index 100% rename from databricks-skills/databricks-iceberg/2-uniform-and-compatibility.md rename to DEPRECATED-databricks-skills/databricks-iceberg/2-uniform-and-compatibility.md diff --git a/databricks-skills/databricks-iceberg/3-iceberg-rest-catalog.md b/DEPRECATED-databricks-skills/databricks-iceberg/3-iceberg-rest-catalog.md similarity index 100% rename from databricks-skills/databricks-iceberg/3-iceberg-rest-catalog.md rename to DEPRECATED-databricks-skills/databricks-iceberg/3-iceberg-rest-catalog.md diff --git a/databricks-skills/databricks-iceberg/4-snowflake-interop.md b/DEPRECATED-databricks-skills/databricks-iceberg/4-snowflake-interop.md similarity index 100% rename from databricks-skills/databricks-iceberg/4-snowflake-interop.md rename to DEPRECATED-databricks-skills/databricks-iceberg/4-snowflake-interop.md diff --git a/databricks-skills/databricks-iceberg/5-external-engine-interop.md b/DEPRECATED-databricks-skills/databricks-iceberg/5-external-engine-interop.md similarity index 100% rename from databricks-skills/databricks-iceberg/5-external-engine-interop.md rename to DEPRECATED-databricks-skills/databricks-iceberg/5-external-engine-interop.md diff --git a/databricks-skills/databricks-iceberg/SKILL.md b/DEPRECATED-databricks-skills/databricks-iceberg/SKILL.md similarity index 100% rename from databricks-skills/databricks-iceberg/SKILL.md rename to DEPRECATED-databricks-skills/databricks-iceberg/SKILL.md diff --git a/databricks-skills/databricks-jobs/SKILL.md b/DEPRECATED-databricks-skills/databricks-jobs/SKILL.md similarity index 100% rename from databricks-skills/databricks-jobs/SKILL.md rename to DEPRECATED-databricks-skills/databricks-jobs/SKILL.md diff --git a/databricks-skills/databricks-jobs/examples.md b/DEPRECATED-databricks-skills/databricks-jobs/examples.md similarity index 100% rename from databricks-skills/databricks-jobs/examples.md rename to DEPRECATED-databricks-skills/databricks-jobs/examples.md diff --git a/databricks-skills/databricks-jobs/notifications-monitoring.md b/DEPRECATED-databricks-skills/databricks-jobs/notifications-monitoring.md similarity index 100% rename from databricks-skills/databricks-jobs/notifications-monitoring.md rename to DEPRECATED-databricks-skills/databricks-jobs/notifications-monitoring.md diff --git a/databricks-skills/databricks-jobs/task-types.md b/DEPRECATED-databricks-skills/databricks-jobs/task-types.md similarity index 100% rename from databricks-skills/databricks-jobs/task-types.md rename to DEPRECATED-databricks-skills/databricks-jobs/task-types.md diff --git a/databricks-skills/databricks-jobs/triggers-schedules.md b/DEPRECATED-databricks-skills/databricks-jobs/triggers-schedules.md similarity index 100% rename from databricks-skills/databricks-jobs/triggers-schedules.md rename to DEPRECATED-databricks-skills/databricks-jobs/triggers-schedules.md diff --git a/databricks-skills/databricks-lakebase-autoscale/SKILL.md b/DEPRECATED-databricks-skills/databricks-lakebase-autoscale/SKILL.md similarity index 100% rename from databricks-skills/databricks-lakebase-autoscale/SKILL.md rename to DEPRECATED-databricks-skills/databricks-lakebase-autoscale/SKILL.md diff --git a/databricks-skills/databricks-lakebase-autoscale/connections.md b/DEPRECATED-databricks-skills/databricks-lakebase-autoscale/connections.md similarity index 100% rename from databricks-skills/databricks-lakebase-autoscale/connections.md rename to DEPRECATED-databricks-skills/databricks-lakebase-autoscale/connections.md diff --git a/databricks-skills/databricks-lakebase-autoscale/operations.md b/DEPRECATED-databricks-skills/databricks-lakebase-autoscale/operations.md similarity index 100% rename from databricks-skills/databricks-lakebase-autoscale/operations.md rename to DEPRECATED-databricks-skills/databricks-lakebase-autoscale/operations.md diff --git a/databricks-skills/databricks-lakebase-autoscale/reverse-etl.md b/DEPRECATED-databricks-skills/databricks-lakebase-autoscale/reverse-etl.md similarity index 100% rename from databricks-skills/databricks-lakebase-autoscale/reverse-etl.md rename to DEPRECATED-databricks-skills/databricks-lakebase-autoscale/reverse-etl.md diff --git a/databricks-skills/databricks-lakebase-provisioned/SKILL.md b/DEPRECATED-databricks-skills/databricks-lakebase-provisioned/SKILL.md similarity index 100% rename from databricks-skills/databricks-lakebase-provisioned/SKILL.md rename to DEPRECATED-databricks-skills/databricks-lakebase-provisioned/SKILL.md diff --git a/databricks-skills/databricks-lakebase-provisioned/connection-patterns.md b/DEPRECATED-databricks-skills/databricks-lakebase-provisioned/connection-patterns.md similarity index 100% rename from databricks-skills/databricks-lakebase-provisioned/connection-patterns.md rename to DEPRECATED-databricks-skills/databricks-lakebase-provisioned/connection-patterns.md diff --git a/databricks-skills/databricks-lakebase-provisioned/reverse-etl.md b/DEPRECATED-databricks-skills/databricks-lakebase-provisioned/reverse-etl.md similarity index 100% rename from databricks-skills/databricks-lakebase-provisioned/reverse-etl.md rename to DEPRECATED-databricks-skills/databricks-lakebase-provisioned/reverse-etl.md diff --git a/databricks-skills/databricks-metric-views/SKILL.md b/DEPRECATED-databricks-skills/databricks-metric-views/SKILL.md similarity index 100% rename from databricks-skills/databricks-metric-views/SKILL.md rename to DEPRECATED-databricks-skills/databricks-metric-views/SKILL.md diff --git a/databricks-skills/databricks-metric-views/patterns.md b/DEPRECATED-databricks-skills/databricks-metric-views/patterns.md similarity index 100% rename from databricks-skills/databricks-metric-views/patterns.md rename to DEPRECATED-databricks-skills/databricks-metric-views/patterns.md diff --git a/databricks-skills/databricks-metric-views/yaml-reference.md b/DEPRECATED-databricks-skills/databricks-metric-views/yaml-reference.md similarity index 100% rename from databricks-skills/databricks-metric-views/yaml-reference.md rename to DEPRECATED-databricks-skills/databricks-metric-views/yaml-reference.md diff --git a/databricks-skills/databricks-mlflow-evaluation/SKILL.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/SKILL.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/SKILL.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/SKILL.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/CRITICAL-interfaces.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/CRITICAL-interfaces.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/CRITICAL-interfaces.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/CRITICAL-interfaces.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/GOTCHAS.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/GOTCHAS.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/GOTCHAS.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/GOTCHAS.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/patterns-context-optimization.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-context-optimization.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/patterns-context-optimization.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-context-optimization.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/patterns-datasets.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-datasets.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/patterns-datasets.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-datasets.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/patterns-evaluation.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-evaluation.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/patterns-evaluation.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-evaluation.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/patterns-judge-alignment.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-judge-alignment.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/patterns-judge-alignment.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-judge-alignment.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/patterns-prompt-optimization.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-prompt-optimization.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/patterns-prompt-optimization.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-prompt-optimization.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/patterns-scorers.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-scorers.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/patterns-scorers.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-scorers.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/patterns-trace-analysis.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-trace-analysis.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/patterns-trace-analysis.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-trace-analysis.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/patterns-trace-ingestion.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-trace-ingestion.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/patterns-trace-ingestion.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/patterns-trace-ingestion.md diff --git a/databricks-skills/databricks-mlflow-evaluation/references/user-journeys.md b/DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/user-journeys.md similarity index 100% rename from databricks-skills/databricks-mlflow-evaluation/references/user-journeys.md rename to DEPRECATED-databricks-skills/databricks-mlflow-evaluation/references/user-journeys.md diff --git a/databricks-skills/databricks-model-serving/1-classical-ml.md b/DEPRECATED-databricks-skills/databricks-model-serving/1-classical-ml.md similarity index 100% rename from databricks-skills/databricks-model-serving/1-classical-ml.md rename to DEPRECATED-databricks-skills/databricks-model-serving/1-classical-ml.md diff --git a/databricks-skills/databricks-model-serving/2-custom-pyfunc.md b/DEPRECATED-databricks-skills/databricks-model-serving/2-custom-pyfunc.md similarity index 100% rename from databricks-skills/databricks-model-serving/2-custom-pyfunc.md rename to DEPRECATED-databricks-skills/databricks-model-serving/2-custom-pyfunc.md diff --git a/databricks-skills/databricks-model-serving/3-genai-agents.md b/DEPRECATED-databricks-skills/databricks-model-serving/3-genai-agents.md similarity index 100% rename from databricks-skills/databricks-model-serving/3-genai-agents.md rename to DEPRECATED-databricks-skills/databricks-model-serving/3-genai-agents.md diff --git a/databricks-skills/databricks-model-serving/4-tools-integration.md b/DEPRECATED-databricks-skills/databricks-model-serving/4-tools-integration.md similarity index 100% rename from databricks-skills/databricks-model-serving/4-tools-integration.md rename to DEPRECATED-databricks-skills/databricks-model-serving/4-tools-integration.md diff --git a/databricks-skills/databricks-model-serving/5-development-testing.md b/DEPRECATED-databricks-skills/databricks-model-serving/5-development-testing.md similarity index 100% rename from databricks-skills/databricks-model-serving/5-development-testing.md rename to DEPRECATED-databricks-skills/databricks-model-serving/5-development-testing.md diff --git a/databricks-skills/databricks-model-serving/6-logging-registration.md b/DEPRECATED-databricks-skills/databricks-model-serving/6-logging-registration.md similarity index 100% rename from databricks-skills/databricks-model-serving/6-logging-registration.md rename to DEPRECATED-databricks-skills/databricks-model-serving/6-logging-registration.md diff --git a/databricks-skills/databricks-model-serving/7-deployment.md b/DEPRECATED-databricks-skills/databricks-model-serving/7-deployment.md similarity index 100% rename from databricks-skills/databricks-model-serving/7-deployment.md rename to DEPRECATED-databricks-skills/databricks-model-serving/7-deployment.md diff --git a/databricks-skills/databricks-model-serving/8-querying-endpoints.md b/DEPRECATED-databricks-skills/databricks-model-serving/8-querying-endpoints.md similarity index 100% rename from databricks-skills/databricks-model-serving/8-querying-endpoints.md rename to DEPRECATED-databricks-skills/databricks-model-serving/8-querying-endpoints.md diff --git a/databricks-skills/databricks-model-serving/9-package-requirements.md b/DEPRECATED-databricks-skills/databricks-model-serving/9-package-requirements.md similarity index 100% rename from databricks-skills/databricks-model-serving/9-package-requirements.md rename to DEPRECATED-databricks-skills/databricks-model-serving/9-package-requirements.md diff --git a/databricks-skills/databricks-model-serving/SKILL.md b/DEPRECATED-databricks-skills/databricks-model-serving/SKILL.md similarity index 100% rename from databricks-skills/databricks-model-serving/SKILL.md rename to DEPRECATED-databricks-skills/databricks-model-serving/SKILL.md diff --git a/databricks-skills/databricks-python-sdk/SKILL.md b/DEPRECATED-databricks-skills/databricks-python-sdk/SKILL.md similarity index 100% rename from databricks-skills/databricks-python-sdk/SKILL.md rename to DEPRECATED-databricks-skills/databricks-python-sdk/SKILL.md diff --git a/databricks-skills/databricks-python-sdk/doc-index.md b/DEPRECATED-databricks-skills/databricks-python-sdk/doc-index.md similarity index 100% rename from databricks-skills/databricks-python-sdk/doc-index.md rename to DEPRECATED-databricks-skills/databricks-python-sdk/doc-index.md diff --git a/databricks-skills/databricks-python-sdk/examples/1-authentication.py b/DEPRECATED-databricks-skills/databricks-python-sdk/examples/1-authentication.py similarity index 100% rename from databricks-skills/databricks-python-sdk/examples/1-authentication.py rename to DEPRECATED-databricks-skills/databricks-python-sdk/examples/1-authentication.py diff --git a/databricks-skills/databricks-python-sdk/examples/2-clusters-and-jobs.py b/DEPRECATED-databricks-skills/databricks-python-sdk/examples/2-clusters-and-jobs.py similarity index 100% rename from databricks-skills/databricks-python-sdk/examples/2-clusters-and-jobs.py rename to DEPRECATED-databricks-skills/databricks-python-sdk/examples/2-clusters-and-jobs.py diff --git a/databricks-skills/databricks-python-sdk/examples/3-sql-and-warehouses.py b/DEPRECATED-databricks-skills/databricks-python-sdk/examples/3-sql-and-warehouses.py similarity index 100% rename from databricks-skills/databricks-python-sdk/examples/3-sql-and-warehouses.py rename to DEPRECATED-databricks-skills/databricks-python-sdk/examples/3-sql-and-warehouses.py diff --git a/databricks-skills/databricks-python-sdk/examples/4-unity-catalog.py b/DEPRECATED-databricks-skills/databricks-python-sdk/examples/4-unity-catalog.py similarity index 100% rename from databricks-skills/databricks-python-sdk/examples/4-unity-catalog.py rename to DEPRECATED-databricks-skills/databricks-python-sdk/examples/4-unity-catalog.py diff --git a/databricks-skills/databricks-python-sdk/examples/5-serving-and-vector-search.py b/DEPRECATED-databricks-skills/databricks-python-sdk/examples/5-serving-and-vector-search.py similarity index 100% rename from databricks-skills/databricks-python-sdk/examples/5-serving-and-vector-search.py rename to DEPRECATED-databricks-skills/databricks-python-sdk/examples/5-serving-and-vector-search.py diff --git a/databricks-skills/databricks-spark-declarative-pipelines/SKILL.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/SKILL.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/SKILL.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/SKILL.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/1-project-initialization.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/1-project-initialization.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/1-project-initialization.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/1-project-initialization.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/2-mcp-approach.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/2-mcp-approach.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/2-mcp-approach.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/2-mcp-approach.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/3-advanced-configuration.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/3-advanced-configuration.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/3-advanced-configuration.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/3-advanced-configuration.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/4-dlt-migration.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/4-dlt-migration.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/4-dlt-migration.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/4-dlt-migration.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/python/1-syntax-basics.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/1-syntax-basics.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/python/1-syntax-basics.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/1-syntax-basics.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/python/2-ingestion.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/2-ingestion.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/python/2-ingestion.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/2-ingestion.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/python/3-streaming-patterns.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/3-streaming-patterns.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/python/3-streaming-patterns.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/3-streaming-patterns.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/python/4-cdc-patterns.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/4-cdc-patterns.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/python/4-cdc-patterns.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/4-cdc-patterns.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/python/5-performance.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/5-performance.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/python/5-performance.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/python/5-performance.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/sql/1-syntax-basics.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/1-syntax-basics.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/sql/1-syntax-basics.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/1-syntax-basics.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/sql/2-ingestion.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/2-ingestion.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/sql/2-ingestion.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/2-ingestion.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/sql/4-cdc-patterns.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/4-cdc-patterns.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/sql/4-cdc-patterns.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/4-cdc-patterns.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/references/sql/5-performance.md b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/5-performance.md similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/references/sql/5-performance.md rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/references/sql/5-performance.md diff --git a/databricks-skills/databricks-spark-declarative-pipelines/scripts/exploration_notebook.py b/DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/scripts/exploration_notebook.py similarity index 100% rename from databricks-skills/databricks-spark-declarative-pipelines/scripts/exploration_notebook.py rename to DEPRECATED-databricks-skills/databricks-spark-declarative-pipelines/scripts/exploration_notebook.py diff --git a/databricks-skills/databricks-spark-structured-streaming/SKILL.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/SKILL.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/SKILL.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/SKILL.md diff --git a/databricks-skills/databricks-spark-structured-streaming/checkpoint-best-practices.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/checkpoint-best-practices.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/checkpoint-best-practices.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/checkpoint-best-practices.md diff --git a/databricks-skills/databricks-spark-structured-streaming/kafka-streaming.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/kafka-streaming.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/kafka-streaming.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/kafka-streaming.md diff --git a/databricks-skills/databricks-spark-structured-streaming/merge-operations.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/merge-operations.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/merge-operations.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/merge-operations.md diff --git a/databricks-skills/databricks-spark-structured-streaming/multi-sink-writes.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/multi-sink-writes.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/multi-sink-writes.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/multi-sink-writes.md diff --git a/databricks-skills/databricks-spark-structured-streaming/stateful-operations.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/stateful-operations.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/stateful-operations.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/stateful-operations.md diff --git a/databricks-skills/databricks-spark-structured-streaming/stream-static-joins.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/stream-static-joins.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/stream-static-joins.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/stream-static-joins.md diff --git a/databricks-skills/databricks-spark-structured-streaming/stream-stream-joins.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/stream-stream-joins.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/stream-stream-joins.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/stream-stream-joins.md diff --git a/databricks-skills/databricks-spark-structured-streaming/streaming-best-practices.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/streaming-best-practices.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/streaming-best-practices.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/streaming-best-practices.md diff --git a/databricks-skills/databricks-spark-structured-streaming/trigger-and-cost-optimization.md b/DEPRECATED-databricks-skills/databricks-spark-structured-streaming/trigger-and-cost-optimization.md similarity index 100% rename from databricks-skills/databricks-spark-structured-streaming/trigger-and-cost-optimization.md rename to DEPRECATED-databricks-skills/databricks-spark-structured-streaming/trigger-and-cost-optimization.md diff --git a/databricks-skills/databricks-synthetic-data-gen/SKILL.md b/DEPRECATED-databricks-skills/databricks-synthetic-data-gen/SKILL.md similarity index 100% rename from databricks-skills/databricks-synthetic-data-gen/SKILL.md rename to DEPRECATED-databricks-skills/databricks-synthetic-data-gen/SKILL.md diff --git a/databricks-skills/databricks-synthetic-data-gen/references/1-data-patterns.md b/DEPRECATED-databricks-skills/databricks-synthetic-data-gen/references/1-data-patterns.md similarity index 100% rename from databricks-skills/databricks-synthetic-data-gen/references/1-data-patterns.md rename to DEPRECATED-databricks-skills/databricks-synthetic-data-gen/references/1-data-patterns.md diff --git a/databricks-skills/databricks-synthetic-data-gen/references/2-troubleshooting.md b/DEPRECATED-databricks-skills/databricks-synthetic-data-gen/references/2-troubleshooting.md similarity index 100% rename from databricks-skills/databricks-synthetic-data-gen/references/2-troubleshooting.md rename to DEPRECATED-databricks-skills/databricks-synthetic-data-gen/references/2-troubleshooting.md diff --git a/databricks-skills/databricks-synthetic-data-gen/scripts/generate_synthetic_data.py b/DEPRECATED-databricks-skills/databricks-synthetic-data-gen/scripts/generate_synthetic_data.py similarity index 100% rename from databricks-skills/databricks-synthetic-data-gen/scripts/generate_synthetic_data.py rename to DEPRECATED-databricks-skills/databricks-synthetic-data-gen/scripts/generate_synthetic_data.py diff --git a/databricks-skills/databricks-unity-catalog/5-system-tables.md b/DEPRECATED-databricks-skills/databricks-unity-catalog/5-system-tables.md similarity index 100% rename from databricks-skills/databricks-unity-catalog/5-system-tables.md rename to DEPRECATED-databricks-skills/databricks-unity-catalog/5-system-tables.md diff --git a/databricks-skills/databricks-unity-catalog/6-volumes.md b/DEPRECATED-databricks-skills/databricks-unity-catalog/6-volumes.md similarity index 100% rename from databricks-skills/databricks-unity-catalog/6-volumes.md rename to DEPRECATED-databricks-skills/databricks-unity-catalog/6-volumes.md diff --git a/databricks-skills/databricks-unity-catalog/7-data-profiling.md b/DEPRECATED-databricks-skills/databricks-unity-catalog/7-data-profiling.md similarity index 100% rename from databricks-skills/databricks-unity-catalog/7-data-profiling.md rename to DEPRECATED-databricks-skills/databricks-unity-catalog/7-data-profiling.md diff --git a/databricks-skills/databricks-unity-catalog/SKILL.md b/DEPRECATED-databricks-skills/databricks-unity-catalog/SKILL.md similarity index 100% rename from databricks-skills/databricks-unity-catalog/SKILL.md rename to DEPRECATED-databricks-skills/databricks-unity-catalog/SKILL.md diff --git a/databricks-skills/databricks-unstructured-pdf-generation/SKILL.md b/DEPRECATED-databricks-skills/databricks-unstructured-pdf-generation/SKILL.md similarity index 100% rename from databricks-skills/databricks-unstructured-pdf-generation/SKILL.md rename to DEPRECATED-databricks-skills/databricks-unstructured-pdf-generation/SKILL.md diff --git a/databricks-skills/databricks-vector-search/SKILL.md b/DEPRECATED-databricks-skills/databricks-vector-search/SKILL.md similarity index 100% rename from databricks-skills/databricks-vector-search/SKILL.md rename to DEPRECATED-databricks-skills/databricks-vector-search/SKILL.md diff --git a/databricks-skills/databricks-vector-search/end-to-end-rag.md b/DEPRECATED-databricks-skills/databricks-vector-search/end-to-end-rag.md similarity index 100% rename from databricks-skills/databricks-vector-search/end-to-end-rag.md rename to DEPRECATED-databricks-skills/databricks-vector-search/end-to-end-rag.md diff --git a/databricks-skills/databricks-vector-search/index-types.md b/DEPRECATED-databricks-skills/databricks-vector-search/index-types.md similarity index 100% rename from databricks-skills/databricks-vector-search/index-types.md rename to DEPRECATED-databricks-skills/databricks-vector-search/index-types.md diff --git a/databricks-skills/databricks-vector-search/search-modes.md b/DEPRECATED-databricks-skills/databricks-vector-search/search-modes.md similarity index 100% rename from databricks-skills/databricks-vector-search/search-modes.md rename to DEPRECATED-databricks-skills/databricks-vector-search/search-modes.md diff --git a/databricks-skills/databricks-vector-search/troubleshooting-and-operations.md b/DEPRECATED-databricks-skills/databricks-vector-search/troubleshooting-and-operations.md similarity index 100% rename from databricks-skills/databricks-vector-search/troubleshooting-and-operations.md rename to DEPRECATED-databricks-skills/databricks-vector-search/troubleshooting-and-operations.md diff --git a/databricks-skills/databricks-zerobus-ingest/1-setup-and-authentication.md b/DEPRECATED-databricks-skills/databricks-zerobus-ingest/1-setup-and-authentication.md similarity index 100% rename from databricks-skills/databricks-zerobus-ingest/1-setup-and-authentication.md rename to DEPRECATED-databricks-skills/databricks-zerobus-ingest/1-setup-and-authentication.md diff --git a/databricks-skills/databricks-zerobus-ingest/2-python-client.md b/DEPRECATED-databricks-skills/databricks-zerobus-ingest/2-python-client.md similarity index 100% rename from databricks-skills/databricks-zerobus-ingest/2-python-client.md rename to DEPRECATED-databricks-skills/databricks-zerobus-ingest/2-python-client.md diff --git a/databricks-skills/databricks-zerobus-ingest/3-multilanguage-clients.md b/DEPRECATED-databricks-skills/databricks-zerobus-ingest/3-multilanguage-clients.md similarity index 100% rename from databricks-skills/databricks-zerobus-ingest/3-multilanguage-clients.md rename to DEPRECATED-databricks-skills/databricks-zerobus-ingest/3-multilanguage-clients.md diff --git a/databricks-skills/databricks-zerobus-ingest/4-protobuf-schema.md b/DEPRECATED-databricks-skills/databricks-zerobus-ingest/4-protobuf-schema.md similarity index 100% rename from databricks-skills/databricks-zerobus-ingest/4-protobuf-schema.md rename to DEPRECATED-databricks-skills/databricks-zerobus-ingest/4-protobuf-schema.md diff --git a/databricks-skills/databricks-zerobus-ingest/5-operations-and-limits.md b/DEPRECATED-databricks-skills/databricks-zerobus-ingest/5-operations-and-limits.md similarity index 100% rename from databricks-skills/databricks-zerobus-ingest/5-operations-and-limits.md rename to DEPRECATED-databricks-skills/databricks-zerobus-ingest/5-operations-and-limits.md diff --git a/databricks-skills/databricks-zerobus-ingest/SKILL.md b/DEPRECATED-databricks-skills/databricks-zerobus-ingest/SKILL.md similarity index 100% rename from databricks-skills/databricks-zerobus-ingest/SKILL.md rename to DEPRECATED-databricks-skills/databricks-zerobus-ingest/SKILL.md diff --git a/databricks-skills/install_skills.sh b/DEPRECATED-databricks-skills/install_skills.sh similarity index 87% rename from databricks-skills/install_skills.sh rename to DEPRECATED-databricks-skills/install_skills.sh index c6d070be..931236cf 100755 --- a/databricks-skills/install_skills.sh +++ b/DEPRECATED-databricks-skills/install_skills.sh @@ -1,24 +1,30 @@ #!/bin/bash # -# Databricks Skills Installer +# Databricks Skills Installer (DEPRECATED) # -# Installs Databricks skills for Claude Code into your project. -# These skills teach Claude how to work with Databricks using MCP tools. +# ⚠️ DEPRECATED: The supported way to install skills is now: +# databricks aitools install (Databricks CLI v1.0.0+) +# or the AI Dev Kit installer, which delegates to it: +# bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) # -# Usage: -# # Install all skills (Databricks + MLflow) -# curl -sSL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/databricks-skills/install_skills.sh | bash +# This script remains only for the Genie Code workspace-upload flow +# (--install-to-genie), which `databricks aitools install` does not cover. +# For that, prefer the notebook uploader install_genie_code_skills.py +# (no local clone needed). When that is not possible, this script's default +# downloads the skills from the last release that shipped the bundled tree +# (git tag v0.1.13). The frozen local copies under DEPRECATED-databricks-skills/ +# are only used with --local. Either way the skills are no longer maintained. # -# # Install specific skills (can mix Databricks and MLflow skills) -# curl -sSL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/databricks-skills/install_skills.sh | bash -s -- databricks-bundles agent-evaluation +# Usage: +# # Install then upload ./.claude/skills to workspace (Genie Code / Assistant) +# ./install_skills.sh --install-to-genie +# ./install_skills.sh --install-to-genie --profile prod --local # -# # Or run locally +# # Other (deprecated) modes # ./install_skills.sh # Install all skills # ./install_skills.sh databricks-bundles agent-evaluation # Install specific skills # ./install_skills.sh --mlflow-version v1.0.0 # Pin MLflow skills version -# ./install_skills.sh --local # Install Databricks skills from local directory -# ./install_skills.sh --install-to-genie # Install then upload ./.claude/skills to workspace (Genie Code / Assistant) -# ./install_skills.sh --install-to-genie --profile prod --local +# ./install_skills.sh --local # Install Databricks skills from DEPRECATED-databricks-skills/ folder # ./install_skills.sh --list # List available skills # ./install_skills.sh --help # Show help # @@ -33,8 +39,12 @@ BLUE='\033[0;34m' NC='\033[0m' # No Color # Configuration +# NOTE: The bundled skill tree was removed from `main` and frozen under +# DEPRECATED-databricks-skills/. The default (non-local) download therefore pins +# to the last release that still bundled the tree at its original paths: v0.1.13. REPO_URL="https://github.com/databricks-solutions/ai-dev-kit" -REPO_RAW_URL="https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main" +REPO_REF="v0.1.13" +REPO_RAW_URL="https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/${REPO_REF}" SKILLS_DIR=".claude/skills" INSTALL_FROM_LOCAL=false INSTALL_TO_GENIE=false @@ -154,6 +164,10 @@ get_mlflow_skill_extra_files() { show_help() { echo -e "${BLUE}Databricks Skills Installer for Claude Code${NC}" echo "" + echo -e "${YELLOW}DEPRECATED:${NC} use 'databricks aitools install' (Databricks CLI v1.0.0+) or" + echo -e "the AI Dev Kit installer (install.sh). This script is kept for the Genie Code" + echo -e "workspace-upload flow (--install-to-genie); skill files are frozen legacy copies." + echo "" echo "Usage:" echo " ./install_skills.sh [options] [skill1 skill2 ...]" echo "" @@ -161,7 +175,7 @@ show_help() { echo " --help, -h Show this help message" echo " --list, -l List all available skills" echo " --all, -a Install all skills (default if no skills specified)" - echo " --local Install from local files instead of downloading" + echo " --local Install from the frozen local copies under DEPRECATED-databricks-skills/ instead of downloading" echo " --install-to-genie After install, upload ./.claude/skills to workspace" echo " /Users//.assistant/skills for Genie Code (uses cwd; requires databricks CLI)" echo " --profile Databricks config profile for workspace upload (default: DEFAULT or \$DATABRICKS_CONFIG_PROFILE)" @@ -312,9 +326,9 @@ download_databricks_skill() { local skill_dir="$SKILLS_DIR/$skill_name" if [ "$INSTALL_FROM_LOCAL" = true ]; then - # Copy from local files - echo -e " Copying from local..." - local source_dir="$SCRIPT_DIR/${skill_name}" + # Copy from the frozen legacy copies under DEPRECATED-databricks-skills/ + echo -e " Copying from local (DEPRECATED-databricks-skills/)..." + local source_dir="$SCRIPT_DIR/../DEPRECATED-databricks-skills/${skill_name}" # Check if source directory exists if [ ! -d "$source_dir" ]; then @@ -349,8 +363,8 @@ download_databricks_skill() { done fi else - # Download from URL - echo -e " Downloading from Databricks repo..." + # Download from URL (pinned to the frozen ${REPO_REF} tag) + echo -e " Downloading from Databricks repo (${REPO_REF})..." # Download SKILL.md (required) if curl -sSL -f "${REPO_RAW_URL}/databricks-skills/${skill_name}/SKILL.md" -o "$skill_dir/SKILL.md" 2>/dev/null; then @@ -525,6 +539,25 @@ echo -e "${BLUE}║ Databricks Skills Installer for Claude Code echo -e "${BLUE}╚════════════════════════════════════════════════════════════╝${NC}" echo "" +# Deprecation banner +echo -e "${YELLOW}────────────────────────────────────────────────────────────${NC}" +echo -e "${YELLOW} ⚠️ DEPRECATED SCRIPT${NC}" +echo -e "${YELLOW}────────────────────────────────────────────────────────────${NC}" +echo -e " The supported way to install editor skills is now:" +echo -e " ${GREEN}databricks aitools install${NC} (Databricks CLI v1.0.0+)" +echo -e " or the AI Dev Kit installer, which delegates to it:" +echo -e " ${GREEN}bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh)${NC}" +echo "" +echo -e " This script is kept only for the ${GREEN}--install-to-genie${NC} workspace" +echo -e " upload flow. For that, the notebook uploader is simpler:" +echo -e " ${GREEN}databricks-skills/install_genie_code_skills.py${NC} (no local clone)" +echo "" +echo -e " Skill files are ${YELLOW}frozen legacy copies${NC}, no longer maintained:" +echo -e " the default downloads them from release tag ${REPO_REF}; ${GREEN}--local${NC} uses" +echo -e " the DEPRECATED-databricks-skills/ folder instead." +echo -e "${YELLOW}────────────────────────────────────────────────────────────${NC}" +echo "" + # Check if we're in a git repo or project directory if [ ! -d ".git" ] && [ ! -f "pyproject.toml" ] && [ ! -f "package.json" ] && [ ! -f "databricks.yml" ]; then echo -e "${YELLOW}Warning: This doesn't look like a project root directory.${NC}" diff --git a/databricks-skills/spark-python-data-source/SKILL.md b/DEPRECATED-databricks-skills/spark-python-data-source/SKILL.md similarity index 100% rename from databricks-skills/spark-python-data-source/SKILL.md rename to DEPRECATED-databricks-skills/spark-python-data-source/SKILL.md diff --git a/databricks-skills/spark-python-data-source/references/authentication-patterns.md b/DEPRECATED-databricks-skills/spark-python-data-source/references/authentication-patterns.md similarity index 100% rename from databricks-skills/spark-python-data-source/references/authentication-patterns.md rename to DEPRECATED-databricks-skills/spark-python-data-source/references/authentication-patterns.md diff --git a/databricks-skills/spark-python-data-source/references/error-handling.md b/DEPRECATED-databricks-skills/spark-python-data-source/references/error-handling.md similarity index 100% rename from databricks-skills/spark-python-data-source/references/error-handling.md rename to DEPRECATED-databricks-skills/spark-python-data-source/references/error-handling.md diff --git a/databricks-skills/spark-python-data-source/references/implementation-template.md b/DEPRECATED-databricks-skills/spark-python-data-source/references/implementation-template.md similarity index 100% rename from databricks-skills/spark-python-data-source/references/implementation-template.md rename to DEPRECATED-databricks-skills/spark-python-data-source/references/implementation-template.md diff --git a/databricks-skills/spark-python-data-source/references/partitioning-patterns.md b/DEPRECATED-databricks-skills/spark-python-data-source/references/partitioning-patterns.md similarity index 100% rename from databricks-skills/spark-python-data-source/references/partitioning-patterns.md rename to DEPRECATED-databricks-skills/spark-python-data-source/references/partitioning-patterns.md diff --git a/databricks-skills/spark-python-data-source/references/production-patterns.md b/DEPRECATED-databricks-skills/spark-python-data-source/references/production-patterns.md similarity index 100% rename from databricks-skills/spark-python-data-source/references/production-patterns.md rename to DEPRECATED-databricks-skills/spark-python-data-source/references/production-patterns.md diff --git a/databricks-skills/spark-python-data-source/references/streaming-patterns.md b/DEPRECATED-databricks-skills/spark-python-data-source/references/streaming-patterns.md similarity index 100% rename from databricks-skills/spark-python-data-source/references/streaming-patterns.md rename to DEPRECATED-databricks-skills/spark-python-data-source/references/streaming-patterns.md diff --git a/databricks-skills/spark-python-data-source/references/testing-patterns.md b/DEPRECATED-databricks-skills/spark-python-data-source/references/testing-patterns.md similarity index 100% rename from databricks-skills/spark-python-data-source/references/testing-patterns.md rename to DEPRECATED-databricks-skills/spark-python-data-source/references/testing-patterns.md diff --git a/databricks-skills/spark-python-data-source/references/type-conversion.md b/DEPRECATED-databricks-skills/spark-python-data-source/references/type-conversion.md similarity index 100% rename from databricks-skills/spark-python-data-source/references/type-conversion.md rename to DEPRECATED-databricks-skills/spark-python-data-source/references/type-conversion.md diff --git a/README.md b/README.md index 123eb7d0..189959b6 100644 --- a/README.md +++ b/README.md @@ -5,29 +5,32 @@

--- -> 📣 **Heads up: a major evolution is coming** +> 📣 **The move is complete: Install now pulls the official Databricks Agent Skills** > -> This will be the **last release where AI Dev Kit skills are installed from the skill files in this -> repository.** AI Dev Kit skills are becoming part of the official, engineering-supported Databricks -> skills set, and the skill files in this repo will soon be deprecated. In the next release you'll -> install these skills directly from the official Databricks set — either straight from the CLI, or -> through the AI Dev Kit installer, which will continue to guide you through the process. +> AI Dev Kit skills are **no longer installed from skill files in this repository.** They are now part +> of the official, engineering-supported Databricks skills set, installed directly from the CLI +> (`databricks aitools install`) or through the AI Dev Kit installer, which delegates to the CLI and +> still guides you through the process. The bundled skill files in this repo are deprecated and kept +> for reference only. > +> **Already installed?** Re-run the installer script in this repo to uninstall older skills and MCP +> server before installing the official skills — see [Where Skills Come From](#where-skills-come-from). > -> **What stays:** The MCP server and Builder App will remain in this repository. The Builder App will -> keep being developed and improved, and the MCP server will be maintained and updated on a -> best-effort basis as GitHub issues are filed. > +> **What stays:** The MCP server and Builder App remain in this repository. The Builder App will keep +> being developed and improved, and the MCP server will be maintained and updated on a best-effort +> basis as GitHub issues are filed. > -> **What's next:** AI Dev Kit will continue to guide you through setting up your AI coding -> environment and be a place to find experimental tools developed by Field Engineering. Beyond the -> skills installs, we plan to add several tutorials to help you get started using coding agents for -> building on Databricks, including getting started with Genie Code or Omnigent. > -> **A few skills will be renamed or merged** in the official install. Most names are unchanged; the +> **What's next:** AI Dev Kit will continue to guide you through setting up your AI coding environment +> and be a place to find experimental tools developed by Field Engineering. We're working on several +> tutorials to help you get started using coding agents for building on Databricks — including getting +> started with Genie Code and Omnigent — which will land in a future update. +> +> **A few skills were renamed or merged** in the official install. Most names are unchanged; the > exceptions are: > -> | Today (AI Dev Kit) | Official Databricks skills | +> | Previously (AI Dev Kit) | Official Databricks skills | > |---|---| > | `databricks-bundles` | `databricks-dabs` | > | `databricks-spark-declarative-pipelines` | `databricks-pipelines` | @@ -55,7 +58,7 @@ Databricks offers two paths for AI-assisted coding. Choose the one that matches **Free, first-party AI coding inside Databricks** -Built into every Databricks workspace at no extra cost, with deep native product context — your notebooks, jobs, and Unity Catalog data are already in scope. Ideal for users who have not started using AI-driven development tools or that are comfortable in Databricks. +Built into every Databricks workspace with deep native product context — your notebooks, jobs, and Unity Catalog data are already in scope. Ideal for users who have not started using AI-driven development tools or that are comfortable in Databricks workspace. @@ -68,7 +71,7 @@ Built into every Databricks workspace at no extra cost, with deep native product **Databricks expertise, in the editor you already use** -Curated by Databricks field experts. Brings the patterns, skills, and 75+ executable tools your AI assistant needs to build on Databricks — wherever you're already coding. +Curated by Databricks field experts. Brings tutorials, patterns, and the official Databricks agent skills to your AI coding tools to build on Databricks — wherever you're already coding.
@@ -123,21 +126,17 @@ Curated by Databricks field experts. Brings the patterns, skills, and 75+ execut | Adventure | Best For | Start Here | |----------------------------------|----------|------------| -| :star: [**Install AI Dev Kit**](#install-in-existing-project) | **Start here!** Follow quick install instructions to add to your existing project folder | [Quick Start (install)](#install-in-existing-project) +| :star: [**Install AI Dev Kit**](#install-in-existing-project) | **Start here!** Follow quick install instructions to add Databricks skills for your user or existing project folder | [Quick Start (install)](#install-in-existing-project) +| [**Genie Code Skills**](databricks-skills/) | Upload selected skills into your workspace for Genie Code. Note: much of the content in the public Databricks skills is already included in Genie Code. | [Genie Code skills (install)](#genie-code-skills) | | [**Visual Builder App**](#visual-builder-app) | Web-based UI for Databricks development | `databricks-builder-app/` | -| [**Builder App + Genie Code MCP**](#visual-builder-app) | Builder UI + MCP server for Genie Code in one deployment | `deploy.sh --enable-mcp` | -| [**Core Library**](#core-library) | Building custom integrations (LangChain, OpenAI, etc.) | `pip install` | -| [**Skills Only**](databricks-skills/) | Provide Databricks patterns and best practices (without MCP functions) | Install skills | -| [**Genie Code Skills**](databricks-skills/install_skills.sh) | Install skills into your workspace for Genie Code (`--install-to-genie`) | [Genie Code skills (install)](#genie-code-skills) | -| [**MCP Tools Only**](databricks-mcp-server/) | Just executable actions (no guidance) | Register MCP server | +| [**MCP Tools**](databricks-mcp-server/) | Standalone MCP server exposing Databricks actions to AI clients. Note: We recommend skills instead, but maintain this as a good foundation if a custom MCP serer is required. | [Register MCP server](databricks-mcp-server/) | --- ## Quick Start ### Prerequisites -- [uv](https://github.com/astral-sh/uv) - Python package manager -- [Databricks CLI](https://docs.databricks.com/aws/en/dev-tools/cli/) - Command line interface for Databricks +- [Databricks CLI](https://docs.databricks.com/aws/en/dev-tools/cli/) **v1.0.0+** - Command line interface for Databricks (v1.0.0+ ships `databricks aitools`, which installs most skills) - AI coding environment (one or more): - [Claude Code](https://claude.ai/code) - [Cursor](https://cursor.com) @@ -151,8 +150,8 @@ Curated by Databricks field experts. Brings the patterns, skills, and 75+ execut ### Install in existing project -By default this will install at a project level rather than a user level. This is often a good fit, but requires you to run your client from the exact directory that was used for the install. -_Note: Project configuration files can be re-used in other projects. You find these configs under .claude, .cursor, .gemini, .codex, .github, .agents, .windsurf, .codeium, .opencode, .kiro, or opencode.json_ +By default this will install at a project level rather than a user level. This is often a good fit, but requires you to run your client from the exact directory that was used for the install. You can install for your user across all projects on the machine by passing `--global` or setting the scope to global in the interactive install. +_Note: Project configuration files can be found under these folder: .claude, .cursor, .gemini, .codex, .github, .agents, .windsurf, .codeium, .opencode, .kiro, or opencode.json_ #### Mac / Linux @@ -171,16 +170,10 @@ bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-ki bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) --global --force ``` -**Specify profile and force reinstall** +**Install specific skills only** ```bash -bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) --profile DEFAULT --force -``` - -**Install for specific tools only** - -```bash -bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) --tools cursor,gemini,antigravity,windsurf,opencode +bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) --skills databricks-core,databricks-bundles,databricks-pipelines ``` @@ -211,12 +204,6 @@ irm https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/insta .\install.ps1 -Global -Force ``` -**Specify profile and force reinstall** - -```powershell -.\install.ps1 -Profile DEFAULT -Force -``` - **Install for specific tools only** ```powershell @@ -228,6 +215,32 @@ irm https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/insta **Next steps:** Respond to interactive prompts and follow the on-screen instructions. - Note: Cursor and Copilot require updating settings manually after install. +### Where Skills Come From + +The installer assembles skills from two upstream sources — nothing is bundled in this repo anymore: + +| Source | Skills | Mechanism | +|--------|--------|-----------| +| [databricks/databricks-agent-skills](https://github.com/databricks/databricks-agent-skills) | Most Databricks skills (jobs, pipelines, DABs, SQL, Unity Catalog, apps, Genie, …) | Delegated to `databricks aitools install` — requires **Databricks CLI v1.0.0+** | +| [mlflow/skills](https://github.com/mlflow/skills) | MLflow skills | Fetched from `main` (override with `MLFLOW_REF`) | + +Skills installed via `databricks aitools` are managed by the CLI afterwards — update them with `databricks aitools update` and remove them with `databricks aitools uninstall`. For tools the CLI can't target yet (Gemini CLI, Windsurf, Kiro), the installer links the same skills into each tool's skills directory. + +Use `--list-skills` to see every skill and profile, and `--dry-run` to preview exactly what an install would do (resolved refs and the `aitools` command) without changing anything. Some skills were renamed or consolidated in the move — see [Breaking change: skill sources and names](#breaking-change-skill-sources-and-names) below. + +
+Installer environment variables (click to expand) + +| Variable | Default | Purpose | +|----------|---------|---------| +| `MLFLOW_REF` | `main` | Ref for the MLflow skills fetch (the repo is tagless) | +| `INCLUDE_PRERELEASES` | `0` | Set to `1` to allow `-rc`/`-beta` tags when resolving `latest` | +| `DRY_RUN` | `false` | Set to `1` to print the install plan and exit | + +The installer also records what it installed (resolved refs, commit SHAs, `aitools` release) in `skills.lock` inside the scope-local `.ai-dev-kit/` state directory. + +
+ ### Uninstall Run the same script with `--uninstall`. It removes the installed skill folders (including ones from older versions), the MCP server runtime (`~/.ai-dev-kit`), and the `databricks` MCP entry from each editor's config — leaving your other MCP servers, skills, and settings untouched. Editor config files are backed up to `.bak` first. @@ -256,30 +269,26 @@ bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-ki Scope mirrors install: a project uninstall touches only the current directory's config; `--global` touches only the per-user (`$HOME`) config. Run it once per scope you installed into. - ### Visual Builder App -Full-stack web application with chat UI for Databricks development. Deploys a Lakebase database and Databricks App with a single command: +Full-stack web application with a chat UI for Databricks development. Its setup is **self-contained** — you do not need to run the kit-wide `install.sh` first; the scripts build their own environment, install the sibling packages, and generate config. ```bash cd ai-dev-kit/databricks-builder-app -# Deploy everything (Lakebase + app + permissions) +# Local development (provisions Lakebase, installs deps, generates .env.local, starts servers) +./scripts/start_local.sh --profile + +# Deploy to Databricks Apps (Lakebase + app + permissions) ./scripts/deploy.sh my-builder-app --profile -# Deploy with MCP Gateway for Genie Code (name must start with mcp-) +# Deploy with the MCP gateway enabled (name must start with mcp- for Genie Code) ./scripts/deploy.sh mcp-builder-app --enable-mcp --profile ``` -With `--enable-mcp`, the app also serves as an **MCP server** at `/mcp`, exposing all 75+ Databricks tools to [Genie Code](https://docs.databricks.com/en/genie/genie-code.html), AI Playground, and other MCP clients. The builder UI and MCP server run in a single deployment. +With `--enable-mcp`, the app also serves as an **MCP server** at `/mcp`, exposing the 40+ Databricks tools to [Genie Code](https://docs.databricks.com/en/genie/genie-code.html), AI Playground, and other MCP clients. The builder UI and MCP server run in a single deployment. -For local development: - -```bash -./scripts/setup.sh # Install dependencies -# Edit .env.local with your credentials -./scripts/start_dev.sh # Start locally at http://localhost:3000 -``` +> **Current dependencies (to be removed later):** the Builder App currently relies on the in-repo MCP server (`databricks-mcp-server` + `databricks-tools-core`) and a bundled, frozen skills snapshot under [`DEPRECATED-databricks-skills/`](DEPRECATED-databricks-skills/). Both are temporary couplings scheduled to be decoupled from the kit. See [`databricks-builder-app/`](databricks-builder-app/) for full documentation. @@ -297,42 +306,56 @@ results = execute_sql("SELECT * FROM my_catalog.schema.table LIMIT 10") Works with LangChain, OpenAI Agents SDK, or any Python framework. See [databricks-tools-core/](databricks-tools-core/) for details. --- -## Genie Code Skills - -Install skills into `./.claude/skills` (relative to the directory where you run the script), then upload them to your workspace at `/Workspace/Users//.assistant/skills` so Genie Code can use them in the UI. Requires the [Databricks CLI](https://docs.databricks.com/aws/en/dev-tools/cli/) authenticated for your workspace. +## Skills -**Always run from the project directory** where you want `.claude/skills` created (for example your repo root or `ai-dev-kit`). - -**From this repo (recommended if you have a clone):** +Skills teach your AI assistant Databricks patterns and best practices. For your **editor**, they are +installed and kept up to date by the Databricks CLI (v1.0.0+), which the AI Dev Kit installer already +delegates to: ```bash -# Databricks skills from this checkout + upload (DEFAULT CLI profile) -./databricks-skills/install_skills.sh --local --install-to-genie - -# Download all skills from GitHub, then upload -./databricks-skills/install_skills.sh --install-to-genie - -# Explicit Databricks CLI profile -./databricks-skills/install_skills.sh --install-to-genie --profile YOUR_PROFILE +databricks aitools install ``` -**Without cloning** (run from the directory that should contain `.claude/skills`): +By default `databricks aitools install` installs the official **`databricks` plugin** through each +agent's own plugin CLI for the agents that support one — **Claude Code, Codex, and GitHub Copilot** — +and writes **raw skill files** for the agents that don't (**Cursor, OpenCode, Antigravity**). Pass +`--skills-only` to force raw skill files for every agent. (This official `databricks` plugin is +separate from — and replaces — the retired `databricks-ai-dev-kit` plugin this repo used to publish; +see [`.claude-plugin/DEPRECATED.md`](.claude-plugin/DEPRECATED.md).) -```bash -curl -sSL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/databricks-skills/install_skills.sh | bash -s -- --install-to-genie -``` +Skills come from [github.com/databricks/databricks-agent-skills](https://github.com/databricks/databricks-agent-skills). +The skill copies that used to be bundled in this repo are deprecated and frozen under +[`DEPRECATED-databricks-skills/`](DEPRECATED-databricks-skills/); if you need the exact historical +files, use git tag `v0.1.13`. Some skills were renamed in the move — see the breaking-change note +below. (APX and Genie-specific skills are no longer bundled here; they live in their own repos.) + +### Genie Code Skills -Combine `--profile`, `--local`, specific skill names, `--mlflow-version`, etc. as needed; see `./databricks-skills/install_skills.sh --help` or [databricks-skills/README.md](databricks-skills/README.md). +To use skills inside **Genie Code** in a Databricks workspace, upload them to +`/Workspace/Users//.assistant/skills`. `databricks aitools install` does not cover this yet, so +use the provided notebook to install: -**From a Databricks notebook** (no local terminal needed): +**From a Databricks notebook (recommended — no local clone):** +Import [`install_genie_code_skills.py`](install_genie_code_skills.py) +into your workspace as a notebook and run it. It downloads skills from GitHub and uploads them via the +Databricks SDK. Works on any compute, including serverless. -Import [`databricks-skills/install_genie_code_skills.py`](databricks-skills/install_genie_code_skills.py) into your workspace as a notebook and run it. It downloads skills from GitHub and uploads them to your workspace using the Databricks SDK. This works on any compute, including serverless. +See [databricks-skills/README.md](databricks-skills/README.md) for details. -**Skill modification or Custom Skill** +**Customizing skills:** after upload, skills live under +`/Workspace/Users//.assistant/skills`. You can modify or remove skills there, or add +your own skill folders (each with a `SKILL.md`) that Genie Code will use automatically in any session. -After the script successfully installs the skills to your workspace, you may find the skills under `/Workspace/Users//.assistant/skills`. +### Breaking change: skill sources and names -This directory is customizable if you wish to only use certain skills or even create custom skills that are related to your organization to make Genie Code even better. You can modify/remove existing skills or create new skills folders that Genie Code will automatically use in any session. +Skills are no longer bundled in this repository — they come from +[databricks-agent-skills](https://github.com/databricks/databricks-agent-skills) via +`databricks aitools install`, and a few were **renamed or consolidated** in the move (see the rename +table in the notice at the top of this README). To see the current skill names, run +`databricks aitools list` (CLI v1.0.0+) or browse the +[databricks-agent-skills](https://github.com/databricks/databricks-agent-skills) repo. To reproduce +the old bundled layout and names exactly, use git tag `v0.1.13` (the frozen copies also remain under +[`DEPRECATED-databricks-skills/`](DEPRECATED-databricks-skills/)). ## Architecture @@ -347,8 +370,8 @@ The AI Dev Kit ships as four composable pieces — install the whole kit, or pic | Component | Description | |-----------|-------------| | [`databricks-tools-core/`](databricks-tools-core/) | Python library with high-level Databricks functions | -| [`databricks-mcp-server/`](databricks-mcp-server/) | MCP server exposing 50+ tools for AI assistants | -| [`databricks-skills/`](databricks-skills/) | 20 markdown skills teaching Databricks patterns | +| [`databricks-mcp-server/`](databricks-mcp-server/) | Standalone MCP server exposing 40+ Databricks tools for AI assistants (installs independently of skills) | +| [`databricks-skills/`](databricks-skills/) | Skills that teach Databricks patterns (installed via `databricks aitools`; bundled copies are deprecated) | | [`databricks-builder-app/`](databricks-builder-app/) | Full-stack web app with Claude Code integration | --- diff --git a/SECURITY.md b/SECURITY.md index 48ae5729..eb6d06ad 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -8,6 +8,12 @@ Please email bugbounty@databricks.com to report any security vulnerabilities. We ## Plugin Trust Model +> **Deprecated:** The Databricks AI Dev Kit is no longer published as a Claude Code plugin (see +> [.claude-plugin/DEPRECATED.md](.claude-plugin/DEPRECATED.md)). Install via the installer +> (`bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh)`) +> or `databricks aitools install` instead. The section below is retained only to describe the retired +> plugin's behavior for anyone who still has it installed. + This section describes what code executes when you install the Databricks AI Dev Kit as a Claude Code plugin. ### What Runs Automatically diff --git a/databricks-builder-app/README.md b/databricks-builder-app/README.md index ba763cc2..f00e3ca0 100644 --- a/databricks-builder-app/README.md +++ b/databricks-builder-app/README.md @@ -4,7 +4,7 @@ A web application that provides a Claude Code agent interface with integrated Databricks tools. Users interact with Claude through a chat interface, and the agent can execute SQL queries, manage pipelines, upload files, and more on their Databricks workspace. -Optionally, the app can also serve as an **MCP server** for [Genie Code](https://docs.databricks.com/en/genie/genie-code.html) and other MCP clients, exposing all 75+ Databricks tools via the MCP protocol at `/mcp`. +Optionally, the app can also serve as an **MCP server** for [Genie Code](https://docs.databricks.com/en/genie/genie-code.html) and other MCP clients, exposing all 40+ Databricks tools via the MCP protocol at `/mcp`. > **✅ Event Loop Fix Implemented** > @@ -172,7 +172,7 @@ Tools are exposed as `mcp__databricks__` and include: Skills provide specialized guidance for Databricks development tasks. They are markdown files with instructions and examples that Claude can load on demand. **Skill loading flow:** -1. On startup, skills are copied from `../databricks-skills/` to `./skills/` +1. On startup, skills are copied from the bundled snapshot (`../DEPRECATED-databricks-skills/`, falling back to `../databricks-skills/`) to `./skills/` 2. When a project is created, skills are copied to `project/.claude/skills/` 3. The agent can invoke skills using the `Skill` tool: `skill: "sdp"` @@ -205,16 +205,40 @@ projects/ ## Setup +> **Standalone setup.** This app is self-contained — you do **not** need to run +> the kit-wide `install.sh` at the repo root first. The scripts under +> `scripts/` build their own Python virtualenv (`uv sync`), install the sibling +> `databricks-tools-core` / `databricks-mcp-server` packages, install skills, +> and generate their own config (`.env.local` / `app.yaml`). Just clone the repo +> and run the commands below from `databricks-builder-app/`. + +### Current dependencies (to be removed later) + +The Builder App is being decoupled from the rest of the kit. Until that work +lands, it still relies on two in-repo dependencies that ship alongside the app: + +- **In-process MCP server** — Databricks tools are loaded from the sibling + `databricks-mcp-server` (backed by `databricks-tools-core`) packages. The + setup/deploy scripts install these from the repo automatically. This is a + temporary coupling; the tool layer will be replaced later. +- **Bundled (frozen) skills snapshot** — Databricks skills are read from the + in-repo snapshot at `DEPRECATED-databricks-skills/` (with a fallback to the + legacy `databricks-skills/` location for older checkouts). This snapshot is + frozen and shipped with the app; MLflow skills are still fetched from + `github.com/mlflow/skills` by `install_skills.sh`. A future change will + migrate the app off these bundled copies (e.g. to `databricks aitools`). + ### Prerequisites - Python 3.11+ - Node.js 18+ - [uv](https://github.com/astral-sh/uv) package manager +- Databricks CLI v0.287.0+ (with an authenticated profile: `databricks auth login --profile `) - Databricks workspace with: - SQL warehouse (for SQL queries) - Cluster (for Python/PySpark execution) - Unity Catalog enabled (recommended) -- PostgreSQL database (Lakebase) for project persistence — autoscale or provisioned + - Lakebase enabled (autoscale or provisioned) for project persistence ### Quick Start (Local Development) @@ -306,7 +330,7 @@ The app supports two authentication modes: #### Skills Configuration -Skills are loaded from `../databricks-skills/` and filtered by the `ENABLED_SKILLS` environment variable: +Skills are loaded from the bundled snapshot (`../DEPRECATED-databricks-skills/`, falling back to `../databricks-skills/`) and filtered by the `ENABLED_SKILLS` environment variable: - `databricks-python-sdk`: Patterns for using the Databricks Python SDK - `databricks-spark-declarative-pipelines`: SDP/DLT pipeline development @@ -314,7 +338,7 @@ Skills are loaded from `../databricks-skills/` and filtered by the `ENABLED_SKIL - `databricks-apps-python`: Python apps with Dash, Streamlit, Flask **Adding custom skills:** -1. Create a new directory in `../databricks-skills/` +1. Create a new directory in the bundled snapshot (`../DEPRECATED-databricks-skills/`) 2. Add a `SKILL.md` file with frontmatter: ```markdown --- @@ -359,7 +383,7 @@ See [EVENT_LOOP_FIX.md](./EVENT_LOOP_FIX.md) for technical details. Check: 1. `ENABLED_SKILLS` environment variable in `.env.local` -2. Skill names match directory names in `../databricks-skills/` +2. Skill names match directory names in the bundled snapshot (`../DEPRECATED-databricks-skills/`, or the legacy `../databricks-skills/`) 3. Each skill has a `SKILL.md` file with proper frontmatter 4. Check logs: `Copied X skills to ./skills` @@ -568,7 +592,7 @@ databricks bundle destroy --profile ### MCP Gateway for Genie Code -The Builder App can optionally serve as an **MCP server** at `/mcp`, exposing all 75+ Databricks tools to [Genie Code](https://docs.databricks.com/en/genie/genie-code.html), AI Playground, and other MCP clients. This turns the app into a dual-purpose deployment: **visual builder UI** at `/` and **MCP server** at `/mcp`. +The Builder App can optionally serve as an **MCP server** at `/mcp`, exposing all 40+ Databricks tools to [Genie Code](https://docs.databricks.com/en/genie/genie-code.html), AI Playground, and other MCP clients. This turns the app into a dual-purpose deployment: **visual builder UI** at `/` and **MCP server** at `/mcp`. #### How It Works @@ -615,8 +639,9 @@ After deploying with `--enable-mcp` and an `mcp-` prefixed name: You can also install skills to the Genie Space for additional context: ```bash -# From the repo root — installs skills to your workspace for Genie Code -./databricks-skills/install_skills_to_genie_code.sh +# Downloads the frozen legacy skills from the last release (v0.1.13), +# then uploads them to your workspace for Genie Code. +bash ../databricks-skills/install_skills.sh --install-to-genie --profile ``` #### Using with Other MCP Clients @@ -640,7 +665,7 @@ The MCP gateway does **not** activate during local development unless you explic ```bash # Normal local dev (no MCP gateway) -./scripts/start_dev.sh +./scripts/start_local.sh --profile # Local dev with MCP gateway (for testing) ENABLE_MCP_GATEWAY=true uvicorn server.app:app --reload --port 8000 --reload-dir server @@ -689,4 +714,4 @@ This provides a minimal working example with setup instructions for integrating - **databricks-tools-core**: Core MCP functionality and SQL operations - **databricks-mcp-server**: MCP server exposing Databricks tools -- **databricks-skills**: Skill definitions for Databricks development +- **DEPRECATED-databricks-skills**: Bundled (frozen) skill definitions the app ships with (falls back to `databricks-skills/` on older checkouts) diff --git a/databricks-builder-app/scripts/_integration-example/setup.sh b/databricks-builder-app/scripts/_integration-example/setup.sh index f830fec1..67d25054 100755 --- a/databricks-builder-app/scripts/_integration-example/setup.sh +++ b/databricks-builder-app/scripts/_integration-example/setup.sh @@ -44,8 +44,13 @@ echo "Installing all skills..." SKILLS_DIR="$SCRIPT_DIR/.claude/skills" mkdir -p "$SKILLS_DIR" -# Copy all skills from databricks-skills (excluding non-skill directories) -SKILLS_SRC="$SCRIPT_DIR/../../../databricks-skills" +# Copy all skills from the bundled skills snapshot (excluding non-skill dirs). +# The bundled skill copies moved to DEPRECATED-databricks-skills/ at the repo +# root; prefer that snapshot and fall back to the legacy databricks-skills/ location. +SKILLS_SRC="$SCRIPT_DIR/../../../DEPRECATED-databricks-skills" +if [ ! -d "$SKILLS_SRC" ]; then + SKILLS_SRC="$SCRIPT_DIR/../../../databricks-skills" +fi if [ -d "$SKILLS_SRC" ]; then # Find all directories containing SKILL.md (these are actual skills) for skill_dir in "$SKILLS_SRC"/*/; do diff --git a/databricks-builder-app/scripts/deploy.sh b/databricks-builder-app/scripts/deploy.sh index 997bbcd4..2c3cf444 100755 --- a/databricks-builder-app/scripts/deploy.sh +++ b/databricks-builder-app/scripts/deploy.sh @@ -18,6 +18,18 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_DIR="$(dirname "$SCRIPT_DIR")" REPO_ROOT="$(dirname "$PROJECT_DIR")" +# Repo-local bundled (frozen) Databricks skills snapshot. The bundled skill +# copies moved to DEPRECATED-databricks-skills/ at the repo root; older checkouts +# still keep them directly under databricks-skills/. Prefer the new location, +# fall back to the legacy one so this works before and after the move. +SKILLS_SNAPSHOT_DIR="$REPO_ROOT/DEPRECATED-databricks-skills" +if [ ! -d "$SKILLS_SNAPSHOT_DIR" ]; then + SKILLS_SNAPSHOT_DIR="$REPO_ROOT/databricks-skills" +fi + +# install_skills.sh (adds MLflow skills fetched from github.com/mlflow/skills). +INSTALL_SKILLS_SCRIPT="$REPO_ROOT/databricks-skills/install_skills.sh" + APP_NAME="${APP_NAME:-}" PROFILE="${PROFILE:-}" STAGING_DIR="" @@ -237,27 +249,52 @@ if [ "$SKIP_SKILLS" = true ] && [ -d "$SKILLS_CACHE_DIR" ] && [ "$(ls -A "$SKILL echo -e " ${GREEN}✓${NC} Reusing cached skills from ${SKILLS_CACHE_DIR} (--skip-skills)" cp -r "$SKILLS_CACHE_DIR"/* "$STAGING_DIR/skills/" else - echo " Installing skills via install_skills.sh..." -INSTALL_SKILLS_SCRIPT="$REPO_ROOT/databricks-skills/install_skills.sh" -if [ ! -f "$INSTALL_SKILLS_SCRIPT" ]; then echo -e "${RED}Error: install_skills.sh not found${NC}"; exit 1; fi - -SKILLS_TEMP_DIR=$(mktemp -d) -trap "rm -rf '$SKILLS_TEMP_DIR'" EXIT -touch "$SKILLS_TEMP_DIR/databricks.yml" -(cd "$SKILLS_TEMP_DIR" && bash "$INSTALL_SKILLS_SCRIPT") - -mkdir -p "$STAGING_DIR/skills" -INSTALLED_SKILLS_DIR="$SKILLS_TEMP_DIR/.claude/skills" -if [ -d "$INSTALLED_SKILLS_DIR" ]; then - for skill_dir in "$INSTALLED_SKILLS_DIR"/*/; do - [ -d "$skill_dir" ] || continue - skill_name=$(basename "$skill_dir") - if [ -f "$skill_dir/SKILL.md" ]; then + mkdir -p "$STAGING_DIR/skills" + + # 1. Ship the bundled (frozen) Databricks skills snapshot. These copies live + # in-repo and work fully offline, so they are the authoritative source for + # Databricks skills regardless of network access. + if [ -d "$SKILLS_SNAPSHOT_DIR" ]; then + echo " Bundling frozen skills snapshot from ${SKILLS_SNAPSHOT_DIR}..." + for skill_dir in "$SKILLS_SNAPSHOT_DIR"/*/; do + [ -d "$skill_dir" ] || continue + [ -f "$skill_dir/SKILL.md" ] || continue + skill_name=$(basename "$skill_dir") + # Skip non-skill scaffolding (TEMPLATE, dotfiles; deprecated kept defensively). + case "$skill_name" in TEMPLATE|deprecated|.*) continue ;; esac mkdir -p "$STAGING_DIR/skills/$skill_name" cp -r "$skill_dir"* "$STAGING_DIR/skills/$skill_name/" + done + else + echo -e " ${YELLOW}⚠${NC} Skills snapshot not found at ${SKILLS_SNAPSHOT_DIR}" + fi + + # 2. Supplement with install_skills.sh (adds MLflow skills fetched from + # github.com/mlflow/skills, plus any skill not already in the snapshot). + # Best-effort: if it is missing or fails, the frozen snapshot still ships. + if [ ! -f "$INSTALL_SKILLS_SCRIPT" ]; then + echo -e " ${YELLOW}⚠${NC} install_skills.sh not found — shipping snapshot skills only" + else + echo " Adding external skills via install_skills.sh..." + SKILLS_TEMP_DIR=$(mktemp -d) + trap "rm -rf '$SKILLS_TEMP_DIR'" EXIT + touch "$SKILLS_TEMP_DIR/databricks.yml" + (cd "$SKILLS_TEMP_DIR" && bash "$INSTALL_SKILLS_SCRIPT") \ + || echo -e " ${YELLOW}⚠${NC} install_skills.sh failed — shipping snapshot skills only" + + INSTALLED_SKILLS_DIR="$SKILLS_TEMP_DIR/.claude/skills" + if [ -d "$INSTALLED_SKILLS_DIR" ]; then + for skill_dir in "$INSTALLED_SKILLS_DIR"/*/; do + [ -d "$skill_dir" ] || continue + skill_name=$(basename "$skill_dir") + # Snapshot wins: only add skills not already bundled. + if [ -f "$skill_dir/SKILL.md" ] && [ ! -d "$STAGING_DIR/skills/$skill_name" ]; then + mkdir -p "$STAGING_DIR/skills/$skill_name" + cp -r "$skill_dir"* "$STAGING_DIR/skills/$skill_name/" + fi + done fi - done -fi + fi # Cache skills for --skip-skills on next run rm -rf "$SKILLS_CACHE_DIR" diff --git a/databricks-builder-app/scripts/start_local.sh b/databricks-builder-app/scripts/start_local.sh index 7b50baa0..c5fac8ad 100755 --- a/databricks-builder-app/scripts/start_local.sh +++ b/databricks-builder-app/scripts/start_local.sh @@ -230,11 +230,11 @@ echo "" # ───────────────────────────────────────────────────────────────────────────── echo -e "${YELLOW}[7/${TOTAL_STEPS}] Installing skills...${NC}" -# Run install_skills.sh from the project dir — installs to .claude/skills/ +# Run install_skills.sh from the project dir — installs to .claude/skills/. INSTALL_SKILLS_SCRIPT="$REPO_ROOT/databricks-skills/install_skills.sh" if [ ! -f "$INSTALL_SKILLS_SCRIPT" ]; then - echo -e " ${YELLOW}⚠${NC} install_skills.sh not found — using existing skills only" + echo -e " ${YELLOW}⚠${NC} install_skills.sh not found — using bundled skills snapshot only" else # Run from PROJECT_DIR so skills install to databricks-builder-app/.claude/skills/ cd "$PROJECT_DIR" @@ -242,16 +242,19 @@ else cd "$PROJECT_DIR" fi -# Scan skills from .claude/skills/ (where install_skills.sh puts them) -# and from ../databricks-skills/ (local repo skills) — union of both +# Scan skills from .claude/skills/ (where install_skills.sh puts them) and from +# the repo-local skills snapshot (DEPRECATED-databricks-skills/ preferred, with +# the legacy databricks-skills/ location as a fallback) — union of all. SKILL_NAMES="" SKILL_COUNT=0 -for skills_root in "$PROJECT_DIR/.claude/skills" "$REPO_ROOT/databricks-skills"; do +for skills_root in "$PROJECT_DIR/.claude/skills" "$REPO_ROOT/DEPRECATED-databricks-skills" "$REPO_ROOT/databricks-skills"; do [ -d "$skills_root" ] || continue for skill_dir in "$skills_root"/*/; do [ -d "$skill_dir" ] || continue if [ -f "$skill_dir/SKILL.md" ]; then name=$(basename "$skill_dir") + # Skip non-skill scaffolding (TEMPLATE, dotfiles; deprecated kept defensively). + case "$name" in TEMPLATE|deprecated|.*) continue ;; esac # Avoid duplicates if ! echo ",$SKILL_NAMES," | grep -q ",$name,"; then if [ -n "$SKILL_NAMES" ]; then SKILL_NAMES="${SKILL_NAMES},${name}"; else SKILL_NAMES="${name}"; fi diff --git a/databricks-builder-app/server/services/skills_manager.py b/databricks-builder-app/server/services/skills_manager.py index 8d30f578..8a2c052b 100644 --- a/databricks-builder-app/server/services/skills_manager.py +++ b/databricks-builder-app/server/services/skills_manager.py @@ -96,11 +96,17 @@ def get_allowed_mcp_tools( # Candidate source directories (checked in priority order): # 1. .claude/skills/ inside the app — populated by install_skills.sh with # the *full* union of Databricks + MLflow skills. -# 2. Sibling ../../databricks-skills — the repo-local Databricks-only skills. +# 2. The repo-local, bundled (frozen) Databricks skills snapshot. The +# bundled skill copies were moved to DEPRECATED-databricks-skills/ at the +# repo root; older checkouts still have them directly under +# databricks-skills/. We prefer the DEPRECATED-databricks-skills/ snapshot +# and fall back to the legacy location so the app works before and after +# that move. # 3. ./skills at app root — the deployed bundle location. _APP_ROOT = Path(__file__).parent.parent.parent _INSTALLED_SKILLS_DIR = _APP_ROOT / '.claude' / 'skills' -_DEV_SKILLS_DIR = _APP_ROOT.parent / 'databricks-skills' +_DEPRECATED_SKILLS_DIR = _APP_ROOT.parent / 'DEPRECATED-databricks-skills' +_LEGACY_DEV_SKILLS_DIR = _APP_ROOT.parent / 'databricks-skills' _DEPLOYED_SKILLS_DIR = _APP_ROOT / 'skills' # Local cache of skills within this app (copied on startup) @@ -109,6 +115,17 @@ def get_allowed_mcp_tools( def _non_empty_dir(p: Path) -> bool: return p.exists() and p.is_dir() and any(p.iterdir()) + +def _resolve_dev_skills_dir() -> Path: + """Resolve the repo-local skills snapshot: DEPRECATED-databricks-skills/ first, else legacy.""" + if _non_empty_dir(_DEPRECATED_SKILLS_DIR): + return _DEPRECATED_SKILLS_DIR + return _LEGACY_DEV_SKILLS_DIR + + +# Repo-local skills snapshot (DEPRECATED-databricks-skills/ preferred, legacy fallback). +_DEV_SKILLS_DIR = _resolve_dev_skills_dir() + # Build an ordered list of source directories. The first directory that # contains a given skill wins, so put the most-complete source first. _SKILLS_SOURCE_DIRS: list[Path] = [] diff --git a/databricks-mcp-server/README.md b/databricks-mcp-server/README.md index f3d7b459..f564f2f2 100644 --- a/databricks-mcp-server/README.md +++ b/databricks-mcp-server/README.md @@ -1,6 +1,21 @@ # Databricks MCP Server -A simple [FastMCP](https://github.com/jlowin/fastmcp) server that exposes Databricks operations as MCP tools for AI assistants like Claude Code. +A simple [FastMCP](https://github.com/jlowin/fastmcp) server that exposes Databricks operations as MCP tools for AI assistants like Claude Code, Cursor, and Genie Code. + +This server is a **self-contained install**. It has no dependency on Databricks skills — follow the steps below and the server runs on its own. (If you *also* want skills, see the optional section at the end.) + +## Prerequisites + +- **[uv](https://docs.astral.sh/uv/)** — used to create the virtual environment and install the packages: + ```bash + curl -LsSf https://astral.sh/uv/install.sh | sh + ``` +- **Python 3.9+** (uv can install one for you with `uv venv --python 3.11`). +- **[Databricks CLI](https://docs.databricks.com/dev-tools/cli/install.html)** with a configured auth profile, so the server can reach your workspace. Verify with: + ```bash + databricks auth profiles + ``` + If you have none, create one with `databricks auth login --host https://your-workspace.cloud.databricks.com`. ## Quick Start @@ -11,68 +26,86 @@ git clone https://github.com/databricks-solutions/ai-dev-kit.git cd ai-dev-kit ``` -### Step 2: Install the packages +### Step 2: Install the server -```bash -# Install the core library -uv pip install -e ./databricks-tools-core +The MCP server depends on the `databricks-tools-core` library, which also lives in this repo. Install both as editable packages into a virtual environment. -# Install the MCP server -uv pip install -e ./databricks-mcp-server +You can do this in one shot with the setup script, which creates a `.venv`, installs both packages, verifies the import, and prints ready-to-paste client config: + +```bash +./databricks-mcp-server/setup.sh ``` -### Step 3: Configure Databricks authentication +Or run the steps manually: ```bash -# Option 1: Environment variables -export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com" -export DATABRICKS_TOKEN="your-token" +# Create and activate a virtual environment +uv venv --python 3.11 +source .venv/bin/activate -# Option 2: Use a profile from ~/.databrickscfg -export DATABRICKS_CONFIG_PROFILE="your-profile" +# Install the core library, then the MCP server (both editable, from this repo) +uv pip install -e ./databricks-tools-core -e ./databricks-mcp-server ``` -### Step 4: Add MCP server to Claude Code +Verify the server imports cleanly: + +```bash +python -c "import databricks_mcp_server; print('OK')" +``` -For Claude Code, add to your project's `.mcp.json` (create the file if it doesn't exist). -For Cursor, add to your project's `.cursor/mcp.json` (create the file if it doesn't exist). +### Step 3: Configure your MCP client + +Point your MCP client at the server's `run_server.py`. Use the Python interpreter from the `.venv` you just created (replace `/path/to/ai-dev-kit` with the absolute path where you cloned the repo). + +**Claude Code** — add to your project's `.mcp.json` (create the file if it doesn't exist): ```json { "mcpServers": { "databricks": { - "command": "uv", - "args": ["run", "--directory", "/path/to/ai-dev-kit", "python", "databricks-mcp-server/run_server.py"], + "command": "/path/to/ai-dev-kit/.venv/bin/python", + "args": ["/path/to/ai-dev-kit/databricks-mcp-server/run_server.py"], "defer_loading": true } } } ``` -**Replace `/path/to/ai-dev-kit`** with the actual path where you cloned the repo. +Or register it from the CLI: + +```bash +claude mcp add-json databricks '{"command":"/path/to/ai-dev-kit/.venv/bin/python","args":["/path/to/ai-dev-kit/databricks-mcp-server/run_server.py"]}' +``` + +**Cursor / Genie Code** — use the same JSON in your client's MCP config (e.g. Cursor's `.cursor/mcp.json`). **Note:** `"defer_loading": true` improves startup time by not loading all tools upfront. -### Step 5 (Recommended): Install Databricks skills +### Step 4: Authenticate -The MCP server works best with **Databricks skills** that teach Claude best practices: +The server uses the Databricks Unified Authentication chain, so it picks up whatever the Databricks CLI/SDK already uses. Choose a profile in one of these ways: ```bash -# In your project directory (not ai-dev-kit) -cd /path/to/your/project -curl -sSL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/databricks-skills/install_skills.sh | bash +# Option 1: Named profile from ~/.databrickscfg (recommended) +export DATABRICKS_CONFIG_PROFILE="your-profile" + +# Option 2: Explicit host + token +export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com" +export DATABRICKS_TOKEN="your-token" ``` -### Step 6: Start Claude Code +To make a profile available to a GUI MCP client, add it to the `env` block of the server config, e.g. `"env": {"DATABRICKS_CONFIG_PROFILE": "your-profile"}`. + +### Step 5: Smoke test + +Confirm the server starts and can reach your workspace: ```bash -cd /path/to/your/project -claude +# The server speaks MCP over stdio; it should start without errors, then Ctrl-C to exit. +.venv/bin/python databricks-mcp-server/run_server.py ``` -Claude now has both: -- **Skills** (knowledge) - patterns and best practices in `.claude/skills/` -- **MCP Tools** (actions) - Databricks operations via the MCP server +Then in your MCP client, ask it to run a lightweight tool such as `list_warehouses` or `get_current_user`. A successful response confirms the server is installed, launched, and authenticated. ## Available Tools @@ -169,13 +202,10 @@ Claude now has both: ``` ┌─────────────────────────────────────────────────────────────┐ -│ Claude Code │ +│ MCP Client (Claude Code / Cursor / …) │ │ │ -│ Skills (knowledge) MCP Tools (actions) │ -│ └── .claude/skills/ └── .claude/mcp.json │ -│ ├── sdp-writer └── databricks server │ -│ ├── databricks-bundles │ -│ └── ... │ +│ MCP Tools (actions) │ +│ └── .mcp.json ──► databricks server │ └──────────────────────────────┬──────────────────────────────┘ │ MCP Protocol (stdio) ▼ @@ -264,6 +294,17 @@ The project name is auto-detected from the git remote URL (no configuration need > **Note:** Audit log entries may take 2–10 minutes to appear. The workspace must have Unity Catalog enabled to query `system.access.audit`. +## Optional: If you also want Databricks skills + +This MCP server runs completely on its own — you do **not** need skills for any of the steps above. Skills are a *separate*, optional add-on that give AI assistants written guidance (patterns and best practices) to complement the executable tools this server provides. + +If you want them, install them separately — do not combine the two installs: + +- **Databricks CLI (recommended):** `databricks aitools install` (requires Databricks CLI v1.0.0+). This is the supported way to get the latest skills. +- **AI Dev Kit installer:** run the repo's top-level `install.sh` and choose the skills option. + +See the [ai-dev-kit README](../README.md) for details. Skills are installed into your own project (e.g. `.claude/skills/`) and are picked up independently of this server. + ## License © Databricks, Inc. See [LICENSE.md](../LICENSE.md). diff --git a/databricks-mcp-server/setup.ps1 b/databricks-mcp-server/setup.ps1 new file mode 100644 index 00000000..836cb23a --- /dev/null +++ b/databricks-mcp-server/setup.ps1 @@ -0,0 +1,164 @@ +# +# Setup script for databricks-mcp-server (Windows). +# +# Creates the Python virtual environment and installs the MCP server (plus its +# databricks-tools-core dependency) in editable mode, then verifies the install. +# +# This is the single source of truth for building the MCP server runtime on +# Windows. The unified installers only write the editor config files that point +# at the venv — when the user opts into the (deprecated) MCP server, they +# delegate the actual environment build here. +# +# Usage: +# .\databricks-mcp-server\setup.ps1 [OPTIONS] +# +# Options: +# -VenvDir DIR Location for the virtual environment (default: \.venv) +# -Python VER Python version to request from uv (default: 3.11) +# -Quiet Suppress progress output (errors still print) +# -Help Show this help +# + +$ErrorActionPreference = "Stop" + +# ─── Parse arguments (accept both -Style and --style for parity with bash) ─── +$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path +$ParentDir = Split-Path -Parent $ScriptDir +$ToolsCoreDir = Join-Path $ParentDir "databricks-tools-core" + +$VenvDir = Join-Path $ScriptDir ".venv" +$PythonVersion = "3.11" +$Quiet = $false + +$i = 0 +while ($i -lt $args.Count) { + switch ($args[$i]) { + { $_ -in "--venv-dir", "-VenvDir" } { $VenvDir = $args[$i + 1]; $i += 2 } + { $_ -in "--python", "-Python" } { $PythonVersion = $args[$i + 1]; $i += 2 } + { $_ -in "--quiet", "-Quiet" } { $Quiet = $true; $i++ } + { $_ -in "-h", "--help", "-Help" } { + Write-Host "Setup the Databricks MCP server runtime (Windows)" + Write-Host "" + Write-Host "Usage: .\databricks-mcp-server\setup.ps1 [OPTIONS]" + Write-Host "" + Write-Host " -VenvDir DIR Virtual environment location (default: