Merge pull request #116 from ProvarTesting/feature/auth-and-quality-hub-api

Feature/auth and quality hub api

Author: mrdailey99

.github/workflows/CI_Execution.yml

@@ -124,7 +124,9 @@ jobs:
           sf plugins link .
           yarn run test:nuts
       - name: Archive NUTS results
+        if: always()
         uses: actions/upload-artifact@v4
         with:
           name: nuts-report-${{ matrix.os }}
           path: mochawesome-report
+          if-no-files-found: warn

README.md

@@ -3,17 +3,20 @@
 [![Version](https://img.shields.io/npm/v/@provartesting/provardx-cli.svg)](https://npmjs.org/package/@provartesting/provardx-cli)
 [![Downloads/week](https://img.shields.io/npm/dw/@provartesting/provardx-cli.svg)](https://npmjs.org/package/@provartesting/provardx-cli)
 [![License](https://img.shields.io/npm/l/@provartesting/provardx-cli.svg)](https://github.com/ProvarTesting/provardx-cli/blob/main/LICENSE.md)
+[![Get Access](https://img.shields.io/badge/Quality%20Hub%20API-Get%20Access-blue)](https://aqqlrlhga7.execute-api.us-east-1.amazonaws.com/dev/auth/request-access)
 
 # What is the ProvarDX CLI?
 
 The Provar DX CLI is a Salesforce CLI plugin for Provar customers who want to automate the execution of tests using Provar Automation, and the reporting of test results and other quality-related metrics to Provar Quality Hub.
 
 # Installation, Update, and Uninstall
 
+**Requires Node.js 18–24 (LTS 22 recommended).** Node 25+ is not yet supported due to a breaking change in a transitive dependency. Check with `node --version`.
+
 Install the plugin
 
 ```sh-session
-$ sf plugins install @provartesting/provardx-cli
+$ sf plugins install @provartesting/provardx-cli@beta
 ```
 
 Update plugins
@@ -30,33 +33,56 @@ $ sf plugins uninstall @provartesting/provardx-cli
 
 # MCP Server (AI-Assisted Quality)
 
-The Provar DX CLI includes a built-in **Model Context Protocol (MCP) server** that connects AI assistants (Claude Desktop, Claude Code, Cursor) directly to your Provar project. Once connected, an AI agent can inspect your project structure, generate Page Objects and test cases, validate every level of the test hierarchy with quality scores that match the Provar Quality Hub API, and work with NitroX (Hybrid Model) component page objects for LWC, Screen Flow, Industry Components, Experience Cloud, and HTML5.
+The Provar DX CLI includes a built-in **Model Context Protocol (MCP) server** that connects AI assistants (Claude Desktop, Claude Code, Cursor) directly to your Provar project. Once connected, an AI agent can inspect your project structure, generate Page Objects and test cases, validate every level of the test hierarchy with quality scores, and work with NitroX (Hybrid Model) component page objects for LWC, Screen Flow, Industry Components, Experience Cloud, and HTML5.
+
+Validation runs in two modes: **local only** (structural rules, no key required) or **Quality Hub API** (170+ rules, quality scoring — requires a `pv_k_` API key). Don't have an account? **[Request access](https://aqqlrlhga7.execute-api.us-east-1.amazonaws.com/dev/auth/request-access)**.
+
+## Quick setup
+
+**Requires:** Provar Automation IDE installed with an activated license.
 
 ```sh
-sf provar mcp start --allowed-paths /path/to/your/provar/project
+# 1. Install the plugin (if not already installed)
+sf plugins install @provartesting/provardx-cli@beta
+
+# 2. (Optional) Authenticate for full 170+ rule validation
+sf provar auth login
 ```
 
-📖 **See [docs/mcp.md](https://github.com/ProvarTesting/provardx-cli/blob/main/docs/mcp.md) for full setup and tool documentation.**
+**Claude Code** — run once to register the server:
 
-## License Validation
+```sh
+claude mcp add provar -s user -- sf provar mcp start --allowed-paths /path/to/your/provar/project
+```
 
-The MCP server verifies your Provar license before accepting any connections. Validation is automatic — no extra flags are required for standard usage.
+**Claude Desktop** — add to your config file and restart the app:
 
-**How it works:**
+- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
+- Windows: `%APPDATA%\Claude\claude_desktop_config.json`
 
-1. **Auto-detection** — the server reads `~/Provar/.licenses/*.properties` (the same files written by Provar's IDE plugins). If a valid, activated license is found the server starts immediately.
-2. **Cache** — successful validations are cached at `~/Provar/.licenses/.mcp-license-cache.json` (2 h TTL). Subsequent starts within the TTL window skip the disk scan.
-3. **Grace fallback** — if the IDE license files cannot be found or read and the cache is stale (but ≤ 48 h old), the server starts with a warning on stderr using the cached result so CI pipelines are not broken by transient local file-access issues.
-4. **Fail closed** — if no valid license is detected the command exits with a non-zero exit code and a clear error message.
+```json
+{
+  "mcpServers": {
+    "provar": {
+      "command": "sf",
+      "args": ["provar", "mcp", "start", "--allowed-paths", "/path/to/your/provar/project"]
+    }
+  }
+}
+```
 
-**`NODE_ENV=test` fast-path:**
+> **Windows (Claude Desktop):** Use `sf.cmd` instead of `sf` if the server fails to start.
 
-When `NODE_ENV=test` the validation step is skipped entirely. This is intended only for the plugin's own unit-test suite.
+📖 **[docs/mcp.md](https://github.com/ProvarTesting/provardx-cli/blob/main/docs/mcp.md) — full setup, all 35+ tools, troubleshooting.**
 
 ---
 
 # Commands
 
+- [`sf provar auth login`](#sf-provar-auth-login)
+- [`sf provar auth rotate`](#sf-provar-auth-rotate)
+- [`sf provar auth status`](#sf-provar-auth-status)
+- [`sf provar auth clear`](#sf-provar-auth-clear)
 - [`sf provar mcp start`](#sf-provar-mcp-start)
 - [`sf provar config get`](#sf-provar-config-get)
 - [`sf provar config set`](#sf-provar-config-set)
@@ -84,6 +110,99 @@ When `NODE_ENV=test` the validation step is skipped entirely. This is intended o
 - [`sf provar manager test run report`](#sf-provar-manager-test-run-report) _(deprecated — use `sf provar quality-hub test run report`)_
 - [`sf provar manager test run abort`](#sf-provar-manager-test-run-abort) _(deprecated — use `sf provar quality-hub test run abort`)_
 
+## `sf provar auth login`
+
+Log in to Provar Quality Hub and store your API key.
+
+```
+USAGE
+  $ sf provar auth login [--url <value>]
+
+FLAGS
+  --url=<value>  Override the Quality Hub API base URL (for non-production environments).
+
+DESCRIPTION
+  Opens a browser to the Provar login page. After you authenticate, your API key is
+  stored at ~/.provar/credentials.json and used automatically by the Provar MCP tools
+  and CI/CD integrations. The key is valid for approximately 90 days.
+
+  For CI/CD pipelines (GitHub Actions, Jenkins, etc.) where a browser cannot open:
+  run sf provar auth login once on your local machine, copy the api_key value from
+  ~/.provar/credentials.json, and store it as the PROVAR_API_KEY environment variable
+  or secret in your pipeline. Rotate the secret every ~90 days when the key expires.
+
+  Don't have an account? Request access at:
+  https://aqqlrlhga7.execute-api.us-east-1.amazonaws.com/dev/auth/request-access
+
+EXAMPLES
+  Log in interactively:
+
+    $ sf provar auth login
+
+  Log in against a staging environment:
+
+    $ sf provar auth login --url https://dev.api.example.com
+```
+
+## `sf provar auth rotate`
+
+Rotate your stored API key without re-authenticating via browser.
+
+```
+USAGE
+  $ sf provar auth rotate
+
+DESCRIPTION
+  Exchanges your current pv_k_ key for a new one atomically. The old key is
+  invalidated immediately. The new key is written to ~/.provar/credentials.json.
+
+  Use this to rotate your key on a regular schedule (~every 90 days) without
+  going through the browser login flow. If your current key is already expired,
+  run sf provar auth login instead.
+
+EXAMPLES
+  Rotate the stored API key:
+
+    $ sf provar auth rotate
+```
+
+## `sf provar auth status`
+
+Show the current API key configuration and validate it against Quality Hub.
+
+```
+USAGE
+  $ sf provar auth status
+
+DESCRIPTION
+  Reports whether an API key is configured, where it came from (environment variable
+  or credentials file), and performs a live check against the Quality Hub API to
+  confirm the key is still valid.
+
+EXAMPLES
+  Check auth status:
+
+    $ sf provar auth status
+```
+
+## `sf provar auth clear`
+
+Remove the stored API key.
+
+```
+USAGE
+  $ sf provar auth clear
+
+DESCRIPTION
+  Deletes ~/.provar/credentials.json and revokes the key server-side. After clearing,
+  the MCP tools fall back to local validation mode. Has no effect if no key is stored.
+
+EXAMPLES
+  Remove the stored key:
+
+    $ sf provar auth clear
+```
+
 ## `sf provar mcp start`
 
 Start a local MCP server for Provar tools over stdio transport.
@@ -136,6 +255,11 @@ TOOLS EXPOSED
   provar.testplan.add-instance         — wire a test case into a plan suite by writing a .testinstance file
   provar.testplan.create-suite         — create a new test suite directory with .planitem inside a plan
   provar.testplan.remove-instance      — remove a .testinstance file from a plan suite
+  provar.nitrox.discover               — discover projects containing NitroX (Hybrid Model) page objects
+  provar.nitrox.read                   — read NitroX .po.json files and return parsed content
+  provar.nitrox.validate               — validate a NitroX .po.json against schema rules
+  provar.nitrox.generate               — generate a new NitroX .po.json from a component description
+  provar.nitrox.patch                  — apply a JSON merge-patch to an existing NitroX .po.json file
 
 EXAMPLES
   Start MCP server (accepts stdio connections from Claude Desktop / Cursor):

docs/mcp-pilot-guide.md

@@ -25,9 +25,9 @@ The server runs **locally on your machine**. It does not phone home, transmit yo
 | --------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
 | Provar Automation IDE       | ≥ 2.x   | Must be installed with an **activated licence** on the same machine. The MCP server reads the licence from `~/Provar/.licenses/`. |
 | Salesforce CLI (`sf`)       | ≥ 2.x   | `npm install -g @salesforce/cli`                                                                                                  |
-| Provar DX CLI plugin        | ≥ 1.5.0 | `sf plugins install @provartesting/provardx-cli`                                                                                  |
+| Provar DX CLI plugin        | ≥ 1.5.0 | `sf plugins install @provartesting/provardx-cli@beta`                                                                                  |
 | An MCP-compatible AI client | —       | Claude Desktop, Claude Code, or Cursor                                                                                            |
-| Node.js                     | ≥ 18    | Installed automatically with the SF CLI                                                                                           |
+| Node.js                     | 18–24   | Installed automatically with the SF CLI. **Node 25+ is not supported** — a transitive dependency crashes on startup. Use Node 22 LTS. |
 
 ---
 
@@ -48,7 +48,7 @@ sf --version
 ### 2. Install the Provar DX CLI plugin
 
 ```sh
-sf plugins install @provartesting/provardx-cli
+sf plugins install @provartesting/provardx-cli@beta
 ```
 
 Verify:
@@ -165,6 +165,9 @@ Prompt your AI assistant:
 - `validity_score` and `quality_score` both returned (0–100)
 - Specific rule violations called out (e.g. TC_010 missing test case ID, TC_001 missing XML declaration)
 - Best-practices suggestions (e.g. hardcoded credentials, missing step descriptions)
+- `validation_source: "local"` if no API key is configured, `"quality_hub"` if authenticated
+
+> **Tip:** Run `sf provar auth login` before this scenario to unlock Quality Hub API validation (170+ rules). Without a key the tool still returns useful results using local rules only.
 
 ---
 
@@ -226,6 +229,24 @@ Pre-requisite: `sf org login web -a MyQHOrg` then `sf provar quality-hub connect
 
 ---
 
+### Scenario 8: Quality Hub API Validation
+
+**Goal:** Confirm that `provar.testcase.validate` upgrades from local rules to the full Quality Hub API ruleset when an API key is present.
+
+**Setup:** Run `sf provar auth login` and complete the browser login, then confirm with `sf provar auth status`.
+
+> "Validate the test case at `/path/to/project/tests/LoginTest.testcase` and tell me what validation_source was used."
+
+**What to look for:**
+
+- `validation_source: "quality_hub"` in the response — confirms the API path is active
+- `quality_score` reflecting the full 170+ rule evaluation
+- If the API is unreachable, `validation_source: "local_fallback"` and a `validation_warning` field explaining why
+
+**To reset and test the fallback:** run `sf provar auth clear`, repeat the prompt, and verify `validation_source` reverts to `"local"`.
+
+---
+
 ### Scenario 7: NitroX (Hybrid Model) Page Object Generation
 
 **Goal:** Have the AI discover, understand, and generate NitroX component page objects.
@@ -323,7 +344,9 @@ The MCP server uses **stdio transport** exclusively. Communication travels over
 
 ### Credential handling
 
-The Quality Hub and Automation tools invoke `sf` subprocesses. Salesforce org credentials are managed entirely by the Salesforce CLI and stored in its own credential store. The Provar MCP server never reads, parses, or transmits those credentials.
+**Salesforce org credentials** — the Quality Hub and Automation tools invoke `sf` subprocesses. Salesforce org credentials are managed entirely by the Salesforce CLI and stored in its own credential store (`~/.sf/`). The Provar MCP server never reads, parses, or transmits those credentials.
+
+**Provar API key** — the `provar.testcase.validate` tool optionally reads a `pv_k_` API key to enable Quality Hub API validation. The key is stored at `~/.provar/credentials.json` (written by `sf provar auth login`) or read from the `PROVAR_API_KEY` environment variable. The key is sent to the Provar Quality Hub API only when a validation request is made — it is never logged or written anywhere other than `~/.provar/credentials.json`.
 
 ### Path policy enforcement
 
@@ -393,7 +416,7 @@ After editing `claude_desktop_config.json`, you must fully restart Claude Deskto
 
 **Server starts but immediately exits**
 
-Check that the SF CLI plugin is installed: `sf plugins | grep provardx`. If missing, run `sf plugins install @provartesting/provardx-cli`.
+Check that the SF CLI plugin is installed: `sf plugins | grep provardx`. If missing, run `sf plugins install @provartesting/provardx-cli@beta`.
 
 ---