[ZEPPELIN-4495] Unify REST and WebSocket authentication with Shiro

[jongyoul]

What is this PR for?

Zeppelin currently authenticates REST requests through Shiro while notebook WebSocket connections use a separate client-provided principal/ticket flow. This PR establishes one server-side Shiro session identity for both transports.

The implementation:

• sends both /api/* and the /ws HTTP upgrade through the same Shiro filter
• installs Jetty's standard WebSocket upgrade filter after Shiro without upgrading Jetty
• captures an immutable authenticated identity during the upgrade and revalidates the exact Shiro session for inbound frames and outbound delivery
• stops treating client-provided principal, roles, and ticket message fields as authentication credentials
• uses the first-match Shiro [urls] rules as the common include/exclude policy, including explicit /ws = authc or /ws = anon
• closes the exact session's WebSockets on logout, relogin, expiry, or identity change
• validates WebSocket Origin consistently with credentialed REST CORS policy
• keeps notebook ACL checks separate from authentication and makes ACL/path authorization snapshots fail closed across concurrent notebook mutations
• preserves rename-to-existing-folder behavior without overwriting data by preflighting note/folder collisions, moving notes individually, compensating partial repository failures, and failing metadata access closed when rollback recovery cannot reconcile state
• upgrades Shiro to 2.2.1 Jakarta and Bouncy Castle to 1.84 while retaining Jetty 11.0.24
• updates the Java client, Angular UI, classic UI, configuration, and upgrade documentation

What type of PR is it?

Improvement

Todos

• Use one Shiro session identity for REST and WebSocket traffic
• Support common include/exclude policy through Shiro URL rules
• Revalidate session and authorization on WebSocket input and output
• Remove client-asserted WebSocket authentication fields
• Cover logout, relogin, expiry, Origin, ACL, and mutation races
• Preserve safe existing-destination folder merges and repository rollback/recovery
• Update both web clients, the Java client, configuration, and documentation
• Complete GitHub Actions validation

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-4495

Related work and overlap:

• ZEPPELIN-6528 / [ZEPPELIN-6528] Upgrade Apache Shiro from 1.13.0 to 2.0.6 #5293 and dependency PRs Bump org.apache.shiro:shiro-core from 1.13.0 to 2.2.1 in /zeppelin-server #5274 and Bump org.apache.shiro:shiro-web from 1.13.0 to 2.2.0 in /zeppelin-server #5281 also upgrade Shiro. This PR includes the migration because the shared session boundary depends on current Shiro 2.2.1 behavior and security fixes.
• [ZEPPELIN-5858] Fix moveNote/saveNote race that duplicates notes #5325 addresses the base move/save race in NoteManager. This PR includes the overlapping serialization plus authorization/metadata generation checks, folder and revision operations, and repository compensation/reconciliation needed to keep a checked identity/ACL bound to one path generation while preserving existing-folder merge behavior.
• [ZEPPELIN-6489] Fix stale Shiro authentication documentation links #5361 also touches conf/shiro.ini.template for an unrelated documentation-link correction.
• CI audit follow-ups keep the unchanged frontend dependency graph secure: the existing brace-expansion override moves from 5.0.8 to 5.0.9 for GHSA-rgw5-rvv9-x895, while the lockfile moves fast-uri to 3.1.5 and undici to 7.29.0 for newly published advisories. npm audit reports 0 vulnerabilities.

This remains intentionally a draft while overlap with those contributions is coordinated.

How should this be tested?

Local verification completed from current apache/master (404866f4171efa27abdf327a4f3d9ffaf3771909):

• packaged-distribution Maven verify: 23/23 reactor modules
• zeppelin-server: 1,693 tests, 0 failures/errors, 21 skipped
• classic web unit tests: 191/191
• isolated plugin build: 117/117
• Git/VFS/NoteManager focused tests: 57/57, including existing-folder merge, collision preflight, partial-move compensation, recovery failure, and no-overwrite coverage
• Java client and WebSocket/Shiro authentication focused tests
• Apache RAT and Maven validate -Pinclude-hadoop
• Angular and classic web lint/build
• independent authentication, concurrency, and frontend audits
• frontend dependency audit follow-up: npm audit reports 0 vulnerabilities, Vitest 14/14, lint and production build pass
• GitHub Actions authentication regression coverage passes in Selenium and both anonymous and authenticated Playwright suites
• all exact-head GitHub Actions workflows pass

Screenshots (if appropriate)

N/A

Questions:

• Does the license files need to update? No.
• Is there breaking changes for older versions? Custom WebSocket clients must retain the authenticated HTTP session cookie and must no longer rely on client-supplied principal/ticket fields. Custom Shiro realms may require the documented Shiro 2.x package migration.
• Does this needs documentation? Yes; configuration, authentication, upgrade, REST, and Java client documentation are included.