Decouple zeppelin-server from Hadoop

[jongyoul]

What is this PR for?

Decouple zeppelin-server from Hadoop so the server can run on a modern JDK without loading Hadoop merely for optional storage, YARN monitoring, Kerberos, group mapping, or JCEKS support.

This draft:

• removes all Hadoop artifacts from the transitive zeppelin-server dependency graph and enforces that boundary with Maven Enforcer;
• moves Hadoop FileSystem config/recovery/notebook storage and its health check into NotebookRepo/FileSystemNotebookRepo;
• moves YARN application monitoring behind a neutral ProcessLaunchObserver SPI provided by Launcher/YarnInterpreterLauncher;
• moves Kerberos/SPNEGO, Hadoop groups, and Hadoop credential-provider/JCEKS code into a new Security/Hadoop plugin;
• adds isolated plugin loading and the TCCL boundaries needed by Hadoop configuration/service discovery; and
• keeps existing configurable implementation class names unchanged where possible.

The intent is not to remove Hadoop from Spark or other interpreters. It is to make Hadoop an optional server integration rather than a server runtime requirement.

What type of PR is it?

Refactoring

Todos

• Enforce a zero-Hadoop dependency graph for zeppelin-server, including -Pinclude-hadoop.
• Exercise filesystem storage, YARN observer discovery, JCEKS resolution, Shiro plugin loading, and TCCL restoration on JDK 17.
• Add MiniKDC/SPNEGO integration coverage.
• Restore equivalent end-to-end recovery coverage inside the filesystem plugin.
• Decide whether distribution packaging should deduplicate the Hadoop API/runtime jars shared by the three plugins.
• Add installation/minimal-distribution documentation before marking the PR ready.

What is the Jira issue?

Not filed yet. This is a draft architecture/prototype PR for discussion before assigning a Zeppelin JIRA.

How should this be tested?

Tests were run with JDK 17.0.19:

./mvnw -pl zeppelin-server,zeppelin-plugins/launcher/yarn -am \
  -Dtest=ZeppelinServerPluginClassLoadingTest,YarnProcessLaunchObserverTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

./mvnw -Pinclude-hadoop -pl zeppelin-server -am validate

./mvnw -Prat \
  -pl zeppelin-server,zeppelin-plugins/notebookrepo/filesystem,zeppelin-plugins/launcher/yarn,zeppelin-plugins/security/hadoop \
  -am apache-rat:check

The three plugin modules also pass clean packaging and focused tests. Production-style isolated-loader smoke tests loaded FileSystemConfigStorage and FileSystemNotebookRepo from the packaged plugin directories. The packaged zeppelin-server/target/lib contains no Hadoop jars, including after an -Pinclude-hadoop build.

A full server-suite pass is not claimed in this draft: the broad local run encountered existing source-tree Spark test-runtime asset requirements and a recovery-test port conflict. The affected focused suites pass.

Screenshots (if appropriate)

N/A

Questions:

• License files: no update expected; all added files pass Apache RAT.
• Compatibility: existing FQCN configuration is preserved, but installations using HDFS storage, YARN monitoring, Kerberos/SPNEGO, Hadoop groups, or JCEKS must retain the corresponding optional plugin directory.
• Packaging: the isolated plugin directories are currently about 49 MiB, 49 MiB, and 54 MiB. Duplicating Hadoop API/runtime in each increases the distribution by roughly 100 MiB versus one shared copy; this should be resolved before the PR is ready.
• Documentation: yes, plugin installation and minimal-distribution documentation is still needed.