diff --git a/AGENTS.md b/AGENTS.md index 090dc4433e1..dcbeaf68568 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,30 +55,24 @@ Apache Zeppelin is a web-based notebook for interactive data analytics. It provi ## Build Gotchas -### Shaded JAR Rebuild Chain +### Interpreter Runtime JAR -The most common build mistake: modifying `zeppelin-interpreter` without rebuilding `zeppelin-interpreter-shaded`. The shaded JAR is an uber JAR that all interpreter processes use. If it's stale, you get `ClassNotFoundException` or `NoSuchMethodError` at runtime. +Packaging `zeppelin-interpreter` produces both the normal Maven artifact and the relocated runtime JAR that every interpreter process uses: ```bash -# After changing zeppelin-interpreter, ALWAYS rebuild in order: -./mvnw clean package -pl zeppelin-interpreter -DskipTests -./mvnw clean package -pl zeppelin-interpreter-shaded -DskipTests -# Then rebuild affected interpreter modules - -# Shorthand: -./mvnw clean package -pl zeppelin-interpreter,zeppelin-interpreter-shaded -DskipTests +./mvnw clean package -pl zeppelin-interpreter --am -DskipTests ``` -The shaded JAR is also copied to `interpreter/` directory by maven-antrun-plugin after packaging. If this directory has a stale JAR, interpreter processes will load old code. +The normal `zeppelin-interpreter-${version}.jar` remains under `zeppelin-interpreter/target` and is installed or deployed as the module artifact. The build stages the runtime-only `zeppelin-interpreter-shaded-${version}.jar` under `target/`, publishes the complete file to `interpreter/`, and then deletes older versions so interpreter launchers never see multiple matching JARs. ### Module Build Order Maven modules are ordered in the root `pom.xml`. Key sequence: ``` -zeppelin-interpreter → zeppelin-interpreter-shaded → zeppelin-server +zeppelin-interpreter → interpreter modules / zeppelin-server ``` -All interpreter modules build after `zeppelin-interpreter-shaded`. A second shading chain exists for Jupyter: +A separate shading chain exists for Jupyter: ``` zeppelin-jupyter-interpreter → zeppelin-jupyter-interpreter-shaded → python ``` @@ -88,17 +82,17 @@ zeppelin-jupyter-interpreter → zeppelin-jupyter-interpreter-shaded → python ### Dependency Flow ``` -zeppelin-interpreter Base API: Interpreter, InterpreterContext, Thrift services - ↓ -zeppelin-interpreter-shaded Uber JAR (maven-shade-plugin, relocated packages) +zeppelin-interpreter Base API + normal Maven JAR + relocated runtime JAR ↓ +interpreter modules Spark, Flink, Python, JDBC, etc. + zeppelin-server Core engine + Jetty 11, REST/WebSocket APIs, HK2 DI, entry point ``` ### Core Modules #### `zeppelin-interpreter/` -The base framework that all interpreters depend on. Defines the interpreter API and the Thrift communication protocol. This module is shaded into an uber JAR (`zeppelin-interpreter-shaded`) and placed on each interpreter process's classpath. +The base framework that all interpreters depend on. Defines the interpreter API and the Thrift communication protocol. Its package phase also builds a relocated runtime JAR under `interpreter/`; that internal file is placed on each interpreter process's classpath but is not installed or deployed as a Maven artifact. Key classes: - `Interpreter` (abstract) / `AbstractInterpreter` — base class every interpreter extends @@ -137,9 +131,6 @@ Engine / runtime (`org.apache.zeppelin.notebook`, `interpreter`, `scheduler`, `s - `RecoveryStorage` — persists interpreter process info for server-restart recovery - `ConfigStorage` — persists interpreter settings to JSON -#### `zeppelin-interpreter-shaded/` -Uses maven-shade-plugin to package `zeppelin-interpreter` + dependencies into an uber JAR with relocated packages (e.g., `org.apache.thrift` → `org.apache.zeppelin.shaded.org.apache.thrift`). This JAR is placed on each interpreter process's classpath. - #### `zeppelin-client/` REST/WebSocket client library for programmatic access to Zeppelin. diff --git a/bin/interpreter.sh b/bin/interpreter.sh index 00ff0307313..c2c74baf3e9 100755 --- a/bin/interpreter.sh +++ b/bin/interpreter.sh @@ -117,8 +117,6 @@ if [[ -d "${ZEPPELIN_HOME}/zeppelin-server/target/test-classes" ]]; then addJarInDirForIntp "${ZEPPELIN_HOME}/zeppelin-server/target/test-classes" fi -addJarInDirForIntp "${ZEPPELIN_HOME}/zeppelin-interpreter-shaded/target" - HOSTNAME=$(hostname) ZEPPELIN_SERVER=org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer diff --git a/docs/setup/operation/upgrading.md b/docs/setup/operation/upgrading.md index 673fcac59c7..0007210a43f 100644 --- a/docs/setup/operation/upgrading.md +++ b/docs/setup/operation/upgrading.md @@ -35,6 +35,10 @@ So, copying `notebook` and `conf` directory should be enough. ## Migration Guide +### Upgrading from Zeppelin 0.12 to 0.13 + + - The `org.apache.zeppelin:zeppelin-interpreter-shaded` Maven artifact is no longer published. Custom interpreters should depend on `org.apache.zeppelin:zeppelin-interpreter` with `provided` scope instead. The Zeppelin distribution continues to provide the internal shaded runtime JAR to interpreter processes; custom interpreters should not depend on classes packaged only in that internal runtime JAR. Custom interpreters that use Commons Configuration, Commons BeanUtils, JSR 305, the Maven Plugin API, or Sisu Plexus must now declare those libraries directly instead of relying on transitive dependencies from `zeppelin-interpreter`. + ### Upgrading from Zeppelin 0.9, 0.10 to 0.11 - From 0.11, The type of `Pegdown` for parsing markdown was deprecated ([ZEPPELIN-5529](https://issues.apache.org/jira/browse/ZEPPELIN-2619)). It will use `Flexmark` instead. diff --git a/jdbc/Dockerfile b/jdbc/Dockerfile index a8865e87389..235715b4c93 100644 --- a/jdbc/Dockerfile +++ b/jdbc/Dockerfile @@ -23,7 +23,7 @@ WORKDIR /zeppelin RUN chmod +x ./mvnw -RUN ./mvnw clean package -am -pl zeppelin-interpreter-shaded,zeppelin-interpreter,jdbc -DskipTests +RUN ./mvnw clean package -am -pl zeppelin-interpreter,jdbc -DskipTests FROM openjdk:11 @@ -32,7 +32,7 @@ COPY --from=builder /zeppelin/bin /zeppelin/bin/ COPY --from=builder /zeppelin/conf /zeppelin/conf COPY --from=builder /zeppelin/interpreter/jdbc /zeppelin/interpreter/jdbc -COPY --from=builder /zeppelin/zeppelin-interpreter-shaded/target /zeppelin/zeppelin-interpreter-shaded/target +COPY --from=builder /zeppelin/interpreter/zeppelin-interpreter-shaded-*.jar /zeppelin/interpreter/ WORKDIR /zeppelin diff --git a/pom.xml b/pom.xml index 8ddaa73ec78..53b22a93dc8 100644 --- a/pom.xml +++ b/pom.xml @@ -55,7 +55,6 @@ build-tools zeppelin-interpreter-parent zeppelin-interpreter - zeppelin-interpreter-shaded zeppelin-jupyter-interpreter zeppelin-jupyter-interpreter-shaded groovy @@ -137,6 +136,7 @@ 4.2.29 1.14.2 3.0.2 + 1.0 3.3.6 provided @@ -362,6 +362,12 @@ ${findbugs.jsr305.version} + + javax.annotation + jsr250-api + ${jsr250.api.version} + + org.apache.shiro diff --git a/python/pom.xml b/python/pom.xml index 3ac4b5f5141..275fa381586 100644 --- a/python/pom.xml +++ b/python/pom.xml @@ -136,10 +136,8 @@ org.apache.maven.plugins @@ -156,11 +154,6 @@ reference.conf - - - org.apache.zeppelin:zeppelin-interpreter-shaded - - ${project.build.directory}/../../interpreter/python/${interpreter.jar.name}-${project.version}.jar diff --git a/shell/Dockerfile b/shell/Dockerfile index 1da36a26338..753e6a4fb2f 100644 --- a/shell/Dockerfile +++ b/shell/Dockerfile @@ -23,7 +23,7 @@ WORKDIR /zeppelin RUN chmod +x ./mvnw -RUN ./mvnw package -am -pl zeppelin-interpreter-shaded,zeppelin-interpreter,shell -DskipTests +RUN ./mvnw package -am -pl zeppelin-interpreter,shell -DskipTests FROM openjdk:11 @@ -32,7 +32,7 @@ COPY --from=builder /zeppelin/bin /zeppelin/bin/ COPY --from=builder /zeppelin/conf /zeppelin/conf COPY --from=builder /zeppelin/interpreter/sh /zeppelin/interpreter/sh -COPY --from=builder /zeppelin/zeppelin-interpreter-shaded/target /zeppelin/zeppelin-interpreter-shaded/target +COPY --from=builder /zeppelin/interpreter/zeppelin-interpreter-shaded-*.jar /zeppelin/interpreter/ WORKDIR /zeppelin diff --git a/spark/interpreter/pom.xml b/spark/interpreter/pom.xml index 0e276a49ba5..8128ff0fe18 100644 --- a/spark/interpreter/pom.xml +++ b/spark/interpreter/pom.xml @@ -335,7 +335,6 @@ org.scala-lang:scala-reflect commons-lang:commons-lang org.apache.commons:commons-compress - org.apache.zeppelin:zeppelin-interpreter-shaded diff --git a/zeppelin-integration/pom.xml b/zeppelin-integration/pom.xml index fe665f2d01e..434c7da9a22 100644 --- a/zeppelin-integration/pom.xml +++ b/zeppelin-integration/pom.xml @@ -198,8 +198,9 @@ true - zeppelin-interpreter-shaded must NOT appear on the zeppelin-integration test classpath. - MiniZeppelinServer instantiates ZeppelinServer in-process; mixing shaded and unshaded + The legacy zeppelin-interpreter-shaded Maven artifact must NOT appear on the + zeppelin-integration test classpath. MiniZeppelinServer instantiates ZeppelinServer + in-process; mixing shaded and unshaded org.eclipse.aether.* in the same JVM causes ClassCastException in InterpreterSettingManager. See ZEPPELIN-6416. diff --git a/zeppelin-interpreter-parent/pom.xml b/zeppelin-interpreter-parent/pom.xml index e80d12dd3a1..9153924cb05 100644 --- a/zeppelin-interpreter-parent/pom.xml +++ b/zeppelin-interpreter-parent/pom.xml @@ -32,13 +32,6 @@ Zeppelin: Interpreter Parent - - ${project.groupId} - zeppelin-interpreter-shaded - ${project.version} - provided - - org.apache.zeppelin zeppelin-interpreter @@ -117,7 +110,6 @@ META-INF/*.SF META-INF/*.DSA META-INF/*.RSA - org.apache.zeppelin:zeppelin-interpreter-shaded @@ -127,11 +119,6 @@ reference.conf - - - org.apache.zeppelin:zeppelin-interpreter-shaded - - ${project.basedir}/../interpreter/${interpreter.name}/${project.artifactId}-${project.version}.jar diff --git a/zeppelin-interpreter-shaded/pom.xml b/zeppelin-interpreter-shaded/pom.xml deleted file mode 100644 index 226f1c06820..00000000000 --- a/zeppelin-interpreter-shaded/pom.xml +++ /dev/null @@ -1,179 +0,0 @@ - - - - - - 4.0.0 - - - zeppelin - org.apache.zeppelin - 0.13.0-SNAPSHOT - - - zeppelin-interpreter-shaded - jar - Zeppelin: Interpreter Shaded - Zeppelin Interpreter Shaded - - - shaded.org.apache.zeppelin - - - - - org.apache.zeppelin - zeppelin-interpreter - 0.13.0-SNAPSHOT - true - - - - - - - org.apache.maven.plugins - maven-shade-plugin - - - - - org.slf4j:* - - log4j:log4j - ch.qos.reload4j:reload4j - - - - - *:* - - META-INF/*.SF - META-INF/*.DSA - META-INF/*.RSA - - - - - - - reference.conf - - - - NOTICE.txt - - - META-INF/LICENSE.txt - ${basedir}/../../LICENSE.txt - - - META-INF/NOTICE.txt - ${basedir}/../../NOTICE.txt - - - - - org - ${shaded.dependency.prefix}.org - - org/apache/zeppelin/* - org/apache/zeppelin/**/* - org/apache/hadoop/* - org/apache/hadoop/** - org/slf4j/* - org/slf4j/**/* - org/apache/commons/logging/* - org/apache/commons/logging/**/* - org/apache/commons/exec/* - org/apache/commons/exec/**/* - org/apache/log4j/* - org/apache/log4j/**/* - org/sonatype/* - org/sonatype/**/* - **/pom.xml - - - org/ietf/jgss/* - org/omg/**/* - org/w3c/dom/* - org/w3c/dom/**/* - org/xml/sax/* - org/xml/sax/**/* - - - org/eclipse/sisu/** - - - - com.google - ${shaded.dependency.prefix}.com.google - - - io - ${shaded.dependency.prefix}.io - - - com.esotericsoftware - ${shaded.dependency.prefix}.com.esotericsoftware - - - - - - package - - shade - - - - - - org.apache.maven.plugins - maven-antrun-plugin - - - copy-to-interpreter - package - - run - - - - ANT TASK - clean files.... - - - - ANT TASK - copying files.... - - - - - - - - - - - - - - diff --git a/zeppelin-interpreter/pom.xml b/zeppelin-interpreter/pom.xml index c86b0890a18..b2e394d41b7 100644 --- a/zeppelin-interpreter/pom.xml +++ b/zeppelin-interpreter/pom.xml @@ -37,8 +37,7 @@ 2.3 1.4.1 - - 0.3.4 + shaded.org.apache.zeppelin 2.14.3 @@ -54,6 +53,11 @@ commons-lang3 + + org.apache.commons + commons-text + + org.apache.thrift libthrift @@ -71,35 +75,13 @@ gson + - com.google.code.findbugs - jsr305 + javax.annotation + jsr250-api + provided - - org.apache.commons - commons-configuration2 - - - - commons-logging - commons-logging - - - - - - commons-beanutils - commons-beanutils - 1.11.0 - - - - commons-logging - commons-logging - - - org.apache.commons commons-exec @@ -132,30 +114,6 @@ - - org.apache.maven - maven-plugin-api - ${maven.version} - - - - org.eclipse.sisu - org.eclipse.sisu.plexus - ${sisu.plexus.version} - - - - org.codehaus.plexus - plexus-classworlds - - - - org.codehaus.plexus - plexus-utils - - - - org.apache.maven maven-resolver-provider @@ -226,6 +184,149 @@ yyyy-MM-dd HH:mm:ss + + + + org.apache.maven.plugins + maven-shade-plugin + + false + ${project.build.directory}/zeppelin-interpreter-shaded-${project.version}.jar + + + + org.slf4j:* + + log4j:log4j + ch.qos.reload4j:reload4j + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + reference.conf + + + + NOTICE.txt + + + META-INF/LICENSE + ${project.build.outputDirectory}/META-INF/LICENSE + + + + + org + ${shaded.dependency.prefix}.org + + org/apache/zeppelin/* + org/apache/zeppelin/**/* + org/apache/hadoop/* + org/apache/hadoop/** + org/slf4j/* + org/slf4j/**/* + org/apache/commons/logging/* + org/apache/commons/logging/**/* + org/apache/commons/exec/* + org/apache/commons/exec/**/* + org/apache/log4j/* + org/apache/log4j/**/* + org/sonatype/* + org/sonatype/**/* + **/pom.xml + + + org/ietf/jgss/* + org/omg/**/* + org/w3c/dom/* + org/w3c/dom/**/* + org/xml/sax/* + org/xml/sax/**/* + + + org/eclipse/sisu/** + + + + com.google + ${shaded.dependency.prefix}.com.google + + + io + ${shaded.dependency.prefix}.io + + + com.esotericsoftware + ${shaded.dependency.prefix}.com.esotericsoftware + + + + + + package + + shade + + + + + + + + org.apache.maven.plugins + maven-antrun-plugin + + + publish-interpreter-runtime + package + + run + + + + + + + + + + + + + + + org.apache.maven.plugins + maven-clean-plugin + + + + ${project.basedir}/../interpreter + + zeppelin-interpreter-shaded-*.jar + + false + + + + diff --git a/zeppelin-jupyter-interpreter-shaded/pom.xml b/zeppelin-jupyter-interpreter-shaded/pom.xml index e43c6471b60..518a47f4ca4 100644 --- a/zeppelin-jupyter-interpreter-shaded/pom.xml +++ b/zeppelin-jupyter-interpreter-shaded/pom.xml @@ -104,7 +104,6 @@ - org.apache.zeppelin:zeppelin-interpreter-shaded log4j:log4j diff --git a/zeppelin-jupyter-interpreter/pom.xml b/zeppelin-jupyter-interpreter/pom.xml index 4651e0f4730..34b3aea4680 100644 --- a/zeppelin-jupyter-interpreter/pom.xml +++ b/zeppelin-jupyter-interpreter/pom.xml @@ -84,13 +84,12 @@ commons-codec - + + + javax.annotation + jsr250-api + provided + ${project.groupId} diff --git a/zeppelin-server/pom.xml b/zeppelin-server/pom.xml index 8dbdb575673..a007b3beb2b 100644 --- a/zeppelin-server/pom.xml +++ b/zeppelin-server/pom.xml @@ -361,6 +361,28 @@ commons-collections + + org.apache.commons + commons-configuration2 + + + + commons-logging + commons-logging + + + + + + com.google.code.findbugs + jsr305 + + + + javax.annotation + jsr250-api + + org.apache.shiro shiro-core @@ -633,8 +655,9 @@ true - zeppelin-interpreter-shaded must NOT appear on the zeppelin-server classpath. - The shaded jar is meant for interpreter JVMs only (loaded from + The legacy zeppelin-interpreter-shaded Maven artifact must NOT appear on the + zeppelin-server classpath. The runtime shaded jar is meant for interpreter JVMs + only (loaded from ${ZEPPELIN_HOME}/interpreter/zeppelin-interpreter-shaded-*.jar). Mixing the shaded and unshaded org.eclipse.aether.* on the server classpath causes ClassCastException in InterpreterSettingManager. See ZEPPELIN-6416.