Skip to content

[improvement](build) Fully exclude OBS/COS deps via --exclude-{obs,cos}-dependencies #66564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zhiqiang-hhhh wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[improvement](build) Fully exclude OBS/COS deps via --exclude-{obs,cos}-dependencies #66564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,11 @@ Usage: $0 <options>
--be-extension-ignore build be-java-extensions package, choose which modules to ignore. Multiple modules separated by commas.
--enable-dynamic-arch enable dynamic CPU detection in OpenBLAS. Default ON.
--disable-dynamic-arch disable dynamic CPU detection in OpenBLAS.
--exclude-obs-dependencies exclude all Huawei Cloud OBS (com.huaweicloud) dependencies and the
fe-filesystem-obs module; nothing from Huawei is resolved, compiled,
or bundled. Use when repo.huaweicloud.com is unreachable or forbidden.
--exclude-cos-dependencies exclude all Tencent Cloud COS dependencies and the fe-filesystem-cos
module; nothing from Tencent COS is resolved, compiled, or bundled.
--clean clean and build target
--compile-bench BE compile-speed benchmark: cold, cache-free BE-only build
(fresh dedicated build dir, ccache disabled) with a per-phase
Expand Down Expand Up @@ -802,6 +807,14 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
modules+=("fe-filesystem/fe-filesystem-api")
modules+=("fe-filesystem/fe-filesystem-spi")
for _fs_mod in s3-base s3 gcs minio ozone oss cos obs azure hdfs-base hdfs oss-hdfs jfs local broker http; do
# Skip the modules whose Maven profile is deactivated so the -pl list stays consistent with
# the reactor: obs is absent under -Ddisable.obs=true, cos under -Ddisable.cos=true.
if [[ "${_fs_mod}" == "obs" && "${BUILD_OBS_DEPENDENCIES}" -eq 0 ]]; then
continue
fi
if [[ "${_fs_mod}" == "cos" && "${BUILD_COS_DEPENDENCIES}" -eq 0 ]]; then
continue
fi
if [[ -d "${DORIS_HOME}/fe/fe-filesystem/fe-filesystem-${_fs_mod}" ]]; then
modules+=("fe-filesystem/fe-filesystem-${_fs_mod}")
fi
Expand Down Expand Up @@ -1070,10 +1083,14 @@ function build_fe_modules() {
extra_mvn_opts=(${MVN_OPT})
fi
if [[ "${BUILD_OBS_DEPENDENCIES}" -eq 0 ]]; then
dependency_mvn_opts+=("-Dobs.dependency.scope=provided")
# Deactivates the `obs` Maven profile in fe-core, hadoop-deps and fe-filesystem, so no
# com.huaweicloud artifact is resolved and the Huawei OBS module is not built or bundled.
dependency_mvn_opts+=("-Ddisable.obs=true")
fi
if [[ "${BUILD_COS_DEPENDENCIES}" -eq 0 ]]; then
dependency_mvn_opts+=("-Dcos.dependency.scope=provided")
# Deactivates the `cos` Maven profile in fe-core and fe-filesystem, so no Tencent COS
# artifact is resolved and the fe-filesystem-cos module is not built or bundled.
dependency_mvn_opts+=("-Ddisable.cos=true")
fi
if [[ -n "${USER_SETTINGS_MVN_REPO}" && -f "${USER_SETTINGS_MVN_REPO}" ]]; then
user_settings_opts=(-gs "${USER_SETTINGS_MVN_REPO}")
Expand Down Expand Up @@ -1191,6 +1208,14 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
if [ ! -d "${fs_module_dir}" ]; then
continue
fi
# These modules are not built when their Maven profile is deactivated, so their plugin zip
# does not exist; skip the unpack to keep packaging consistent with the reactor.
if [[ "${fs_module}" == "obs" && "${BUILD_OBS_DEPENDENCIES}" -eq 0 ]]; then
continue
fi
if [[ "${fs_module}" == "cos" && "${BUILD_COS_DEPENDENCIES}" -eq 0 ]]; then
continue
fi
mkdir -p "${fs_plugin_target}"
# Unpack the self-contained plugin zip produced by maven-assembly-plugin.
# Layout inside the zip: <plugin>.jar at root + lib/*.jar for runtime deps.
Expand Down
39 changes: 28 additions & 11 deletions fe/be-java-extensions/hadoop-deps/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,8 @@ under the License.
<groupId>org.apache.hadoop.thirdparty</groupId>
<artifactId>hadoop-shaded-protobuf_3_25</artifactId>
</dependency>
<dependency>
<groupId>com.huaweicloud</groupId>
<artifactId>hadoop-huaweicloud</artifactId>
</dependency>
<!-- Huawei OBS: hadoop-huaweicloud lives in the `obs` profile below;
disabled via -Ddisable.obs=true (build.sh exclude-obs-dependencies flag). -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
Expand All @@ -95,13 +93,32 @@ under the License.
</dependency>
</dependencies>

<repositories>
<!-- for huawei obs sdk -->
<repository>
<id>huawei-obs-sdk</id>
<url>https://repo.huaweicloud.com/repository/maven/huaweicloudsdk/</url>
</repository>
</repositories>
<!-- Huawei OBS support. Active by default; disabled by -Ddisable.obs=true
(build.sh exclude-obs-dependencies flag), which drops the hadoop-huaweicloud dependency
and the Huawei Maven repository so nothing from com.huaweicloud is resolved or bundled. -->
<profiles>
<profile>
<id>obs</id>
<activation>
<property>
<name>!disable.obs</name>
</property>
</activation>
<dependencies>
<dependency>
<groupId>com.huaweicloud</groupId>
<artifactId>hadoop-huaweicloud</artifactId>
</dependency>
</dependencies>
<repositories>
<!-- for huawei obs sdk -->
<repository>
<id>huawei-obs-sdk</id>
<url>https://repo.huaweicloud.com/repository/maven/huaweicloudsdk/</url>
</repository>
</repositories>
</profile>
</profiles>

<build>
<plugins>
Expand Down
42 changes: 30 additions & 12 deletions fe/be-java-extensions/preload-extensions/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,18 +168,8 @@ under the License.
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
</dependency>
<!-- For BE CosN Access -->
<dependency>
<groupId>com.qcloud.cos</groupId>
<artifactId>hadoop-cos</artifactId>
<version>${tencentcos.version}</version>
<exclusions>
<exclusion>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- For BE CosN Access: hadoop-cos lives in the `cos` profile below;
disabled via -Ddisable.cos=true (build.sh exclude-cos-dependencies flag). -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
Expand Down Expand Up @@ -236,6 +226,34 @@ under the License.
</dependency>
</dependencies>

<!-- Tencent COS support. Active by default; disabled by -Ddisable.cos=true
(build.sh exclude-cos-dependencies flag), which drops hadoop-cos so nothing from
com.qcloud.cos is resolved or bundled into preload-extensions. -->
<profiles>
<profile>
<id>cos</id>
<activation>
<property>
<name>!disable.cos</name>
</property>
</activation>
<dependencies>
<!-- For BE CosN Access -->
<dependency>
<groupId>com.qcloud.cos</groupId>
<artifactId>hadoop-cos</artifactId>
<version>${tencentcos.version}</version>
<exclusions>
<exclusion>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</profile>
</profiles>

<build>
<finalName>preload-extensions</finalName>
<directory>${project.basedir}/target/</directory>
Expand Down
75 changes: 46 additions & 29 deletions fe/fe-connector/fe-connector-paimon/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,24 +192,8 @@ under the License.
<artifactId>hadoop-aws</artifactId>
</dependency>

<!-- hadoop-huaweicloud supplies org.apache.hadoop.fs.obs.OBSFileSystem for obs:// warehouses, the
SAME self-contained reasoning as hadoop-aws above: the plugin runs child-first with hadoop-common
(FileSystem) bundled in the child loader. Without hadoop-huaweicloud here, OBSFileSystem resolved
from the parent 'app' loader while FileSystem resolved from the child ChildFirstClassLoader, so
FileSystem.createFileSystem's cast threw "OBSFileSystem cannot be cast to FileSystem"
(FIX-PAIMON-OBS-SELFCONTAINED, CI 968994 paimon_base_filesystem). Bundling it child-first
co-locates OBSFileSystem with the plugin's own FileSystem so the cast succeeds. runtime scope
(mirrors fe-core: OBSFileSystem is loaded reflectively via the Hadoop FileSystem SPI, not
referenced at compile time) — still bundled by plugin-zip.xml, which packs the runtime closure.
The -hw-46 jar is a fat jar self-containing the esdk-obs SDK (com/obs/*); hadoop-common stays the
plugin's own direct (depth-1) copy via Maven mediation, so no duplicate FileSystem is introduced.
NOTE: the -hw-46 artifact is NOT in Maven Central — it lives in the huawei-obs-sdk repo declared
in <repositories> below (fe-core declares the same repo for the same dep). -->
<dependency>
<groupId>com.huaweicloud</groupId>
<artifactId>hadoop-huaweicloud</artifactId>
<scope>runtime</scope>
</dependency>
<!-- Huawei OBS: hadoop-huaweicloud (runtime OBSFileSystem) and its Maven repository live in the
`obs` profile below; disabled via -Ddisable.obs=true (build.sh exclude-obs-dependencies flag). -->

<!-- AWS SDK v2 for the child-bundled hadoop-aws S3AFileSystem (RC-3, completes
FIX-PAIMON-HADOOP-CLASSLOADER). Without it the plugin bundles S3AFileSystem child-first but NO
Expand Down Expand Up @@ -291,17 +275,50 @@ under the License.
</dependency>
</dependencies>

<!-- FIX-PAIMON-OBS-SELFCONTAINED (CI 968994): com.huaweicloud:hadoop-huaweicloud:3.1.1-hw-46 is NOT
published to Maven Central / the Apache repos, so a connector-module build resolves only against
central+apache-snapshots and fails ("hadoop-huaweicloud ... was not found in repo.maven.apache.org").
fe-core declares this exact repo to resolve the same dep; the connector module does not inherit it
(fe-connector / fe declare no repositories), so it must be declared here too. -->
<repositories>
<repository>
<id>huawei-obs-sdk</id>
<url>https://repo.huaweicloud.com/repository/maven/huaweicloudsdk/</url>
</repository>
</repositories>
<!-- Huawei OBS support. Active by default; disabled by -Ddisable.obs=true
(build.sh exclude-obs-dependencies flag), which drops hadoop-huaweicloud and the Huawei
Maven repository so nothing from com.huaweicloud is resolved or bundled into the paimon plugin. -->
<profiles>
<profile>
<id>obs</id>
<activation>
<property>
<name>!disable.obs</name>
</property>
</activation>
<dependencies>
<!-- hadoop-huaweicloud supplies org.apache.hadoop.fs.obs.OBSFileSystem for obs:// warehouses, the
SAME self-contained reasoning as hadoop-aws: the plugin runs child-first with hadoop-common
(FileSystem) bundled in the child loader. Without hadoop-huaweicloud here, OBSFileSystem resolved
from the parent 'app' loader while FileSystem resolved from the child ChildFirstClassLoader, so
FileSystem.createFileSystem's cast threw "OBSFileSystem cannot be cast to FileSystem"
(FIX-PAIMON-OBS-SELFCONTAINED, CI 968994 paimon_base_filesystem). Bundling it child-first
co-locates OBSFileSystem with the plugin's own FileSystem so the cast succeeds. runtime scope
(mirrors fe-core: OBSFileSystem is loaded reflectively via the Hadoop FileSystem SPI, not
referenced at compile time) — still bundled by plugin-zip.xml, which packs the runtime closure.
The -hw-46 jar is a fat jar self-containing the esdk-obs SDK (com/obs/*); hadoop-common stays the
plugin's own direct (depth-1) copy via Maven mediation, so no duplicate FileSystem is introduced.
NOTE: the -hw-46 artifact is NOT in Maven Central — it lives in the huawei-obs-sdk repo declared
in <repositories> below (fe-core declares the same repo for the same dep). -->
<dependency>
<groupId>com.huaweicloud</groupId>
<artifactId>hadoop-huaweicloud</artifactId>
<scope>runtime</scope>
</dependency>
</dependencies>
<repositories>
<!-- FIX-PAIMON-OBS-SELFCONTAINED (CI 968994): com.huaweicloud:hadoop-huaweicloud:3.1.1-hw-46 is NOT
published to Maven Central / the Apache repos, so a connector-module build resolves only against
central+apache-snapshots and fails ("hadoop-huaweicloud ... was not found in repo.maven.apache.org").
fe-core declares this exact repo to resolve the same dep; the connector module does not inherit it
(fe-connector / fe declare no repositories), so it must be declared here too. -->
<repository>
<id>huawei-obs-sdk</id>
<url>https://repo.huaweicloud.com/repository/maven/huaweicloudsdk/</url>
</repository>
</repositories>
</profile>
</profiles>

<build>
<finalName>doris-fe-connector-paimon</finalName>
Expand Down
104 changes: 68 additions & 36 deletions fe/fe-core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,70 @@ under the License.
<fe_ut_parallel>${env.FE_UT_PARALLEL}</fe_ut_parallel>
</properties>
</profile>
<!-- Huawei Cloud OBS support. Active by default; disabled by passing
-Ddisable.obs=true (build.sh exclude-obs-dependencies flag), which drops the
fe-filesystem-obs test coupling so nothing from com.huaweicloud reaches the test
classpath and the Huawei OBS jars (only on repo.huaweicloud.com) are never resolved. -->
<profile>
<id>obs</id>
<activation>
<property>
<name>!disable.obs</name>
</property>
</activation>
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>fe-filesystem-obs</artifactId>
<version>${project.version}</version>
<scope>test</scope>
<!-- The cloud SDK's okhttp (3.x/5.x) must not shadow fe-core's own okhttp on the
test classpath; the bind-parity tests never open SDK http connections. -->
<exclusions>
<exclusion>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>com.squareup.okio</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</profile>
<!-- Tencent Cloud COS support. Active by default; disabled by passing
-Ddisable.cos=true (build.sh exclude-cos-dependencies flag), which drops the
fe-filesystem-cos test coupling so the Tencent COS SDK is never pulled onto the
test classpath. -->
<profile>
<id>cos</id>
<activation>
<property>
<name>!disable.cos</name>
</property>
</activation>
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>fe-filesystem-cos</artifactId>
<version>${project.version}</version>
<scope>test</scope>
<!-- The cloud SDK's okhttp (3.x/5.x) must not shadow fe-core's own okhttp on the
test classpath; the bind-parity tests never open SDK http connections. -->
<exclusions>
<exclusion>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>com.squareup.okio</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</profile>
</profiles>
<dependencies>
<dependency>
Expand Down Expand Up @@ -219,42 +283,10 @@ under the License.
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>fe-filesystem-obs</artifactId>
<version>${project.version}</version>
<scope>test</scope>
<!-- The cloud SDK's okhttp (3.x/5.x) must not shadow fe-core's own okhttp on the
test classpath; the bind-parity tests never open SDK http connections. -->
<exclusions>
<exclusion>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>com.squareup.okio</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>fe-filesystem-cos</artifactId>
<version>${project.version}</version>
<scope>test</scope>
<!-- The cloud SDK's okhttp (3.x/5.x) must not shadow fe-core's own okhttp on the
test classpath; the bind-parity tests never open SDK http connections. -->
<exclusions>
<exclusion>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>com.squareup.okio</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- fe-filesystem-obs (test scope) lives in the `obs` profile above;
disabled via -Ddisable.obs=true (build.sh exclude-obs-dependencies flag). -->
<!-- fe-filesystem-cos (test scope) lives in the `cos` profile above;
disabled via -Ddisable.cos=true (build.sh exclude-cos-dependencies flag). -->
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>fe-filesystem-gcs</artifactId>
Expand Down
Loading
Loading