Skip to content

Dev 2.0.0 webank#5439

Open
aiceflower wants to merge 38 commits into
apache:masterfrom
WeDataSphere:dev-2.0.0-webank
Open

Dev 2.0.0 webank#5439
aiceflower wants to merge 38 commits into
apache:masterfrom
WeDataSphere:dev-2.0.0-webank

Conversation

@aiceflower

@aiceflower aiceflower commented Jun 11, 2026

Copy link
Copy Markdown
Member

What is the purpose of the change

EngineConn-Core defines the the abstractions and interfaces of the EngineConn core functions.
The Engine Service in Linkis 0.x is refactored, EngineConn will handle the engine connection
and session management.

Related issues/PRs

Related issues: close #590 close #591
Related pr:#591

Brief change log

  • Define the core abstraction and interfaces of the EngineConn Factory;
  • Define the core abstraction and interfaces of Executor Manager.

Checklist

  • I have read the Contributing Guidelines on pull requests.
  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
  • If this is a code change: I have written unit tests to fully verify the new behavior.

aiceflower and others added 30 commits April 4, 2026 06:42
@aiceflower @claude
#AI commit# chore: Bump version from 1.9.0 to 2.0.0
0f69c9f 
Update project version to 2.0.0 in pom.xml and database scripts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@v-kkhuang
[fix][CGS][entrance] parallelgroup cache not updated when entrance in…
6648906 
…stance goes offline (#966)

* #AI commit# 开发阶段： *  修复异常场景，当Entrance实例offline后，ParallelGroup缓存未更新，导致用户任务并发数计算错误

* #AI commit# 开发阶段： *  修复异常场景，当Entrance实例offline后，ParallelGroup缓存未更新，导致用户任务并发数计算错误

* #AI commit# 开发阶段： *  增加异常保护

* #AI commit# 开发阶段： * 修复异常场景，当Entrance实例offline后，ParallelGroup缓存未更新，导致用户任务并发数计算错误
@v-kkhuang @casionone
[feat][CGS][hive] add security control to prevent location clause usa…
57e014c 
…ge in hive tasks (#968)

* #AI commit# 开发阶段： Hive任务禁止使用LOCATION功能

* #AI commit# 开发阶段： Hive任务禁止使用LOCATION功能，测试报告相关文件提交

* #AI commit# 开发阶段： *  hive禁止location正则优化

* #AI commit# 开发阶段： entrance配置文件优化

* #AI commit# 开发阶段： 去除code打印

---------

Co-authored-by: Casion <casionone@gmail.com>
@v-kkhuang
[feat][EC][hive] add support for yarn tag and username in hive engine (…
27f6b88 
…#970)

* #AI commit# 开发阶段： * 修复sr任务重试导致加载init_sql异常bug
* 优化spark参数获取
* 管理台支持展示引擎版本
*

* #AI commit# 开发阶段： *  增加开关和异常保护
@v-kkhuang
[perf][COM][kerberos] optimize keytab file caching to prevent full gc…
00a099c 
… issues (#972)

* #AI commit# 优化：修复keytab文件缓存导致的Full GC问题，添加keytabFile缓存机制

* #AI commit# 优化：修复keytab文件缓存导致的Full GC问题，添加keytabFile缓存机制

* #AI commit# 开发阶段： 增加异常保护
@v-kkhuang
[feat][COM][logging] optimize log printing and add hadoop client usag…
31bb257 
…e logs (#974)

* #AI commit# 开发阶段： *  optimize log printing and token masking

* #AI commit# 开发阶段： *  增加hadoop客户端使用日志
@v-kkhuang @casionone
[feat][ETS][monitor] optimize monitor module with log cleanup and con…
297faf4 
…nection pool expansion (#976)

* #AI commit# 开发阶段：Monitor模块优化 - 诊断日志自动清理、诊断功能拆分、连接池扩容

主要变更：
1. 新增DiagnosisLogClear.java - 诊断日志定时清理功能
2. 诊断功能配置化拆分 - 新增配置开关控制
3. Alert连接池扩容 - 5→20线程
4. MonitorConfig新增5个配置常量

* #AI commit# 开发阶段： 相关文档提交

* #AI commit# 开发阶段： monitor ai 相关文档提交

* #AI commit# 开发阶段： *  monitor参数优化

* #AI commit# 开发阶段： *  新增诊断表数据清理

* #AI commit# 开发阶段： 去除日志地址

---------

Co-authored-by: Casion <casionone@gmail.com>
@v-kkhuang
#AI commit# 开发阶段： * 资源重置接口逻辑优化 (#978)
56478dc
@v-kkhuang
[fix][CGS][entrance] fix dependency injection failure in entrance con…
614b0f3 
…figuration (#980)

* #AI commit# 开发阶段： 异常注释修复

* #AI commit# 开发阶段： 修复依赖注入失败
@aiceflower @claude
#AI commit# fix: Restore linkis-io_file-client dependency in linkis-p…
340ae29 
…es-publicservice

- Restore linkis-io_file-client dependency that was accidentally removed
- Remove hbase-server dependency that was incorrectly added
- This fixes missing jars in linkis-ps-publicservice lib directory:
  - linkis-io_file-client
  - linkis-computation-orchestrator
  - linkis-code-orchestrator
  - linkis-orchestrator-core

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@aiceflower @claude
#AI commit# [fix][PES][datasource] fix DB2 schema query to use correc…
23fe635 
…t SQL instead of invalid command (#983)

- Fix getAllDatabases() method in DB2 SqlConnection classes
- Replace invalid "list database directory" command with proper SQL query
- Add configurable schema query SQL with default filtering system schemas
- Query SYSCAT.SCHEMATA to get actual schema list from DB2

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@v-kkhuang
[docs] reorganize 2.0.0 feature documentation structure (#994)
6ea5a7d 
* #AI commit# 开发阶段： dev 2.0.0 特性文档整理

* #AI commit# 开发阶段： 增加Licensed
@v-kkhuang
[feat][CGS][engineconn-manager] add support for hadoop3 engine materi…
dc3897e 
…al refresh (#996)

* #AI commit# 开发阶段： 支持hadoop3引擎物料刷新

* #AI commit# 开发阶段： 优化引擎脚本生成逻辑
@v-kkhuang
#AI commit# 开发阶段： 代码格式化 (#998)
9b2d2d8
@v-kkhuang
#AI commit# 开发阶段： 优化禁止引擎打印code隐私信息 (#1000)
b0b5866
@v-kkhuang
#AI commit# 开发阶段： 加密日志打印token (#1002)
a0c9a0f
@v-kkhuang
[fix][EC][all-engines] mask user code in engine execution logs to pre…
8086e26 
…vent security leakage (#1003)

* #AI commit# 开发阶段： 加密日志打印用户运行code

* #AI commit# 开发阶段： 加密日志打印用户运行code
@v-kkhuang
[style][CGS][core] improve code formatting for unit tests and configu…
49fbb65 
…ration files (#1004)

* #AI commit# 开发阶段： 代码格式化

* #AI commit# 开发阶段： 单元测试&配置文件优化
@v-kkhuang
#AI commit# 开发阶段： spark scala version update (#1006)
2e0772c
@v-kkhuang
#AI commit# 开发阶段： add week variable support for date calculation (#1008)
a0ac6f5
@v-kkhuang
#AI commit# 开发阶段： add permission check for secondary queue selection (#…
1fc405c 
…1010)
@aiceflower @claude
#AI COMMIT# [fix][CGS] Add MDC jobId tracing for task-related logs (#…
a2ea93b 
…1011)

Add setJobIdMDC/removeJobIdMDC at key points where task ID is available
but not set to MDC, so that task-related logs include [JobId-xxx] for
cross-service log tracing.

Changes:
- EntranceServer: move setJobIdMDC before first logger.info call
- EntranceRestfulApi: add MDC in submit() and execute() methods
- EntranceJob: override transition() to set MDC for status change logs
- JobHistoryQueryServiceImpl: add MDC in change() method
- TaskExecutionServiceImpl: add MDC in dealRequestTaskKill/Status/Pause/Resume

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
@v-kkhuang
#AI commit# 开发阶段： 同步flink版本 (#1013)
53b4003
@v-kkhuang
#AI commit# 开发阶段： 增加label service 修改entrance 服务标签触发广播 (#1017)
cc92eeb
@v-kkhuang
[fix][ETS][et-monitor] fix log printing exception and optimize monito…
8835052 
…r cleanup process (#1015)

* #AI commit# 开发阶段： 异常注释修复

* #AI commit# 开发阶段： 修复依赖注入失败

* #AI commit# 开发阶段： 优化2.0.0特性文档

* #AI commit# 开发阶段： 日志打印优化，spark init sql 异常优化

* #AI commit# 开发阶段： 支持hadoop3引擎物料刷新

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程
@v-kkhuang
#AI commit# 开发阶段： 安全工单升级commons-lang移除，commons-lang3升级3.18.0 (#1019)
fa31f0d
@v-kkhuang
#AI commit# 开发阶段： 优化引擎日志输出用户执行code (#1021)
67f0bef
@v-kkhuang
#AI commit# 开发阶段： 支持keytab环境，启动命令行增安全参数 (#1020)
9c20c54
@aiceflower @claude @casionone
[fix][EC][JDBC] Add configurable validation query for different datab…
936168a 
…ase types (#984)

* update version to 1.9.0

* #AI commit# refactor: 调整 GatewaySpringConfiguration 中 SecurityFilter 设置逻辑

- 添加 userRestful 字段注入
- 将 SecurityFilter.setUserRestful 调用从 createUserRestful 方法移至 @PostConstruct init 方法

* #AI commit# refactor: 升级 Knife4j 和 Springfox 到 3.x 版本

- 在 apache profile 中升级 knife4j.version 从 2.0.9 到 3.0.3
- 在 apache profile 中升级 springfox.version 从 2.10.5 到 3.0.0
- 添加 hbase-shaded-jetty 和 hbase-protocol-shaded 排除依赖，解决 Jetty 冲突
- SwaggerBeanPostProcessor 保持兼容 Springfox 3.x API

修复问题：
- 解决 publicservice 服务启动失败问题
- 解决 DynamicResponseModelReader 依赖 TypeNameExtractor 错误
- 解决 Jetty ServletHolder.setClassFrom IllegalStateException 错误

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* #AI COMMIT# Exclude hbase dependencies from JDBC and Hive engine connectors

Add exclusions for hbase-shaded-jetty and hbase-protocol-shaded to prevent
Jetty class conflicts in JDBC and Hive EngineConn plugins.

Modified files:
- linkis-engineconn-plugins/jdbc/pom.xml
- linkis-engineconn-plugins/hive/pom.xml

* #AI COMMIT# Exclude hbase-shaded-jetty from assembly packaging

Add hbase-shaded-jetty and hbase-protocol-shaded exclusions to distribution.xml
files to prevent Jetty class conflicts at runtime.

Modified files:
- linkis-public-enhancements/distribution.xml
- linkis-engineconn-plugins/hive/src/main/assembly/distribution.xml
- linkis-engineconn-plugins/jdbc/src/main/assembly/distribution.xml

* #AI COMMIT# Use wildcard pattern to exclude all hbase-shaded jars

Change from excluding individual hbase-shaded-* jars to using wildcard
pattern to exclude all hbase-shaded packages at once.

Modified files:
- linkis-public-enhancements/distribution.xml
- linkis-engineconn-plugins/hive/src/main/assembly/distribution.xml
- linkis-engineconn-plugins/jdbc/src/main/assembly/distribution.xml

* #AI COMMIT# Exclude all hbase-shaded jars explicitly

List each hbase-shaded jar explicitly instead of using wildcard pattern
which doesn't work with Maven assembly plugin.

Modified files:
- linkis-public-enhancements/distribution.xml
- linkis-engineconn-plugins/hive/src/main/assembly/distribution.xml
- linkis-engineconn-plugins/jdbc/src/main/assembly/distribution.xml

* #AI COMMIT# Exclude all hbase jars from linkis-public-enhancements assembly

Add comprehensive exclusions for all hbase-related jars to prevent
Jetty class conflicts and reduce unnecessary dependencies.

Modified:
- linkis-public-enhancements/distribution.xml

* # AI COMMIT# Optimize linkis scripts - service alias and status output format

## Changes

### 1. Service Alias Mapping (alias 1-6)
- alias [1] --> mg-eureka
- alias [2] --> mg-gateway
- alias [3] --> ps-publicservice
- alias [4] --> cg-linkismanager
- alias [5] --> cg-entrance
- alias [6] --> cg-engineconnmanager

### 2. Status Output Format
- Unified format: alias->[X]  [service-name]  [OK/FAILED]
- Service names left-aligned in 22-character width
- Green [OK] for running services
- Red [FAILED] for failed services

### 3. Files Modified
- linkis-dist/package/sbin/linkis
- linkis-dist/package/sbin/linkis-start-all.sh

### 4. Help Information
Updated help message to show new alias format

* #AI COMMIT# feat: Add build-time parameter for Python Module feature toggle

- Add PYTHON_MODULE environment variable support in vue.config.js
- Create separate python-module-config virtual module to avoid circular dependency
- Modify router.js to conditionally register pythonModule route
- Update index.vue to control menu visibility based on feature flag
- Default: enabled, use PYTHON_MODULE=false npm run build to disable

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* #AI COMMIT# docs: Remove Python Module documentation files

These documentation files are not required by the project and can be removed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* #AI COMMIT# fix: Fix i18n key for add button in datasourceAccess module

- Change 'message.linkis.basedata.add' to 'message.linkis.basedataManagement.add'
- Keep consistent with other modules like datasourceEnv
- Fix Chinese translation not displaying for add button

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* #AI COMMIT# feat: Add build-time parameter for disallow system user login feature

Add compile-time toggle for disallow login feature, similar to python module:
- Default enabled (feature ON)
- Use DISALLOW_LOGIN=false npm run build to disable
- Create virtual module disallow-login-config.js for config injection

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* #AI COMMIT# fix: Add DB2 validation query support for JDBC engine connection pool

- Add DB2-specific validation query "SELECT 1 FROM SYSIBM.SYSDUMMY1"
- Same pattern as existing Oracle validation query handling
- Fixes DB2 connection pool validation error when using datasource name

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* #AI COMMIT# fix: Exclude conflicting jars from JDBC plugin assembly

- Exclude Jetty jars to avoid version conflicts with public-module
- Exclude SLF4J bindings to prevent multiple binding errors
- Exclude Hadoop jars to use shared public-module versions

These exclusions prevent class loading conflicts when JDBC engine
starts and uses jars from both plugin lib and public-module.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* #AI COMMIT# refactor: Make validation query configurable for different database types

- Add configuration parameter 'wds.linkis.jdbc.validation.query.mapping'
- Default mapping: oracle:SELECT 1 FROM DUAL,db2:SELECT 1 FROM SYSIBM.SYSDUMMY1
- Remove hardcoded database-specific validation query logic
- To add new database type, just update configuration, no code change needed

Benefits:
- Configuration-driven approach for extensibility
- Easy to add support for new databases without modifying source code
- Default values work for most common databases

Usage example:
  Add to linkis-engineconn.properties:
  wds.linkis.jdbc.validation.query.mapping=oracle:SELECT 1 FROM DUAL,db2:SELECT 1 FROM SYSIBM.SYSDUMMY1,postgresql:SELECT 1

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* #AI commit# Remove hbase-server dependency from pom.xml

Removed hbase-server dependency and its exclusions from pom.xml.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Casion <casionone@gmail.com>
@v-kkhuang
[fix][EC][spark] fix spark and hive engine execution task error excep…
39f0e37 
…tion handling (#1023)

* #AI commit# 开发阶段： 异常注释修复

* #AI commit# 开发阶段： 修复依赖注入失败

* #AI commit# 开发阶段： 优化2.0.0特性文档

* #AI commit# 开发阶段： 日志打印优化，spark init sql 异常优化

* #AI commit# 开发阶段： 支持hadoop3引擎物料刷新

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复spark/hive引擎执行任务error异常
v-kkhuang and others added 8 commits May 9, 2026 17:04
@v-kkhuang
Dev 2.0.0 bug fix (#1026)
04a9535 
* #AI commit# 开发阶段： 异常注释修复

* #AI commit# 开发阶段： 修复依赖注入失败

* #AI commit# 开发阶段： 优化2.0.0特性文档

* #AI commit# 开发阶段： 日志打印优化，spark init sql 异常优化

* #AI commit# 开发阶段： 支持hadoop3引擎物料刷新

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复spark/hive引擎执行任务error异常

* #AI commit# 开发阶段： 修复开源代码合并异常
@v-kkhuang
[refactor][CGS][entrance] remove duplicate log printing code in entra…
dbec63e 
…nce server (#1028)

* #AI commit# 开发阶段： 异常注释修复

* #AI commit# 开发阶段： 修复依赖注入失败

* #AI commit# 开发阶段： 优化2.0.0特性文档

* #AI commit# 开发阶段： 日志打印优化，spark init sql 异常优化

* #AI commit# 开发阶段： 支持hadoop3引擎物料刷新

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复spark/hive引擎执行任务error异常

* #AI commit# 开发阶段： 修复开源代码合并异常

* #AI commit# 开发阶段： 去除日志打印重复代码

* #AI commit# 开发阶段： 适配行内版本
@v-kkhuang
[fix][MGS][gateway] fix compilation errors in security filter and ada…
71c21ce 
…pt to inline version (#1030)

* #AI commit# 开发阶段： 异常注释修复

* #AI commit# 开发阶段： 修复依赖注入失败

* #AI commit# 开发阶段： 优化2.0.0特性文档

* #AI commit# 开发阶段： 日志打印优化，spark init sql 异常优化

* #AI commit# 开发阶段： 支持hadoop3引擎物料刷新

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复日志打印异常，优化monitor清理流程

* #AI commit# 开发阶段： 修复spark/hive引擎执行任务error异常

* #AI commit# 开发阶段： 修复开源代码合并异常

* #AI commit# 开发阶段： 去除日志打印重复代码

* #AI commit# 开发阶段： 适配行内版本

* #AI commit# 开发阶段： 修复编译报错
@v-kkhuang
#AI commit# 开发阶段：Revert community code (#1032)
a4d31e5
@v-kkhuang
#AI commit# refactor:忽略Angent系统文件
a23c5c8
@v-kkhuang
#AI commit# refactor:修复linkismanager 慢sql (#1042)
814447e
@v-kkhuang
#AI commit# refactor:修复datasource oom异常 (#1040)
65fddab
@casionone
Merge branch 'master' into dev-2.0.0-webank
1843e45
casionone
casionone approved these changes Jun 15, 2026
View reviewed changes

@casionone casionone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@casionone casionone casionone approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

3 participants

@aiceflower @casionone @v-kkhuang