[Improvement]: Upgrade Iceberg with per-engine compatibility modules

[xxubai]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

Amoro's Iceberg upgrade path currently conflicts across engines because Flink/Spark/Hive depend on different Iceberg artifact availability and API levels. A single global iceberg.version causes incompatibilities in build and runtime behavior.

How should we improve?

Adopt per-module iceberg.version overrides and engine-specific compatibility modules:

• Keep older engines on compatible Iceberg versions when upstream artifacts/APIs are limited.
• Upgrade newer engines to newer Iceberg versions.
• Maintain separate compatibility modules for Flink 1.17 paths (common + iceberg bridge).
• Adapt mixed-hive, flink bridge, and spark parquet reader code paths for newer Iceberg APIs.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

• Upgrade Iceberg versions with per-module override strategy.
• Add Flink 1.17 compatibility modules.
• Adapt mixed-hive/flink/spark code for Iceberg API changes.
• Final CI verification and follow-up fixes if needed.

Code of Conduct

• I agree to follow this project's Code of Conduct