From 83cf9e76499d907bee59574009dca80b047c95c0 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Sat, 8 Aug 2026 13:22:08 -0600 Subject: [PATCH] docs: correct Spark 4.2 version and CI test status in installation guide The installation guide listed the experimental Spark 4.2 support as targeting `4.2.0-preview4` and claimed no Comet tests run in CI. Both are stale: - The `spark-4.2` Maven profile targets the `4.2.0` final release. - `pr_build_linux.yml` includes a "Spark 4.2, JDK 17" job in the `linux-test` matrix, so Comet tests do run in CI. Spark SQL tests still do not run for 4.2 (no `spark_4_2` job in `ci.yml`), so that column stays "No" and the version remains experimental. Also refresh the two `4.2.0-preview4` examples in the contributor guide, which cited a version Comet no longer targets, to use a hypothetical future release instead. --- docs/source/contributor-guide/adding_a_new_spark_version.md | 5 +++-- .../user-guide/latest/compatibility/spark-versions.md | 6 +++--- docs/source/user-guide/latest/installation.md | 6 +++--- 3 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/source/contributor-guide/adding_a_new_spark_version.md b/docs/source/contributor-guide/adding_a_new_spark_version.md index 45c9ebc15c..64f90bec5c 100644 --- a/docs/source/contributor-guide/adding_a_new_spark_version.md +++ b/docs/source/contributor-guide/adding_a_new_spark_version.md @@ -62,7 +62,8 @@ existing profile (for example `spark-4.1`) and update the version properties: - `spark.version`: the full upstream version, including any qualifier - (for example `4.2.0-preview4`). + (for example `4.3.0` for a final release, or `4.3.0-preview1` while only a + preview has been published). - `spark.version.short`: the major.minor (for example `4.2`). - `parquet.version`, `slf4j.version`, `scala.version`, `scala.binary.version`, `java.version`: align with what the new Spark @@ -317,7 +318,7 @@ issue keeps the diff small and makes regressions easy to bisect. The user guide currently uses two tiers, "Supported" and "Experimental". Comet uses "Experimental" to describe its confidence in its own integration with a Spark version, distinct from Spark's "preview" tag -(which refers to upstream release qualifiers like `4.2.0-preview4`). The +(which refers to upstream release qualifiers like `4.3.0-preview1`). The term is already established in `installation.md`, `operators.md`, and `datasources.md`, so keep using it rather than introducing a new label. diff --git a/docs/source/user-guide/latest/compatibility/spark-versions.md b/docs/source/user-guide/latest/compatibility/spark-versions.md index 0675aa83ec..090a1df85e 100644 --- a/docs/source/user-guide/latest/compatibility/spark-versions.md +++ b/docs/source/user-guide/latest/compatibility/spark-versions.md @@ -91,9 +91,9 @@ Spark 4.1.3 is supported with Java 17/21 and Scala 2.13. ## Spark 4.2 (Experimental) -Spark 4.2.0-preview4 is provided as experimental support with Java 17 and Scala 2.13. +Spark 4.2.0 is provided as experimental support with Java 17 and Scala 2.13. ```{warning} -Spark 4.2 support is experimental and targets a preview release of Spark. It is intended for early -evaluation only and should not be used in production. +Spark 4.2 support is experimental. Comet tests run in CI for this version, but the Spark SQL tests +do not yet. It is intended for early evaluation only and should not be used in production. ``` diff --git a/docs/source/user-guide/latest/installation.md b/docs/source/user-guide/latest/installation.md index 9fc350c25f..7fc4a81d89 100644 --- a/docs/source/user-guide/latest/installation.md +++ b/docs/source/user-guide/latest/installation.md @@ -62,9 +62,9 @@ Note that we do not test the full matrix of supported Java and Scala versions in Experimental support is provided for the following versions of Apache Spark and is intended for development/testing use only and should not be used in production yet. -| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI | -| -------------- | ------------ | ------------- | ----------------- | --------------------- | -| 4.2.0-preview4 | 17 | 2.13 | No | No | +| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark SQL Tests in CI | +| ------------- | ------------ | ------------- | ----------------- | --------------------- | +| 4.2.0 | 17 | 2.13 | Yes | No | Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by Cloud Service Providers.