[SPARK-58167][SPARK-57393][3.5] Build: PySpark and SparkR source distributions are missing LICENSE and NOTICE files - #57427
Conversation
This reverts commit bc180d3.
… R and Python dist" This reverts commit 5e69fb2.
…ons are missing LICENSE and NOTICE files Make the PySpark and SparkR source distributions include top-level LICENSE and NOTICE files: - Add include LICENSE / include NOTICE to python/MANIFEST.in. - In dev/make-distribution.sh, copy LICENSE/NOTICE into python/ and R/pkg/ before building the sdists (with an EXIT trap to clean them up). - The classic pyspark sdist bundles the assembly jars, so it ships the binary license variants (LICENSE-binary/NOTICE-binary plus the full licenses-binary set), mirroring the binary distribution. The pyspark_connect, pyspark_client, and SparkR artifacts bundle no jars, so they ship the plain source LICENSE/NOTICE. packaging/classic/setup.py falls back to licenses/ when licenses-binary/ is absent (RELEASE-mode builds). - During the SparkR build, + file LICENSE is added to DESCRIPTION temporarily (restored after the build) so R CMD check --as-cran does not warn that the bundled LICENSE is not mentioned. The committed DESCRIPTION is unchanged, so SparkR CI is unaffected. The "Non-standard file/directory found at top level: 'NOTICE'" NOTE cannot be silenced this way and is expected in release-build logs. - Add tar-listing regression guards after the sdist and SparkR builds that fail the release build if LICENSE or NOTICE ever goes missing again, instead of relying on an RC vote to catch it. ASF release policy requires every distributed artifact to include LICENSE and NOTICE. The pyspark, pyspark_connect, pyspark_client, and SparkR source tarballs currently don't, which was raised as a -1 during the Spark 4.2.0 RC1 vote. No Built the source distributions locally and confirmed each contains LICENSE and NOTICE at the package root Yes, drafted with assistance from Cursor. Closes apache#56453 from huaxingao/fix_license. Authored-by: Huaxin Gao <huaxin.gao11@gmail.com> Co-Authored-by: Huaxin Gao <huaxin.gao11@gmail.com> Signed-off-by: huaxin-gao_snow <huaxin.gao@snowflake.com> (cherry picked from commit b88952c) Signed-off-by: huaxin-gao_snow <huaxin.gao@snowflake.com> Backported-By: Holden Karau <holden@pigscanfly.ca>
| grep -qE "^[^/]+/$required\$" <<< "$listing" || \ | ||
| { echo "ERROR: $f is missing $required at the package root"; exit 1; } | ||
| done | ||
| done |
There was a problem hiding this comment.
Missing the explicit rm -f LICENSE NOTICE before popd that master carries (master L298). On master the removal runs before cp -r "$SPARK_HOME/python" "$DISTDIR" (L340), so the assembled binary tarball's python/ is clean. Here the files are removed only by the EXIT trap, which fires at script end; after the cp -r python DISTDIR — so DISTDIR/python/ ships stray LICENSE/NOTICE (harmless duplicates of the top-level ones, but a divergence from master's design intent), and the adjacent comment "These are removed again after the sdists are built" is inaccurate. Note the R block just below does remove pkg/LICENSE pkg/NOTICE explicitly, so the asymmetry looks like an oversight. Fix: add rm -f LICENSE NOTICE immediately before the PySpark popd.
| # Ship the Apache LICENSE and NOTICE inside the PySpark source distributions | ||
| # (see MANIFEST.in). These are removed again after the sdists are built. | ||
| # | ||
| # The classic pyspark sdist bundles the assembly jars (packaging/classic/setup.py |
There was a problem hiding this comment.
Comments inherited verbatim from master describe a layout that does not exist on 3.5: "packaging/classic/setup.py builds a deps/jars symlink farm" and "The connect and client sdists bundle no jars and ship the plain source LICENSE/NOTICE." Confirmed branch-3.5 has no python/packaging/ directory (only setup.py), so a single classic pyspark sdist is produced. The logic is correct for 3.5, but the shipped comments will mislead a future 3.5 maintainer; trim the connect/client/ packaging references for the 3.5 context.
uros-b
left a comment
uros-b
left a comment
There was a problem hiding this comment.
Left a few comments, but otherwise LGTM
…ranch-3.5 comments - Add explicit `rm -f LICENSE NOTICE` before `popd` in the PySpark section to match the R block's pattern and avoid relying solely on the EXIT trap. - Remove references to packaging/classic/setup.py, connect, and client sdist variants that don't exist in branch-3.5; only a single classic pyspark sdist is produced here. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
4 participants
Changes proposed:
Backport of [SPARK-57393] Build: PySpark and SparkR source distributions are missing LICENSE and NOTICE files
Make the PySpark and SparkR source distributions include top-level LICENSE and NOTICE files:
Why are they needed?
ASF release policy requires every distributed artifact to include LICENSE and NOTICE. The pyspark, pyspark_connect, pyspark_client, and SparkR source tarballs currently don't, which was raised as a -1 during the Spark 3.5.9 RC1
No
Built the source distributions locally and confirmed each contains LICENSE and NOTICE at the package root