[SPARK-51975][SQL] Add variant_from_arrays and variant_from_entries

[SreeramaYeshwanthGowd]

What changes were proposed in this pull request?

Add two built in scalar SQL functions that construct a Variant object directly from collections:

• variant_from_arrays(keys, values) builds a Variant object from an array of string keys and an array of values.
• variant_from_entries(entries) builds a Variant object from an array of key/value struct entries (a two field struct per entry).

> SELECT variant_from_arrays(array('a', 'b'), array(1, 2));
 {"a":1,"b":2}
> SELECT variant_from_entries(array(struct('a', 1), struct('b', 2)));
 {"a":1,"b":2}

API surface added:

• SQL: variant_from_arrays, variant_from_entries
• Scala DataFrame: functions.variant_from_arrays(keys, values), functions.variant_from_entries(entries)
• PySpark, classic and Spark Connect: pyspark.sql.functions.variant_from_arrays / variant_from_entries

Implementation notes:

• Both are native expressions in variantExpressions.scala next to to_variant_object, delegating (eval and codegen) to two new VariantExpressionEvalUtils helpers that build the object in a single pass with VariantBuilder (addKey / finishWritingObject) and reuse the existing buildVariant for each value. This avoids materializing an intermediate MapType, which is the point of the ticket.
• Semantics, chosen to match the existing to_variant_object and map_from_* precedents:
  • Object keys must be non-null strings. A non-string key type is rejected at analysis (as to_variant_object does for non-string-keyed maps, it is not implicitly cast); a null key raises NULL_MAP_KEY.
  • Duplicate keys raise VARIANT_DUPLICATE_KEY, matching to_variant_object (Variant objects cannot hold duplicate keys). This is stricter than map_from_* under the non default spark.sql.mapKeyDedupPolicy=LAST_WIN.
  • Null values are kept as Variant null. A null array argument yields NULL, and for variant_from_entries a null entry yields NULL (as map_from_entries does).
  • For variant_from_arrays, a keys/values length mismatch reuses the same error map_from_arrays raises.

Why are the changes needed?

Today, creating a Variant object from arrays or entries requires a two step expression: to_variant_object(map_from_arrays(...)) or to_variant_object(map_from_entries(...)). That materializes an intermediate MapType only to immediately convert it. variant_from_arrays / variant_from_entries build the Variant object directly, reducing allocation and CPU, which matters for large maps and high volume workloads. They also mirror Spark's own map_from_arrays / map_from_entries naming, so the Variant surface stays consistent.

Does this PR introduce any user-facing change?

Yes. It adds two new built in SQL functions and their Scala and PySpark DataFrame API entries. No existing behavior changes.

How was this patch tested?

• Catalyst unit tests in VariantExpressionSuite covering object construction, key sorting, empty input, null values, nested values, null array / null entry inputs returning null, and the null key / duplicate key / length mismatch errors.
• An end to end test in VariantEndToEndSuite exercising the DataFrame API under both CODEGEN_ONLY and NO_CODEGEN over a non foldable relation.
• SQL golden tests in variant/variant-from-arrays-entries.sql, covering each JSON type, null handling, and the null key / duplicate key / length mismatch / non-string key / unsupported value type errors.
• PySpark doctests and a test_functions.py test, and regenerated sql-expression-schema.md.

Was this patch authored or co-authored using generative AI tooling? No