From 823830b88e35cd5408901807d5d9b28f72985018 Mon Sep 17 00:00:00 2001 From: Jorge Molina <70665012+Joorgem@users.noreply.github.com> Date: Tue, 28 Jul 2026 15:27:32 -0300 Subject: [PATCH] [MINOR][DOCS] Document CSV read behaviour when multiLine is disabled The multiLine entry documents only what enabling the option does. This adds what the default costs: a line break inside a quoted value terminates the record, remaining schema fields become null, the rest of the value starts a new record, and both halves are malformed -- so FAILFAST raises, DROPMALFORMED discards both halves and loses the whole source record, and PERMISSIVE keeps both silently unless columnNameOfCorruptRecord is declared. Also notes that the split increases the record count and that an action needing no columns (a bare count()) may surface none of it, because of parser column pruning. Verified against pyspark 3.5.9. --- docs/sql-data-sources-csv.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md index 9dfe9739b7ac7..03a2eac0cb431 100644 --- a/docs/sql-data-sources-csv.md +++ b/docs/sql-data-sources-csv.md @@ -231,7 +231,8 @@ Data source options of CSV can be set via: multiLine false - Allows a row to span multiple lines, by parsing line breaks within quoted values as part of the value itself. CSV built-in functions ignore this option. + Allows a row to span multiple lines, by parsing line breaks within quoted values as part of the value itself. CSV built-in functions ignore this option.
+ When this option is disabled, a line break inside a quoted value terminates the record at that break: the value is truncated, the remaining fields of the schema are set to null, and the rest of the value begins a new record. Both halves are malformed records, so what happens next follows mode: FAILFAST raises an error, DROPMALFORMED discards both halves and therefore loses the whole source record, and PERMISSIVE retains both, without any signal unless the schema declares columnNameOfCorruptRecord. Note also that the split increases the number of records, and that an action requiring no columns (a bare count(), for instance) may surface none of this, because of parser column pruning. read