From 823830b88e35cd5408901807d5d9b28f72985018 Mon Sep 17 00:00:00 2001 From: Jorge Molina <70665012+Joorgem@users.noreply.github.com> Date: Tue, 28 Jul 2026 15:27:32 -0300 Subject: [PATCH] [MINOR][DOCS] Document CSV read behaviour when multiLine is disabled The multiLine entry documents only what enabling the option does. This adds what the default costs: a line break inside a quoted value terminates the record, remaining schema fields become null, the rest of the value starts a new record, and both halves are malformed -- so FAILFAST raises, DROPMALFORMED discards both halves and loses the whole source record, and PERMISSIVE keeps both silently unless columnNameOfCorruptRecord is declared. Also notes that the split increases the record count and that an action needing no columns (a bare count()) may surface none of it, because of parser column pruning. Verified against pyspark 3.5.9. --- docs/sql-data-sources-csv.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md index 9dfe9739b7ac7..03a2eac0cb431 100644 --- a/docs/sql-data-sources-csv.md +++ b/docs/sql-data-sources-csv.md @@ -231,7 +231,8 @@ Data source options of CSV can be set via:
multiLinenull, and the rest of the value begins a new record. Both halves are malformed records, so what happens next follows mode: FAILFAST raises an error, DROPMALFORMED discards both halves and therefore loses the whole source record, and PERMISSIVE retains both, without any signal unless the schema declares columnNameOfCorruptRecord. Note also that the split increases the number of records, and that an action requiring no columns (a bare count(), for instance) may surface none of this, because of parser column pruning.