[MINOR][DOCS] Document CSV read behaviour when multiLine is disabled

[Joorgem]

What changes were proposed in this pull request?

Extends the multiLine entry in the CSV data source options table to describe what happens when the option is left at its default of false and a quoted value contains a line break.

The entry currently documents only what enabling the option does:

Allows a row to span multiple lines, by parsing line breaks within quoted values as part of the value itself. CSV built-in functions ignore this option.

The added sentence states that the record is terminated at the line break, that the remaining schema fields are set to null, that the rest of the value begins a new record, and how each mode then behaves.

One table row changed; no other files touched.

Why are the changes needed?

A line break inside a quoted field is valid CSV (RFC 4180 §2.6) and occurs in real published datasets, so a reader can meet this without doing anything unusual. The entry says what multiLine=true enables but never what the default costs, and the behaviour is not guessable from it:

• the record is split in two, which increases the record count;
• the leading half keeps plausible values for its first fields and null for the rest, so it can satisfy schema and not-null style checks;
• under the default PERMISSIVE mode there is no signal at all unless the schema happens to declare columnNameOfCorruptRecord;
• DROPMALFORMED discards both halves, losing the entire source record rather than a bad line;
• a bare count() reports the inflated number without surfacing a malformed record, because column pruning means no column is parsed.

The last two were the surprises worth writing down, and neither is inferable from the current text.

SPARK-21356 was closed as Invalid because "the workaround looks so easy". That is true, and is exactly the argument for documenting the default: the workaround only helps a reader who knows they need it.

Does this PR introduce any user-facing change?

No. Documentation only; no behaviour change.

How was this patch tested?

Documentation-only, so no tests were added. The described behaviour was verified against pyspark 3.5.9 using a three-record file whose second record contains a quoted line break, read with an explicit three-column schema:

read	result
default (multiLine=false)	4 rows from 3 records: 2, 'EMPRESA COM', null plus a fragment 'QUEBRA DE LINHA"', 'RJ', null
multiLine=true	3 rows, values intact
mode=PERMISSIVE, collect()	4 rows, no error
mode=DROPMALFORMED, collect()	2 rows — both halves dropped
mode=FAILFAST, collect()	raises
any mode, bare count()	4 rows, no error

With columnNameOfCorruptRecord declared, both halves are captured as corrupt, with raw text 2,"EMPRESA COM and QUEBRA DE LINHA",RJ.

git diff --check is clean.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

[szehon-ho]

Thanks for documenting this — the default-multiLine gotcha is worth calling out. Suggestion on placement: keep the option-specific fact under multiLine, and move the mode / column-pruning surprises next to mode, which already documents corrupt handling and pruning.

multiLine — keep only the split behavior:

<td>Allows a row to span multiple lines, by parsing line breaks within quoted values as part of the value itself. CSV built-in functions ignore this option.<br>
When this option is disabled (the default), a line break inside a quoted value terminates the record at that break: the value is truncated, the remaining fields of the schema are set to <code>null</code>, and the rest of the value begins a new record. Both resulting records are malformed; how they are handled is controlled by <code>mode</code>.</td>

mode — append after the existing column-pruning sentence (before the <ul>):

<td>Allows a mode for dealing with corrupt records during parsing. It supports the following case-insensitive modes. Note that Spark tries to parse only required columns in CSV under column pruning. Therefore, corrupt records can be different based on required set of fields. This behavior can be controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled by default). In particular, when <code>multiLine</code> is disabled, a quoted value that contains a line break is split into two malformed records, which increases the record count: <code>DROPMALFORMED</code> discards both halves and therefore loses the whole source record, <code>FAILFAST</code> raises an error, and <code>PERMISSIVE</code> retains both without a signal unless the schema declares <code>columnNameOfCorruptRecord</code>. An action that requires no columns (for example a bare <code>count()</code>) may surface none of this because of column pruning.<br>
<ul>
  <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the malformed string into a field configured by <code>columnNameOfCorruptRecord</code>, and sets malformed fields to <code>null</code>. To keep corrupt records, an user can set a string type field named <code>columnNameOfCorruptRecord</code> in an user-defined schema. If a schema does not have the field, it drops corrupt records during parsing. A record with less/more tokens than schema is not a corrupted record to CSV. When it meets a record having fewer tokens than the length of the schema, sets <code>null</code> to extra fields. When the record has more tokens than the length of the schema, it drops extra tokens.</li>
  <li><code>DROPMALFORMED</code>: ignores the whole corrupted records. This mode is unsupported in the CSV built-in functions.</li>
  <li><code>FAILFAST</code>: throws an exception when it meets corrupted records.</li>
</ul>
</td>

That keeps multiLine to what the default does to the record, and puts the DROPMALFORMED / PERMISSIVE / count() caveats with the option that already covers corrupt handling.