Skip to content

[MINOR][DOCS] Document CSV read behaviour when multiLine is disabled #57608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Joorgem wants to merge 1 commit into
base: master
Choose a base branch
from
+2 −1
Open

[MINOR][DOCS] Document CSV read behaviour when multiLine is disabled #57608

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/sql-data-sources-csv.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,8 @@ Data source options of CSV can be set via:
<tr>
<td><code>multiLine</code></td>
<td>false</td>
<td>Allows a row to span multiple lines, by parsing line breaks within quoted values as part of the value itself. CSV built-in functions ignore this option.</td>
<td>Allows a row to span multiple lines, by parsing line breaks within quoted values as part of the value itself. CSV built-in functions ignore this option.<br>
When this option is disabled, a line break inside a quoted value terminates the record at that break: the value is truncated, the remaining fields of the schema are set to <code>null</code>, and the rest of the value begins a new record. Both halves are malformed records, so what happens next follows <code>mode</code>: <code>FAILFAST</code> raises an error, <code>DROPMALFORMED</code> discards both halves and therefore loses the whole source record, and <code>PERMISSIVE</code> retains both, without any signal unless the schema declares <code>columnNameOfCorruptRecord</code>. Note also that the split <em>increases</em> the number of records, and that an action requiring no columns (a bare <code>count()</code>, for instance) may surface none of this, because of parser column pruning.</td>

@szehon-ho szehon-ho Jul 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for documenting this — the default-multiLine gotcha is worth calling out. Suggestion on placement: keep the option-specific fact under multiLine, and move the mode / column-pruning surprises next to mode, which already documents corrupt handling and pruning.

multiLine — keep only the split behavior:

<td>Allows a row to span multiple lines, by parsing line breaks within quoted values as part of the value itself. CSV built-in functions ignore this option.<br>
When this option is disabled (the default), a line break inside a quoted value terminates the record at that break: the value is truncated, the remaining fields of the schema are set to <code>null</code>, and the rest of the value begins a new record. Both resulting records are malformed; how they are handled is controlled by <code>mode</code>.</td>

mode — append after the existing column-pruning sentence (before the <ul>):

<td>Allows a mode for dealing with corrupt records during parsing. It supports the following case-insensitive modes. Note that Spark tries to parse only required columns in CSV under column pruning. Therefore, corrupt records can be different based on required set of fields. This behavior can be controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled by default). In particular, when <code>multiLine</code> is disabled, a quoted value that contains a line break is split into two malformed records, which increases the record count: <code>DROPMALFORMED</code> discards both halves and therefore loses the whole source record, <code>FAILFAST</code> raises an error, and <code>PERMISSIVE</code> retains both without a signal unless the schema declares <code>columnNameOfCorruptRecord</code>. An action that requires no columns (for example a bare <code>count()</code>) may surface none of this because of column pruning.<br>
<ul>
  <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the malformed string into a field configured by <code>columnNameOfCorruptRecord</code>, and sets malformed fields to <code>null</code>. To keep corrupt records, an user can set a string type field named <code>columnNameOfCorruptRecord</code> in an user-defined schema. If a schema does not have the field, it drops corrupt records during parsing. A record with less/more tokens than schema is not a corrupted record to CSV. When it meets a record having fewer tokens than the length of the schema, sets <code>null</code> to extra fields. When the record has more tokens than the length of the schema, it drops extra tokens.</li>
  <li><code>DROPMALFORMED</code>: ignores the whole corrupted records. This mode is unsupported in the CSV built-in functions.</li>
  <li><code>FAILFAST</code>: throws an exception when it meets corrupted records.</li>
</ul>
</td>

That keeps multiLine to what the default does to the record, and puts the DROPMALFORMED / PERMISSIVE / count() caveats with the option that already covers corrupt handling.

<td>read</td>
</tr>
<tr>
Expand Down