[R] read_ipc_stream fails to unify nested uint64 fields inside a Struct array across record batches

[jo-vogel]

When reading an Arrow IPC stream that contains a nested Struct array named regions, some of its fields — e.g. regional_key — are explicitly typed as uint64 in the source API schema (described here: https://md.umwelt.info/swagger-ui/ at the bottom in Dataset/regions/RegionalKey).

Across different record batches in the stream, the R arrow package inconsistently casts this nested uint64 field:
In chunks with smaller values, it is converted into an R integer.
In chunks with larger values, it is converted into an R double to preserve precision.

Because this type unification fails globally across chunks for nested elements, the resulting R data frame ends up with conflicting types row-by-row within the list-column. This causes standard tidyverse tools like tidyr::unnest_wider() to fail due to a loss of precision mismatch.

In contrast, Python's pyarrow handling of the same IPC stream correctly unifies the entire nested regional_key field into a consistent float type across all records.

According to https://arrow.apache.org/docs/r/articles/data_types.html there is the option to use
options(arrow.int64_downcast = FALSE)
but this seems to work only at the top level and not for nested structures.

Minimal Reproducible Example

library(arrow)
library(httr2)
library(tidyr)

# 1. Fetch the exact IPC stream that contains the heterogeneous nested structures
req <- httr2::request("https://md.umwelt.info/search/all?format=arrow_ipc") |>
  httr2::req_url_query(
    query = "type:'/Daten und Messstellen/Wasser/Flüsse' AND measuring_station:true", 
    language = "de"
  )

resp <- httr2::req_perform(req)
raw_bytes <- httr2::resp_body_raw(resp)

# 2. Parse the stream into an R data frame via Arrow
df <- as.data.frame(arrow::read_ipc_stream(raw_bytes))

# 3. Attempt to unnest the 'regions' struct column
# This fails because 'regional_key' inside 'regions' alternates between integer and double across rows
df |> unnest_wider(any_of(c("regions")), names_sep = "_")

# Throws:
# Error in `unnest_wider()`:
# ! Can't convert from `..1` <double> to <integer> due to loss of precision.
# • Locations: 2

Expected behavior
If a uint64 field inside a regions struct chunk requires upcasting to an R double, the entire nested column across all record batches should be uniformly converted to double to maintain data structure integrity upon data frame extraction.

Environment

OS: Ubuntu 26.04 LTS

R Version: 4.5.2

arrow R Package Version: 24.0.0

Component(s)

R

[thisisnic]

I had Claude deep dive into the codebase and here's whati says is going on:

The root cause is in Converter_List::Ingest_some_nulls. Each list element is converted independently via Converter::Convert(slice), which creates a fresh converter for that element's data. For UINT64, Converter::Make calls ArraysCanFitInteger() to decide whether to use R integer or double — but it's checking each element's values in isolation. So row 1 with regional_key = 5 gets integer, while row 2 with regional_key = 9999999999 gets double.

This wasn't always a problem. List conversion was added in 2019 (ARROW-5586) when UINT64 always converted to double. The value-dependent downcasting (ArraysCanFitInteger) was added later in 2020 (ARROW-9083), and nobody reconnected it with the list path.

For context, Converter_Struct doesn't have this problem — it uses Table::FromChunkedStructArray() to gather all chunks into one ChunkedArray per field, so ArraysCanFitInteger checks all values together and makes one consistent decision. But Converter_List converts each element independently because each list element is a separate slice.

Also worth noting: arrow.int64_downcast only governs INT64, not UINT64 — so that option can't be used as a workaround here.

Python avoids this entirely by using the schema type to determine output type rather than inspecting values. For lists, it flattens all values into one batch and converts them together.

I think the fix for R is to not do value-dependent downcasting inside list elements and instead use the declared Arrow type to make the conversion decision. The "fit into int32 if possible" optimization makes sense at the top level where you have a large column and the memory savings matter, but inside list elements the overhead of the list + data frame per row dwarfs any savings from using integer vs double. And the consistency is what matters — it's what makes unnest_wider() and similar tools work.