ReadStat produces unreadable files when long-string ghost variable names collide with real variable names

[jhorzek]

Thank you very much for this great package! We use it extensively to work with .sav in R and have had an amazing experience so far :).

The only issue we experience from time to time is that ReadStat writes a .sav file that then cannot be read into R again. I think haven is using ReadStat 1.1.9, but it looks like the same issue could still be present here.

I spent some time trying to understand what is going on and I think I could pinpoint the issue with some help from Claude.

Issue description

When storing long strings, ReadStat can create multiple variables with the same shortname that clash on read. I think the issue is with how the hash table is created:

1. Every variable's short name is its name truncated to 8 bytes and uppercased.
2. If there is a long string, the variable is split into multiple variables - the base and some additional ghost variables. However, I think ReadStat does not check if the names of the ghost variables are already taken

Example using R:

df <- tibble::tibble(
  # LONGA's ghost segment is named "LONGA" + "1" = "LONGA1", duplicating the
  # short name of the real column LONGA1
  LONGA  = paste0(strrep("a", 254), "क"),  # 257 bytes
  LONGA1 = paste0(strrep("b", 254), "क")   # 257 bytes
)
haven::write_sav(df, path = temp_file)
haven::read_sav(temp_file)
#> Error: Failed to parse ...: Unable to convert string to the requested
#> encoding (invalid byte sequence).

latin1 <- haven::read_sav(temp_file, encoding = "latin1")
#> New names:
#> • `LONGA1` -> `LONGA1...2`
#> • `LONGA1` -> `LONGA1...3`
colnames(latin1)
# "LONGA" "LONGA1...2" "LONGA1...3"

The fix would be to only accept names that are not already taken.

I have had Claude prepare a fix for this because I am not very versed in C, but won't open a pull request because I lack the C skills to review Claude's solution. See here: jhorzek#1