Unicode letter identifiers cause compilation failures in recent AOT toolchains

[charlesroddie]

Android application type

Classic Xamarin.Android (MonoAndroid12.0, etc.)

Affected platform version

Android NDK 23.2.8568313

Description

Error message [aot-compiler stderr] ... invalid character in input.
This seems to result from using unicode characters in identifiers, including σ and Ѫ. Both of these are letter characters.

There was a breaking change in C++ and/or LLVM llvm/llvm-project#54732 which reduced the allowable identifiers, but every indication there is that letter characters are still supported, so that issue wouldn't seem to explain this problem, unless some implementation removed not only unicode non-letters but all non-ascii unicode letters by mistake.

Non-working Android NDKs:
23.2.8568313
24.0.8215888
25.1.8937393

Working Android NDKs:
22.1.7171670

Currently this is blocking compilation of our app on Azure Devops, which has the non-working versions. But we can build in a VS locally which has the older working version.

This is in a dotnet Xamarin.Forms app with full mono AOT/LLVM on.

If this is the wrong repo for the bug report please let me know.

Steps to Reproduce

Build https://github.com/FoggyFinder/XFAMCVE7336

The main branch displays the build failure in the simplest way (without specifying NDK), and the consistency branch has it fail only with newer NDKs.

Did you find any workaround?

No response

Relevant log output

[aot-compiler stdout] Executing the native linker: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Xamarin\Android\binutils\bin\x86_64-linux-android-ld.CMD" -Bsymbolic -shared -o obj\Release\130\aot\x86_64\libaot-40AppStorage.dll.so.tmp "obj\Release\130\aot\x86_64\40AppStorage.dll\temp-llvm.o" obj\Release\130\aot\x86_64\40AppStorage.dll\temp.s.o "-LC:\Program Files (x86)\Android\android-sdk
dk\25.1.8937393\toolchains\llvm\prebuilt\windows-x86_64\sysroot\usr\lib\x86_64-linux-android\21" "C:\Program Files (x86)\Android\android-sdk
dk\25.1.8937393\toolchains\llvm\prebuilt\windows-x86_64\sysroot\usr\lib\x86_64-linux-android\21\libc.so" "C:\Program Files (x86)\Android\android-sdk
dk\25.1.8937393\toolchains\llvm\prebuilt\windows-x86_64\sysroot\usr\lib\x86_64-linux-android\21\libm.so" -s
[aot-compiler stdout] JIT time: 141 ms, Generation time: 1321 ms, Assembly+Link time: 525 ms.
[aot-compiler stderr] obj\\Release\\130\\aot\\x86_64\\40AppStorage.dll\\temp.s:2446:40: error: invalid character in input

[charlesroddie]

Info from Discord

[dascandy]

This is not related to the bug mentioned above. Can you check if the input file is in Unicode normalization form NFC? This is a thing that was proposed for C++ and may have taken impact here.

[charlesroddie]

I can ask my colleague who knows the android build to look at this, but I doubt that normalization is a factor as we know that the following characters are problematic, and their normalizations are all the same:

open System
open System.Text
for str in [ "Ѫ"; "δ"; "σ" ] do
    Console.WriteLine(str)
    for nf in [ NormalizationForm.FormC; NormalizationForm.FormD; NormalizationForm.FormKC; NormalizationForm.FormKD ] do
        printfn "%A" (str.Normalize(nf) |> Seq.toArray |> Array.map int)

Output:

Ѫ
[|1130|]
[|1130|]
[|1130|]
[|1130|]
δ
[|948|]
[|948|]
[|948|]
[|948|]
σ
[|963|]
[|963|]
[|963|]
[|963|]

I also tried ensuring that of our code files (.cs, .csproj, .fs, .fsproj) were UTF-8 with BOM, and also UTF-8 without BOM, and it didn't make a difference. Unsurprisingly as .dlls are generated so it must be part of the dll -> native android step that is generating bad encodings or failing to understand good encodings.

[charlesroddie]

@FoggyFinder made a minimal repro here: https://github.com/FoggyFinder/XFAMCVE7336

[kevcrooks]

We had thought that the commit 346a933 would have solved the issue.

This commit was included in the xamarin-android release 13.1.0.0, and 13.1.0.1 which is now in Visual Studio 17.4.0.

However we still have the same problem in this latest Visual Studio version, so the underlying problem must be different.

[vilinet]

``The issue is still present even with the very latest VS 2022 Preview! And it is stopping us from using newer version of VS for projects that using libraries that trigger this issue...
I am using VS2022 17.5.0 Preview 3.0 on windows. Target is MonoAndroid11.

The AOT compiler just seems to choke up on some chars that are totally valid utf8 chars.

Like character in the PublicHoliday nuget package: ō in "public static DateTime ShōwaDay(int year)"

https://github.com/martinjw/Holiday/blob/c3a3007358d8ca4a7de5cb2dd004505728c459b6/src/PublicHoliday/JapanPublicHoliday.cs#L70

Build Log:
[aot-compiler stderr] obj\\QA-Release\\110\\aot\\arm64-v8a\\PublicHoliday.dll\\temp.s:28136:44: error: unexpected token in directive (TaskId:393) [aot-compiler stderr] .local PublicHoliday_JapanPublicHoliday_Sh+_waDay_int (TaskId:393) [aot-compiler stderr] obj\\QA-Release\\110\\aot\\arm64-v8a\\PublicHoliday.dll\\temp.s:28137:43: error: expected STT_<TYPE_IN_UPPER_CASE>, '#<type>', '@<type>', '%<type>' or "<type>" (TaskId:393) [aot-compiler stderr] .type PublicHoliday_JapanPublicHoliday_Sh+_waDay_int,@function (TaskId:393) [aot-compiler stderr] obj\\QA-Release\\110\\aot\\arm64-v8a\\PublicHoliday.dll\\temp.s:28138:36: error: invalid operand (TaskId:393) [aot-compiler stderr] PublicHoliday_JapanPublicHoliday_Sh+_waDay_int: (TaskId:393) [aot-compiler stderr] obj\\QA-Release\\110\\aot\\arm64-v8a\\PublicHoliday.dll\\temp.s:28168:43: error: unexpected token in directive (TaskId:393) [aot-compiler stderr] .size PublicHoliday_JapanPublicHoliday_Sh+_waDay_int,.-PublicHoliday_JapanPublicHoliday_Sh+_waDay_int (TaskId:393) [aot-compiler stderr] obj\\QA-Release\\110\\aot\\arm64-v8a\\PublicHoliday.dll\\temp.s:62781:48: error: unexpected token in directive (TaskId:393) [aot-compiler stderr] .local plt_PublicHoliday_JapanPublicHoliday_Sh+_waDay_int (TaskId:393) [aot-compiler stderr] obj\\QA-Release\\110\\aot\\arm64-v8a\\PublicHoliday.dll\\temp.s:62782:47: error: expected STT_<TYPE_IN_UPPER_CASE>, '#<type>', '@<type>', '%<type>' or "<type>" (TaskId:393) [aot-compiler stderr] .type plt_PublicHoliday_JapanPublicHoliday_Sh+_waDay_int,@function (TaskId:393) [aot-compiler stderr] obj\\QA-Release\\110\\aot\\arm64-v8a\\PublicHoliday.dll\\temp.s:62783:40: error: invalid operand (TaskId:393) [aot-compiler stderr] plt_PublicHoliday_JapanPublicHoliday_Sh+_waDay_int: (TaskId:393) [aot-compiler stderr] obj\\QA-Release\\110\\aot\\arm64-v8a\\PublicHoliday.dll\\temp.s:62790:47: error: unexpected token in directive (TaskId:393) [aot-compiler stderr] .size plt_PublicHoliday_JapanPublicHoliday_Sh+_waDay_int,.-plt_PublicHoliday_JapanPublicHoliday_Sh+_waDay_int (TaskId:393) [aot-compiler stderr] ld: error: cannot open obj\QA-Release\110\aot\arm64-v8a\PublicHoliday.dll\temp.s.o: no such file or directory (TaskId:393)

[charlesroddie]

@grendello @jonathanpeppers is either of you able to look at the reproduction above?

[vilinet]

I have just tested the small, reproduction solution linked above( https://github.com/FoggyFinder/XFAMCVE7336 ),
and it is still a problem in latest VS 2022 Preview also in VS 17.4.4.

[grendello]

@charlesroddie I'm unable to test this on Windows, alas, atm. The repro given by @vilinet works fine on Unix, expectedly.

@kevcrooks the switch to LLVM binutils does deal with issues in file names (on Windows), but it can't do much about code that's fed to its assembler, which is the case here.

The code that trips the AOT compilation process up is native assembler code generated by the Mono AOT compiler. Apparently, somewhere in its code it incorrectly treats UTF-8 input as ASCII, thus leading to the above error where the assembler cannot accept a + in identifier name. My guess is that the Windows environment in which the compilation takes place is not configured to use UTF-8 as the default console encoding.

@lambdageek @vargaz would you be able to look at this? LLVM built-in assembler should be able to accept UTF-8 input, but if not, perhaps some form of encoding of such identifiers in ASCII without reserved characters is necessary?

[grendello]

On Unix, the class ѪFoo name is properly encoded, replacing the UTF-8 character (not the entire byte sequence) with a single underscore character, like so:

.text 0
        .align 2
.Lm_1:
        .local EmptyXF_Droid___Foo__cctor
        .type EmptyXF_Droid___Foo__cctor,#function
EmptyXF_Droid___Foo__cctor:

This is with Xamarin.Android/main which uses mono/mono 2020-02@6dd9def57ce969ca04a0ecd9ef72c0a8f069112d