[Python] Improve Extension Types Support in PyArrow (umbrella issue)

[AlenkaF]

Describe the enhancement requested

This is an umbrella issue tracking open bugs and improvements related to extension types support in PyArrow.

1. Extension types in nested arrays (struct/list):

• [Python] Python extension types aren't usable in struct arrays #34985
• [C++][Python] Casting extension types with list-based storage segfaults #37004

2. Compute support:

• [C++][Python] Support ExtensionType arrays in more kernels #22304
• [C++][Python] Allow an ExtensionType to register or implement custom casts #33103

3. Parquet/IPC (I/O) interoperability:

• [Python][Parquet] read_schema drops extension types (UUID returned as fixed_size_binary[16]) #48254
• [C++][Python] IPC serialization roundtrip of extension type with storage type pa.null() reconstructs invalid buffers #15068
• [C++][Parquet] Writing DictionaryArrays with ExtensionType to Parquet #30080, related though closed as stale issues: [C++][Python] Support for pandas Categoricals with Intervals #30119 and [C++][Parquet] Support direct dictionary decoding of types other than BYTE_ARRAY #22534

Stale but relevant:

• [C++] ExtensionType subclass for "unknown" types? #22572

4. Pandas and other integrations:

• [Python] Use ExtensionScalar.as_py() as fallback in ExtensionArray to_pandas? #33134
• [Python] FixedShapedTensor to_pandas_dtype is returning NotImplementedError #49907
• [Python] List<Extension> arrays aren't supported in to_pandas calls #32791

5. Internal Python and API improvements:

• [Python] Clean up ExtensionType.__reduce__ #39094
• [C++] Provide a default implementation for ExtensionType::ExtensionEquals #24348

6. FixedShapeTensor specifics:

• [Python] Convert multidimensional arrays into FixedShapeTensorArrays automatically in pyarrow.array #35647

7. Docs and warnings:

• [Docs] Warn against execution of arbitrary code in use of extension types #39007

If any additional cases related to extension types are not covered here, feel free to add them to the list.

Component(s)

Python

[AlenkaF]

cc @raulcd @rok @pitrou @jorisvandenbossche