Describe the bug
Standardization keeps metadata of root fields, but does not preserve metadata of nested fields.
To Reproduce
Run standardization of a dataframe with a following schema:
{
"type": "struct",
"fields": [
{
"name": "STRING_FIELD",
"type": "string",
"nullable": true,
"metadata": {
"maxLength": 8
}
},
{
"name": "NESTED_STRUCT",
"type": {
"type": "struct",
"fields": [
{
"name": "NESTED_STRING",
"type": "string",
"nullable": true,
"metadata": {
"maxLength": 5
}
}
]
}
}
]
}
The result will look like this (nested field's metadata is lost):
{
"type": "struct",
"fields": [
{
"name": "STRING_FIELD",
"type": "string",
"nullable": true,
"metadata": {
"maxLength": 8
}
},
{
"name": "NESTED_STRUCT",
"type": {
"type": "struct",
"fields": [
{
"name": "NESTED_STRING",
"type": "string",
"nullable": true,
"metadata": { }
}
]
}
}
]
}
STRING_FIELD has metadata retained.
NESTED_STRING has list metadata field "maxLength": 5
Expected behavior
Standardization should preserve all metadata.
Screenshots
--
Desktop (please complete the following information):
--
Additional context
--
Describe the bug
Standardization keeps metadata of root fields, but does not preserve metadata of nested fields.
To Reproduce
Run standardization of a dataframe with a following schema:
{ "type": "struct", "fields": [ { "name": "STRING_FIELD", "type": "string", "nullable": true, "metadata": { "maxLength": 8 } }, { "name": "NESTED_STRUCT", "type": { "type": "struct", "fields": [ { "name": "NESTED_STRING", "type": "string", "nullable": true, "metadata": { "maxLength": 5 } } ] } } ] }The result will look like this (nested field's metadata is lost):
{ "type": "struct", "fields": [ { "name": "STRING_FIELD", "type": "string", "nullable": true, "metadata": { "maxLength": 8 } }, { "name": "NESTED_STRUCT", "type": { "type": "struct", "fields": [ { "name": "NESTED_STRING", "type": "string", "nullable": true, "metadata": { } } ] } } ] }STRING_FIELDhas metadata retained.NESTED_STRINGhas list metadata field"maxLength": 5Expected behavior
Standardization should preserve all metadata.
Screenshots
--
Desktop (please complete the following information):
--
Additional context
--