RecordBatch normalization (flattening)
#6758
Conversation
ngli-me
changed the title
ngli-me
changed the title
RecordBatch normalization (flattening)
… iterative function for `RecordBatch`. Not sure which one is better currently.
There was a problem hiding this comment.
I had some questions regarding the implementation of this, since the one example from PyArrow doesn't seem to clarify on the edge cases here. Normalizing the Schema seems fairly straight forward to me, I'm just not sure on
- Whether the iterative or recursive approach is better (or something I missed)
- If
DataType::Structis the onlyDataTypethat requires flattening. To me, it looks like that's the only one that can contained nestedFields.
(I'm also not sure if I'm missing something with unwrapping like a List<Struct>)
Any feedback/help would be appreciated!
| let field_name = field.name().as_str(); | ||
| new_fields = [ | ||
| new_fields, | ||
| Self::normalizer( |
There was a problem hiding this comment.
Not sure if it's better to have it be recursive or iterative.
| (c.clone(), Arc::new(c_field.clone()) as ArrayRef), | ||
| ])); | ||
|
|
||
| /*let exclamation_field = Arc::new(StructArray::from(vec![ |
There was a problem hiding this comment.
It looks like this should work to me, based on the schema, but I get an error when trying to construct this Field, not sure what I'm missing here.
error[E0277]: the trait boundstd::sync::Arc<struct_array::StructArray>: array::Arrayis not satisfied --> arrow-array/src/record_batch.rs:1353:27 | 1353 | (one.clone(), Arc::new(one_field.clone()) as ArrayRef), | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ the traitarray::Arrayis not implemented forstd::sync::Arc<struct_array::StructArray>| = help: the traitarray::Arrayis implemented forstd::sync::Arc<(dyn array::Array + 'static)>= note: required for the cast fromstd::sync::Arc<std::sync::Arc<struct_array::StructArray>>tostd::sync::Arc<(dyn array::Array + 'static)>
Which issue does this PR close?
Closes #6369.
Rationale for this change
Adds normalization (flattening) for
RecordBatch, with normalization viaSchema. Based on pandas/pola-rs.What changes are included in this PR?
Are there any user-facing changes?