[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[jira] [Created] (ARROW-4131) [Python] Coerce mixed columns to String

Leo Meyerovich created ARROW-4131:

             Summary: [Python] Coerce mixed columns to String
                 Key: ARROW-4131
             Project: Apache Arrow
          Issue Type: Improvement
            Reporter: Leo Meyerovich

Continuing [] ;




I'm seeing variants of this elsewhere (e.g., [wesm/feather#349|] ;) --

Not all Pandas tables coerce to Arrow tables, and when they fail, not in a way that is conducive to automation:


{{mixed_df = pd.DataFrame(\{'mixed': [1, 'b']}) pa.Table.from_pandas(mixed_df) => ArrowInvalid: ('Could not convert b with type str: tried to convert to double', 'Conversion failed for column mixed with type object') }}

I would have expected behaviors more like the following:
 * Coerce {{toString}} by default, with a default-off option to disallow toString coercions

 * Provide a default-off option to {{from_pandas}} to auto-coerce

 * Name the exception so it is clear that this is a column coercion failure, and include the column name(s), making this predictable and clearly handleable by both library writers & users

I lean towards:
 * Defaults auto-coerce, improving life of early users, `coerce_mixed_columns_to_strings=True`
 * For less frequent yet more advanced library implementors, allow them to override to `False`
 * In their case, create a predictable & machine-readable exception, `MixedColumnException(mixed_columns=['a', 'b', ...], msg="....")`

This message was sent by Atlassian JIRA