带有缺失值的 Pandas 转换为 Int64答案

【问题标题】：Pandas conversion to Int64 with missing values带有缺失值的 Pandas 转换为 Int64
【发布时间】：2021-01-02 15:14:46
【问题描述】：

请注意，我使用的是 panda 1.1.2 和 numpy 1.19.2

非工作场景

我有一个provider_frame['NEQ'] 系列，在数值中包含pd.NA 数据。系列的类型是object。

在阅读pandas documentation 关于 Int64 时，我知道我们应该使用 pandas.NA 来处理缺失值。在确保我的系列包含 pd.NA 或数字后，我正在尝试以下操作：


$ provider_frame['NEQ'] = provider_frame['NEQ'].astype('Int64')

output :
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
----> 1 provider_frame['NEQ'] = provider_frame['NEQ'].astype('Int64')

/usr/local/lib/python3.8/site-packages/pandas/core/generic.py in astype(self, dtype, copy, errors)
   5541         else:
   5542             # else, only a single dtype is given
-> 5543             new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
   5544             return self._constructor(new_data).__finalize__(self, method="astype")
   5545 

/usr/local/lib/python3.8/site-packages/pandas/core/internals/managers.py in astype(self, dtype, copy, errors)
    593         self, dtype, copy: bool = False, errors: str = "raise"
    594     ) -> "BlockManager":
--> 595         return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
    596 
    597     def convert(

/usr/local/lib/python3.8/site-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, **kwargs)
    404                 applied = b.apply(f, **kwargs)
    405             else:
--> 406                 applied = getattr(b, f)(**kwargs)
    407             result_blocks = _extend_blocks(applied, result_blocks)
    408 

/usr/local/lib/python3.8/site-packages/pandas/core/internals/blocks.py in astype(self, dtype, copy, errors)
    592             vals1d = values.ravel()
    593             try:
--> 594                 values = astype_nansafe(vals1d, dtype, copy=True)
    595             except (ValueError, TypeError):
    596                 # e.g. astype_nansafe can fail on object-dtype of strings

/usr/local/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in astype_nansafe(arr, dtype, copy, skipna)
    912     # dispatch on extension dtype if needed
    913     if is_extension_array_dtype(dtype):
--> 914         return dtype.construct_array_type()._from_sequence(arr, dtype=dtype, copy=copy)
    915 
    916     if not isinstance(dtype, np.dtype):

/usr/local/lib/python3.8/site-packages/pandas/core/arrays/integer.py in _from_sequence(cls, scalars, dtype, copy)
    367     @classmethod
    368     def _from_sequence(cls, scalars, dtype=None, copy: bool = False) -> "IntegerArray":
--> 369         return integer_array(scalars, dtype=dtype, copy=copy)
    370 
    371     @classmethod

/usr/local/lib/python3.8/site-packages/pandas/core/arrays/integer.py in integer_array(values, dtype, copy)
    158     TypeError if incompatible types
    159     """
--> 160     values, mask = coerce_to_array(values, dtype=dtype, copy=copy)
    161     return IntegerArray(values, mask)
    162 

/usr/local/lib/python3.8/site-packages/pandas/core/arrays/integer.py in coerce_to_array(values, dtype, mask, copy)
    242             "mixed-integer-float",
    243         ]:
--> 244             raise TypeError(f"{values.dtype} cannot be converted to an IntegerDtype")
    245 
    246     elif is_bool_dtype(values) and is_integer_dtype(dtype):

TypeError: object cannot be converted to an IntegerDtype

从post 了解到，由于 pandas 的问题，我们应该进行两次转换，首先是浮点数，然后是 Int64。

尝试转换时，我有以下回溯：

$ provider_frame['NEQ'] = provider_frame['NEQ'].astype('float')

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
----> 1 provider_frame['NEQ'] = provider_frame['NEQ'].astype('float')

/usr/local/lib/python3.8/site-packages/pandas/core/generic.py in astype(self, dtype, copy, errors)
   5541         else:
   5542             # else, only a single dtype is given
-> 5543             new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
   5544             return self._constructor(new_data).__finalize__(self, method="astype")
   5545 

/usr/local/lib/python3.8/site-packages/pandas/core/internals/managers.py in astype(self, dtype, copy, errors)
    593         self, dtype, copy: bool = False, errors: str = "raise"
    594     ) -> "BlockManager":
--> 595         return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
    596 
    597     def convert(

/usr/local/lib/python3.8/site-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, **kwargs)
    404                 applied = b.apply(f, **kwargs)
    405             else:
--> 406                 applied = getattr(b, f)(**kwargs)
    407             result_blocks = _extend_blocks(applied, result_blocks)
    408 

/usr/local/lib/python3.8/site-packages/pandas/core/internals/blocks.py in astype(self, dtype, copy, errors)
    592             vals1d = values.ravel()
    593             try:
--> 594                 values = astype_nansafe(vals1d, dtype, copy=True)
    595             except (ValueError, TypeError):
    596                 # e.g. astype_nansafe can fail on object-dtype of strings

/usr/local/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in astype_nansafe(arr, dtype, copy, skipna)
    988     if copy or is_object_dtype(arr) or is_object_dtype(dtype):
    989         # Explicit copy, or required since NumPy can't view from / to object.
--> 990         return arr.astype(dtype, copy=True)
    991 
    992     return arr.view(dtype)

TypeError: float() argument must be a string or a number, not 'NAType'

我了解 float 类型不喜欢 pd.NA 类型。

工作场景

现在，我将使用provider_frame.loc[provider_frame['NEQ'].isna() == True, 'NEQ']=np.NaN 将我所有的pd.NA 行转换为np.nan

然后，我将执行以下代码：

provider_frame['NEQ'] = provider_frame['NEQ'].astype('float')
provider_frame['NEQ'] = provider_frame['NEQ'].astype('Int64')

And my conversion will work successfully. If I had tried directly to doprovider_frame['NEQ'] = provider_frame['NEQ'].astype('Int64')` 没有浮点步骤，输出为：

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
----> 1 provider_frame['NEQ'] = provider_frame['NEQ'].astype('Int64')

/usr/local/lib/python3.8/site-packages/pandas/core/generic.py in astype(self, dtype, copy, errors)
   5541         else:
   5542             # else, only a single dtype is given
-> 5543             new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors,)
   5544             return self._constructor(new_data).__finalize__(self, method="astype")
   5545 

/usr/local/lib/python3.8/site-packages/pandas/core/internals/managers.py in astype(self, dtype, copy, errors)
    593         self, dtype, copy: bool = False, errors: str = "raise"
    594     ) -> "BlockManager":
--> 595         return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
    596 
    597     def convert(

/usr/local/lib/python3.8/site-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, **kwargs)
    404                 applied = b.apply(f, **kwargs)
    405             else:
--> 406                 applied = getattr(b, f)(**kwargs)
    407             result_blocks = _extend_blocks(applied, result_blocks)
    408 

/usr/local/lib/python3.8/site-packages/pandas/core/internals/blocks.py in astype(self, dtype, copy, errors)
    592             vals1d = values.ravel()
    593             try:
--> 594                 values = astype_nansafe(vals1d, dtype, copy=True)
    595             except (ValueError, TypeError):
    596                 # e.g. astype_nansafe can fail on object-dtype of strings

/usr/local/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in astype_nansafe(arr, dtype, copy, skipna)
    912     # dispatch on extension dtype if needed
    913     if is_extension_array_dtype(dtype):
--> 914         return dtype.construct_array_type()._from_sequence(arr, dtype=dtype, copy=copy)
    915 
    916     if not isinstance(dtype, np.dtype):

/usr/local/lib/python3.8/site-packages/pandas/core/arrays/integer.py in _from_sequence(cls, scalars, dtype, copy)
    367     @classmethod
    368     def _from_sequence(cls, scalars, dtype=None, copy: bool = False) -> "IntegerArray":
--> 369         return integer_array(scalars, dtype=dtype, copy=copy)
    370 
    371     @classmethod

/usr/local/lib/python3.8/site-packages/pandas/core/arrays/integer.py in integer_array(values, dtype, copy)
    158     TypeError if incompatible types
    159     """
--> 160     values, mask = coerce_to_array(values, dtype=dtype, copy=copy)
    161     return IntegerArray(values, mask)
    162 

/usr/local/lib/python3.8/site-packages/pandas/core/arrays/integer.py in coerce_to_array(values, dtype, mask, copy)
    242             "mixed-integer-float",
    243         ]:
--> 244             raise TypeError(f"{values.dtype} cannot be converted to an IntegerDtype")
    245 
    246     elif is_bool_dtype(values) and is_integer_dtype(dtype):

TypeError: object cannot be converted to an IntegerDtype

问题

转换过程显然有一些我不明白的地方。将包含 pd.NA 行的系列转换为 Int64 类型的最佳方法是什么？ pd.NA 的目的不就是为了方便地转换成 Int64 类型吗？

【问题讨论】：

标签： pandas type-conversion missing-data

【解决方案1】：

这是一种将数字的字符串版本转换为 Int64 的方法：

import pandas as pd

df = pd.DataFrame({'x': ['10', '20', None, '40']})  # list of strings + None
df['x'] = pd.to_numeric(df['x'], downcast='float', errors='raise').astype('Int64')

print(df)
      x
0    10
1    20
2  <NA>
3    40

您也许可以消除to_numeric() 中的 downcast 和/或错误参数，但这个版本可以工作（pandas 版本 1.1.0）。

【讨论】：