Vaex 日期时间错误未知变量或列答案

【问题标题】：Vaex datetime error unknown variables or columnVaex 日期时间错误未知变量或列
【发布时间】：2022-01-17 09:49:55
【问题描述】：

我有一个名为df 的vaex.dataframe.DataFrame 持有一个名为timestamp 的字符串类型的时间列。我将列转换为日期时间如下

import numpy as np
from pandas.api.types import is_datetime64_any_dtype as is_datetime

if not is_datetime(df['timestamp']):
    df['timestamp'] = df['timestamp'].apply(np.datetime64)

然后我只想选择df 的行，其中timestamp 在特定范围内。让我们说

sliced_df = df[(df['timestamp'] > np.datetime64("2022-01-01"))]

我在 Sagemaker 中这样做，它会抛出一个巨大的错误，主要是说以下错误消息

Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/site-packages/vaex/scopes.py", line 106, in evaluate
    result = self[expression]
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/site-packages/vaex/scopes.py", line 166, in __getitem__
    raise KeyError("Unknown variables or column: %r" % (variable,))
KeyError: "Unknown variables or column: 'datetime64(__timestamp)'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/site-packages/vaex/expression.py", line 1327, in _apply
    scalar_result = self.f(*[fix_type(k[i]) for k in args], **{key: value[i] for key, value in kwargs.items()})
ValueError: Error parsing datetime string "nan" at position 0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/site-packages/vaex/scopes.py", line 265, in __getitem__
    values = self.evaluate(expression)  # , out=self.buffers[variable])
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/site-packages/vaex/scopes.py", line 188, in evaluate
    result = eval(expression, expression_namespace, self)
  File "<string>", line 1, in <module>
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/site-packages/vaex/arrow/numpy_dispatch.py", line 136, in wrapper
    result = f(*args, **kwargs)
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/site-packages/vaex/expression.py", line 1312, in __call__
    return vaex.multiprocessing.apply(self._apply, args, kwargs, self.multiprocessing)
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/site-packages/vaex/multiprocessing.py", line 32, in apply
    result = _get_pool().apply(f, args, kwargs)
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/multiprocessing/pool.py", line 261, in apply
    return self.apply_async(func, args, kwds).get()
  File "/home/ec2-user/SageMaker/custom-miniconda/miniconda/envs/custom_python/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
ValueError: Error parsing datetime string "nan" at position 0
ERROR:MainThread:vaex.scopes:error in evaluating: 'timestamp'
"""

df 在timestamp 列下保存与这些类似的值

<pyarrow.lib.StringArray object at 0x7f569e5f54b0>
[
  "2021-12-19 06:01:10.789",
  "2021-12-20 07:02:11.89",
  "2022-01-01 08:02:12.678",
  "2022-01-02 09:03:13.567",
  "2022-01-03 10:04:14.456"
]

时间戳在我看来很好。我与之前比较有效的数据进行了比较，似乎没有什么不同。我不知道为什么现在这不再起作用了。几天来我一直在努力解决它，但真的找不到它抛出该错误的原因。

当我检查时

df[df.timestamp.isna()]

它什么也不返回。所以我不明白为什么它发现nan 在上面的错误消息中所述的第一个位置。

感谢您的帮助。提前致谢！

【问题讨论】：

标签： python dataframe numpy time vaex

【解决方案1】：

这可能是因为您将箭头时间戳与 numpy 时间戳进行比较。您需要选择一个框架并使用它。

vaex 的 github 上的这个问题讨论了你所面临的问题，所以它可能会更清楚一些事情： https://github.com/vaexio/vaex/issues/1704

【讨论】：