胶水python shell作业无法导入s3fs

Question

我是 Glue 工作的新手，并按照以下链接配置 whl 文件的方式

Import failure of s3fs library in AWS Glue

AWS Glue Python - 3 作业出现以下错误

WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fdac61a8080>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/fsspec/

ERROR: Could not find a version that satisfies the requirement fsspec==2021.07.0 (from s3fs==2021.7.0) (from versions: none)

ERROR: No matching distribution found for fsspec==2021.07.0 (from s3fs==2021.7.0)

Traceback (most recent call last):
  File "/tmp/runscript.py", line 207, in <module>
    download_and_install(args.extra_py_files, timeout=time_out)
  File "/tmp/runscript.py", line 141, in download_and_install
    install_package(file_extension, install_path, local_file_path, timeout=timeout)
  File "/tmp/runscript.py", line 115, in install_package
    install_wheel_with_retry(install_path, local_file_path, timeout=timeout)
  File "/tmp/runscript.py", line 108, in install_wheel_with_retry
    retry(run_check_call, install_path, local_file_path, timeout, exceptions=TimeoutExpired, message="{} installation".format(local_file_path))
  File "/tmp/runscript.py", line 35, in retry
    return func(*func_args)
  File "/tmp/runscript.py", line 106, in run_check_call
    check_call([sys.executable, "-m", "pip", "install", "--target={}".format(_install_path), _local_file_path], timeout=_timeout)
  File "/usr/local/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--target=/glue/lib/installation', '/tmp/glue-python-libs-lw4p0gdn/s3fs-2021.7.0-py3-none-any.whl']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/tmp/runscript.py", line 230, in <module>
    raise e_type(e_value).with_traceback(new_stack)
TypeError: __init__() missing 1 required positional argument: 'cmd'

Answer 1

我认为您只是在尝试安装的依赖项中有错字。

比较：

2021.07.0
2021.7.0

所以更新你要安装的s3fs的版本。

见：https://pypi.org/project/s3fs/