paramiko 在下载 20 MB 文件后挂起答案

【问题标题】：paramiko hangs on get after ownloading 20 MB of fileparamiko 在下载 20 MB 文件后挂起
【发布时间】：2018-02-04 02:28:48
【问题描述】：

我需要 python sftp 客户端从 sftp 服务器下载文件。我开始使用 Paramiko。 KB 中的小文件效果很好，但是当我尝试下载 600 MB 的文件时，它在下载 20 MB 的文件后无限期挂起。无法弄清楚问题是什么。增加窗口大小也没有解决。任何帮助将不胜感激！

host = config.getsafe(section, "host")
username = config.getsafe(section, "username")
port = config.getsafe(section, "port")
remote_dir = config.getsafe(section, "remote_dir")
download_dir = config.getsafe(section, "download_dir")
archive_dir = config.getsafe(section, "archive_dir") if config.has_option(section, "archive_dir") else \
    None
password = config.getsafe(section, "password") if config.has_option(section, "password") else None
file_pattern = config.getsafe(section, "file_pattern") if config.has_option(section, "file_pattern") \
    else "*"
passphrase = config.getsafe(section, "passphrase") if config.has_option(section, "passphrase") else None
gnupg_home = config.getsafe(section, "gnupg_home") if config.has_option(section, "gnupg_home") else None

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=host, port=int(port), username=username, password=password)

sftp = ssh.open_sftp()
sftp.sshclient = ssh

sftp.get("/SFTP/PL_DEV/test.dat", "C:/import/download/test.dat")

【问题讨论】：

执行数据包捕获（使用 Wireshark）以准确了解发生了什么。还可以尝试使用独立的 SFTP 客户端下载相同的文件，看看是否可行。
像 filezilla 这样的独立 sftp 客户端完美运行

标签： python-3.x ssh paramiko

【解决方案1】：

一段时间以来，我一直在努力解决这个问题，在提取了大约四个不同的建议并将它们混合在一起之后，这是我必须为我工作的方法：

首先（连接到您的 sftp 并遍历与您的请求匹配的文件列表）：

def getzipfiles(directory):
    # configuration file collection used to build my 
    # custom classlib.dataconnection json files
    configfilename = [fname for fname in configfiles if 'verifty_get' in fname]
    sftp_get = Configs.get_sftp_settings(configfilename[0])

    print("got here")
    try:
        cnopts = pysftp.CnOpts()
        cnopts.hostkeys = None #debug in dev (set your hostkeys!!!)
        sftpconn_get = pysftp.Connection(sftp_get.hostname,
                                         username=sftp_get.username,
                                         password=sftp_get.password,
                                         cnopts=cnopts)

        filelist = sftpconn_get.listdir()
        sftpconn_get.close()

        for filename in filelist:
            matchval = re.search(r'D*********_(?P<date>\d{8})_(?P<time>(\d{2}-?){3}.\d{1,8}).zip', filename, re.I)
            if matchval:
                getlargezipfiles(directory, filename)

    except:
        e = sys.exc_info()
        sftp_exception = e
        print("SFTP listdir failed, exception: {}".format(e))

第二个（传入你要保存文件的目录和文件名）

def getlargezipfiles(directory, filename):
    configfilename = [fname for fname in configfiles if 'verifty_get' in fname]
    sftp_get = Configs.get_sftp_settings(configfilename[0])

    MAX_RETRIES = 2

    port = 22
    sftp_file = filename
    local_file = "{}{}".format(directory,filename)
    ssh_conn = sftp_client = None
    start_time = time.time()

    for retry in range(MAX_RETRIES):
        try:
            ssh_conn = paramiko.Transport((sftp_get.hostname, port))
            ssh_conn.packetizer.REKEY_BYTES = pow(2, 40)  # 1TB max, this is a security degradation!
            ssh_conn.packetizer.REKEY_PACKETS = pow(2, 40)  # 1TB max, this is a security degradation!
            ssh_conn.default_window_size = paramiko.common.MAX_WINDOW_SIZE
            ssh_conn.connect(username=sftp_get.username, password=sftp_get.password)
            sftp_client = paramiko.SFTPClient.from_transport(ssh_conn)
            filesize = sftp_client.stat(sftp_file).st_size
            sftp_client.get_channel().in_window_size = 2097152
            sftp_client.get_channel().out_window_size = 2097152
            sftp_client.get_channel().in_max_packet_size = 2097152
            sftp_client.get_channel().out_max_packet_size = 2097152

            print("Getting {} size [{}] at {}".format(sftp_file, filesize, datetime.now()))
            sftp_client.get(sftp_file, local_file)
            
            break
        except (EOFError, paramiko.ssh_exception.SSHException, OSError) as x:
            retry += 1
            print("%s %s - > retrying %s..." % (type(x), x, retry))
            time.sleep(abs(retry) * 10)
            # back off in steps of 10, 20.. seconds
        finally:
            if hasattr(sftp_client, "close") and callable(sftp_client.close):
                sftp_client.close()
            if hasattr(ssh_conn, "close") and callable(ssh_conn.close):
                ssh_conn.close()

    print("Loading File %s Took %d seconds " % (sftp_file, time.time() - start_time))

【讨论】：

【解决方案2】：

使用最新的 paramiko 2.4.2，我遇到了类似的问题。就我而言，我们的供应商几天前将他们的 SFTP 提供商从 Globalscape (SSH-2.0-1.82_sshlib Globalscape) 切换到了 Cerberus (SSH-2.0-CerberusFTPServer_10.0)。从那以后，paramiko 一直无法下载大约 450MB 的文件。

症状如下：下载速度极慢。下载 20~30MB 后，总是报错： Server connection dropped msg.

这里是日志 (Globalscape) - 成功下载：

"paramiko.transport", "DEBUG", "starting thread (client mode): 0x160096d8"
"paramiko.transport", "DEBUG", "Local version/idstring: SSH-2.0-paramiko_2.4.1"
"paramiko.transport", "DEBUG", "Remote version/idstring: SSH-2.0-1.82_sshlib Globalscape"
"paramiko.transport", "INFO", "Connected (version 2.0, client 1.82_sshlib)"
"paramiko.transport", "DEBUG", "kex algos:['diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa'] client encrypt:['twofish256-cbc', 'twofish-cbc', 'twofish128-cbc', 'blowfish-cbc', '3des-cbc', 'arcfour', 'cast128-cbc', 'aes256-cbc', 'aes128-cbc', 'aes256-ctr', 'aes128-ctr'] server encrypt:['twofish256-cbc', 'twofish-cbc', 'twofish128-cbc', 'blowfish-cbc', '3des-cbc', 'arcfour', 'cast128-cbc', 'aes256-cbc', 'aes128-cbc', 'aes256-ctr', 'aes128-ctr'] client mac:['hmac-sha1', 'hmac-md5', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-sha1', 'hmac-md5', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['zlib', 'none'] server compress:['zlib', 'none'] client lang:[''] server lang:[''] kex follows?False"
"paramiko.transport", "DEBUG", "HostKey agreed: ssh-rsa"
"paramiko.transport", "DEBUG", "Cipher agreed: aes128-ctr"
"paramiko.transport", "DEBUG", "MAC agreed: hmac-sha1"
"paramiko.transport", "DEBUG", "Compression agreed: none"
"paramiko.transport", "DEBUG", "Got server p (2048 bits)"
"paramiko.transport", "DEBUG", "kex engine KexGex specified hash_algo <built-in function openssl_sha1>"
"paramiko.transport", "DEBUG", "Switch to new keys ..."
"paramiko.transport", "DEBUG", "Attempting public-key auth..."
"paramiko.transport", "DEBUG", "userauth is OK"
"paramiko.transport", "INFO", "Auth banner: b'Welcome to the our Secure FTP Server'"
"paramiko.transport", "INFO", "Authentication (publickey) successful!"
"paramiko.transport", "DEBUG", "[chan 0] Max packet in: 32768 bytes"
"paramiko.transport", "DEBUG", "[chan 0] Max packet out: 35840 bytes"
"paramiko.transport", "DEBUG", "Secsh channel 0 opened."
"paramiko.transport", "DEBUG", "[chan 0] Sesch channel 0 request ok"
"paramiko.transport.sftp", "INFO", "[chan 0] Opened sftp connection (server version 3)"
"paramiko.transport.sftp", "DEBUG", "[chan 0] stat(b'data.csv')"
"paramiko.transport.sftp", "DEBUG", "[chan 0] open(b'data.csv', 'rb')"
"paramiko.transport.sftp", "DEBUG", "[chan 0] open(b'data.csv', 'rb') -> 31"
"paramiko.transport.sftp", "DEBUG", "[chan 0] close(31)"
"paramiko.transport.sftp", "INFO", "[chan 0] sftp session closed."
"paramiko.transport", "DEBUG", "[chan 0] EOF sent (0)"
"paramiko.transport", "DEBUG", "EOF in transport thread"

这是日志 (Cerberus) - 下载失败：

"paramiko.transport", "DEBUG", "starting thread (client mode): 0x119706d8"
"paramiko.transport", "DEBUG", "Local version/idstring: SSH-2.0-paramiko_2.4.1"
"paramiko.transport", "DEBUG", "Remote version/idstring: SSH-2.0-CerberusFTPServer_10.0"
"paramiko.transport", "INFO", "Connected (version 2.0, client CerberusFTPServer_10.0)"
"paramiko.transport", "DEBUG", "kex algos:['ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa'] client encrypt:['aes128-ctr', 'aes128-cbc', 'aes192-ctr', 'aes192-cbc', 'aes256-ctr', 'aes256-cbc', '3des-cbc'] server encrypt:['aes128-ctr', 'aes128-cbc', 'aes192-ctr', 'aes192-cbc', 'aes256-ctr', 'aes256-cbc', '3des-cbc'] client mac:['hmac-sha1', 'hmac-sha1-96', 'hmac-sha2-256', 'hmac-sha2-256-96', 'hmac-sha2-512', 'hmac-sha2-512-96', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-md5'] server mac:['hmac-sha1', 'hmac-sha1-96', 'hmac-sha2-256', 'hmac-sha2-256-96', 'hmac-sha2-512', 'hmac-sha2-512-96', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-md5'] client compress:['none'] server compress:['none'] client lang:['en-US'] server lang:['en-US'] kex follows?False"
"paramiko.transport", "DEBUG", "Kex agreed: ecdh-sha2-nistp256"
"paramiko.transport", "DEBUG", "HostKey agreed: ssh-rsa"
"paramiko.transport", "DEBUG", "Cipher agreed: aes128-ctr"
"paramiko.transport", "DEBUG", "MAC agreed: hmac-sha2-256"
"paramiko.transport", "DEBUG", "Compression agreed: none"
"paramiko.transport", "DEBUG", "kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>"
"paramiko.transport", "DEBUG", "Switch to new keys ..."
"paramiko.transport", "DEBUG", "Attempting public-key auth..."
"paramiko.transport", "DEBUG", "userauth is OK"
"paramiko.transport", "INFO", "Authentication (publickey) successful!"
"paramiko.transport", "DEBUG", "[chan 0] Max packet in: 32768 bytes"
"paramiko.transport", "DEBUG", "[chan 0] Max packet out: 32768 bytes"
"paramiko.transport", "DEBUG", "Secsh channel 0 opened."
"paramiko.transport", "DEBUG", "[chan 0] Sesch channel 0 request ok"
"paramiko.transport.sftp", "INFO", "[chan 0] Opened sftp connection (server version 3)"
"paramiko.transport.sftp", "DEBUG", "[chan 0] stat(b'data.csv')"
"paramiko.transport.sftp", "DEBUG", "[chan 0] open(b'data.csv', 'rb')"
"paramiko.transport.sftp", "DEBUG", "[chan 0] open(b'data.csv', 'rb') -> 7b45394343333830462d383832352d343436342d393831302d4444373838314237303433367d"
"paramiko.transport", "DEBUG", "EOF in transport thread"

添加

transport.default_window_size = paramiko.common.MAX_WINDOW_SIZE

对我有用（至少现在）。不确定如果文件大小从 ~450MB 增加到 >>0.5GB 会发生什么。

【讨论】：

【解决方案3】：

如下增加 default_max_packet_size 和 default_window_size 对我有用：

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.load_system_host_keys()

client.connect(hostname, username=username, password=password, port=port)
tr = client.get_transport()
tr.default_max_packet_size = 100000000
tr.default_window_size = 100000000
sftp = client.open_sftp()
sftp.get(remote_file, local_filepath)

client.close()

【讨论】：

【解决方案4】：

我做了两件事来解决类似的问题：

increase window size - 你说你也试过了；对我来说，这有助于从几十 MB 到半 GB，但没有更多

effectively disable rekeying - 这可能有安全隐患，但帮助我从奇怪的 Windows sftp 服务器获取超过 GB 的文件

with paramiko.Transport((_SFTP['host'], 22)) as transport:
    # SFTP FIXES
    transport.default_window_size=paramiko.common.MAX_WINDOW_SIZE
    transport.packetizer.REKEY_BYTES = pow(2, 40)  # 1TB max, this is a security degradation!
    transport.packetizer.REKEY_PACKETS = pow(2, 40)  # 1TB max, this is a security degradation!
    # / SFTP FIXES

    transport.connect(username=_SFTP['user'], password=_SFTP['password'])
        with paramiko.SFTPClient.from_transport(transport) as sftp:
            listdir = sftp.listdir()
            # ...
            sftp.get(remotepath=filename, localpath=localpath)

【讨论】：

让我补充一点，在 ~2GB 之后，使用这种方法传输仍然会变慢:(
你能解释一下# 1TB max, this is a security degradation!吗？这是什么意思？
@YuChen SSH 应该在传输一定数量的数据（默认值：1GB）后重做密钥交换机制（即同意新的加密密钥）以使连接更加安全。这种变通方法大大提高了限制，以至于它有效地禁用了密钥更新，即整个传输将使用客户端和服务器之间传输开始时商定的相同加密密钥。一些旧服务器不支持重新加密，这就是为什么需要这种解决方法。
更多解释见security.stackexchange.com/questions/74634/…