使用 Python ftplib 限制 FTP 下载答案

【问题标题】：Throttling FTP download with Python ftplib使用 Python ftplib 限制 FTP 下载
【发布时间】：2017-03-20 20:01:50
【问题描述】：

如何使用 Python ftplib 限制 FTP 下载？例如将速度限制为 20Mb/s？

我正在使用以下代码通过 Python ftplib 下载文件：

from ftplib import FTP
import os

download_list = 'testlist.txt' # inital list of directories to be downloaded
path_list = [] # initalize a list of all the pathes from download_list
local_folder = 'testStorage' #where files are going to be downloaded to
downloaded_list = 'completedownload.txt' # list of completed downloads
error_list = 'incomplete_downloads.txt' # list of paths that are incomplete


ftp=FTP("ftp.address.com")
ftp.login("user_name","password") #login to FTP account
print "Successfully logged in"

# make a list of files to download from a file
with open(download_list, 'r') as f:
    content = f.readlines()
    path_list = [x.strip() for x in content]

for path in path_list:
    path = path.replace("*","") # strips the * found in the source file
    print '\nChanging directory to ' + path + ':\n'
    #ftp.cwd('/AAA/BBB/CCC/logic-1/')    #the format to change into path note the * is omitted
    #if ftp.cwd(path) == True:


    try:    # tries the path in the file
        ftp.cwd(path)
        #ftp.retrlines('LIST')
        filenames = ftp.nlst()
        for filename in filenames:
            local_directory = local_folder+path # create the local path ie : testStorage/AAA/BBB/CCC/logic-1/
            local_filename = os.path.join(local_directory,filename) #

            if os.path.exists(local_filename) == False: # checks if file already exists
                if not os.path.exists(local_directory): # mimic the remote path locally
                    os.makedirs(local_directory)

                file = open(local_filename,'wb')
                ftp.retrbinary('RETR '+ filename, file.write)
                print filename
                file.close()

            elif os.path.exists(local_filename) == True: # skip the file if it exits
                print 'File ' +filename + ' already exists, skipping this file'

    except: #if path in text file does not exist write to error_list.txt

        print 'Path ' + path + ' does not exist writing path to error_list.txt'
        with open(error_list, 'a') as f2:
            f2.write(path+'\n')
        continue

print "all done closing connection"
ftp.close()  #CLOSE THE FTP CONNECTION

【问题讨论】：

标签： python ftp ftplib

【解决方案1】：

要限制下载，只需根据需要实现一个执行file.write 和time.sleep 的函数。将该函数作为callback（而不是直接file.write）传递给ftp.retrbinary。

这个伪代码（我不做 Python）应该会给你一些想法：

total_length = 0
start_time = time.time()

def write_and_sleep(buf):
    global file
    global total_length
    global start_time
    file.write(buf)
    total_length += sys.getsizeof(buf)
    while (total_length / (time.time() - start_time)) > 100000000:
        time.sleep(0.1)

ftp.retrbinary('RETR '+ filename, write_and_sleep)

减少maxblocksize（ftp.retrbinary 的第三个参数）可能有助于实现更平滑的“下载曲线”。

【讨论】：