从目录批量导入文件到python脚本

Question

我想将特定目录中的所有 jpg 图像导入我的 python 脚本，但不是一次全部导入，而是每次导入 500 张图像。

一种可能的解决方案如下：

from glob import glob

i = 0
batch= 500
# Read images from file
for filename in glob('Directory/*.jpg'):
    i = i + 1
    if i % batch == 0:
        # apply an algorithm with this data-batch #

这对吗？

有没有更有效的方法来做到这一点？

Answer 1

from os import listdir

directory = 'Directory/*.jpg'

fnames = list(fname for fname in listdir(directory) if fname.endswith('.jpg'))

batchsize = 500
l_index, r_index = 0, batchsize
batch = fnames[l_index:r_index]

while batch:

    for i in batch:
        import_function(i) 
    l_index, r_index = r_index, r_index + batchsize
    batch = fnames[l_index:r_index]

Answer 2

batch_size = 500
filenames = glob(...)    # fill with your own details
nfiles = len(filenames)
nbatches, remainder = divmod(nfiles, batch_size)
for i in xrange(nbatches):   # or range() for Python 3
    batch = filenames[batch_size * i:batch_size * (i + 1)]
    do_something_with(batch)
if remainder:
    do_something_with(filenames[batch_size * nbatches:])

---

使用生成器从可能的非结束迭代中获取每个 N 元素的版本：

def every(thing, n):
    """every(ABCDEFG, 2) --> AB CD EF G"""                                      
    toexit = False
    it = iter(thing)
    while not toexit:
        batch = []
        for i in xrange(n):
            try:
                batch.append(it.next())
            except StopIteration:
                toexit = True
        if not batch:
            break
        yield batch


filenames_i = glob.iglob("...")
for batch in every(filenames_i, 500):
    do_something_with(batch)

这将使批次本身的迭代更加简洁（此代码 sn-p 中的 for batch in every()）。