拆分结构化输入文件中的生成器'yield'会产生不同步的结果

Question

在这里参考@georg 的最佳答案（我在下面进行了改编）： Split one file into multiple files based on pattern (cut can occur within lines)

我发现这是一种基于初始分隔符将文件拆分为多个文件的潜在有用模式。 但是，正如评论者指出的那样，它首先创建了一个空白文件，其原因尚不清楚。我认为这与我遇到的问题有关。

在我的（笨拙，我不是 python 大师！）改编中，我尝试通过解析分隔符后面的行来设置文件名，然后通过调用 output=next(fs) 生成器打开新的输出文件。

但是，难题当然是直到分隔符后面的行才知道域名。我最终得到的文件名与包含的数据不同步。

输入文件包含 100+ xml'trees'，每一个都以一个标准开头

<?xml version='1.0' encoding='UTF-8'?>

后跟这样的一行，其中包含域名

<ns2:domain ... name="atypi.org" ...">

这是我当前的脚本：

#!/usr/bin/python2.7

import re

def files():
    n = 0 
    while n<12 :
         n += 1
         print "**DEBUG** in generator nameFile=%s n=%d \r" % (nameFile, n) 
         yield open('/Users/peterf/Google Drive/2015 Projects-Strategy/Domain Admin/RackDomains/%s.part.xml' % nameFile, 'w')


filename='/Users/peterf/Google Drive/2015 Projects-Strategy/Domain Admin/RackspaceListDomain.output.xml'
nameFile=''
pat ='<?xml'
namePat=re.compile('<ns2:domain.+ name="(.+?)".+>')
fs = files()
outfile = next(fs)

with open(filename) as infile:
     for line in infile:   
        m=namePat.search(line)
        if m:
           nameFile=m.group(1)
           print "<---\rin 'if m:' nameFile=%s\r" % (nameFile)   
        if pat not in line: 
#           print "\rin 'pat not in line' line=%s\r" % (line)       
           outfile.write(line)
        else:
            items = line.split(pat)
            outfile.write(items[0])
            for item in items[1:]:
                print "in 'for item' pre next(fs) nameFile=%s\r" % (nameFile)
                outfile = next(fs)
                print "in 'for item' post next(fs) nameFile=%s --->\r" % (nameFile)
                outfile.write(pat + item)

我的调试列表显示：

**DEBUG** in generator nameFile= n=1 

in 'for item' pre next(fs) nameFile=

**DEBUG** in generator nameFile= n=2 

in 'for item' post next(fs) nameFile= --->

<---
in 'if m:' nameFile=addressing.com

in 'for item' pre next(fs) nameFile=addressing.com

**DEBUG** in generator nameFile=addressing.com n=3 

in 'for item' post next(fs) nameFile=addressing.com --->

<---
in 'if m:' nameFile=alicemcmahon.com

in 'for item' pre next(fs) nameFile=alicemcmahon.com

**DEBUG** in generator nameFile=alicemcmahon.com n=4 

in 'for item' post next(fs) nameFile=alicemcmahon.com --->

<---
in 'if m:' nameFile=alphabets.com

in 'for item' pre next(fs) nameFile=alphabets.com

**DEBUG** in generator nameFile=alphabets.com n=5 

in 'for item' post next(fs) nameFile=alphabets.com --->

输出目录包含这些文件名，从我猜的第一个“yield”中截断的名称开始......

.part.xml (this has data from 'addressing.com')
addressing.com.part.xml
alicemcmahon.com.part.xml
alphabets.com.part.xml
americanletterpress.com.part.xml
americanwoodtype.com.part.xml
amyshoemaker.com.part.xml
archaicrevivalbooks.com.part.xml
archaicrevivalfonts.com.part.xml
archaicrevivalimages.com.part.xml
astroteddies.com.part.xml

我不知道如何解决这个问题，生成器在我得到一个合适的文件名之前就生成了一个输出文件。

以下是输入文件的一些代表性部分：

<?xml version='1.0' encoding='utf-8'?>
<ns2:domain xmlns:ns3="http://www.w3.org/2005/Atom" xmlns:ns2="http://docs.rackspacecloud.com/dns/api/v1.0" xmlns="http://docs.rackspacecloud.com/dns/api/management/v1.0" id="1204245"  name="addressing.com" ttl="300" emailAddress="ipadmin@stabletransit.com" updated="2012-10-10T21:33:36Z" created="2009-07-25T15:05:39Z">
    <ns2:nameservers>
        <ns2:nameserver name="dns1.stabletransit.com" />
        <ns2:nameserver name="dns2.stabletransit.com" />
    </ns2:nameservers>
    <ns2:recordsList totalEntries="5">
        <ns2:record id="A-2542579" type="A" name="addressing.com" data="198.101.155.141" ttl="300" updated="2012-10-10T21:33:35Z" created="2010-02-17T05:02:16Z" />
    </ns2:recordsList>
</ns2:domain>
<?xml version='1.0' encoding='UTF-8'?>
<ns2:domain xmlns:ns3="http://www.w3.org/2005/Atom" xmlns:ns2="http://docs.rackspacecloud.com/dns/api/v1.0" xmlns="http://docs.rackspacecloud.com/dns/api/management/v1.0" id="2776403"  name="alicemcmahon.com" ttl="300" emailAddress="ipadmin@stabletransit.com" updated="2013-10-21T16:43:17Z" created="2011-05-01T03:01:51Z">
    <ns2:nameservers>
        <ns2:nameserver name="dns1.stabletransit.com" />
        <ns2:nameserver name="dns2.stabletransit.com" />
    </ns2:nameservers>
    <ns2:recordsList totalEntries="10">
        <ns2:record id="A-6895108" type="A" name="alicemcmahon.com" data="216.185.152.144" ttl="300" updated="2013-10-21T16:43:17Z" created="2011-05-01T03:01:51Z" />
    </ns2:recordsList>
</ns2:domain>
<?xml version='1.0' encoding='UTF-8'?>
<ns2:domain xmlns:ns3="http://www.w3.org/2005/Atom" xmlns:ns2="http://docs.rackspacecloud.com/dns/api/v1.0" xmlns="http://docs.rackspacecloud.com/dns/api/management/v1.0" id="1204247"  name="americanletterpress.com" ttl="300" emailAddress="ipadmin@stabletransit.com" updated="2012-10-10T21:33:37Z" created="2009-07-25T15:05:41Z">
    <ns2:nameservers>
        <ns2:nameserver name="dns1.stabletransit.com" />
        <ns2:nameserver name="dns2.stabletransit.com" />
    </ns2:nameservers>
    <ns2:recordsList totalEntries="5">
        <ns2:record id="A-2542581" type="A" name="americanletterpress.com" data="198.101.155.141" ttl="300" updated="2012-10-10T21:33:36Z" created="2010-02-17T05:02:16Z" />        
    </ns2:recordsList>
</ns2:domain>
<?xml version='1.0' encoding='UTF-8'?>
<ns2:domain xmlns:ns3="http://www.w3.org/2005/Atom" xmlns:ns2="http://docs.rackspacecloud.com/dns/api/v1.0" xmlns="http://docs.rackspacecloud.com/dns/api/management/v1.0" id="1204249"  name="americanwoodtype.com" ttl="300" emailAddress="ipadmin@stabletransit.com" updated="2012-10-10T21:33:38Z" created="2009-07-25T15:05:42Z">
    <ns2:nameservers>
        <ns2:nameserver name="dns1.stabletransit.com" />
        <ns2:nameserver name="dns2.stabletransit.com" />
    </ns2:nameservers>
    <ns2:recordsList totalEntries="5">
        <ns2:record id="A-2542583" type="A" name="americanwoodtype.com" data="198.101.155.141" ttl="300" updated="2012-10-10T21:33:37Z" created="2010-02-17T05:02:16Z" />
    </ns2:recordsList>
</ns2:domain>

Answer 1

您要求生成器在一开始就生成一个输出文件：

nameFile=''
# ...
outfile = next(fs)

那是你的空白文件名。推迟调用 next(fs)，直到您获得 nameFile 的值，而不是之前。

您可以改为设置outfile = None 并在编写之前测试None：

if pat not in line:
    if outfile is not None: 
        outfile.write(line)
else:
    items = line.split(pat)
    if outfile is not None:
        outfile.write(items[0])

如果您需要在找到第一个文件名之前处理行，请将这些行存储在缓冲区中，并在首次创建新文件时清除缓冲区。

并不是说我认为您应该根本使用生成器，使用生成器确实使事情变得过于复杂。只需直接在循环中创建新的文件对象，这样会更清晰。

如果您所做的只是拆分文件，请使用缓冲区直到获得文件名：

buffer = []
out_name = '/Users/peterf/Google Drive/2015 Projects-Strategy/Domain Admin/RackDomains/%s.part.xml'

outfile = None

with open(filename) as infile:
    for line in infile:
        # look for a filename to write to if we don't have one yet
        if outfile is None:
            match = namePat.search(line)
            if match:
                # New filename, open a file object
                outfile = open(out_name % match.group(1), 'w')
                # clear out the buffer, we'll write directly to 
                # the file after this.
                outfile.writelines(buffer)
                buffer = []

        if '<?xml' in line:
            # new XML doc, close off the previous one
            if outfile is not None:
                outfile.close()
            outfile = None

        # line handling
        if outfile is None:
            buffer.append(line)
        else:
            outfile.write(line)

if outfile is not None:
    outfile.close()
# All lines processed, if there is a buffer left, then we have unhandled lines
if buffer:
    print('There were trailing lines without a name')
    print(*buffer, sep='')