逻辑流程 - 尝试使用 BeautifulSoup 和 CSV Writer 遍历网站页面答案

【问题标题】：Logic flow - trying to iterate through website pages with BeautifulSoup and CSV Writer逻辑流程 - 尝试使用 BeautifulSoup 和 CSV Writer 遍历网站页面
【发布时间】：2015-05-20 20:42:58
【问题描述】：

我似乎无法找出正确的缩进/子句位置来让这个循环超过 1 页。此代码当前可以很好地打印出 CSV 文件，但只针对第一页。

#THIS WORKS BUT ONLY PRINTS THE FIRST PAGE

from bs4 import BeautifulSoup
from urllib2 import urlopen
import csv

page_num = 1
total_pages = 20

with open("MegaMillions.tsv","w") as f:
    fieldnames = ['date', 'numbers', 'moneyball']
    writer = csv.writer(f, delimiter = '\t')
    writer.writerow(fieldnames)

    while page_num < total_pages:
        page_num = str(page_num)
        soup = BeautifulSoup(urlopen('http://www.usamega.com/mega-millions-history.asp?p='+page_num).read())

    for row in soup('table',{'bgcolor':'white'})[0].findAll('tr'):

        tds = row('td')
        if tds[1].a is not None:
            date = tds[1].a.string.encode("utf-8")
            if tds[3].b is not None:
                uglynumber = tds[3].b.string.split()
                betternumber = [int(uglynumber[i]) for i in range(len(uglynumber)) if i%2==0]
                moneyball = tds[3].strong.string.encode("utf-8")

                writer.writerow([date, betternumber, moneyball])
        page_num = int(page_num)
        page_num += 1

print 'We\'re done here.'

当然，这只会打印最后一页：

#THIS WORKS BUT ONLY PRINTS THE LAST PAGE

from bs4 import BeautifulSoup
from urllib2 import urlopen
import csv

page_num = 1
total_pages = 20

while page_num < total_pages:
    page_num = str(page_num)
    soup = BeautifulSoup(urlopen('http://www.usamega.com/mega-millions-history.asp?p='+page_num).read())

    with open("MegaMillions.tsv","w") as f:
        fieldnames = ['date', 'numbers', 'moneyball']
        writer = csv.writer(f, delimiter = '\t')
        writer.writerow(fieldnames)

        for row in soup('table',{'bgcolor':'white'})[0].findAll('tr'):

            tds = row('td')
            if tds[1].a is not None:
                date = tds[1].a.string.encode("utf-8")
                if tds[3].b is not None:
                    uglynumber = tds[3].b.string.split()
                    betternumber = [int(uglynumber[i]) for i in range(len(uglynumber)) if i%2==0]
                    moneyball = tds[3].strong.string.encode("utf-8")

                    writer.writerow([date, betternumber, moneyball])
        page_num = int(page_num)
        page_num += 1

print 'We\'re done here.'

【问题讨论】：

将代码提取到函数中并单独排除故障，而不是大量代码。

标签： python csv beautifulsoup

【解决方案1】：

您的第二个代码示例的问题是您每次都在覆盖您的文件。而不是

open("MegaMillions.tsv","w")

使用

open("MegaMillions.tsv","a")

“a”打开文件进行追加，这是你想要做的

【讨论】：

愚蠢的我。这样可行。但是如果我们想使用 'w' 而不是 'a'，有没有办法在 while/fors/etc 周围进行洗牌以使其工作？我觉得我迷失在控制流中，这让我很烦。
是的，您可以在 while 循环之前移动 open。这样，您只需打开文件一次，就可以写到所有页面都完成
这不是我在第一个代码中所做的吗？在while循环之前打开？
如果我取出 while 块中的缩进以将其与第一个示例中的开放块对齐，它会给我一个 ValueError: I/O operation on closed file
如果您的第一个代码示例是准确的，我认为它不会退出 while 循环，因为 page_num 将始终小于 total_pages，因为您没有在该循环中增加 page_num。我的猜测是你的 for 循环需要再缩进一级，以便它在你的 while 循环中。现在，由于其缩进级别，for 循环正在获取 page_num 增量而不是 while 循环。

【解决方案2】：

感谢您的建议，以下是一种可行的变体：

from bs4 import BeautifulSoup
from urllib2 import urlopen
import csv

page_num = 1
total_pages = 73

with open("MegaMillions.tsv","w") as f:
    fieldnames = ['date', 'numbers', 'moneyball']
    writer = csv.writer(f, delimiter = '\t')
    writer.writerow(fieldnames)

    while page_num <= total_pages:
        page_num = str(page_num)
        soup = BeautifulSoup(urlopen('http://www.usamega.com/mega-millions-history.asp?p='+page_num).read())

        for row in soup('table',{'bgcolor':'white'})[0].findAll('tr'):

            tds = row('td')
            if tds[1].a is not None:
                date = tds[1].a.string.encode("utf-8")
                if tds[3].b is not None:
                    uglynumber = tds[3].b.string.split()
                    betternumber = [int(uglynumber[i]) for i in range(len(uglynumber)) if i%2==0]
                    moneyball = tds[3].strong.string.encode("utf-8")

                    writer.writerow([date, betternumber, moneyball])
        page_num = int(page_num)
        page_num += 1

print 'We\'re done here.'

选择了这个而不是“a”，因为这样标题行就会为每一页写入。

【讨论】：