【问题标题】：How to scrape data properly in Python and BS4?如何在 Python 和 BS4 中正确抓取数据？
【发布时间】：2017-05-08 23:52:26
【问题描述】：

这是所需的输出。包含 2 行的 CSV 文件：

1639, 06/05/17, 08,09,16,26,37,50
1639, 06/05/17, 13,28,32,33,37,38

今天，我只有这个，但是使用 VBA Excel 代码来清理/组织数据：

08,09,16,26,37,50
13,28,32,33,37,38

print screen

在第一行 '1639, 06/05/17' 来自 Resultado <span>Concurso 1639 (06/05/2017)</span> 和 '08,09,16,26,37,50' 来自下面提供的标签：

<ul class="numbers dupla-sena">
<h6>1º sorteio</ <h6>1º sorteio</h6>
<li>08</li><li>09</li><li>16</li><li>26</li><li>37</li><li>50</li>    
</ul>

在第二行中，我们可能可以从第 1 行复制 '1639, 06/05/17'，而 '13,28,32,33,37,38' 来自另一个标签：

<ul class="numbers dupla-sena">
<h6>2º sorteio</h6>
<li>13</li><li>28</li><li>32</li><li>33</li><li>37</li><li>38</li>
</ul>

下面是我的代码：

import requests
from bs4 import BeautifulSoup as soup 

url = 'http://loterias.caixa.gov.br/wps/portal/loterias/landing/duplasena/'

r = requests.get(url)
ltr = soup(r.text, "xml")
ltr.findAll("div",{"class":"content-section section-text with-box no-margin-bottom"})

filename = "ds_1640.csv"
f=open(filename,"w")

使用下面的命令我想我可以得到我想要的所有东西，但我不知道如何以我需要的方式提取数据：

ltr.findAll("div",{"class":"content-section section-text with-box no-margin-bottom"})

所以，我尝试了另一种方法来捕获 '1º sorteio da dupla-sena' 中的值

print('-----------------dupla-sena 1º sorteio-----------------------------')
d1 = ltr.findAll("ul",{"class":"numbers dupla-sena"})[0].text.strip()
print(ltr.findAll("ul",{"class":"numbers dupla-sena"})[0].text.strip())

输出 1

1º sorteio
080916263750

将两个数字分开

d1 = '0'+ d1 if len(d1)%2 else d1    
gi = [iter(d1)]*2   
r = [''.join(dz1) for dz1 in zip(*gi)]
d3=",".join(r)

结果

08,09,16,26,37,50

第二次提取也是如此

print('-----------------dupla-sena 2º sorteio-----------------------------')
dd1 = ltr.findAll("ul",{"class":"numbers dupla-sena"})[1].text.strip()
print(ltr.findAll("ul",{"class":"numbers dupla-sena"})[1].text.strip())

输出 2

2º sorteio
132832333738

将两个数字分开

dd1 = '0'+ dd1 if len(dd1)%2 else dd1    
gi = [iter(dd1)]*2    
r1 = [''.join(ddz1) for ddz1 in zip(*gi)]    
dd3=",".join(r1)

那么我们有

13,28,32,33,37,38

将数据保存到 csv 文件

f.write(d3 + ',' + dd3 +'\n')
f.close()

输出：当前目录下的一个csv文件：

01,º ,so,rt,ei,o
,08,09,16,26,37,50,02,º ,so,rt,ei,o
,13,28,32,33,37,38

我可以使用上述方法/输出，但我必须使用 VBA excel 来处理这些杂乱的数据，但我尽量避免使用 vba 代码。实际上我对学习 Python 更感兴趣，并且越来越多地使用这个强大的工具。有了这个解决方案，我只实现了我想要的一个部分，即：

08,09,16,26,37,50
13,28,32,33,37,38

但是，正如我们所知，所需的输出是：

1639, 06/05/17, 08,09,16,26,37,50
1639, 06/05/17, 13,28,32,33,37,38

我在 MAC OS X Yosemite(10.10.5) 中使用 Python 3.6.1 (v3.6.1:)、Jupyter notebook。

我怎样才能做到这一点？我不知道如何提取 '1639, 06/05/17' 并将其放入 csv 文件中，有没有更好的方法来提取六个数字（08、09、16、26、37、50 和 13 ,28,32,33,37,38) 不使用下面的代码也不使用vba？

分隔两个数字：

d1 = '0'+ d1 if len(d1)%2 else d1
gi = [iter(d1)]*2
r = [''.join(dz1) for dz1 in zip(*gi)]

更新问题

import requests
from bs4 import BeautifulSoup  
import re
import csv

url = 'http://loterias.caixa.gov.br/wps/portal/loterias/landing/duplasena/'

r = requests.get(url)

soup = BeautifulSoup(r.text, "lxml")  ## "lxml" to avoid the warning

pat = re.compile(r'(?i)(?<=concurso)\s*(?P<concurso>\d+)\s*\((?P<data>.+?)(?=\))')
concurso_e_data = soup.find(id='resultados').h2.span.text
match = pat.search(concurso_e_data)


# first I would do the above part differently seeing as how you want the end data to look
if match:
    concurso, data = match.groups()
    nums = soup.find_all("ul", {"class": "numbers dupla-sena"})
    num_headers = (','.join(['numero%d']*6) % tuple(range(1,7))).split(',')
    # unpack numheaders into field names
    field_names = ['sena', 'data', *num_headers]
    # above gives you this
    # field_names = [
    #    'sena',               ## I've changed "seria"for "sena"
    #    'data',
    #    'numero1',
    #    'numero2',
    #    'numero3',
    #    'numero4',
    #    'numero5',
    #    'numero6',
    # ]
    
    
rows = []
# then add the numbers
# nums is all the `ul` list elements contains the drawing numbers
for group in nums:
    # start each row with the shared concurso, data elements
    row = [concurso, data]
    # for each `ul` get all the `li` elements containing the individual number
    for num in group.findAll('li'):
        # add each number
        row.append(int(num.text))
    # get [('sena', '1234'), ('data', '12/13'2017'),...]
    row_title_value_pairs = zip(field_names, row)
    # turn into dict {'sena': '1234', 'data': '12/13/2017', ...}
    row_dict = dict(row_title_value_pairs)
    rows.append(row_dict)
    
    
    # so now rows looks like: [{
    #   'sena': '1234', 
    #   'data': '12/13/2017', 
    #   'numero1': 1, 
    #   'numero2': 2, 
    #   'numero3': 3, 
    #   'numero4': 4, 
    #   'numero5': 5, 
    #   'numero6': 6
    #   }, ...]



    with open('file_v5.csv', 'w', encoding='utf-8') as csvfile:
        csv_writer = csv.DictWriter(
            csvfile,
            fieldnames=field_names,
            dialect='excel',
            extrasaction='ignore', # drop extra fields if not in field_names not necessary but just in case
            quoting=csv.QUOTE_NONNUMERIC  # quote anything thats not a number, again just in case
        )
        csv_writer.writeheader()
        for row in rows:
            csv_writer.writerow(row_dict)

输出

# "sena","data","numero1","numero2","numero3","numero4","numero5","numero6"
# "1641","11/05/2017",1,5,15,28,30,43
# "1641","11/05/2017",1,5,15,28,30,43  #This comes from 1. drawing and not 
from the corcect one (2.)

更新的问题 2

import requests
from bs4 import BeautifulSoup  
import re
import csv

url = 'http://loterias.caixa.gov.br/wps/portal/loterias/landing/duplasena/'

r = requests.get(url)

soup = BeautifulSoup(r.text, "lxml")  ## "lxml" to avoid the warning

pat = re.compile(r'(?i)(?<=concurso)\s*(?P<concurso>\d+)\s*\((?P<data>.+?)(?=\))')
concurso_e_data = soup.find(id='resultados').h2.span.text
match = pat.search(concurso_e_data)

# everything should be indented under this block since 
# if there is no match then none of the below code should run
if match:  
    concurso, data = match.groups()
    nums = soup.find_all("ul", {"class": "numbers dupla-sena"})
    num_headers = (','.join(['numero%d']*6) % tuple(range(1,7))).split(',')
    field_names = ['sena', 'data', *num_headers]

    # PROBLEM 1
    # all this should be indented into the `if match:` block above
    # none of this should run if there is no match
    # you cannot build the rows without the match for sena and data
    # Let's add some print statements to see whats going on
    rows = []
    for group in nums:
        # here each group is a full `sena` row from the site
        print('Pulling sena: ', group.text)
        row = [concurso, data]
        print('Adding concurso + data to row: ', row)
        for num in group.findAll('li'):
            row.append(int(num.text))
            print('Adding {} to row.'.format(num))
        print('Row complete: ', row)
        row_title_value_pairs = zip(field_names, row)
        print('Transform row to header, value pairs: ', row_title_value_pairs)
        row_dict = dict(row_title_value_pairs)
        print('Row dictionary: ', row_dict)
        rows.append(row_dict)
        print('Rows: ', rows)

    # PROBLEM 2
    # It would seem that you've confused this section when switching
    # out the original list comprehension with the more explicit 
    # for loop in building the rows.
    
# The below block should be indented to this level.
# Still under the `if match:`, but out of the the 
# `for group in nums:` above

    # the below block loops over rows, but you are still building
    # the rows in the for loop
    # you are effectively double looping over the values in `row`

    
    with open('ds_v4_copy5.csv', 'w', encoding='utf-8') as csvfile:
        csv_writer = csv.DictWriter(
        csvfile,
        fieldnames=field_names,
        dialect='excel',
        extrasaction='ignore', # drop extra fields if not in field_names not necessary but just in case
        quoting=csv.QUOTE_NONNUMERIC  # quote anything thats not a number, again just in case
        )
        csv_writer.writeheader()
        # this is where you are looping extra because this block is in the `for` loop mentioned in my above notes
        #for row in rows:  ### I tried here to avoid the looping extra
            #print('Adding row to CSV: ', row)
        csv_writer.writerow(row_dict)

我想我听从了您的指示。但到目前为止我们得到了这个：

"sena","data","numero1","numero2","numero3","numero4","numero5","numero6"
"1643","16/05/2017",3,4,9,19,21,26  #which is "1º sorteio"

仍然缺少“2º sorteio”。我知道我做错了什么，因为“1º sorteio”和“2º sorteio”都在：

print(rows[0]) --> {'sena': '1643', 'data': '16/05/2017', 'numero1': 1, 'numero2': 21, 'numero3': 22, 'numero4': 43, 'numero5': 47, 'numero6': 50} 

 print(rows[1]) --> {'sena': '1643', 'data': '16/05/2017', 'numero1': 3, 'numero2': 4, 'numero3': 9, 'numero4': 19, 'numero5': 21, 'numero6': 26}

但是，当我尝试将 row_dict 的内容存储在 csv 中时（只有 row[0] 出现在 row_dict 中。我试图弄清楚如何包含丢失的内容。也许我错了，但我认为“1º sorteio”和“2º sorteio”都应该包含在row_dict中，但是当我们看到这个时，代码并没有确认它（这是一个猜测）：

print(row_dict)
{'sena': '1643', 'data': '16/05/2017', 'numero1': 3, 'numero2': 4, 'numero3': 9, 'numero4': 19, 'numero5': 21, 'numero6': 26}

我看不出我做错了什么。我知道这个答案需要很长时间，但我在这个过程中和你一起学习了很多。并且已经在使用我和你一起学习的几种工具（re、concepts、dict、zip）。

【问题讨论】：

添加解决方案和解释
这里的问题是您复制并粘贴了代码的不同部分，而没有花时间去理解它的作用。我试图用它的功能描述来标记每一行，但如果你不花时间阅读它，那么这些错误就会发生。检查我的第二次更新...
你们俩都做得很好，因为他们坚持不懈地获得了所需的结果，干得好。但是，这里的一篇文章中可能塞满了太多问题，而让问题比这更简单一点是 Stack Exchange 的原则。首先，Fabio，Verbal_Kint 的回答非常友善，但大多数回答者可能会要求您（非常正确）开始一个新问题。
这样做的主要原因是一个问题可能对未来的读者有用；复杂的问题可以分解，这样新读者就不必完全摸索上一个阶段来理解当前的阶段。然而，同一个问题中的多阶段问题对于提问者来说变得相当独特，因此它可能没有那么多未来的适用性。

标签： python bs4

【解决方案1】：

免责声明：我对美汤不是很熟悉，我通常使用lxml，也就是说......

soup = BeautifulSoup(response.text)  # <-- edit showing how i assigned soup
pat = re.compile(r'(?i)(?<=concurso)\s*(?P<concurso>\d+)\s*\((?P<data>.+?)(?=\))')
concurso_e_data = soup.find(id='resultados').h2.span.text
match = pat.search(concurso_e_data)
if match:
    concurso, data = match.groups()
    nums = soup.find_all("ul", {"class": "numbers dupla-sena"})
    numeros = []
    for i in nums:
        numeros.append(','.join(j.text for j in i.findAll('li')))
    rows = []
    for n in numeros:
        rows.append(','.join([concurso, data, n]))

print(rows)
['1639,06/05/2017,08,09,16,26,37,50', '1639,06/05/2017,13,28,32,33,37,38']

虽然这是您要求的格式，但在数字组中使用逗号（列分隔符）不是一个坏主意。您应该用另一个字符分隔或用空格分隔数字。

更新 1：

在 cmets 部分写入并不是最好的方法...假设您真正想要的格式是 8 行，如下所示 (seria, data, num1, num2, ... num6) 其中 seria 和 data 是字符串，数字是 @ 987654325@s:

# first I would do the above part differently seeing as how you want the end data to look
...
if match:
    concurso, data = match.groups()
    nums = soup.find_all("ul", {"class": "numbers dupla-sena"})
    num_headers = (','.join(['numero%d']*6) % tuple(range(1,7))).split(',')
    # unpack numheaders into field names
    field_names = ['seria', 'data', *num_headers]
    # above gives you this
    # field_names = [
    #    'seria',
    #    'data',
    #    'numero1',
    #    'numero2',
    #    'numero3',
    #    'numero4',
    #    'numero5',
    #    'numero6',
    # ]
    rows = [
        dict(zip(
            field_names, 
            [concurso, data, *[int(num.text) for num in group.findAll('li')]]
        )) 
        for group in nums]
    # so now rows looks like: [{
    #   'seria': '1234', 
    #   'data': '12/13/2017', 
    #   'numero1': 1, 
    #   'numero2': 2, 
    #   'numero3': 3, 
    #   'numero4': 4, 
    #   'numero5': 5, 
    #   'numero6': 6
    #   }, ...]
    with open('file.csv', 'a', encoding='utf-8') as csvfile:
        csv_writer = csv.DictWriter(
            csvfile,
            fieldnames=field_names,
            dialect='excel',
            extrasaction='ignore', # drop extra fields if not in field_names not necessary but just in case
            quoting=csv.QUOTE_NONNUMERIC  # quote anything thats not a number, again just in case
        )
        csv_writer.writeheader()
        for row in rows:
            csv_writer.writerow(row_dict)

这部分有点杂乱：

rows = [
    dict(zip(
        field_names, 
        [concurso, data, *[int(num.text) for num in group.findAll('li')]
    )) 
    for group in nums]

那我换个方式写吧：

rows = []
# then add the numbers
# nums is all the `ul` list elements contains the drawing numbers
for group in nums:
    # start each row with the shared concurso, data elements
    row = [concurso, data]
    # for each `ul` get all the `li` elements containing the individual number
    for num in group.findAll('li'):
        # add each number
        row.append(int(num.text))
    # get [('seria', '1234'), ('data', '12/13'2017'),...]
    row_title_value_pairs = zip(field_names, row)
    # turn into dict {'seria': '1234', 'data': '12/13/2017', ...}
    row_dict = dict(row_title_value_pairs)
    rows.append(row_dict)
    # or just write the csv here instead of appending to rows and re-looping over the values
    ...

更新 2：

我希望您从中学到的一件事是在学习时使用print 语句，这样您就可以理解代码的作用。我不会进行更正，但我会指出它们并在每个发生重大变化的位置添加 print 语句...

match = pat.search(concurso_e_data)


# everything should be indented under this block since 
# if there is no match then none of the below code should run
if match:  
    concurso, data = match.groups()
    nums = soup.find_all("ul", {"class": "numbers dupla-sena"})
    num_headers = (','.join(['numero%d']*6) % tuple(range(1,7))).split(',')
    field_names = ['sena', 'data', *num_headers]

# PROBLEM 1
# all this should be indented into the `if match:` block above
# none of this should run if there is no match
# you cannot build the rows without the match for sena and data
# Let's add some print statements to see whats going on
rows = []
for group in nums:
    # here each group is a full `sena` row from the site
    print('Pulling sena: ', group.text)
    row = [concurso, data]
    print('Adding concurso + data to row: ', row)
    for num in group.findAll('li'):
        row.append(int(num.text))
        print('Adding {} to row.'.format(num))
    print('Row complete: ', row)
    row_title_value_pairs = zip(field_names, row)
    print('Transform row to header, value pairs: ', row_title_value_pairs)
    row_dict = dict(row_title_value_pairs)
    print('Row dictionary: ', row_dict)
    rows.append(row_dict)
    print('Rows: ', rows)

    # PROBLEM 2
    # It would seem that you've confused this section when switching
    # out the original list comprehension with the more explicit 
    # for loop in building the rows.
# The below block should be indented to this level.
# Still under the `if match:`, but out of the the 
# `for group in nums:` above

    # the below block loops over rows, but you are still building
    # the rows in the for loop
    # you are effectively double looping over the values in `row`
    with open('file_v5.csv', 'w', encoding='utf-8') as csvfile:
        csv_writer = csv.DictWriter(
            csvfile,
            fieldnames=field_names,
            dialect='excel',
            extrasaction='ignore', # drop extra fields if not in field_names not necessary but just in case
            quoting=csv.QUOTE_NONNUMERIC  # quote anything thats not a number, again just in case
        )
        csv_writer.writeheader()
        # this is where you are looping extra because this block is in the `for` loop mentioned in my above notes
        for row in rows:
            print('Adding row to CSV: ', row)
            csv_writer.writerow(row_dict)

运行它，看看打印语句会显示什么。但是也要阅读注释，因为如果 sena, data 不匹配，会导致错误。

提示：缩进，然后在最后的if match: 块下添加else: print('No sena, data match!')...但首先运行它并检查它打印的内容。

【讨论】：

嗨@Verbal_Kint，感谢您的帮助！我试过并收到了这条消息：TypeError: find() missing 1 required positional argument: 'self'
@fabio Voce nao pode simplesmente me dar o erro sem o traceback。马斯，欧盟 acho que sei qual e o problema。 Meu 变量 "soup" e sua "soup" sao definidas diferente。 Eu mudei a primeira linha do meu 代码。
博阿推荐！！ Deu certo, consegui chegar na solução parcial。 Como exportar adequadamente？ Usei esse metodo abaixo e ele transporta para o csv mas ele ignora o formato desejado (todos os dados em 2 linhas e 8 colunas: concurso-1coluna, data do sorteio-1coluna e resultado da extracao (6 dezenas - 6 colunas): file = 'ds_stack.csv' with open(file,'a') as f: writer = csv.writer(f) writer.writerow(rows)
foi falha minha na declaracao do questiona, talvez nao tenha sido claro quanto a necessidade de deixar os dados em duas linhas e 6 colunas。 Tentei concatenar, mas nao se pode (str to list)。 Veja o traceback:filename = "ds_stack_2.csv" f=open(filename,"w") f.write(rows+'\n') f.close() TypeError Traceback (最近一次调用最后一次) 在 () 1 文件名 = "ds_stack_2.csv" 2 f=open(filename,"w") ----> 3 f.write(rows+'\n') 4 f.close() TypeError：只能将列表（不是“str”）连接到列表
@fabio nao precisa o '\n': f.write(rows+'\n') --> f.write(rows)。 Mais porque nao esta usando csvwriter? O problma e qui rows e tipo list e '\n' e tipo string

【解决方案2】：

（代表 OP 发布）。

在@Verbal_Kint 的帮助下，我们成功了！输出我需要的方式！我已将输出更改为：

sena;data;numero1;numero2;numero3;numero4;numero5;numero6
1644;18/05/2017;4;6;31;39;47;49
1644;18/05/2017;20;37;44;45;46;50

所以，按照他们对“，”和“;”的关注在 Excel 中，我决定将 "," 更改为 ";" 以在 Excel 中打开列而不会出现任何问题。

import requests
from bs4 import BeautifulSoup  
import re
import csv

url = 'http://loterias.caixa.gov.br/wps/portal/loterias/landing/duplasena/'

r = requests.get(url)

soup = BeautifulSoup(r.text, "lxml")  ## "lxml" to avoid the warning

pat = re.compile(r'(?i)(?<=concurso)\s*(?P<concurso>\d+)\s*\((?P<data>.+?)(?=\))')
concurso_e_data = soup.find(id='resultados').h2.span.text
match = pat.search(concurso_e_data)

if match:  
    concurso, data = match.groups()
    nums = soup.find_all("ul", {"class": "numbers dupla-sena"})
    num_headers = (','.join(['numero%d']*6) % tuple(range(1,7))).split(',')
    field_names = ['sena', 'data', *num_headers]

    rows = []
    for group in nums:
        row = [concurso, data]
        for num in group.findAll('li'):
            row.append(int(num.text))
        row_title_value_pairs = zip(field_names, row)
        row_dict = dict(row_title_value_pairs)
        rows.append(row_dict)

    with open('ds_v10.csv', 'w', encoding='utf-8') as csvfile:
        csv_writer = csv.DictWriter(
        csvfile,
        fieldnames=field_names,             
        dialect='excel',       
            delimiter = ';',  #to handle column issue in excel!
        )
        csv_writer.writeheader()
        csv_writer.writerow(rows[0])
        csv_writer.writerow(rows[1])

【讨论】：

感谢@halfer 的帮助！现在好多了！