使用 bs4 和请求抓取多个链接

Question

我是新手，请放轻松。所以我想抓取多个页面的名称和数量，但只抓取第一个 url。此外，该代码仅抓取名称或电话号码中的一项。名称在“h2”类“c411ListedName”中，号码在“span”类“c411Phone”中

import csv
import requests
from bs4 import BeautifulSoup

urls = ['https://www.canada411.ca/search/si/1/kumar/Canada/?pgLen=100','https://www.canada411.ca/search/si/2/kumar/Canada/?pgLen=100']


for url in urls:
    response = requests.get(url)
    html = requests.get(url).text
    soup = BeautifulSoup(response.content, "html.parser")
    products = soup.findAll('span','h2', class_=['c411Phone', 'c411ListedName'])

for div in products: 
    print(div.text)

Answer 1

会发生什么

您请求两个 url，但只处理最后一个，因为在您的第一个循环完成后，您的第二个循环将不会执行。

解决方案

改进你的缩进并将第二个循环放在第一个

import csv
import requests
from bs4 import BeautifulSoup

urls = ['https://www.canada411.ca/search/si/1/kumar/Canada/?pgLen=100','https://www.canada411.ca/search/si/2/kumar/Canada/?pgLen=100']


for url in urls:
    response = requests.get(url)
    html = requests.get(url).text
    soup = BeautifulSoup(response.content, "html.parser")
    products = soup.findAll('span','h2', class_=['c411Phone', 'c411ListedName'])

    for div in products: 
        
        print(url,div.text)

其他解决方案

选择并处理封闭的<div>

import csv
import requests
from bs4 import BeautifulSoup

urls = ['https://www.canada411.ca/search/si/1/kumar/Canada/?pgLen=100','https://www.canada411.ca/search/si/2/kumar/Canada/?pgLen=100']

data=[]

for url in urls:
    response = requests.get(url)
    html = requests.get(url).text
    soup = BeautifulSoup(response.content, "html.parser")
    
    for row in soup.select('div.c411Listing.jsResultsList > div.listing__row:nth-of-type(1)'):
        data.append({
            'name':row.h2.a.text,
            'number':row.span.text
        })

data

Answer 2

你应该做两件事：

1. 将products 作为一个数组从您的 for 中取出，以便能够存储所有 url 的内容
2. 对每个 h2 和 span 分别使用 findall

代码如下：

import requests
from bs4 import BeautifulSoup

urls = ['https://www.canada411.ca/search/si/1/kumar/Canada/?pgLen=100',
        'https://www.canada411.ca/search/si/2/kumar/Canada/?pgLen=100']

products_h = []
products_span = []

for url in urls:
    response = requests.get(url)
    soup = BeautifulSoup(response.content, "html.parser")
    products_h.append(soup.findAll('span', class_=['c411Phone']))
    products_span.append(soup.findAll('h2', class_=['c411ListedName']))

for i in range(len(urls)):
    for span, h in zip(products_span[i], products_h[i]):
        print(f'{span.text},{h.text}')

这是我得到的结果：

B Kumar,(416) 444-5781
V Roy Kumar,(613) 226-2668
C Kumar,(514) 388-4472
Ananth Kumar,(905) 389-2429
ASHOK KUMAR,(905) 575-7795
R Kumar,(905) 820-2447
A Kumar,(613) 736-1882
S R Kumar,(705) 426-9866
R Kumar,(519) 886-9588
S & H Kumar,(905) 303-6949
V KUMAR,(613) 834-1453
Ashwani Kumar,(867) 633-5854
S Kumar,(514) 620-1780
V Kumar,(514) 421-8659
B C M Kumar,(905) 335-4480
Vasant Kumar,(613) 723-3485
MICHAEL KUMAR,(519) 824-8555
M Kumar,(613) 747-3790
G Kumar,(905) 265-2017
T Kumar,(905) 824-8625
A Kumar,(905) 508-1561
U Kumar,(905) 433-2165
J Kumar,(905) 426-5249
Priya Kumar,(905) 827-0090
C Kumar,(905) 216-4078
M Kumar,(905) 456-6423
R Kumar,(647) 827-5029
N Kumar,(647) 350-2730
Sanjai Kumar,(416) 913-3648
R Kumar,(905) 916-2177
S Kumar,(289) 232-5477
J Kumar,(416) 238-9504
P Kumar,(905) 956-1699
R Kumar,(905) 793-9551
S Kumar,(905) 848-8282
Naresh Kumar,(905) 257-6640
K Kumar-Telang,(905) 450-1745
Poonam Kumar,(905) 937-6120
R Kumar,(416) 741-1314
A Kumar,(905) 454-4490
A Kumar,(905) 799-8742
Subodh Kumar,(416) 239-4590
R Kumar,(905) 888-0007
S Kumar,(416) 421-2311
R Kumar,(905) 654-8063
N Kumar,(905) 295-4343
R Kumar,(905) 303-4532
S Kumar,(519) 653-0474
Aneal Kumar,(905) 792-1051
Unesh Kumar,(905) 451-3614
Virender Kumar,(416) 483-9116
R Kumar,(905) 796-3413
M Kumar,(519) 691-1176
S SAMPATH-kUMAR,(416) 745-7911
S Kumar,(416) 724-5511
R KUMAR,(416) 256-7991
C Kumar,(905) 509-0211
A Kumar,(905) 455-2071
Ram Kumar,(905) 829-4686
D Kumar,(905) 608-1849
R Kumar,(416) 283-9395
M Kumar,(905) 456-0497
P Kumar,(416) 284-5028
M Kumar,(705) 724-3507
Pradeep Kumar,(613) 548-4619
R Kumar,(416) 481-6271
V Kumar,(905) 451-0797
Rakesh Kumar,(416) 626-8349
J Kumar,(416) 604-4806
A Kumar,(416) 759-7526
A Kumar,(905) 460-9268
Parapurath S Kumar,(905) 794-3673
N Kumar,(905) 642-9708
Praveen Kumar,(647) 827-1562
V Kumar,(905) 488-7531
A Kumar,(905) 901-0050
R Kumar,(905) 216-7945
Praveen Kumar,(416) 238-1247
J Kumar,(905) 265-7537
Praveen Kumar,(416) 855-3744
S Kumar,(905) 792-7671
Ragesh Kumar,(647) 727-5826
Goutham Kumar,(905) 670-9268
V Kumar,(519) 472-2394
Dharmana Kumar,(905) 908-2216
K Kumar,(905) 654-7177
J Kumar,(905) 554-2210
Praveen Kumar,(416) 855-3091
D Kumar,(416) 901-7977
Parmod Kumar,(905) 456-2286
S Kumar,(416) 674-5392
Suby Kumar,(905) 831-8886
A Kumar,(416) 282-0182
A Kumar,(905) 915-7124
Surinder Kumar,(905) 264-7743
Rajesh Kumar,(416) 855-1081
Selva Kumar,(905) 769-4456
Santhosh Kumar,(905) 769-4503
Praveen Kumar,(416) 855-1880
K Kumar,(416) 467-6727
J Kumar,(905) 453-9596
C Kumar,(905) 683-5983
k kumar,(416) 746-2705
A Kumar,(416) 667-9363
Praveen Kumar,(416) 855-0367
J Kumar,(905) 666-4988
Ashok N Kumar,(905) 877-2302
V Chandra Kumar,(416) 724-0156
D Kumar,(416) 332-8825
S Kumar,(905) 476-1995
Maria Kumar,(905) 499-4918
S KUMAR,(905) 814-0518
P Kumar,(905) 522-1386
SURESH KUMAR,(514) 693-0564
S Kumar,(514) 932-1504
Santosh Kumar,(416) 855-3747
p kumar,(905) 604-0670
V R Kumar,(416) 766-3335
Satish Kumar,(416) 406-2340
J Kumar,(905) 854-2337
Sam Kumar,(905) 854-2342
R Kumar,(416) 593-4306
M Kumar,(905) 338-7543
Nita Kumar,(905) 653-0474
Rajender Kumar,(705) 522-2213
R Kumar,(416) 438-3855
V KUMAR,(705) 522-6114
Raman Kumar,(905) 886-0558
B Kumar,(416) 445-4501
S Kumar,(416) 246-9104
S Kumar,(905) 792-7124
S Kumar,(905) 495-3315
D Kumar,(416) 742-8025
R Kumar,(905) 824-8170
Pj Kumar,(905) 785-3110
Rai Bal Kumar,(418) 522-8497
S Kumar,(514) 620-0626
S Kumar,(705) 352-0418
V KUMAR,(647) 345-3130
R Kumar,(416) 208-7965
R Kumar,(226) 663-3309
R Kumar,(905) 553-6099
K Kumar,(905) 566-5855
P Kumar Rudra,(905) 565-6553
Rashmi Kumar,(905) 363-7578
Anil Kumar,(905) 497-6514
Shiv Kumar,(416) 265-5763
Aprul Kumar,(416) 240-9379
A Kumar,(289) 752-7462
P Kumar,(416) 744-3493
T Melburn-Kumar,(905) 524-5529
V Kumar,(416) 724-7624
Poonam Kumar,(905) 430-1763
C Kumar,(905) 470-1296
V Kumar,(905) 237-8979
M KUMAR,(905) 458-1234
R Kumar,(416) 335-0731
A Kumar,(416) 213-9003
M Kumar,(905) 564-5779
S Kumar,(905) 890-3061
P & R Kumar,(905) 726-4444
L Kumar,(416) 438-3283
R Kumar,(613) 841-4638
R KUMAR,(416) 490-9860
K Kumar,(226) 647-1075
J Kumar,(905) 654-1532
S Kumar,(905) 472-4343
D Kumar,(905) 956-7503
Jagjit Kumar,(905) 915-8826
Vimala Krishna Kumar,(416) 551-3021
R Kumar,(647) 340-0236
A Kumar,(905) 791-8114
Santosh Kumar,(416) 431-2661
Vijay Kumar,(905) 487-9475
Dr Vinod Dr Uma Kumar,(613) 737-4113
S Kumar,(905) 281-8837
L KUMAR,(905) 860-2970
Mifan Kumar,(905) 201-0539
S Kumar,(416) 746-5671
Padin Kumar,(905) 881-3870
Shiue Kumar,(905) 499-0950
M Kumar,(905) 216-8303
P Kumar,(289) 752-6480
A Kumar,(613) 837-3511
R Kumar,(905) 846-7810
R Kumar,(416) 286-5743
E Kumar,(613) 440-2647
S Kumar,(905) 840-1426
V Kumar,(819) 681-6376
M Kumar,(416) 287-1812
D Kumar,(705) 352-2627
S Kumar,(905) 455-2242
Raj Kumar,(438) 288-3934
A Kumar,(289) 752-2619
S Kumar,(416) 284-0542
S Kumar,(705) 352-5001
V Kumar,(905) 820-6636
P KUMAR,(514) 342-4183
A Kumar,(519) 256-2012
P Kumar,(416) 412-7972

希望对您有所帮助。