BeautifulSoup 4 HTML Web Scraping - 查找 Mailto 链接并导出到电子表格

Question

我正在尝试从该索引页面抓取所有电子邮件地址 - http://www.uschess.org/assets/msa_joomla/AffiliateSearch/clubresultsnew.php?st=AL

我修改了一个python脚本来定义字符串，用BS4解析内容并将每个唯一地址保存到一个xls文件中：

import requests
from bs4 import BeautifulSoup
import xlwt

wb = xlwt.Workbook()
ws = wb.add_sheet('Emails')
ws.write(0,0,'Emails')

emailList= []
r=0

#add url of the page you want to scrape to urlString
urlString='http://www.uschess.org/assets/msa_joomla/AffiliateSearch/clubresultsnew.php?st=AL'

#function that extracts all emails from a page you provided and stores them in a list
def emailExtractor(urlString):
getH=requests.get(urlString)
h=getH.content
soup=BeautifulSoup(h,'html.parser')
mailtos = soup.select('a[href^=mailto]')
for i in mailtos:
    href=i['href']
    try:
        str1, str2 = href.split(':')
    except ValueError:
        break
    
    emailList.append(str2)
    
   emailExtractor(urlString)
   #adding scraped emails to an excel sheet
   for email in emailList:
   r=r+1
   ws.write(r,0,email)

    wb.save('emails.xls')

xls 文件按预期导出，但没有电子邮件值。如果有人能解释为什么或如何简化此解决方案，将不胜感激！

Answer 1

因为电子邮件受到保护。我只添加电子邮件抓取部分。并且不添加 excel 部分，因为您对此没有问题。将受保护的电子邮件转换为文本信用转到https://stackoverflow.com/a/36913154/7518304

emailList= []
r=0

#add url of the page you want to scrape to urlString
urlString='http://www.uschess.org/assets/msa_joomla/AffiliateSearch/clubresultsnew.php?st=AL'
def decodeEmail(e): #https://stackoverflow.com/a/36913154/7518304
    de = ""
    k = int(e[:2], 16)

    for i in range(2, len(e)-1, 2):
        de += chr(int(e[i:i+2], 16)^k)

    return de

#function that extracts all emails from a page you provided and stores them in a list
def emailExtractor(urlString):
    getH=requests.get(urlString)
    h=getH.content
    soup=BeautifulSoup(h,'html.parser')
    mailtos = soup.select('a[href]')
    for i in mailtos:
        href=i['href']
        if "email-protect" in href:
            emailList.append(decodeEmail(href.split("#")[1]))

emailExtractor(urlString)
emailList

Answer 2

您可以为此使用pandas。完整代码如下：

from bs4 import BeautifulSoup
import requests
import re
import pandas as pd

urlString = 'http://www.uschess.org/assets/msa_joomla/AffiliateSearch/clubresultsnew.php?st=AL'


# function that extracts all emails from a page you provided and stores them in a list
def emailExtractor(urlString):
   
    emailList = []
    getH = requests.get(urlString)
    h = getH.content
    soup = BeautifulSoup(h, 'html.parser')

    mailtos = soup.find_all('a')

    href_lst = []
    for i in mailtos:
        href_lst.append(i['href'])

    for href in href_lst:
        if ':' in href:
            emailList.append(href)
    print(emailList)

    s = pd.Series(emailList)

    s = s.rename('Emails')
    
    s.to_excel('D:\\Emails.xls',index=False)

emailExtractor(urlString)

输出：

['http://msa.uschess.org/AffDtlMain.php?T6006791', 'https://alabamachess.org', 'http://msa.uschess.org/AffDtlMain.php?A6029262', 'http://www.caesarchess.com/', 'http://msa.uschess.org/AffDtlMain.php?A6045660', 'http://msa.uschess.org/AffDtlMain.php?H6046485', 'http://msa.uschess.org/AffDtlMain.php?A6040580']

Excel 工作表截图：

如果您希望将链接作为hyperlinks 输出到Excel 工作表（单击链接后将被重定向到网站），请将emailList.append(href) 更改为emailList.append('=HYPERLINK("'+href+'")')。 同时，您还应该将文件扩展名更改为.xlsx。只有这样，您才能以超链接的形式获取链接。

输出：

希望这会有所帮助！