仅抓取 PDF 嵌入 URL 中包含某些单词的段落

Question

我目前正在开发一些代码来从网站上抓取文本。我不想刮掉整个页面，而只是刮掉包含某些单词的页面部分。对于大多数使用 .find_all("p") 命令的 URL，我已经设法做到了这一点，但这不适用于定向到 PDF 的 URL。

我似乎找不到将 PDF 作为文本打开然后将文本分成段落的方法。这就是我想做的事情：首先 1）将 PDF 嵌入 URL 作为文本打开，2）将此文本分成多个段落。这样，我可以只抓取包含某些单词的段落。

下面是我目前用来为“普通”网址抓取包含某些单词的段落的代码。非常感谢任何使这项工作适用于 PDF 嵌入式 URL 的提示（例如变量 'url2'，下面的代码）！

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
import re

url1 = "https://brainybackpackers.com/best-places-for-whale-watching-in-the-world/"
url2 = "https://www.environment.gov.au/system/files/resources/7f15bfc1-ed3d-40b6-a177-c81349028ef6/files/aust-national-guidelines-whale-dolphin-watching-2017.pdf"
url = url1
req = Request(url, headers={"User-Agent": 'Mozilla/5.0'})
page = urlopen(req, timeout = 5) # Open page within 5 seconds. This line skips 'empty' websites
htmlParse = BeautifulSoup(page.read(), 'lxml') 
SearchWords = ["orca", "killer whale", "humpback"] # text must contain these words

# Check if the article text mentions the SearchWord(s). If so, continue the analysis. 
if any(word in htmlParse.text for word in SearchWords):
    textP = ""
    text = ""
    
    # Look for paragraphs ("p") that contain a SearchWord
    for word in SearchWords:
        print(word)
        for para in htmlParse.find_all("p", text = re.compile(word)): 
            textParagraph = para.get_text()
            textP = textP + textParagraph
    text= text + textP
    print(text)

Answer 1

您可以尝试的一件事是pdfminer.six package。导入后，我们可以使用pdfminer.high_level.extract_text() 函数。通过导入它，我们可以抓取一个 pdf：

import pdfminer.high_level as pdfminer

infile = "my/file/path.pdf" # file you want to turn into text

out_text = pdfminer.extract_text(infile) # extract the text to out_file var

# out_text now contains a string of your pdf contents

应该注意extract_text 函数适用于本地文件，因此我们需要将 pdf 保存到某个本地缓冲区，以便以后删除。如果您使用的是类 Unix 操作系统，我会说类似 /tmp/。

转到您的实施，我相信您会想要这样的东西：

import pdfminer.high_level as pdfminer
import requests

# get the pdf and save it
url = "https://www.environment.gov.au/system/files/resources/7f15bfc1-ed3d-40b6-a177-c81349028ef6/files/aust-national-guidelines-whale-dolphin-watching-2017.pdf"
response = requests.get(url)
pdf_name = url.split('/')[-1] # everything right of the last slash
pdf_path = "/tmp/" + pdf_name # CHANGE TO WHATEVER "BUFFER" FOLDER YOU WANT

# save the pdf locally to be used with the pdf parser
with open(pdf_path,'wb') as outfile:
    outfile.write(response.content)

# read the contents of the pdf into the out_text var
out_text = pdfminer.extract_text(pdf_path)

# out_text now contains a string of your pdf contents

从这里您应该可以随意抓取所有内容。

Answer 2

您可以阅读 PDF 并在页面中搜索您想要的内容：

# pip install pyPDF2

import io
import requests
import PyPDF2


URI = "https://www.environment.gov.au/system/files/resources/7f15bfc1-ed3d-40b6-a177-c81349028ef6/files/aust-national-guidelines-whale-dolphin-watching-2017.pdf"

r = requests.get(URI)
with io.BytesIO(r.content) as f:
  reader = PyPDF2.PdfFileReader(f)
  num_pages = reader.numPages
  
  data = []
  # place page text to data
  for page in range(num_pages):
    page_data = reader.getPage(page)
    data.append(page_data.extractText())

# look up
search_words = set(["orca", "killer whale", "humpback"])

# get pages containing your lookup
wanted_page = []
for page_contents in data:
     for word in search_words:
         if word in page_contents.lower():
             wanted_page.append(page_contents)

             
print(wanted_page)