使用 DDG 获取响应 200 而不是 <418 I'm a Teapot>答案

【问题标题】：Get response 200 instead of <418 I'm a Teapot>, using DDG使用 DDG 获取响应 200 而不是 <418 I'm a Teapot>
【发布时间】：2020-10-21 18:43:42
【问题描述】：

前几天我试图从 DDG 抓取搜索结果，但我不断收到响应 418。我怎样才能让它响应 200 或从中获取结果？这是我的代码。

import requests
from bs4 import BeautifulSoup
import urllib
    
while True:
    
    query = input("Enter Search Text: ")

    a = query.replace(' ', '+')

    url = 'https://duckduckgo.com/?q=random' +a
    
    headers = {"User-Agent": "Mozilla/5.0 (Linux; Android 6.0.1; SHIELD Tablet K1 Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Safari/537.36"}
    
    r = requests.get(url)
    print(r)

    soup = BeautifulSoup(r.content,'lxml')
    
    tags = soup.findAll('h2', class_="result__title")
    print(tags)```

【问题讨论】：

标签： python web-scraping beautifulsoup python-requests duckduckgo

【解决方案1】：

您可以使用本地化 (https://html.duckduckgo.com/html/) 的仅 HTML 版本的 DDG 来获取结果。

例如：

import requests
from bs4 import BeautifulSoup


url = 'https://html.duckduckgo.com/html/'
params = {'q': 'python'}
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

soup = BeautifulSoup(requests.get(url, params=params, headers=headers).content, 'html.parser')

for t in soup.select('h2.result__title'):
    print(t.get_text(strip=True))

打印：

LearnPythonOnline - Start Today & Change Your LifeAdViewing ads is privacy protected by DuckDuckGo. Ad clicks are managed by Microsoft's ad network (more info).
PythonOnline - FreePythonIntro by DataCampAdViewing ads is privacy protected by DuckDuckGo. Ad clicks are managed by Microsoft's ad network (more info).
Welcome toPython.org
How Modulo (%) works inPython: Explained with 6 Examples
Python(programming language) - Wikipedia
python- How do I pass a variable by reference? - Stack ...
PythonTutorial - W3Schools
Python- Basic Operators - Tutorialspoint
DownloadPython|Python.org
PythonFor Beginners |Python.org
Introduction toPython- W3Schools
PythonReleases for Windows |Python.org
PythonReleasePython3.8.2 |Python.org
ThePythonTutorial —Python3.8.3 documentation
Python
Python- Free download and software reviews - CNET ...
ThePythonLanguage Reference —Python3.8.3 documentation
After 19 Years,PythonMay Finally Get a Pattern Matching ...
PythonTutorial - Tutorialspoint
Python| snake group | Britannica
FrontPage -PythonWiki
BeginnersGuide -PythonWiki
syntax - What is :: (double colon) inPythonwhen ...
PythonOperators - W3Schools
PythonTutorial: LearnPythonFor Free | Codecademy
uuid — UUID objects according to RFC 4122 —Python3.8.3 ...
LearnPython- Free InteractivePythonTutorial
PythonString find() Method - W3Schools
PythonPIP - W3Schools
How to Use thePythonor Operator - RealPython
Python2.7.18 documentation
What is the result of % inPython? - Stack Overflow

【讨论】：