Python requests.get 和 urllib.urlopen 向浏览器返回不同的 HTML

Question

我确信这是一个非常常见的问题，但我只是想了解正在发生的事情，以便了解情况。

问题：运行 urllib.urlopen 和 requests.get 都返回不同的 HTML 到浏览器中返回的内容。我已禁用 Javascript，以防万一它通过 AJAX 或其他方式运行，但问题仍然存在。我还检查了 XHR 调用的开发人员工具或其他方法，但一无所获。

Python 代码：

headers = {'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate, sdch',
'Accept-Language':'en-GB,en-US;q=0.8,en;q=0.6',
'Cache-Control':'max-age=0',
'Connection':'keep-alive',
'Cookie':'tgaLiveAssistantVisible=false; ASP.NET_SessionId=c5wxbmboygytpn20gubpbh0o; .ASPXANONYMOUS=lLuVr11J1Tbq9zbQPk-ZpOFMQQ4gOlePIJpfGJwAnsFNV-1gUyvJpfb2ubnIPGZg8dQnlygyrKDj1KGf14vUehOx6Iwolslm0NZZkVbfdRvv9nBZhrmAiC-2MQijnBlmZPwinqBVVVrmBmVQavyOqUVzKWr8qapeBUO4nHWgzEy_1MqPx2njtCs0DvZ3IUdz0; __utmt=1; __utma=185625580.34928295.1483869478.1484065473.1484068673.4; __utmb=185625580.17.10.1484068673; __utmc=185625580; __utmz=185625580.1484065473.3.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)',
'Host':'training.gov.au',
'Upgrade-Insecure-Requests':'1',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
}
# Load the RTO page
r = urllib.urlopen('https://training.gov.au/Organisation/Details/45001')
# r = requests.get('https://training.gov.au/Organisation/Details/45001', headers=headers)
soup = BeautifulSoup(r.read(),"html.parser")
print(soup)

在 HTML 底部返回：

<div class="display-row">
<div class="display-label">ABN:</div>
<div class="display-field-no-width">
</div>
</div>
</div>

而在浏览器中：

Request URL:http://training.gov.au/Organisation/Details/45001
Request Method:GET
Status Code:200 OK
Remote Address:117.53.170.212:80

返回 HTML：

<div class="display-row">
<div class="display-label">ABN:</div>
<div class="display-field-no-width">
<a href="http://www.abr.business.gov.au/search.aspx?SearchText=90608470113"title="View organisation on ABN Lookup"rel="external">90 608 470 113
<span class="accessibilityOnly">(external link)</span>
<img src="/Content/images/openNewWindow.png" alt=""/>
</a>
</div>
</div>

有人可以向我解释为什么会发生这种情况吗？

Answer 1

使用请求：

import requests

url = 'https://training.gov.au/Organisation/Details/45001'
r = requests.get(url).text

出来：

<div class="display-row">
                <div class="display-label">ABN:</div>
                <div class="display-field-no-width">
                <a href="http://www.abr.business.gov.au/search.aspx?SearchText=90608470113"title="View organisation on ABN Lookup"rel="external">90 608 470 113
<span class="accessibilityOnly">(external link)</span>
<img src="/Content/images/openNewWindow.png" alt=""/>
</a>

                </div>
            </div>