网络抓取确实显示了不同的结果-Python

Question

我正在尝试为美国的工作搜索 Indeed。我有以下代码

url = "https://www.indeed.com/jobs?q=&l=United+states"
headers = {'Indeed': 'Mozilla/5.0'}
page = requests.get(url, headers=headers)
soup1 = BeautifulSoup(page.text, 'html.parser') 

location=[]
title = []
for item in soup1.findAll('div', {'class': 'location accessible-contrast-color-location'}):
            result = [item.get_text(strip=True, separator=" ")]
            location.append(result)
for item in soup1.findAll('h2', {'class': 'title'}):
            result = [item.get_text(strip=True, separator=" ")]
            title.append(result)

当我打印位置和标题时，结果与网站上实际显示的不同。

print(title[1]) should give **Special Agent (FBI)**

where as it gives **Amazon Prime Shopper**

你能告诉我我哪里出错了

Answer 1

import requests
import re

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0'
}


def main(url):
    r = requests.get(url)
    match = re.finditer(r"loc:'(.+?)'.title:'(.+?)'", r.text)
    for item in match:
        print(item.group(2), item.group(1))


main("https://www.indeed.com/jobs?q=&l=United+states")

输出：

Work-at-Home - Customer Care Support - 90 Day Contract Remote',country:'US',zip:'',city:'Remote     
QA Tester: Work from Home (contract) Baton Rouge, LA 70820',country:'US',zip:'',city:'Baton Rouge   
Professional Test Scorer Remote',country:'US',zip:'',city:'Remote
Work part-time every other weekend. Gross pay is $230 a day. Haul is from Five Star Dairy United States',country:'US',zip:'',city:'
Remote Copywriter Remote',country:'US',zip:'',city:'Remote
Freelance Subtitle Translators & Captioners New York, NY 10021',country:'US',zip:'',city:'New York
Amazon Customer Support United States',country:'US',zip:'',city:'
FIREFIGHTER 2112 (Application period begins February 11, 2020) City of Los Angeles, CA',country:'US',zip:'',city:'City of Los Angeles
COVID-19 - 100% Remote Call Center Agent Frankfort, KY',country:'US',zip:'',city:'Frankfort
Functional Game Tester Eden Prairie, MN',country:'US',zip:'',city:'Eden Prairie