从包含类名但不包含 ID 的美丽 Soup 对象中获取所有类

Question

main_url = 'https://www.indeed.com'
url_to_list = ["https://www.indeed.com/jobs?q=Data%20Scientist&advn=9634682979760233"]

for tab_number in range(1,101):
    url_temp = f'https://www.indeed.com/jobs?q=Data+Scientist&start={tab_number}0'
    url_to_list.append(url_temp)

url = url_to_list[0]
resp = requests.get(url)
soup = bs4.BeautifulSoup(resp.text, 'html.parser')
#class_titles = soup.findAll("div", {"class": "title"}) #Contains Descrition URL and Title 
#class_sjcl = soup.findAll("div", {"class": "sjcl"}) #contains Ratings, company name, area
#class_salarySnippet_holisticSalar = soup.findAll("div",{"class": "salarySnippet holisticSalary"}) #contains pay
main_class = soup.findAll('div', class_='jobsearch-SerpJobCard unifiedRow row result clickcard')

这应该是网络爬虫，但我被卡住了。我希望它给我一个标题、描述等。困难的部分是，如果没有可用的薪水，我希望它返回 None。一切都包含在class_='jobsearch-SerpJobCard unifiedRow row result clickcard'

但是它在每个容器中都有不同的 ID，所以 findAll() 方法返回空。我一直在尝试其他类似问题的解决方案，但它一直返回一个空列表。

Answer 1

此脚本将遍历页面并获取有关每个结果的一些信息。如果结果没有，例如薪水 - 它会将'-' 放入数据中（如果需要，可以将其更改为None）：

import requests
from bs4 import BeautifulSoup

base_url = 'https://www.indeed.com/jobs?q=Data+Scientist&start={}'

data = []
for p in range(0, 100, 10):
    print('Scraping results {}...'.format(p))
    soup = BeautifulSoup(requests.get(base_url.format(p)).content, 'html.parser')

    for result in soup.select('.result'):
        title = result.select_one('.title').get_text(strip=True)
        job_url = result.select_one('.title a')['href']
        company = result.select_one('.company').get_text(strip=True) if result.select_one('.company') else '-'
        rating = result.select_one('.ratingsDisplay').get_text(strip=True) if result.select_one('.ratingsDisplay') else '-'
        location = result.select_one('.location').get_text(strip=True) if result.select_one('.location') else '-'
        salary = result.select_one('.salary').get_text(strip=True) if result.select_one('.salary') else '-'

        data.append((title, company, rating, location, salary, job_url))

# just print the data for now:    
print('{:<65} {:<50} {:<10} {:<65} {:<10}'.format(*'Title Company Rating Location Salary'.split()))
for row in data:
    print('{:<65} {:<50} {:<10} {:<65} {:<10}'.format(*row[:-1]))

打印：

Scraping results 0...
Scraping results 10...
Scraping results 20...
...

Title                                                             Company                                            Rating     Location                                                          Salary    
Data Scientist – Pricing Optimization                             Delta                                              4.2        Atlanta, GA                                                       -         
Data Scientist - Entry Level                                      Numerdox                                           -          Sacramento, CA                                                    -         
Data Scientist                                                    RTI International                                  3.7        Durham, NC 27709                                                  -         
Entry Level Data Scientist                                        IBM                                                3.9        United States                                                     -         
Data Scientist - Economic Data                                    Zillow                                             3.8        Seattle, WA 98101(Downtown area)                                  -         
Data Scientist                                                    FCA                                                4.0        Detroit, MI 48201                                                 -         
Data Scientist, Analytics (University Grad)                       Facebook                                           4.2        New York, NY 10017                                                -         
Data Scientist                                                    Oath Inc                                           3.8        Champaign, IL                                                     -         
...and so on.