网页抓取，使用机械化获取整个页面

Question

我的目标是从页面中获取所有项目。不过，我只得到了 25 个中的前 10 个。我认为它与表有关，我认为它是某种类型的小部件？我是初学者，还在学习基础知识。

import mechanize,time
from bs4 import BeautifulSoup

br = mechanize.Browser()  
br.set_handle_robots(False)  
br.addheaders = [("User-agent", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13")]  

sign_in = br.open('https://sellercentral.amazon.com/gp/homepage.html?')  

br.select_form(name="signinWidget")  
br["username"] = 'spam' 
br["password"] = 'eggs'
logged_in = br.submit() 

orders_html = br.open("https://sellercentral.amazon.com/hz/inventory/ref=ag_invmgr_dnav_xx_?tbla_myitable=sort:{%22sortOrder%22%3A%22DESCENDING%22%2C%22sortedColumnId%22%3A%22date%22};search:;pagination:1;")

print('Login complete...')
time.sleep(5)

soup = BeautifulSoup(orders_html,'html.parser')
partNums = soup.find_all('span', {'class': 'mt-text-content mt-table-main'})

 print(partNums)

 for part in partNums:
  print(part.text)  

 print('Process Complete.')

Answer 1

您可以通过br.response().read()获取页面的HTML

orders_page = br.open("https://sellercentral.amazon.com/hz/inventory/ref=ag_invmgr_dnav_xx_?tbla_myitable=sort:{%22sortOrder%22%3A%22DESCENDING%22%2C%22sortedColumnId%22%3A%22date%22};search:;pagination:1;")  # loads page
orders_html = br.response().read()  # saves page source