当 HTML 不变时，如何使用 Python 进行网络抓取？

Question

我目前正在使用 Selenium 和 BeautifulSoup 来尝试从 Google 财经中抓取财务报表数据。例如：

http://www.google.com/finance?q=GOOG&fstype=ii

打开 Google 损益表。当我让 Selenium 点击页面顶部的“Balance Statement”和“Cash Flow”按钮时，页面上的图表和表格发生了变化，但 url 没有改变，当我拉页面源时，它是损益表表的原始页面。我的代码贴在下面：

driver = webdriver.Firefox()
driver.get("http://www.google.com/finance?q=" + ticker[0] + "&fstype=ii")

url1 = driver.page_source
soup1 = BeautifulSoup(url1)

element = driver.find_element_by_xpath('//*[@id=":1"]/a/b/b')
element.click()

driver.implicity_wait(3.0)
url2 = driver.page_source
soup2 = BeautifulSoup(url2)

element = driver.find_element_by_xpath('//*[@id=":2"]/a/b/b')
element.click()

driver.implicity_wait(3.0)
url3 = driver.page_source
soup3 = BeautifulSoup(url3)

driver.quit()

感谢任何帮助。谢谢。

Answer 1

这里不需要BeautifulSoup HTML 解析器。 Selenium 本身在navigating on the page and getting elements by almost everything you can imagine 中已经足够强大了。

您需要的表格数据位于div 元素中，具有不同的ids。激活每个选项卡并从适当的 div 中获取数据。

这是一个打印出所有选项卡内表格标题的示例：

from selenium import webdriver

def print_header(element):
    table = element.find_element_by_id('fs-table')
    for row in table.find_elements_by_tag_name('th'):
        print row.text


driver = webdriver.Firefox()
driver.get('http://www.google.com/finance?q=GOOG&fstype=ii')

print_header(driver.find_element_by_id('incinterimdiv'))
print "----"

# activate Balance Sheet
element = driver.find_element_by_xpath('//*[@id=":1"]/a/b/b')
element.click()

print_header(driver.find_element_by_id('balinterimdiv'))
print "----"

# activate Cash Flow
element = driver.find_element_by_xpath('//*[@id=":2"]/a/b/b')
element.click()

print_header(driver.find_element_by_id('casinterimdiv'))

driver.quit()

打印：

In Millions of USD (except for per share items)
3 months ending 2014-03-31
3 months ending 2013-12-31
3 months ending 2013-09-30
3 months ending 2013-06-30
3 months ending 2013-03-31
----
In Millions of USD (except for per share items)
As of 2014-03-31
As of 2013-12-31
As of 2013-09-30
As of 2013-06-30
As of 2013-03-31
----
In Millions of USD (except for per share items)
3 months ending 2014-03-31
12 months ending 2013-12-31
9 months ending 2013-09-30
6 months ending 2013-06-30
3 months ending 2013-03-31