在 Python 中抓取更新的 JavaScript 页面

Question

我一直在从事一个研究项目，希望从巴西 Hemeroteca 获得参考文章列表（所需的页面参考：http://memoria.bn.br/DocReader/720887x/839，需要从以下页面上的两个隐藏元素中收集：@ 987654322@)。几周前我问了一个问题，这个问题得到了回答，我能够让事情顺利进行，但现在我遇到了一个新的障碍，我不确定如何解决它。

问题是，在填写第一个表单后，页面会重定向到第二个页面，这是一个启用 JavaScript/AJAX 的页面，我需要通过单击来完成所有匹配项页面顶部的按钮。我遇到的问题是，当单击下一页按钮时，我正在处理页面上正在更新的元素，这会导致过时的元素。我尝试实现几段代码来检测何时出现这种“陈旧”效果以指示页面已更改，但这并没有提供太多运气。这是我实现的代码：

import urllib
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
import time

saveDir = "C:/tmp"

print("Opening Page...")

browser = webdriver.Chrome()
url = "http://bndigital.bn.gov.br/hemeroteca-digital/"
browser.get(url)

print("Searching for elements")

fLink = ""
fails = 0

frame_ref = browser.find_elements_by_tag_name("iframe")[0]
iframe = browser.switch_to.frame(frame_ref)
journal = browser.find_element_by_id("PeriodicoCmb1_Input")

search_journal = "Relatorios dos Presidentes dos Estados Brasileiros (BA)"
search_timeRange = "1890 - 1899"
search_text = "Milho"

xpath_form = "//input[@name=\'PesquisarBtn1\']"
xpath_journal = "//li[text()=\'"+search_journal+"\']"
xpath_timeRange = "//input[@name=\'PeriodoCmb1\' and not(@disabled)]"
xpath_timeSelect = "//li[text()=\'"+search_timeRange+"\']"
xpath_searchTerm = "//input[@name=\'PesquisaTxt1\']"

print("Locating Journal/Periodical")
journal.click()
dropDownJournal = WebDriverWait(browser, 60).until(EC.presence_of_element_located((By.XPATH, xpath_journal)))
dropDownJournal.click()
print("Waiting for Time Selection")
try:
    timeRange = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, xpath_timeRange)))
    timeRange.click()
    time.sleep(1)
    print("Locating Time Range")    
    dropDownTime = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, xpath_timeSelect)))
    dropDownTime.click()
    time.sleep(1)
except:
    print("Failed...")
print("Adding Search Term")

searchTerm = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, xpath_searchTerm)))
searchTerm.clear()
searchTerm.send_keys(search_text)
time.sleep(5)

print("Perform search")

submitButton = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, xpath_form)))
submitButton.click()

# Wait for the second page to load, pull what we need from it.
download_list = []

browser.switch_to_window(browser.window_handles[-1])
print("Waiting for next page to load...")

matches = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//span[@id=\'OcorNroLbl\']")))
print("Next page ready, found match element... counting")
countText = matches.text
countTotal = int(countText[countText.find("/")+1:])
print("A total of " + str(countTotal) + " matches have been found, standing by for page load.")
for i in range(1, countTotal+2):               
    print("Waiting for page " + str(i-1) + " to load...")
    while(fLink in download_list):
        try:
            jIDElement = browser.find_element_by_xpath("//input[@name=\'HiddenBibAlias\']")
            jPageElement = browser.find_element_by_xpath("//input[@name=\'hPagFis\']")
            fLink = "http://memoria.bn.br/DocReader/" + jIDElement.get_attribute('value') + "/" + jPageElement.get_attribute('value') + "&pesq=" + search_text         
        except:
            fails += 1
            time.sleep(1)
            if(fails == 10):
                print("Locked on a page, attempting to push to next.")
                nextPageButton = WebDriverWait(browser, 5).until(EC.presence_of_element_located((By.XPATH, "//input[@id=\'OcorPosBtn\']")))
                nextPageButton.click()                    
            #raise
        while(fLink == ""):
            jIDElement = browser.find_element_by_xpath("//input[@name=\'HiddenBibAlias\']")
            jPageElement = browser.find_element_by_xpath("//input[@name=\'hPagFis\']")
            fLink = "http://memoria.bn.br/DocReader/" + jIDElement.get_attribute('value') + "/" + jPageElement.get_attribute('value') + "&pesq=" + search_text                     
    fails = 0
    print("Link obtained: " + fLink)
    download_list.append(fLink)

    if(i != countTotal):
        print("Moving to next page...")
        nextPageButton = WebDriverWait(browser, 5).until(EC.presence_of_element_located((By.XPATH, "//input[@id=\'OcorPosBtn\']")))
        nextPageButton.click()

我试图用这个块解决两个“错误”。首先，第一页总是在循环中被跳过（IE：fLink =“”），即使那里有一个测试，我不知道为什么会发生这种情况。另一个错误是代码会完全随机地挂在特定页面上，唯一的出路就是中断代码执行。

这个块已经被修改了几次，所以我知道它不是最“优雅”的解决方案，但我开始没时间了。

Answer 1

从这件事中抽出一天的时间来考虑一下（并多睡一会），我能够弄清楚发生了什么。上面的代码有三个“大毛病”。首先是它不处理 StaleElementException 与 NoSuchElementException，后者可能在页面移动时发生。其次，循环条件迭代地检查页面不在列表中，当进入第一次运行时，允许空白条件直接加载，因为循环在第一次运行时从未执行（应该使用 do-while那里，但我做了更多的修改）。最后，我犯了一个愚蠢的错误，即只检查第一个隐藏元素是否发生了变化，而实际上那是期刊 ID，并且几乎始终保持不变。

修订开始于修改this other SO article 上的代码以实现“保留”条件，直到其中一个隐藏元素发生更改：

from selenium.common.exceptions import StaleElementReferenceException
from selenium.common.exceptions import NoSuchElementException
def hold_until_element_changed(driver, element1_xpath, element2_xpath, old_element1_text, old_element2_text):
    while True:
        try:
            element1 = driver.find_element_by_xpath(element1_xpath)
            element2 = driver.find_element_by_xpath(element2_xpath)
            if (element1.get_attribute('value') != old_element1_text) or (element2.get_attribute('value') != old_element2_text):
                break
        except StaleElementReferenceException:
            break
        except NoSuchElementException:
            return False
        time.sleep(1)
    return True    

然后我修改了原始循环条件，返回到我创建的没有内部循环的原始“for循环”计数器，而不是调用上述函数来创建“保持”，直到页面翻转，并且瞧，工作就像一个魅力。 （注意：我还提高了下一页按钮的超时时间，因为这是导致锁定条件的原因）

for i in range(1, countTotal+1):               
    print("Waiting for page " + str(i) + " to load...")
    bibxpath = "//input[@name=\'HiddenBibAlias\']"
    pagexpath = "//input[@name=\'hPagFis\']"
    jIDElement = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, bibxpath)))
    jPageElement = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, pagexpath)))
    jidtext = jIDElement.get_attribute('value')
    jpagetext = jPageElement.get_attribute('value')
    fLink = "http://memoria.bn.br/DocReader/" + jidtext + "/" + jpagetext + "&pesq=" + search_text         
    print("Link obtained: " + fLink)
    download_list.append(fLink)

    if(i != countTotal):
        print("Moving to next page...")
        nextPageButton = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//input[@id=\'OcorPosBtn\']")))
        nextPageButton.click()
        # Wait for next page to be ready
        change = hold_until_element_changed(browser, bibxpath, pagexpath, jidtext, jpagetext)
        if(change == False):
            print("Something went wrong.")

总而言之，这是一个很好的思考练习和一些有用的链接，供我在发布未来问题时考虑。谢谢！