【发布时间】:2016-04-18 12:00:30
【问题描述】:
我刚刚编写了一个简单的网页抓取脚本,为我提供了特定网站页面上的所有剧集链接。脚本运行良好,但现在它坏了。我没有改变任何东西。
试试这个网址(用于抓取):- http://www.crunchyroll.com/tabi-machi-late-show
现在,脚本在中途运行并给我一个错误说明,“在缓存中找不到元素 - 页面可能在查找后发生了变化”
我在网上查了一下,有人说在某些地方使用“隐式等待”命令。我做到了,仍然没有运气。
更新:我在降级桌面上尝试了这个脚本,它在那里工作没有任何问题。
这是我的脚本:-
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import os
import time
from subprocess import Popen
#------------------------------------------------
try:
Link = raw_input("Please enter your Link : ")
if not Link:
raise ValueError('Please Enter A Link To The Anime Page. This Application Will now Exit in 5 Seconds.')
except ValueError as e:
print(e)
time.sleep(5)
exit()
print 'Analyzing the Page. Hold on a minute.'
driver = webdriver.Firefox()
driver.get(Link)
assert "Crunchyroll" in driver.title
driver.implicitly_wait(5) # <-- I tried removing this lines as well. No luck.
elem = driver.find_elements_by_xpath("//*[@href]")
driver.implicitly_wait(10) # <-- I tried removing this lines as well. No luck.
text_file = open("BatchLink.txt", "w")
print 'Fetching The Links, please wait.'
for elem in elem:
x = elem.get_attribute("href")
#print x
text_file.write(x+'\n')
print 'Links have been fetched. Just doing the final cleaning now.'
text_file.close()
CleanFile = open("queue.txt", "w")
with open('BatchLink.txt') as f:
mylist = f.read().splitlines()
#print mylist
with open('BatchLink.txt', 'r') as inF:
for line in inF:
if 'episode' in line:
CleanFile.write(line)
print 'Please Check the file named queue.txt'
CleanFile.close()
os.remove('BatchLink.txt')
driver.close()
这是错误的屏幕截图(可能会有所帮助): http://i.imgur.com/SaANlsg.png
【问题讨论】:
标签: python-2.7 selenium web-scraping