无法使用scrapy抓取数据

Question

我曾使用各种方法从angel.co 中抓取数据

但仍然无法使用每次获取空列表来抓取数据

results = self.driver.find_elements_by_css_selector(".results > div") for result in results: name = result.find_element_by_css_selector(".name") print(name.text)

另一个是

soup = BeautifulSoup(response.body) val = soup.findAll('div.name')

for post in response.xpath('.//div[@class="base startup"]'): item = {} item['title'] =post.xpath('.//div[@class="name"]//text()').extract()[0] print item 如果有其他建议，这些都是我尝试过的，然后帮助我抓取页面 完整蜘蛛的链接是 here

Answer 1

您需要wait 才能加载搜索结果，然后才能提取它们：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(self.driver, 10)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".startup")))

results = self.driver.find_elements_by_css_selector(".results > div")
for result in results:
    name = result.find_element_by_css_selector(".name")
    print(name.text)