需要帮助循环浏览硒上的网页链接

Question

我想循环浏览this 网页中的一些链接，但我不知道该怎么做。我想从中获取链接的部分是这个：

这基本上是页脚。因此，对于我正在尝试做的事情的简要描述，我想刮掉表中列出的证券的所有链接，然后循环浏览页脚，以便我可以更改页面并从这些页面中刮掉链接：

所以目前这是第 1 页，我可以抓取我想要的内容，但我不知道如何进入第二页并继续该过程。稍后我会告诉你原因：

from selenium import webdriver
from bs4 import BeautifulSoup as bs
import time
from requests import get

driver = webdriver.Firefox()
driver.get("http://www.nse.com.ng/market-data/trading-statistics/equities")

time.sleep(30)

html = driver.page_source
soup = bs(html,"html.parser")
driver.close()

以上是我的代码。现在，如果我想找到页脚部分，我会这样做：

foot = soup.find("ul",class_="pagination no-top-pad").find_all("a")

它真正返回页脚/页面列表：

但是，如您所见，它并没有真正使用链接。每个页面的href 下方都有一个简单的"#" 符号。所以我不能只是获取链接并循环浏览它们。这是我的问题。你如何建议我从第 1 页移到第 2 页等等？任何帮助将不胜感激。

Answer 1

页面正在通过 JavaScript 从外部 URL 加载数据，您可以使用此示例将页面加载到数据框中：

import pandas as pd


url = 'http://www.nse.com.ng/REST/api/statistics/equities/?market=&sector=&orderby=&pageSize=10&pageNo={page}'

for page in range(0, 10):
    df = pd.read_json(url.format(page=page))
    print(df)

打印：

...

   $id   Id      Symbol  ...      Market              Sector        Company2
0    1  123        JULI  ...        ASeM            SERVICES      JULI [MRF]
1    2   96      LASACO  ...  Main Board  FINANCIAL SERVICES         LASACO 
2    3  150    LAWUNION  ...  Main Board  FINANCIAL SERVICES  LAWUNION [DIP]
3    4  116  LEARNAFRCA  ...  Main Board            SERVICES     LEARNAFRCA 
4    5   92  LINKASSURE  ...  Main Board  FINANCIAL SERVICES     LINKASSURE 
5    6   48   LIVESTOCK  ...  Main Board         AGRICULTURE      LIVESTOCK 
6    7  106     MANSARD  ...  Main Board  FINANCIAL SERVICES        MANSARD 
7    8  141    MAYBAKER  ...  Main Board          HEALTHCARE       MAYBAKER 
8    9   23    MBENEFIT  ...  Main Board  FINANCIAL SERVICES       MBENEFIT 
9   10   47   MCNICHOLS  ...        ASeM      CONSUMER GOODS      MCNICHOLS 

[10 rows x 16 columns]
   $id   Id      Symbol  ...         Market             Sector          Company2
0    1  153  MEDVIEWAIR  ...     Main Board           SERVICES  MEDVIEWAIR [BMF]
1    2  146       MEYER  ...     Main Board   INDUSTRIAL GOODS            MEYER 
2    3   89       MOBIL  ...     Main Board        OIL AND GAS            MOBIL 
3    4   98     MORISON  ...     Main Board         HEALTHCARE          MORISON 
4    5   18         MRS  ...     Main Board        OIL AND GAS              MRS 
5    6  162        MTNN  ...  Premium Board                ICT             MTNN 
6    7  148   MULTITREX  ...     Main Board     CONSUMER GOODS   MULTITREX [BMR]
7    8   49  MULTIVERSE  ...     Main Board  NATURAL RESOURCES       MULTIVERSE 
8    9  143       NAHCO  ...     Main Board           SERVICES            NAHCO 
9   10  158      NASCON  ...     Main Board     CONSUMER GOODS           NASCON 

[10 rows x 16 columns]


...