如何从整个 json 响应 api 调用中获取特定项目

Question

我只想从整个 json 数据中获取 Symbol 和 Company Name 项目，但得到 所有数据。如何获取上述数据并存储在 pandas DataFrame。

Base_url

我的代码：

import requests
import pandas as pd

params = {
    'sectorID': 'All',
    '_': '1630217365368'}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'}


def main(url):
    with requests.Session() as req:
        req.headers.update(headers)
        sym = []
        name = []
        r = req.get(url, params=params, headers =headers)
        for item in r.json()['data']:
            print(item)
        #     sym.append(item['symbol']),
        #     name.append(item['lonaName'])

        # df = pd.DataFrame(sym, name, columns=[["Symble","Company name"]])
        # print(df)
        

main('https://www.saudiexchange.sa/wps/portal/tadawul/market-participants/issuers/issuers-directory/!ut/p/z1/04_Sj9CPykssy0xPLMnMz0vMAfIjo8zi_Tx8nD0MLIy8DTyMXAwczVy9vV2cTY0MnEz1w8EKjIycLQwtTQx8DHzMDYEK3A08A31NjA0CjfWjSNLv7ulnbuAY6OgR5hYWYgzUQpl-AxPi9BvgAI4GhPVHgZXgCwFUBVi8iFcByA9gBXgcWZAbGhoaYZDpma6oCABqndOv/p0/IZ7_NHLCH082KOAG20A6BDUU6K3082=CZ6_NHLCH082K0H2D0A6EKKDC520B5=N/')

Answer 1

如果您将数据存储在 pandas DataFrame 中并稍后处理它会更快。

示例代码：

import requests
import pandas as pd

params = {
    'sectorID': 'All',
    '_': '1630217365368'}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'}


def main(url):
    with requests.Session() as req:
        req.headers.update(headers)
        r = req.get(url, params=params, headers =headers)
        data = r.json()['data']
        df_main = pd.DataFrame(data)
        df_min = df_main.iloc[:, 0:2]
        df_min.columns = ['Symbol', 'Company name']
        print(df_min)
        

main('https://www.saudiexchange.sa/wps/portal/tadawul/market-participants/issuers/issuers-directory/!ut/p/z1/04_Sj9CPykssy0xPLMnMz0vMAfIjo8zi_Tx8nD0MLIy8DTyMXAwczVy9vV2cTY0MnEz1w8EKjIycLQwtTQx8DHzMDYEK3A08A31NjA0CjfWjSNLv7ulnbuAY6OgR5hYWYgzUQpl-AxPi9BvgAI4GhPVHgZXgCwFUBVi8iFcByA9gBXgcWZAbGhoaYZDpma6oCABqndOv/p0/IZ7_NHLCH082KOAG20A6BDUU6K3082=CZ6_NHLCH082K0H2D0A6EKKDC520B5=N/')

输出：

Answer 2

要从网站获取所有数据，您可以使用他们的 API：

import requests
import pandas as pd

url = "https://www.saudiexchange.sa/tadawul.eportal.theme.helper/TickerServlet"

data = requests.get(url).json()
# print(json.dumps(data, indent=4))

df = pd.json_normalize(data["stockData"])
print(df)

打印：

    pk_rf_company     companyShortNameEn       companyShortNameAr                                             companyLongNameEn                                     companyLongNameAr highPrice lowPrice  noOfTrades previousClosePrice todaysOpen transactionDate      turnOver  volumeTraded  aveTradeSize  change  changePercent  lastTradePrice transactionDateStr
0            4700       Alkhabeer Income             الخبير للدخل                     Al Khabeer Diversified Income Traded Fund                   صندوق الخبير للدخل المتنوع المتداول      None     None         308               None       None            None  1.293560e+06        142791        463.61    0.01           0.11            9.07               None
1            2030                  SARCO                  المصافي                                   Saudi Arabia Refineries Co.                         شركة المصافي العربية السعودية      None     None         877               None       None            None  1.352797e+07         83391         95.09   -0.40          -0.25          162.20               None
2            2222           SAUDI ARAMCO          أرامكو السعودية                                         Saudi Arabian Oil Co.                           شركة الزيت العربية السعودية      None     None        4054               None       None            None  6.034732e+07       1731463        427.10    0.05           0.14           34.90               None


...and so on.

---

仅获取符号/公司名称：

print(df[["pk_rf_company", "companyLongNameEn"]])

    pk_rf_company                                             companyLongNameEn
0            4700                     Al Khabeer Diversified Income Traded Fund
1            2030                                   Saudi Arabia Refineries Co.
2            2222                                         Saudi Arabian Oil Co.

...and so on.

Answer 3

您需要修复创建数据框的方式：

import requests
import pandas as pd

params = {
    'sectorID': 'All',
    '_': '1630217365368'}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'}


def main(url):
    with requests.Session() as req:
        req.headers.update(headers)
        sym = []
        name = []
        r = req.get(url, params=params, headers =headers)
        for item in r.json()['data']:
            # print(item)
            sym.append(item['symbol']),
            name.append(item['lonaName'])

        df = pd.DataFrame({'symbol':sym , 'longName':name})
        print(df)
        

main('https://www.saudiexchange.sa/wps/portal/tadawul/market-participants/issuers/issuers-directory/!ut/p/z1/04_Sj9CPykssy0xPLMnMz0vMAfIjo8zi_Tx8nD0MLIy8DTyMXAwczVy9vV2cTY0MnEz1w8EKjIycLQwtTQx8DHzMDYEK3A08A31NjA0CjfWjSNLv7ulnbuAY6OgR5hYWYgzUQpl-AxPi9BvgAI4GhPVHgZXgCwFUBVi8iFcByA9gBXgcWZAbGhoaYZDpma6oCABqndOv/p0/IZ7_NHLCH082KOAG20A6BDUU6K3082=CZ6_NHLCH082K0H2D0A6EKKDC520B5=N/')

    symbol                                           longName
0     1330                 Abdullah A. M. Al-Khodari Sons Co.
1     4001                     Abdullah Al Othaim Markets Co.
2     4191  Abdullah Saad Mohammed Abo Moati for Bookstore...
3     1820  Abdulmohsen Alhokair Group for Tourism and Dev...
4     2330                         Advanced Petrochemical Co.
..     ...                                                ...
199   3020                                  Yamama Cement Co.
200   3060                                   Yanbu Cement Co.
201   2290                   Yanbu National Petrochemical Co.
202   3007                     Zahrat Al Waha for Trading Co.
203   2240                    Zamil Industrial Investment Co.