为什么用 BeautifulSoup 看不到这个 html 表？答案

【问题标题】：Why this html table is not seen with BeautifulSoup?为什么用 BeautifulSoup 看不到这个 html 表？
【发布时间】：2023-03-18 09:29:01
【问题描述】：

我正在尝试抓取下表

url = https://www.bitmex.com/app/settlementHistory?start=100

但在尝试了多种方式后 - 没有成功。

这是我尝试过的：

尝试 1

import pandas as pd

pd.read_html(url)

这将返回一个空列表 1（不包含有问题的表）。

尝试 2

我尝试遍历表行 (tr)，但是当我这样做时：

from bs4 import BeautifulSoup

response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
soup.find_all('tr')

我没有得到实际表格的表格行...

尝试 3

我尝试查找所有带有soup.find_all('table') 的表，但结果与上述类似。

谁能帮我把它加载到pd.DataFrame，更重要的是解释为什么以前的尝试根本不起作用？

【问题讨论】：

标签： python pandas beautifulsoup html-table

【解决方案1】：

数据是从外部 URL 以 Json 格式加载的。将其加载到数据框的最简单方法是使用pd.read_json()：

df = pd.read_json(
    "https://www.bitmex.com/api/v1/settlement?count=100&start=100&reverse=true&filter="
)
print(df)

打印：

                   timestamp      symbol settlementType  settledPrice  optionStrikePrice  optionUnderlyingPrice  bankrupt  taxBase  taxRate
0  2019-10-04 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             8500.0                8144.71       NaN      NaN      NaN
1  2019-09-27 12:00:00+00:00      TRXU19     Settlement      0.000002                NaN                    NaN       NaN      NaN      NaN
2  2019-09-27 12:00:00+00:00      EOSU19     Settlement      0.000346                NaN                    NaN       NaN      NaN      NaN
3  2019-09-27 12:00:00+00:00      ADAU19     Settlement      0.000005                NaN                    NaN       NaN      NaN      NaN
4  2019-09-27 12:00:00+00:00      BCHU19     Settlement      0.026730                NaN                    NaN       NaN      NaN      NaN
5  2019-09-27 12:00:00+00:00      XRPU19     Settlement      0.000030                NaN                    NaN       NaN      NaN      NaN
6  2019-09-27 12:00:00+00:00      LTCU19     Settlement      0.006842                NaN                    NaN       NaN      NaN      NaN
7  2019-09-27 12:00:00+00:00      ETHU19     Settlement      0.020660                NaN                    NaN       NaN      NaN      NaN
8  2019-09-27 12:00:00+00:00      XBTU19     Settlement   7979.550000                NaN                    NaN       NaN      NaN      NaN
9  2019-09-27 12:00:00+00:00   XBT7D_D95     Settlement      0.022190             9750.0                7979.55       NaN      NaN      NaN
10 2019-09-27 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            10750.0                7979.55       NaN      NaN      NaN
11 2019-09-20 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             9750.0               10151.10       NaN      NaN      NaN
12 2019-09-20 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            10750.0               10151.10       NaN      NaN      NaN
13 2019-09-13 12:00:00+00:00   XBT7D_D95     Settlement      0.000000            10250.0               10301.01       NaN      NaN      NaN
14 2019-09-13 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            11500.0               10301.01       NaN      NaN      NaN
15 2019-09-06 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             9000.0               10874.27       NaN      NaN      NaN
16 2019-09-06 12:00:00+00:00  XBT7D_U105     Settlement      0.008040            10000.0               10874.27       NaN      NaN      NaN
17 2019-08-30 12:00:00+00:00   XBT7D_D95     Settlement      0.001780             9750.0                9579.34       NaN      NaN      NaN
18 2019-08-30 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            10750.0                9579.34       NaN      NaN      NaN
19 2019-08-23 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             9500.0               10153.85       NaN      NaN      NaN
20 2019-08-23 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            10500.0               10153.85       NaN      NaN      NaN
21 2019-08-16 12:00:00+00:00   XBT7D_D95     Settlement      0.012340            11250.0               10014.39       NaN      NaN      NaN
22 2019-08-16 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            12250.0               10014.39       NaN      NaN      NaN
23 2019-08-09 12:00:00+00:00   XBT7D_D95     Settlement      0.000000            10000.0               11722.96       NaN      NaN      NaN
24 2019-08-09 12:00:00+00:00  XBT7D_U105     Settlement      0.006170            11000.0               11722.96       NaN      NaN      NaN
25 2019-08-02 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             9250.0               10505.43       NaN      NaN      NaN
26 2019-08-02 12:00:00+00:00  XBT7D_U105     Settlement      0.002430            10250.0               10505.43       NaN      NaN      NaN
27 2019-07-26 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             9750.0                9809.83       NaN      NaN      NaN
28 2019-07-26 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            10750.0                9809.83       NaN      NaN      NaN
29 2019-07-19 12:00:00+00:00   XBT7D_D95     Settlement      0.006720            11000.0               10307.18       NaN      NaN      NaN
30 2019-07-19 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            12250.0               10307.18       NaN      NaN      NaN
31 2019-07-12 12:00:00+00:00   XBT7D_D95     Settlement      0.000000            10750.0               11685.10       NaN      NaN      NaN
32 2019-07-12 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            11750.0               11685.10       NaN      NaN      NaN
33 2019-07-05 12:00:00+00:00   XBT7D_D95     Settlement      0.000000            11250.0               11256.95       NaN      NaN      NaN
34 2019-07-05 12:00:00+00:00  XBT7D_U105     Settlement      0.000000            12500.0               11256.95       NaN      NaN      NaN
35 2019-06-28 12:00:00+00:00      TRXM19     Settlement      0.000003                NaN                    NaN       NaN      NaN      NaN
36 2019-06-28 12:00:00+00:00      EOSM19     Settlement      0.000499                NaN                    NaN       NaN      NaN      NaN
37 2019-06-28 12:00:00+00:00      ADAM19     Settlement      0.000007                NaN                    NaN       NaN      NaN      NaN
38 2019-06-28 12:00:00+00:00      BCHM19     Settlement      0.035550                NaN                    NaN       NaN      NaN      NaN
39 2019-06-28 12:00:00+00:00      XRPM19     Settlement      0.000034                NaN                    NaN       NaN      NaN      NaN
40 2019-06-28 12:00:00+00:00      LTCM19     Settlement      0.009907                NaN                    NaN       NaN      NaN      NaN
41 2019-06-28 12:00:00+00:00      ETHM19     Settlement      0.025620                NaN                    NaN       NaN      NaN      NaN
42 2019-06-28 12:00:00+00:00      XBTM19     Settlement  11918.300000                NaN                    NaN       NaN      NaN      NaN
43 2019-06-28 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             9250.0               11918.30       NaN      NaN      NaN
44 2019-06-28 12:00:00+00:00  XBT7D_U105     Settlement      0.014000            10250.0               11918.30       NaN      NaN      NaN
45 2019-06-21 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             8000.0                9821.26       NaN      NaN      NaN
46 2019-06-21 12:00:00+00:00  XBT7D_U105     Settlement      0.010910             8750.0                9821.26       NaN      NaN      NaN
47 2019-06-14 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             7500.0                8290.34       NaN      NaN      NaN
48 2019-06-14 12:00:00+00:00  XBT7D_U105     Settlement      0.000490             8250.0                8290.34       NaN      NaN      NaN
49 2019-06-07 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             7750.0                7931.67       NaN      NaN      NaN
50 2019-06-07 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             8750.0                7931.67       NaN      NaN      NaN
51 2019-05-31 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             7750.0                8229.91       NaN      NaN      NaN
52 2019-05-31 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             8500.0                8229.91       NaN      NaN      NaN
53 2019-05-24 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             6750.0                8038.43       NaN      NaN      NaN
54 2019-05-24 12:00:00+00:00  XBT7D_U105     Settlement      0.006700             7500.0                8038.43       NaN      NaN      NaN
55 2019-05-17 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             6000.0                7139.66       NaN      NaN      NaN
56 2019-05-17 12:00:00+00:00  XBT7D_U105     Settlement      0.008960             6500.0                7139.66       NaN      NaN      NaN
57 2019-05-10 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             5500.0                6281.85       NaN      NaN      NaN
58 2019-05-10 12:00:00+00:00  XBT7D_U105     Settlement      0.004490             6000.0                6281.85       NaN      NaN      NaN
59 2019-05-03 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             5000.0                5707.68       NaN      NaN      NaN
60 2019-05-03 12:00:00+00:00  XBT7D_U105     Settlement      0.003640             5500.0                5707.68       NaN      NaN      NaN
61 2019-04-26 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             5000.0                5160.06       NaN      NaN      NaN
62 2019-04-26 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             5500.0                5160.06       NaN      NaN      NaN
63 2019-04-19 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             4750.0                5261.75       NaN      NaN      NaN
64 2019-04-19 12:00:00+00:00  XBT7D_U105     Settlement      0.000220             5250.0                5261.75       NaN      NaN      NaN
65 2019-04-12 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             4750.0                5074.01       NaN      NaN      NaN
66 2019-04-12 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             5250.0                5074.01       NaN      NaN      NaN
67 2019-04-05 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             4000.0                5003.48       NaN      NaN      NaN
68 2019-04-05 12:00:00+00:00  XBT7D_U105     Settlement      0.015060             4250.0                5003.48       NaN      NaN      NaN
69 2019-03-29 12:00:00+00:00      TRXH19     Settlement      0.000006                NaN                    NaN       NaN      NaN      NaN
70 2019-03-29 12:00:00+00:00      BCHH19     Settlement      0.041210                NaN                    NaN       NaN      NaN      NaN
71 2019-03-29 12:00:00+00:00      XRPH19     Settlement      0.000074                NaN                    NaN       NaN      NaN      NaN
72 2019-03-29 12:00:00+00:00      EOSH19     Settlement      0.001055                NaN                    NaN       NaN      NaN      NaN
73 2019-03-29 12:00:00+00:00      ADAH19     Settlement      0.000017                NaN                    NaN       NaN      NaN      NaN
74 2019-03-29 12:00:00+00:00      LTCH19     Settlement      0.014920                NaN                    NaN       NaN      NaN      NaN
75 2019-03-29 12:00:00+00:00      ETHH19     Settlement      0.034360                NaN                    NaN       NaN      NaN      NaN
76 2019-03-29 12:00:00+00:00      XBTH19     Settlement   4095.130000                NaN                    NaN       NaN      NaN      NaN
77 2019-03-29 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3750.0                4095.13       NaN      NaN      NaN
78 2019-03-29 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             4250.0                4095.13       NaN      NaN      NaN
79 2019-03-22 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3750.0                3982.17       NaN      NaN      NaN
80 2019-03-22 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             4000.0                3982.17       NaN      NaN      NaN
81 2019-03-15 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3750.0                3881.87       NaN      NaN      NaN
82 2019-03-15 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             4000.0                3881.87       NaN      NaN      NaN
83 2019-03-08 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3500.0                3876.24       NaN      NaN      NaN
84 2019-03-08 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             4000.0                3876.24       NaN      NaN      NaN
85 2019-03-01 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3750.0                3810.54       NaN      NaN      NaN
86 2019-03-01 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             4000.0                3810.54       NaN      NaN      NaN
87 2019-02-22 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3500.0                3923.44       NaN      NaN      NaN
88 2019-02-22 12:00:00+00:00  XBT7D_U105     Settlement      0.004420             3750.0                3923.44       NaN      NaN      NaN
89 2019-02-15 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3250.0                3574.36       NaN      NaN      NaN
90 2019-02-15 12:00:00+00:00  XBT7D_U105     Settlement      0.002080             3500.0                3574.36       NaN      NaN      NaN
91 2019-02-08 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3250.0                3390.14       NaN      NaN      NaN
92 2019-02-08 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             3500.0                3390.14       NaN      NaN      NaN
93 2019-02-01 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3250.0                3423.25       NaN      NaN      NaN
94 2019-02-01 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             3750.0                3423.25       NaN      NaN      NaN
95 2019-01-25 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3500.0                3535.21       NaN      NaN      NaN
96 2019-01-25 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             3750.0                3535.21       NaN      NaN      NaN
97 2019-01-18 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3500.0                3611.65       NaN      NaN      NaN
98 2019-01-18 12:00:00+00:00  XBT7D_U105     Settlement      0.000000             3750.0                3611.65       NaN      NaN      NaN
99 2019-01-11 12:00:00+00:00   XBT7D_D95     Settlement      0.000000             3500.0                3633.44       NaN      NaN      NaN

【讨论】：

谢谢！这对你来说很明显，但是（愚蠢的问题）你怎么看到它是 json 而不是 html？内容确实看起来像 html，所以我有点困惑。
我明白了，您找到了 api 端点并直接从那里拿走了它！容易得多：）
@Newskooler 首先，我检查了 HTML 源代码是否存在表（Ctrl + U）。如果不存在，则必须从某处加载数据。所以我检查了 Developer Tools->Network 选项卡是否有任何 Ajax 请求。