如何根据选择抓取具有不同值的网站表

Question

我正在尝试刮：

https://id.investing.com/commodities/gold-historical-data 

2010-2020 的表格，但问题是默认日期和我选择的日期之间的链接仍然相同。那么如何告诉 python 从 2010 年到 2020 年抓取数据呢？请帮助我，我正在使用 python 3。

这是我的代码：

import requests, bs4

url = 'https://id.investing.com/commodities/gold-historical-data'
headers = {"User-Agent":"Mozilla/5.0"}
response = requests.get(url, headers=headers)
soup = bs4.BeautifulSoup(response.text, 'lxml')
tables = soup.find_all('table')

print(soup)

with open('emasfile.csv','w') as csv:
    for row in tables[1].find_all('tr'):
        line = ""
        for td in row.find_all(['td', 'th']):
            line += '"' + td.text + '",'
        csv.write(line + '\n')

Answer 1

此页面使用JavaScript 和AJAX 从中获取数据

https://id.investing.com/instruments/HistoricalDataAjax

它发送带有额外数据的 POST 请求 - 开始日期和结束日期（"st_date"，"end_date"）

您可以尝试使用01/01/2010、12/31/2020，但我使用for-loop 来分别获取每一年。

我从 Chrome/Firefox 中的 DevTool（标签“网络”）获取所有信息。

import requests
from bs4 import BeautifulSoup
import csv

url = 'https://id.investing.com/instruments/HistoricalDataAjax'

payload = {
    "curr_id": "8830",
    "smlID": "300004",
    "header": "Data+Historis+Emas+Berjangka",
    "st_date": "01/30/2020",
    "end_date": "12/31/2020",
    "interval_sec": "Daily",
    "sort_col": "date",
    "sort_ord": "DESC",
    "action":"historical_data"
}

headers = {
    #"Referer": "https://id.investing.com/commodities/gold-historical-data",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0",
    "X-Requested-With": "XMLHttpRequest"
}

fh = open('output.csv', 'w')
csv_writer = csv.writer(fh)

for year in range(2010, 2021):
    print('year:', year)
    
    payload["st_date"] = f"01/01/{year}"
    payload["end_date"] = f"12/31/{year}"
    
    r = requests.post(url, data=payload, headers=headers)
    #print(r.text)
    
    soup = BeautifulSoup(r.text, 'lxml')
    table = soup.find('table')
    for row in table.find_all('tr')[1:]: # [1:] to skip header
        row_data = [item.text for item in row.find_all('td')]
        print(row_data)
        csv_writer.writerow(row_data)
        
fh.close()