使用 AJAX 抓取 - 如何获取数据？

Question

我正在尝试从https://www.anre.ro/ro/info-consumatori/comparator-oferte-tip-de-furnizare-a-gn 中抓取数据，该数据通过 Ajax 获取输入（请求 URL 为 https://www.anre.ro/ro/ajax/comparator/get_results_gaz）。

然而，我可以看到的是，将数据以一种形式 - 的 tip_client = casnic＆modalitate_racordare = sistem_de_distributie＆transee_de_consum = B1＆tip_pret_unitar = cu_reglementate＆id_judet = ALBA＆id_siruta = 1222＆consum_mwh =＆pret_furnizare_mwh =＆componenta_fixa =＆suplimentar_componenta_fixa =＆termen_plata =＆durata_contractului =＆garantii =＆frecventa_emitere_factura =＆tip_pret =（如果我在 Chrome 中查看源代码）。如何将此传递给 scrapy 或任何其他模块以检索所需的网页？

到目前为止，我有这个（考虑到表单数据，json 格式是否正确？）：

class ExSpider(scrapy.Spider):
    name = 'ExSpider'
    allowed_domains = ['anre.ro']

    def start_requests(self):
        params = {
            "tip_client":"casnic",
            "modalitate_racordare":"sistem_de_distributie",
            "transee_de_consum":"b1",
            "tip_pret_unitar":"cu_reglementate",
            "id_judet":"ALBA",
            "id_siruta":"1222",
            "consum_mwh":"",
            "pret_furnizare_mwh":"",
            "componenta_fixa":"",
            "suplimentar_componenta_fixa":"",
            "termen_plata":"",
            "durata_contractului":"",
            "garantii":"",
            "frecventa_emitere_factura":"",
            "tip_pret":""
        }
        r = scrapy.FormRequest('https://www.anre.ro/ro/ajax/comparator/get_results_gaz', method = "POST",formdata=params)
        print(r)

Answer 1

以下内容应从您希望从中获取数据的页面产生所需的响应。

class ExSpider(scrapy.Spider):
    name = "exspider"

    url = 'https://www.anre.ro/ro/ajax/comparator/get_results_gaz'

    payload = {
        'tip_client': 'casnic',
        'modalitate_racordare': 'sistem_de_distributie',
        'transee_de_consum': 'b2',
        'tip_pret_unitar': 'cu_reglementate',
        'id_judet': 'ALBA',
        'id_siruta': '1222',
        'consum_mwh': '',
        'pret_furnizare_mwh': '',
        'componenta_fixa': '',
        'suplimentar_componenta_fixa': '',
        'termen_plata': '',
        'durata_contractului': '',
        'garantii': '',
        'frecventa_emitere_factura': '',
        'tip_pret': ''
    }

    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36',
        'X-Requested-With': 'XMLHttpRequest',
        'Referer': 'https://www.anre.ro/ro/info-consumatori/comparator-oferte-tip-de-furnizare-a-gn'
    }

    def start_requests(self):
        yield scrapy.FormRequest(
                self.url, 
                formdata=self.payload, 
                headers=self.headers, 
                callback=self.parse
            )

    def parse(self, response):
        print(response.text)