Web 将 json 源中的数据抓取到 csv 中

Question

我正在尝试从带有 python 的网站、网上商店获取一些信息。

我试过这个：

my_url = requests.get(https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL)

data = my_url.json()

name = data['MainContent'][0]['contents'][0]['productList']['products'][0]['productModel']["displayName"]
price = data['MainContent'][0]['contents'][0]['productList']['products'][0]['priceInfo']['priceItemSale']["gross"]
url= data['MainContent'][0]['contents'][0]['productList']['products'][0]['productModel']["url"]

for mc in data['MainContent']:
    for co in mc:
        for prod in co['productList']['products']:
            name = prod['productModel']['displayName']
            price = prod['priceItemSale']['gross']
            url = prod['productModel']['url']

filename = "test.csv"
csv_writer = csv.writer(open(filename, 'w'))
headers = "Name, Price, Link\n"
f.write(headers)
f.close()

在这个网上商店有很多具有这些属性“productModel”的产品，但我怎样才能得到这些并写入 csv？我想在不同的单元格中对这个页面的名称、价格和网址进行网络抓取，但它不起作用。

* 编辑：

def proba():

    my_url = requests.get('https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL')
    data = my_url.json()
    results = []
    products = data['MainContent'][0]['contents'][0]['productList']['products']
    for product in products:
        name = product['productModel']['displayName']
        try:
            priceGross = product['priceInfo']['priceItemSale']['gross']
        except:
            priceGross = product['priceInfo']['priceItemToBase']['gross']
        url = product['productModel']['url']
        results.append([name, priceGross, url])
    df = pd.DataFrame(results, columns = ['Name', 'Price', 'Url'])    
# print(df)  ## print df
    df.to_csv(r'/usr/src/Python-2.7.13/test.csv', sep=',', encoding='utf-8-sig',index = False )

while True:
    mytime=datetime.now().strftime("%H:%M:%S")
    while mytime < "23:59:59":
    print mytime
    proba()
    mytime=datetime.now().strftime("%H:%M:%S")

Answer 1

并非所有项目都具有相同数量的密钥，因此您无法始终使用prod['priceItemSale']['gross'] 访问。当价值不存在时，您需要决定从哪里获得价值。示例如下：

import requests
import pandas as pd

my_url = requests.get('https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL')
data = my_url.json()
results = []
products = data['MainContent'][0]['contents'][0]['productList']['products']
for product in products:
    name = product['productModel']['displayName']
    try:
        priceGross = product['priceInfo']['priceItemSale']['gross']
    except:
        priceGross = product['priceInfo']['priceItemToBase']['gross']
    url = product['productModel']['url']
    results.append([name, priceGross, url])
df = pd.DataFrame(results, columns = ['Name', 'Price', 'Url'])    
# print(df)  ## print df
df.to_csv(r'C:\Users\User\Desktop\Data.csv', sep=',', encoding='utf-8-sig',index = False )

---

输出：

Answer 2

import requests
my_url = requests.get("https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL")

data = my_url.json()
datas=data["MainContent"][0]["contents"]
finaldata=[]
for mc in datas:
  plist=mc["productList"]["products"]
  for p in plist:
    name =p['productModel']['displayName']
    try:
      price = p['priceInfo']['priceItemToBase']['gross']
    except:
      price=p['priceInfo']['priceItemSale']['gross']
    url = p['productModel']['url']
    finaldata.append([name,price,url])
files=open("data.csv",'w+')
columns = ['Name', 'Price', 'Url']
files.write(",".join(columns))
files.write("\n")
for f in finaldata:
 files.write('{},{},{}\n'.format(f[0],f[1],f[2]))