Scrapy json修改数组

Question

使用 Scrapy，我从网站上获得了我想要的 json 形式的数据，最终结果是我想使用 rest api 将此数据发布到数据库，因此它必须具有某种形式的某些键。

我正在尝试将数据转换为这种 json 形式； ONLY 数组可以在属性下，所有其他数组必须是一对一的键和值。

{
"Event":{
"date":"2015-11-25",
"threat_level_id":"1",
"info":"TEST",
"analysis":"0",
"distribution":"0",
"orgc":"Malware, Inc",
"Attribute": [
{
"type":"md5",
"category":"Payload delivery",
"to_ids":true,
"distribution":"3",
"value":"35b759347aee663e36f5b91877749349"},
{"type":"url",
"category":"Network activity",
"to_ids":true,
"distribution":"3",
"value":"conf.f.360.cn"},
{
"type":"ip-dst",
"category":"Network activity",
"to_ids":true,
"distribution":"3",
"value":"1.1.1.1"}]
}
}

*数组“属性”可以包含多个项目。我遇到的问题是我无法为多个对象创建“属性”数组，我不确定如何添加看起来相似并包含相同“键”的多种类型。

这行不通……

def parse_items(self, response):
          self.log("Hi, this is an item page! %s" % response.url)
          item = Website()
          item["date"] = "current date"
          item["threat_level_id"] = "1"
          item["info"] = "Malware"
          item["analysis"] = 0
          item["distribution"] = 0
          item["orgc"] = "Malware"
          item["Attribute"] = {}
          item["Attribute"]['type'] = "ip-dst"
          item["Attribute"]["category"] ="Network activity"
          item["Attribute"]["to-ids"] = True
          item["Attribute"]["distribution"] = "3"
          item["Attribute"]['ip']  = response.xpath('//*[@id="contacted-   hosts"]//tr[1]/td[1]/text()').extract()
          item["Attribute"]['type'] = "domain"
          item["Attribute"]["category"] ="Network activity"
          item["Attribute"]["to-ids"] = True
          item["Attribute"]["distribution"] = "3"
          item["Attribute"]['domain'] = response.xpath('//*[@id="dns-requests"]//a/text()').extract()
          item["Attribute"]['type'] = "md5"
          item["Attribute"]["category"] ="Payload delivery"
          item["Attribute"]["to-ids"] = True
          item["Attribute"]["distribution"] = "3"
          item["Attribute"]['md5'] = response.xpath('//*[contains(text(), "MD5")]/following-sibling::dd[1]/text()')[0].extract()

这会返回一个像这样的混乱

[ 
[
[
{"info": "Website",
"orgc": "Malware",
"analysis": 0,
"Attribute": {"category": "Payload delivery",
"domain": ["anatolio69.no-ip.biz", "lucidspung.com", "swashsepal.com"],        #Gotta get this array to a one to one value as well
"to-ids": true,
"ip": ["190.127.234.120"],
"distribution": "3",
"type": "md5",
"md5": "28e59d011c6103b5f5330b5ad042bdba"},
"date": "current date",
"distribution": 0,
"threat_level_id": "1"},

我如何分解“ip”和“域”返回的数组并获得“属性”以保存所有相同的值--

{
"Event":{
"date":"2015-11-25",
"threat_level_id":"1",
"info":"TEST",
"analysis":"0",
"distribution":"0",
"orgc":"Malware, Inc",
"Attribute": [
{
"type":"md5",
"category":"Payload delivery",
"to_ids":true,
"distribution":"3",
"value":"35b759347aee663e36f5b91877749349"},
{"type":"url",
"category":"Network activity",
"to_ids":true,
"distribution":"3",
"value":"conf.f.360.cn"},
{
"type":"ip-dst",
"category":"Network activity",
"to_ids":true,
"distribution":"3",
"value":"1.1.1.1"}]
}
}

我会很感激...我真的不知道如何开始。

Answer 1

Attribute JSON 中的键应该包含一个字典数组。您应该首先初始化数组，然后分配如下所示的值：

item["Attribute"] = []
item["Attribute"].append({'type':"ip-dst"})
item["Attribute"][0]["category"] ="Network activity"
item["Attribute"][0]["to-ids"] = True
item["Attribute"][0]["distribution"] = "3"
item["Attribute"][0]['ip']  = response.xpath('//*[@id="contacted-   hosts"]//tr[1]/td[1]/text()').extract()

item["Attribute"].append({'type':"domain"})
item["Attribute"][1]["category"] ="Network activity"
item["Attribute"][1]["to-ids"] = True
item["Attribute"][1]["distribution"] = "3"
item["Attribute"][1]['domain'] = response.xpath('//*[@id="dns-requests"]//a/text()').extract()

item["Attribute"].append({'type':"md5"})
item["Attribute"][2]["category"] ="Payload delivery"
item["Attribute"][2]["to-ids"] = True
item["Attribute"][2]["distribution"] = "3"
item["Attribute"][2]['md5'] = response.xpath('//*[contains(text(), "MD5")]/following-sibling::dd[1]/text()')[0].extract()