需要从 ip-ranges 文件中获取 Amazon S3 部分答案

【问题标题】：Need to grab Amazon S3 sections from ip-ranges file需要从 ip-ranges 文件中获取 Amazon S3 部分
【发布时间】：2019-10-26 06:59:16
【问题描述】：

我正在尝试从下面的站点获取 ip 我能够使用 python beautiful soup 获取整个站点并使用 python regex 模块来获取 ip4 地址但我遇到了一个问题，我只需要 ipv4 ip 在它说“S3”的部分中，任何帮助将不胜感激

https://ip-ranges.amazonaws.com/ip-ranges.json

akamai_feed = urlopen('https://ip-ranges.amazonaws.com/ip-ranges.json').read() 
soup = BeautifulSoup(akamai_feed, 'html.parser')
ip_addr = re.findall(r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}.\d{1,3}.\b', soup.get_text())

【问题讨论】：

标签： python regex beautifulsoup

【解决方案1】：

此提要是 Json 文件，因此您可以使用 Python 标准库中的 json 模块：

from urllib.request import urlopen
import json

akamai_feed = json.loads( urlopen('https://ip-ranges.amazonaws.com/ip-ranges.json').read() )

for prefix in akamai_feed['prefixes']:
    if prefix['service'] == 'S3':
        print(prefix['ip_prefix'])

打印：

...

52.95.163.0/24
52.95.145.0/24
52.92.40.0/21
52.219.32.0/21
52.95.136.0/23
52.219.62.0/23
52.95.175.0/24

... and so on

【讨论】：

【解决方案2】：

该网站包含一个 JSON 格式的嵌套字典，因此您需要先读取该 JSON 数据：

>>> import urllib.request
>>> import json
>>> akamai_feed = urllib.request.urlopen('https://ip-ranges.amazonaws.com/ip-ranges.json').read()
>>> akamai_json = json.loads(akamai_feed)

现在您有了实际数据，您可以使用例如筛选filter() 函数：

>>> list(filter(lambda _: _['service'] == 'S3', akamai_json['prefixes']))
[{'ip_prefix': '52.95.154.0/23', 'region': 'eu-west-3', 'service': 'S3'}, {'ip_prefix': '52.219.64.0/22', 'region': 'ap-south-1', 'service': 'S3'}, ...]

这将为您提供'service' 为'S3' 的字典列表。或者，您也可以使用列表推导：

>>> [_ for _ in json.loads(akamai_feed)['prefixes'] if _['service'] == 'S3']
[{'ip_prefix': '52.95.154.0/23', 'region': 'eu-west-3', 'service': 'S3'}, {'ip_prefix': '52.219.64.0/22', 'region': 'ap-south-1', 'service': 'S3'}, ...]

如果您只对 IP 地址感兴趣，那么……

>>> [_['ip_prefix'][:-3] for _ in json.loads(akamai_feed)['prefixes'] if _['service'] == 'S3']
['52.95.154.0', '52.219.64.0', ...]

如果/xx 可以是任意位数，或者如果IP 地址的格式不是IPv4 字符串，那么regex 将有助于过滤字符串。

【讨论】：

【解决方案3】：

这里不需要正则表达式，因为 IP 可能不需要验证，如果我理解问题正确，我们只想获取 "ip_prefix" 值，但是如果您希望使用正则表达式来做到这一点，这可能就足够了：

"ip_prefix": "(.+?)"

Demo

测试

# coding=utf8
# the above tag defines encoding for this document and is for Python 2.x compatibility

import re

regex = r"\"ip_prefix\": \"(.+?)\""

test_str = ("{\n"
    "  \"syncToken\": \"1560279544\",\n"
    "  \"createDate\": \"2019-06-11-18-59-04\",\n"
    "  \"prefixes\": [\n"
    "    {\n"
    "      \"ip_prefix\": \"18.208.0.0/13\",\n"
    "      \"region\": \"us-east-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.95.245.0/24\",\n"
    "      \"region\": \"us-east-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.194.0.0/15\",\n"
    "      \"region\": \"ap-northeast-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"54.155.0.0/16\",\n"
    "      \"region\": \"eu-west-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"54.196.0.0/15\",\n"
    "      \"region\": \"us-east-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"99.78.170.0/23\",\n"
    "      \"region\": \"ap-southeast-2\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.94.22.0/24\",\n"
    "      \"region\": \"us-gov-east-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.95.255.112/28\",\n"
    "      \"region\": \"us-west-2\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"13.210.0.0/15\",\n"
    "      \"region\": \"ap-southeast-2\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.94.17.0/24\",\n"
    "      \"region\": \"eu-central-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.95.154.0/23\",\n"
    "      \"region\": \"eu-west-3\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"52.95.212.0/22\",\n"
    "      \"region\": \"ap-southeast-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"54.239.0.240/28\",\n"
    "      \"region\": \"eu-west-2\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"54.241.0.0/16\",\n"
    "      \"region\": \"us-west-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"184.169.128.0/17\",\n"
    "      \"region\": \"us-west-1\",\n"
    "      \"service\": \"AMAZON\"\n"
    "    },\n"
    "    {\n"
    "      \"ip_prefix\": \"216.182.224.0/21\",\n"
    "      \"region\": \"us-east-1\",\n"
    "      \"service\": \"AMAZON\"\n\n"
    "...")

matches = re.finditer(regex, test_str, re.MULTILINE)

for matchNum, match in enumerate(matches, start=1):

    print ("Match {matchNum} was found at {start}-{end}: {match}".format(matchNum = matchNum, start = match.start(), end = match.end(), match = match.group()))

    for groupNum in range(0, len(match.groups())):
        groupNum = groupNum + 1

        print ("Group {groupNum} found at {start}-{end}: {group}".format(groupNum = groupNum, start = match.start(groupNum), end = match.end(groupNum), group = match.group(groupNum)))

# Note: for Python 2.7 compatibility, use ur"" to prefix the regex and u"" to prefix the test string and substitution.

【讨论】：