使用 boto3 从 AWS Glue 获取表答案

【问题标题】：Get tables from AWS Glue using boto3使用 boto3 从 AWS Glue 获取表
【发布时间】：2021-03-09 10:33:43
【问题描述】：

我需要从 AWS Glue 爬虫元数据目录中获取表和列名。我使用了boto3，但不断获得 100 张桌子的数量，即使还有更多。设置NextToken 没有帮助。如果可能，请提供帮助。

想要的结果如下：

lst = [table_one.col_one, table_one.col_two, table_two.col_one....table_n.col_n]

def harvest_aws_crawler():
glue = boto3.client('glue', region_name='')
response = glue.get_tables(DatabaseName='', NextToken = '')

#response syntax:
#https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/glue.html#Glue.Client.get_tables
crawler_list_tables = []

for tables in response['TableList']:
    while (response.get('NextToken') is not None):
        crawler_list_tables.append(tables['Name'])
        break
print(len(crawler_list_tables))

harvest_aws_crawler()

更新后的代码，还是需要有tablename+columnname：

def harvest_aws_crawler():
glue = boto3.client('glue', region_name='')
next_token = ""

#response syntax:
#https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/glue.html#Glue.Client.get_tables
response = glue.get_tables(DatabaseName='', NextToken = next_token)

tables_from_crawler = []
while True:
    table_list = response['TableList']
    for table_dict in table_list:
        table_name = table_dict['Name']
        
        #append table_name+column_name
        for columns in table_name['StorageDescriptor']['Columns']:
            tables_from_crawler.append(table_name + '.' + columns['Name'])
                
        #tables_from_crawler.append(table_name)
    next_token = response.get('NextToken')
    if next_token is None:
        break
print(tables_from_crawler)

harvest_aws_crawler()

【问题讨论】：

下一个请求不要使用NextToken，否则总是会得到前100个。或者使用get_tables的分页器：boto3.amazonaws.com/v1/documentation/api/latest/reference/…
@luk2302 - 谢谢，但我在使用get_paginator时仍然获得了 100 张桌子@

标签： python amazon-web-services pyspark boto3 aws-glue

【解决方案1】：

您可以使用paginator 选项尝试以下方法：

def get_tables_for_database(database):
    starting_token = None
    next_page = True
    tables = []
    while next_page:
        paginator = glue_client.get_paginator(operation_name="get_tables")
        response_iterator = paginator.paginate(
            DatabaseName=database,
            PaginationConfig={"PageSize": 100, "StartingToken": starting_token},
        )
        for elem in response_iterator:
            tables += [
                {
                    "name": table["Name"],
                }
                for table in elem["TableList"]
            ]
            try:
                starting_token = elem["NextToken"]
            except:
                next_page = False
    return tables

然后调用该方法列出给定数据库的表：

for table in get_tables_for_database(database):
        print(f"Table: {table['name']}")

如果您想在 Glue 中列出每个数据库的表，您可能必须执行一个额外的 for 循环才能首先检索数据库，然后使用上述 sn-p 作为内部循环提取表每个数据库。

【讨论】：

谢谢！你知道如何用它的列名获取表名吗？

【解决方案2】：

添加子循环可以得到表格+列的结果。

#harvest aws crawler metadata
next_token = ""
client = boto3.client('glue',region_name='us-east-1')
crawler_tables = []

while True:
  response = client.get_tables(DatabaseName = '', NextToken = next_token)
  for tables in response['TableList']:
    for columns in tables['StorageDescriptor']['Columns']:
        crawler_tables.append(tables['Name'] + '.' + columns['Name'])
  next_token = response.get('NextToken')
  if next_token is None:
    break
print(crawler_tables)

【讨论】：