插入 jsonb 类型列

Question

import psycopg2, json, requests, hidden

# Load secrets
secrets = hidden.secrets()

conn = psycopg2.connect(host=secrets['host'],port=secrets['port'],
       ....,connect_timeout=3)

cur = conn.cursor()

defaulturl = 'https://pokeapi.co/api/v2/pokemon?limit=100&offset=0'


sql = '''
CREATE TABLE IF NOT EXISTS pokeapi
(id INTEGER, body JSONB);
'''
print(sql)
cur.execute(sql)

response = requests.get(defaulturl)
js = json.loads(response.text)

# js is a library and i'm interested in the values of 'results' key.
results = js['results'] 

# 'results' is a list of libraries and i want to loop through each element of the list
# and extract the value of 'url' key
# I NEED TO INSERT EACH VALUE INTO pokeapi (body), note that 'body' is of type JSONB

for x in range(len(results)):
    body = requests.get(results[x]['url'])
    js_body = json.loads(body.text)
    sql = f"INSERT INTO pokeapi (body) VALUES ('{js_body}')::JSONB";
    cur.execute(sql, (defaulturl))

print('Closing database connection...')
conn.commit()
cur.close() 

此脚本不断抛出错误：

如果不存在则创建表 pokeapi（id 整数，正文）；追溯 （最近一次通话最后）：文件“pokeapi.py”，第 45 行，在 cur.execute(sql, (defaulturl)) psycopg2.errors.SyntaxError: "{" LINE 1 或附近的语法错误: INSERT INTO pokeapi (body) VALUES {'abilities': [{'ability':...

我尝试插入 pokeapi (body) 而不转换为 jsonb，但我不断收到相同的错误。有没有我遗漏的基础知识？

Answer 1

您应该正常传递 JSON 字符串而不是解析它，并且不带引号和强制转换：

js_body = body.text
sql = "INSERT INTO pokeapi (body) VALUES (%s)";
cur.execute(sql, [js_body])

重要提示：请勿对随机互联网数据使用格式！始终使用 psycopg2 的内置参数处理。它将正确处理 SQL 注射风险。

目前你没有使用defaulturl，如果你想插入它，那么你需要一个列来插入它。此外，您需要使 id 自动递增：

sql = '''
    CREATE TABLE IF NOT EXISTS pokeapi
    ("id" int8 NOT NULL GENERATED BY DEFAULT AS IDENTITY, body JSONB);
'''

如果没有，您将必须在正文中提供一个 id。

最后，您通常应该避免尝试在每个循环中执行一次。如果你有足够的内存，你应该只循环有效载荷然后使用execute_values():https://www.psycopg.org/docs/extras.html

rows = list()

for result in results:
    response = requests.get(result['url'])
    rows.append([response.text])

sql = "INSERT INTO pokeapi (body) VALUES %s";
sql_template = "(%s)"
execute_values(cur, sql, rows, sql_template)

(另外，为了将来参考，requests 库在响应上有一个 .json() 方法，它可以为你将 json 字符串加载到 python 原语中。也就是说，在这种情况下你不需要解析 json .https://docs.python-requests.org/en/master/user/quickstart/#json-response-content)

Answer 2

这是我最终想出的解决方案。我学到的是需要了解对正确 python 字典的“响应”的反序列化，然后在将其转换为 JSONB 类型之前对 python 字典进行序列化。

import psycopg2, json, requests, hidden

# Load secrets
secrets = hidden.secrets()

conn = psycopg2.connect(host=secrets['host'],port=secrets['port'],
       ....,connect_timeout=3)

cur = conn.cursor()

defaulturl = 'https://pokeapi.co/api/v2/pokemon?limit=100&offset=0'


sql = '''
CREATE TABLE IF NOT EXISTS pokeapi
(id SERIAL, body JSONB); # <== CREATING id OF SERIAL TYPE HELPS AUTO- 
                         #     GENERATE ids of INTEGER TYPE.
'''
print(sql)
cur.execute(sql)

response = requests.get(defaulturl)
js = response.json() # <== THIS IS ONE OF THE CORRECTIONS, I NEEDED TO DE- 
                     #     SERIALIZE THE RESPONSE SO THAT IT'S A PROPER 
                     #     PYTHON DICTIONERY

# js is a library and i'm interested in the values of 'results' key.
results = js['results'] 

# 'results' is a list of libraries and i want to loop through each element of the list
# and extract the value of 'url' key
# I NEED TO INSERT EACH VALUE INTO pokeapi (body), note that 'body' is of type JSONB

for x in range(len(results)):
    body = requests.get(results[x]['url'])
    js_body = json.dumps(body) # <== 2ND MAJOR CORRECTION, I HAVE TO 
                               #     SERIALIZE THE PYTHON DICTIONERY/LIST 
                               #     TO BE ABLE TO CAST IT TO JSONB BELLOW
    sql = f"INSERT INTO pokeapi (body) VALUES ('{js_body}'::JSONB)";
    cur.execute(sql, (defaulturl))

print('Closing database connection...')
conn.commit()
cur.close()