使用记录分页对 API 的 OPENJSON SQL 调用

Question

我正在尝试使用 API，解析数据，然后将其放入表中。我做了几个测试，因为API有分页，所以我无法获得所有记录。我看过其他帖子，我找不到解决方案

DECLARE @token INT;
DECLARE @ret INT;
DECLARE @url NVARCHAR(MAX);
DECLARE @json AS TABLE(Json_Table NVARCHAR(MAX))

-- Define the URL
SET @url = 'https://api.xpto.io/v1/catalog?X-API-KEY=ABCD123456&limit=10000&page=1'

-- This creates the new object.
EXEC @ret = sp_OACreate 'MSXML2.XMLHTTP', @token OUT;
IF @ret <> 0 RAISERROR('Unable to open HTTP connection.', 10, 1);

-- This calls the necessary methods.
EXEC @ret = sp_OAMethod @token, 'open', NULL, 'GET', @url, 'false';
EXEC @ret = sp_OAMethod @token, 'send'

-- Grab the responseText property, and insert the JSON string into a table temporarily. This is very important, if you don't do this step you'll run into problems.
INSERT into @json (Json_Table) EXEC sp_OAGetProperty @token, 'responseText'

-- Select the JSON string from the Table we just inserted it into. You'll also be able to see the entire string with this statement.
SELECT * FROM @json

-- Display all the data we just parsed, keep in mind you can negate certain columns we parsed. There is no requirement to display all the columns.
SELECT
    *
FROM OPENJSON((SELECT * FROM @json))  -- USE OPENJSON to begin the parse.

此代码返回我可以解析的所有数据

image1

如果我们解析项目，它会返回 10000 条记录。

DECLARE @token INT;
DECLARE @ret INT;
DECLARE @url NVARCHAR(MAX);
DECLARE @json AS TABLE(Json_Table NVARCHAR(MAX))

-- Define the URL
SET @url = 'https://api.xpto.io/v1/catalog?X-API-KEY=ABCD123456&limit=10000&page=1'

-- This creates the new object.
EXEC @ret = sp_OACreate 'MSXML2.XMLHTTP', @token OUT;
IF @ret <> 0 RAISERROR('Unable to open HTTP connection.', 10, 1);

-- This calls the necessary methods.
EXEC @ret = sp_OAMethod @token, 'open', NULL, 'GET', @url, 'false';
EXEC @ret = sp_OAMethod @token, 'send'

-- Grab the responseText property, and insert the JSON string into a table temporarily. This is very important, if you don't do this step you'll run into problems.
INSERT into @json (Json_Table) EXEC sp_OAGetProperty @token, 'responseText'

-- Select the JSON string from the Table we just inserted it into. You'll also be able to see the entire string with this statement.
SELECT * FROM @json

-- Display all the data we just parsed, keep in mind you can negate certain columns we parsed. There is no requirement to display all the columns.
SELECT
    metadata.[id],
    metadata.[sku],
    metadata.[status],
    metadata.[highlight],
    metadata.[new],
    metadata.[stock],
    prices.[price_table]
    FROM OPENJSON((SELECT * FROM @json))  -- USE OPENJSON to begin the parse.

-- At the highest level we n parts
WITH (
    [items] NVARCHAR(MAX) AS JSON
) AS  Data

-- Parse the Metadata
CROSS APPLY OPENJSON([Data].[items])
WITH(
    [id] NVARCHAR(MAX),
    [sku] NVARCHAR(MAX),
    [status] NVARCHAR(MAX),
    [highlight] NVARCHAR(MAX),
    [new] NVARCHAR(MAX),
    [stock] NVARCHAR(MAX),
    [prices] NVARCHAR(MAX) AS JSON
) AS metadata

CROSS APPLY OPENJSON([Metadata].[prices])
WITH(
    [price_table] NVARCHAR(MAX)
) AS prices

我不能做的是添加一个分页循环，以便它返回 13494 条记录。有人可以帮助我吗？

Answer 1

我建议您不要使用sp_OA 程序，因为它们有很多问题。

相反，您可以使用 Powershell、C# 或 Python 之类的工具来执行此操作。

算法很简单：

• 下载第一页。
• 将其作为参数推送到 SQL 查询中。这将使用OPENJSON 将其切碎并将其存储在一个表中。
• SQL 还将返回下一页的 URL（如果有）。
• 如果还有其他页面，则循环回到开头。

所以你需要这样的东西，例如在 Powershell 中：

$query = "
INSERT YourTable (id, sku, status, highlight, [new], stock, price_table)
SELECT
    metadata.id,
    metadata.sku,
    metadata.status,
    metadata.highlight,
    metadata.[new],
    metadata.stock,
    prices.price_table

FROM OPENJSON(@json, '$.items')    -- use a Json path
WITH(
    id NVARCHAR(MAX),
    sku NVARCHAR(MAX),
    status NVARCHAR(MAX),
    highlight NVARCHAR(MAX),
    [new] NVARCHAR(MAX),
    stock NVARCHAR(MAX),
    prices NVARCHAR(MAX) AS JSON
) AS metadata

CROSS APPLY OPENJSON([Metadata].[prices])
WITH(
    price_table NVARCHAR(MAX)
) AS prices;

SET @nextUrl = JSON_VALUE(@json, '$.paging.next');
";

$url = "https://api.xpto.io/v1/catalog?X-API-KEY=ABCD123456&limit=10000&page=1";

while ($url -ne "") {
    try {
        $response = Invoke-WebRequest -URI $url;
        $json = $response.Content;
    } finally {
        if ($response -ne $null) { $response.Dispose(); }
    }

    $jsonParam = New-DbaSqlParameter -ParameterName nextUrl -SqlDbType NVarChar -Size -1 -Value $json
    $output = New-DbaSqlParameter -ParameterName nextUrl -SqlDbType NVarChar -Size -1 -Direction Output;

    Invoke-DbaQuery -SqlInstance YourServer -Database YourDB -CommandType StoredProcedure -Query $query -SqlParameter $output,$jsonParam;
    $url = [string]$output.Value;
}

Invoke-DbaQuery 由单独的 DBA-Tools 模块提供，您可以下载。

我建议您重新考虑您的数据类型：例如，stock 应该是 nvarchar(max) 还是 int？