数据工厂 polybase - 零行复制到接收器答案

【问题标题】：Data factory polybase - zero rows copied to sink数据工厂 polybase - 零行复制到接收器
【发布时间】：2019-12-11 03:51:55
【问题描述】：

使用来自源数据仓库的复制活动的数据工厂管道 -> 暂存 Blob 存储 -> 接收器数据仓库。

从源到 blob 的复制按预期工作（复制行）。从 staging 到 sink 的复制失败 - 复制了 0 行

禁用 Polybase ，并使用批量插入工作。

{
    "name": "PI_TEST",
    "properties": {
        "activities": [
            {
                "name": "MaterializedEventIdFilter_Copy",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [
                    {
                        "name": "Destination",
                        "value": "[formigration].[MaterializedEventIdFilter]"
                    }
                ],
                "typeProperties": {
                    "source": {
                        "type": "SqlDWSource",
                        "sqlReaderStoredProcedureName": "[formigration].[proc_GetStgMaterializedEventIdFilter]"
                    },
                    "sink": {
                        "type": "SqlDWSink",
                        "allowPolyBase": true,
                        "writeBatchSize": 100000,
                        "polyBaseSettings": {
                            "rejectValue": 0,
                            "rejectType": "value",
                            "useTypeDefault": false
                        }
                    },
                    "enableStaging": true,
                    "stagingSettings": {
                        "linkedServiceName": {
                            "referenceName": "riskstoreprd",
                            "type": "LinkedServiceReference"
                        },
                        "enableCompression": true
                    }
                },
                "inputs": [
                    {
                        "referenceName": "ioPrePrdMaterializedEventIdFilter",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "CloudPrdMaterializedEventIdFilter",
                        "type": "DatasetReference"
                    }
                ]
            },
            {
                "name": "MaterialisedEvent",
                "type": "SqlServerStoredProcedure",
                "dependsOn": [
                    {
                        "activity": "MaterializedEventIdFilter_Copy",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 2,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "storedProcedureName": "[formigration].[proc_SetStgMaterializedEventIdFilter]"
                },
                "linkedServiceName": {
                    "referenceName": "cloud_prd",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "annotations": []
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

我希望来自 blob 的数据进入接收器，但没有复制任何行。

编辑 1：检查数据仓库（接收器）建立连接，我可以在一秒钟内观察从 blob 存储创建的外部表等，但没有数据被复制。

INSERT INTO [formigration].[MaterializedEventIdFilter] SELECT * FROM [ADFCopyGeneratedExternalTable_307e2c7f-a56f-4b75-86fb-10ab0cb94548]

【问题讨论】：

标签： sql azure-data-factory-2 polybase azure-sql-data-warehouse

【解决方案1】：

在 polybase 中，外部表只是对 blob 存储文件夹/文件的引用，它们没有任何行。如果您想实际将数据复制到仓库中，请创建一个常规表并将其用作复制活动中的接收器！

希望这有帮助！

【讨论】：