从 Blob 复制 Azure ADF 失败答案

【问题标题】：Azure DF CopyFromBlob Failing从 Blob 复制 Azure ADF 失败
【发布时间】：2018-06-12 07:00:35
【问题描述】：

我正在尝试使用数据工厂将文件从 Azure Blob 复制到 Azure Data Lake。我一直遇到这个错误，没有找到关于参数“baseURI”映射到什么的任何信息：

"errorCode": "2200",
"message": "ErrorCode=InvalidParameter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The value of the property &apos;baseUri&apos; is invalid: &apos;Value cannot be null.\r\nParameter name: baseUri&apos;.,Source=,''Type=System.ArgumentNullException,Message=Value cannot be null.\r\nParameter name: baseUri,Source=Microsoft.DataTransfer.MsiStoreServiceClient,'",
"failureType": "UserError",
"target": "CopyFromBlob"

我正在使用带有 json 文件的 Powershell。我在这里遗漏了什么明显的东西？

Azure 数据湖链接服务

{
    "name": "<redacted>",
    "properties": {
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "<redacted>",
            "tenant": "<redacted>",
            "subscriptionId": "<redacted>",
            "resourceGroupName": "<redacted>"
        },
        "connectVia": {
            "referenceName": "<redacted>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

Azure Blob 链接服务：

{
    "name": "<redacted>",
    "properties": {
        "type": "AzureStorage",
        "typeProperties": {
            "connectionString": {
                "type": "SecureString",
                "value": "DefaultEndpointsProtocol=https;AccountName=<redacted>;AccountKey=<redacted>"
            }
        },
        "connectVia": {
           "referenceName": "<redacted>",
           "type": "IntegrationRuntimeReference"
        }   
    }
}

数据湖数据集

{
    "name": "<redacted>",
    "properties": {
        "type": "AzureDataLakeStoreFile",
        "linkedServiceName":{
            "referenceName": "<redacted>",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "folderPath": "<redacted>"
        }
    }
}

Blob 数据集

{
    "name": "<redacted>",
    "properties": {
        "type": "AzureBlob",
        "linkedServiceName": {
            "referenceName": "<redacted>",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "folderPath": "<redacted>",
        }
    }
}

管道

{
"name": "<redacted>",
"properties": {
"activities":[

   {
        "name": "CopyFromBlob",
    "type": "Copy",
        "inputs": [
            {
                "referenceName": "<redacted>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<redacted>",
                "type": "DatasetReference"
            }
        ],

        "typeProperties": {
            "source": {
                "type": "BlobSource"
            },
            "sink": {
                "type": "AzureDataLakeStoreSink"
            }
        }
    }] 
}}

Powershell 执行以下操作：

1. Create Data Factory
2. Create Azure Integration Runtime
3. Create Azure Data Lake Linked Service
4. Create Azure Blob Linked Service
5. Create Azure Blob Dataset
6. Create Azure Data Lake Dataset
7. Create pipeline
8. Invoke pipeline

【问题讨论】：

请参阅Minimal Complete Verifiable Example 并包含尽可能小的代码以重现您的问题。
奇怪的是，我包含的其中一个 JSON 脚本存在格式问题。每个都是构建 Azure DF 管道并调用它所必需的。应该去掉哪一部分？

标签： azure-blob-storage azure-data-factory azure-data-lake

【解决方案1】：

检查您的 Azure 集成运行时 powershell 命令 - 您是否定义了位置和类型？当我忘记为我的集成运行时定义一个位置时，我遇到了一个非常相似的错误（我的空参数是“字典”）。

希望这会有所帮助！

【讨论】：

这是我使用的：Set-AzureRmDataFactoryV2IntegrationRuntime -DataFactoryName "" -name "" -ResourceGroupName "szweather" -Type managed -location "East US"