Azure 数据工厂 V2 数据集动态文件夹答案

【问题标题】：Azure Data Factory V2 Dataset Dynamic FolderAzure 数据工厂 V2 数据集动态文件夹
【发布时间】：2018-01-22 00:35:30
【问题描述】：

在 Azure 数据工厂 (V1) 中，我能够创建幻灯片并将输出存储到特定文件夹（即 {Year}/{Month}/{Day}。请参阅下面的代码。

如何在 Azure 数据工厂 V2 中创建相同类型的切片？我确实发现您必须创建一个参数。是的，我无法弄清楚如何传递参数。

 "folderPath": "@{dataset().path}",
"parameters": {
  "path": {
    "type": "String"

这是原始 ADF V1 代码。

{
    "name": "EMS_EMSActivations_L1_Snapshot",
    "properties": {
        "published": false,
        "type": "AzureDataLakeStore",
        "linkedServiceName": "SalesIntelligence_ADLS_LS",
        "typeProperties": {
            "fileName": "EMS.FACT_EMSActivations_WTA.tsv",
            "folderPath": "/Snapshots/EMS/FACT_EMSActivations_WTA/{Year}/{Month}/{Day}",
            "format": {
                "type": "TextFormat",
                "rowDelimiter": "␀",
                "columnDelimiter": "\t",
                "nullValue": "#NULL#",
                "quoteChar": "\""
            },
            "partitionedBy": [
                {
                    "name": "Year",
                    "value": {
                        "type": "DateTime",
                        "date": "SliceStart",
                        "format": "yyyy"
                    }
                },
                {
                    "name": "Month",
                    "value": {
                        "type": "DateTime",
                        "date": "SliceStart",
                        "format": "MM"
                    }
                },
                {
                    "name": "Day",
                    "value": {
                        "type": "DateTime",
                        "date": "SliceStart",
                        "format": "dd"
                    }
                },
                {
                    "name": "Hour",
                    "value": {
                        "type": "DateTime",
                        "date": "SliceStart",
                        "format": "HH"
                    }
                },
                {
                    "name": "Minute",
                    "value": {
                        "type": "DateTime",
                        "date": "SliceStart",
                        "format": "mm"
                    }
                }
            ]
        },
        "availability": {
            "frequency": "Day",
            "interval": 1
        }
    }
}

【问题讨论】：

标签： azure-data-factory

【解决方案1】：

以下是在将数据从 SQL 导入 ADL 时创建动态文件夹路径的方法。查看 folderPath 行。

{
    "name": "EBC_BriefingActivitySummary_L1_Snapshot",
    "properties": {
        "linkedServiceName": {
            "referenceName": "SIAzureDataLakeStore",
            "type": "LinkedServiceReference"
        },
        "type": "AzureDataLakeStoreFile",
        "typeProperties": {
            "format": {
                "type": "TextFormat",
                "columnDelimiter": ",",
                "rowDelimiter": "",
                "nullValue": "\\N",
                "treatEmptyAsNull": false,
                "firstRowAsHeader": false
            },
            "fileName": {
                "value": "EBC.rpt_BriefingActivitySummary.tsv",
                "type": "Expression"
            },
            "folderPath": {
                "value": "@concat('/Snapshots/EBC/rpt_BriefingActivitySummary/', formatDateTime(pipeline().parameters.scheduledRunTime, 'yyyy'), '/', formatDateTime(pipeline().parameters.scheduledRunTime, 'MM'), '/', formatDateTime(pipeline().parameters.scheduledRunTime, 'dd'), '/')",
                "type": "Expression"
            }
        }
    }
}

【讨论】：

【解决方案2】：

第 1 步： 在文件夹路径中使用 WindowStartTime / WindowEndTime

"folderPath": {
"value": "<<path>>/@{formatDateTime(pipeline().parameters.windowStart,'yyyy')}-@{formatDateTime(pipeline().parameters.windowStart,'MM')}-@{formatDateTime(pipeline().parameters.windowStart,'dd')}/@{formatDateTime(pipeline().parameters.windowStart,'HH')}/",
"type": "Expression"
}

第 2 步：添加管道 JSON

"parameters": {
    "windowStart": {
        "type": "String"
    },
    "windowEnd": {
        "type": "String"
    }
}

Step3 : 在 TumblingWindow 触发器中添加运行参数（这在第 2 步中提到）

   "parameters": {
        "windowStart": {
            "type": "Expression",
            "value": "@trigger().outputs.windowStartTime"
        },
        "windowEnd": {
            "type": "Expression",
            "value": "@trigger().outputs.windowEndTime"
        }
    }

欲了解更多详情，请参考

参考此链接。

https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/data-factory/how-to-create-tumbling-window-trigger.md

【讨论】：