通过 ARM 模板在 Azure 数据湖存储帐户中创建文件系统

Question

我使用 ARM 模板创建了 Azure Data Lake gen 2。但现在我试图弄清楚如何在 ARM 中创建数据湖文件系统，但似乎找不到执行此操作的 API。这不可用吗，可以通过其他方式吗？

尝试手动创建文件系统并在门户中导出模板，但似乎没有看到文件系统资源。只有这个警告。

因为这表明文件系统可能在 API 中作为“blobservices/containers”被监听，所以我尝试将此资源添加到 ARM 模板中

 {
            "name": "[concat( parameters('DataLakeName'), '/default/input')]",
            "type": "Microsoft.Storage/storageAccounts/blobServices/containers",
            "dependsOn": [
                "[concat('Microsoft.Storage/storageAccounts/', parameters('DataLakeName'))]"
            ],
            "apiVersion": "2018-07-01",
            "properties": {
                "publicAccess": "None",
                "metadata": {}
            },
            "resources": []
        }

但不幸的是，这不起作用并提供了以下错误消息： Blob API is not yet supported for hierarchical namespace accounts. 这让我想到这甚至可能无法通过 ARM 进行部署。有没有人试过这个？

我的数据湖存储帐户的 ARM 模板资源块作为上下文完成：

 {
            "name": "[parameters('DataLakeName')]",
            "type": "Microsoft.Storage/storageAccounts",
            "apiVersion": "2018-07-01",
            "location": "[parameters('location')]",
            "tags": {},
            "properties": {
                "accessTier": "[parameters('accessTier')]",
                "supportsHttpsTrafficOnly": "[parameters('supportsHttpsTrafficOnly')]",
                "isHnsEnabled": true
            },
            "resources": [
                {
                    "type": "providers/advancedThreatProtectionSettings",
                    "name": "Microsoft.Security/current",
                    "apiVersion": "2017-08-01-preview",
                    "dependsOn": [
                        "[resourceId('Microsoft.Storage/storageAccounts/', parameters('DataLakeName'))]"
                    ],
                    "properties": {
                        "isEnabled": true
                    }
                }
            ],
            "dependsOn": [],
            "sku": {
                "name": "[parameters('accountType')]"
            },
            "kind": "StorageV2"
        }

Answer 1

我在github issue answer 上找到了这个解决方案。 从一个发现还没有任何解决方案的人那里，所以这里要做的基本事情是手动调用一个rest api来这样做。解释和博客可以在这个链接中找到： http://sql.pawlikowski.pro/2019/03/10/connecting-to-azure-data-lake-storage-gen2-from-powershell-using-rest-api-a-step-by-step-guide/

这是创建文件系统的 Powershell 脚本，以防链接将被弃用：

感谢 Michał Pawlikowski 的所有功劳，感谢您创作了这个脚本，效果非常棒。

[CmdletBinding()]
Param(
  [Parameter(Mandatory=$true,Position=1)] [string] $StorageAccountName,
  [Parameter(Mandatory=$True,Position=2)] [string] $FilesystemName,
  [Parameter(Mandatory=$True,Position=3)] [string] $AccessKey
)

# Rest documentation:
# https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/filesystem/create

$date = [System.DateTime]::UtcNow.ToString("R") # ex: Sun, 10 Mar 2019 11:50:10 GMT

$n = "`n"
$method = "PUT"

$stringToSign = "$method$n" #VERB
$stringToSign += "$n" # Content-Encoding + "\n" +  
$stringToSign += "$n" # Content-Language + "\n" +  
$stringToSign += "$n" # Content-Length + "\n" +  
$stringToSign += "$n" # Content-MD5 + "\n" +  
$stringToSign += "$n" # Content-Type + "\n" +  
$stringToSign += "$n" # Date + "\n" +  
$stringToSign += "$n" # If-Modified-Since + "\n" +  
$stringToSign += "$n" # If-Match + "\n" +  
$stringToSign += "$n" # If-None-Match + "\n" +  
$stringToSign += "$n" # If-Unmodified-Since + "\n" +  
$stringToSign += "$n" # Range + "\n" + 
$stringToSign +=    
                    <# SECTION: CanonicalizedHeaders + "\n" #>
                    "x-ms-date:$date" + $n + 
                    "x-ms-version:2018-11-09" + $n # 
                    <# SECTION: CanonicalizedHeaders + "\n" #>

$stringToSign +=    
                    <# SECTION: CanonicalizedResource + "\n" #>
                    "/$StorageAccountName/$FilesystemName" + $n + 
                    "resource:filesystem"# 
                    <# SECTION: CanonicalizedResource + "\n" #>

$sharedKey = [System.Convert]::FromBase64String($AccessKey)
$hasher = New-Object System.Security.Cryptography.HMACSHA256
$hasher.Key = $sharedKey

$signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))


$authHeader = "SharedKey ${StorageAccountName}:$signedSignature"

$headers = @{"x-ms-date"=$date} 
$headers.Add("x-ms-version","2018-11-09")
$headers.Add("Authorization",$authHeader)

$URI = "https://$StorageAccountName.dfs.core.windows.net/" + $FilesystemName + "?resource=filesystem"

Try {
    Invoke-RestMethod -method $method -Uri $URI -Headers $headers # returns empty response
}
catch {
    $ErrorMessage = $_.Exception.Message
    $StatusDescription = $_.Exception.Response.StatusDescription
    $false

    Throw $ErrorMessage + " " + $StatusDescription
}

Answer 2

查看 az storage fs (https://docs.microsoft.com/en-us/cli/azure/storage/fs?view=azure-cli-latest)。这使您可以管理 Azure Data Lake Gen2 中的文件系统。

这样你就可以先通过 ARM 部署具有分层命名空间的存储帐户，然后运行脚本来创建文件系统（脚本的执行也可以包含在 ARM 模板中 -> https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-script-template?tabs=PowerShell