使用 Get-AzureStorageBlobContent 从 ADL Gen2 下载镶木地板文件

Question

我正在尝试使用 powershell 命令将 parquet 文件从 ADL Gen 2 下载到本地系统。 下面是代码sn-p

#this appid has access to ADL
[string] $AppID  = "bbb88818-aaaa-44fb-q2345678901y" 
 [string] $TenantId  = "ttt88888-xxxx-yyyy-q2345678901y"
 [string] $SubscriptionName  = "Sub Sample"
 [string] $LocalTargetFilePathName  = "D:\MoveToModern"


Write-Host "AppID = " $AppID
Write-Host "TenantId = " $TenantId
Write-Host "SubscriptionName = " $SubscriptionName
Write-Host "AzureDataLakeAccountName = " AzureDataLakeAccountName
Write-Host "AzureDataLakeSrcFilePath = " $AzureDataLakeSrcFilePath
Write-Host "LocalTargetFilePathName = " $LocalTargetFilePathName



#this is the access key of the appid
$AccessKeyValue = "1234567=u-r.testabcdefaORYsw5AN5"

$azurePassword    = ConvertTo-SecureString $AccessKeyValue -AsPlainText -Force
$psCred           = New-Object System.Management.Automation.PSCredential($AppID, $azurePassword)
Login-AzureRmAccount -Credential $psCred -ServicePrincipal -Tenant $TenantId

Get-AzureRmSubscription

Get-AzureRmSubscription -SubscriptionName $SubscriptionName  | Set-AzureRmContext
Get-AzureStorageBlobContent -Container "/Test/Partner/Account/" -Blob "Account.parquet" -Destination "D:\MoveToModern"

但是我遇到了错误

可能我们必须设置存储上下文。您能否让我知道如何使用服务主体设置存储上下文。 （我有服务主体的应用程序 ID 和应用程序密钥。W.r.t ADL Gen2 源代码，我只有路径详细信息。源代码团队提供了对服务主体的访问权限）

Answer 1

如果你想从 Azure Data Lake Gen2 下载文件，我建议你使用 PowerShell 模块Az.Storage。同时，关于如何使用服务主体来实现它，您有两种选择。

1.使用 Azure RABC 角色

如果使用 Azure RABC 角色，则需要将特殊角色（Storage Blob Data Reader）分配给 sp。

例如

$AppID  = "" 
$AccessKeyValue  = ""
$TenantId=""
$SubscriptionName  = ""
#1. Assign role at the storage account level
#please use owner account to login
Connect-AzAccount -Tenant $TenantId -Subscription $SubscriptionName
New-AzRoleAssignment -ApplicationId $AppID  -RoleDefinitionName "Storage Blob Data Reader" `
      -Scope "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>"

# download

$azurePassword    = ConvertTo-SecureString $AccessKeyValue -AsPlainText -Force
$psCred           = New-Object System.Management.Automation.PSCredential($AppID, $azurePassword)
Connect-AzAccount -Credential $psCred -ServicePrincipal -Tenant $TenantId -Subscription $SubscriptionName

$AzureDataLakeAccountName  = "testadls05"

$ctx =New-AzStorageContext -StorageAccountName $AzureDataLakeAccountName -UseConnectedAccount 

$LocalTargetFilePathName  = "D:\test.parquet"
$filesystemName="test"
$path="2020/10/28/test.parquet"
Get-AzDataLakeGen2ItemContent -Context $ctx -FileSystem $filesystemName -Path $path -Destination $LocalTargetFilePathName 

1. 使用访问控制列表

如果您使用该方法来授予安全主体对文件的读取权限，您需要授予安全主体对容器以及指向该文件的文件夹层次结构中的每个文件夹的执行权限。

例如

$AppID  = "" 
$AccessKeyValue  = ""
$TenantId=""


$azurePassword    = ConvertTo-SecureString $AccessKeyValue -AsPlainText -Force
$psCred           = New-Object System.Management.Automation.PSCredential($AppID, $azurePassword)
Connect-AzAccount -Credential $psCred -ServicePrincipal -Tenant $TenantId


$AzureDataLakeAccountName  = "testadls05"
$ctx =New-AzStorageContext -StorageAccountName $AzureDataLakeAccountName -UseConnectedAccount 

$filesystemName="test"
$path="2020/10/28/test.parquet"
$LocalTargetFilePathName  = "D:\test1.parquet"
Get-AzDataLakeGen2ItemContent -Context $ctx -FileSystem $filesystemName -Path $path -Destination $LocalTargetFilePathName 

更多详情请参考

https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control

https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control-model

https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-powershell