将文本输出导出为 csv 格式，以便使用 Powershell 插入数据库答案

【问题标题】：Export text ouput into csv format ready for insert into databases using Powershell将文本输出导出为 csv 格式，以便使用 Powershell 插入数据库
【发布时间】：2021-06-02 18:54:37
【问题描述】：

我希望将显示在屏幕上的 aws cli 输出作为 powershell 会话的文本输出传输到 csv 格式的文本文件中。

我从以下文章中研究了Export-CSV cmdlet：

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/export-csv?view=powershell-7.1

我看不出如何使用它来帮助我实现目标。根据我的测试，它似乎只适用于特定的 windows 程序，而不是一般的文本输出。

本网站上的一篇文章展示了如何使用 unix 命令实现我的目标，方法是将空格替换为逗号。

Output AWS CLI command with filters to CSV without jq

unix 的答案是在命令末尾使用 sed，如下所示：

aws rds describe-db-instance-automated-backups --query 'DBInstanceAutomatedBackups[*].{ARN:DBInstanceArn,EarliestTime:RestoreWindow.EarliestTime,LatestTime:RestoreWindow.LatestTime}' --output text | sed -E 's/\s+/,/g'

Export-csv` 似乎无法做到这一点。

有谁知道我如何使用 powershell 复制 sed 在这里所做的事情？

这是我想要的 csv 格式输出示例：

arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod     2019-03-03T09:54:29.402Z        2019-03-05T01:25:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf    2019-03-01T09:04:31.477Z        2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb   2019-02-01T09:07:30.648Z        2019-03-05T01:27
:20Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb    2019-02-02T09:04:30.771Z        2019-03-05T01:28
:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault   2019-02-26T14:14:30.254Z        2019-03-05T01:29
:13Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault   2019-02-01T14:05:40.456Z        2019-03-05T01:31
:05Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod    2019-02-02T14:06:26.050Z        2019-03-
05T01:27:02Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod  2019-02-01T14:12:05.286Z        2019-03-05T01:26
:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage 2019-03-02T09:54:29.053Z        2019-03-05T01:29
:11Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod     2019-02-02T22:09:00.673Z        2019-03-05T01:29:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod       2019-03-02T09:54:20.729Z        2019-03-05T01:30:21Z

【问题讨论】：

你能给我们举一个文本输出的例子吗？
我已经这样做了西奥谢谢。
我明白了，但这只是所需的输出。我真正的意思是 aws rds ... 命令的 raw 输出，您需要将其转换为 CSV
No mate that IS 原始输出...这是执行命令后屏幕上出现的内容

标签： windows powershell csv aws-cli

【解决方案1】：

您可能正在使用制表符分隔的文本文件，没有标题。制表符分隔符在屏幕上显示时可能看起来像多个空格。

如果是这种情况，如果是这样，您实际上可以使用 import-csv 读取此文件，但您必须使用 -header 参数提供您自己的字段名称，并使用 -delimiter 字符使用制表符作为分隔符.必须使用反引号转义机制指定制表符。

有关详细信息，请参阅this question 的已接受答案。

如果您可以控制自己的数据馈送，还有其他选择。 aws cli 界面有一个以 JSON 格式格式化输出的选项。这种格式将更容易以您可以使用的形式导入 Powershell。

编辑：

以下脚本使用 Theo 提供的模型，只是多个空格已替换为制表符。它使用的是 ConvertFrom-Csv 而不是 Import-Csv，但思路是一样的：

$awsReturn = @"
arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod   2019-03-03T09:54:29.402Z    2019-03-05T01:25:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf   2019-03-01T09:04:31.477Z    2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb    2019-02-01T09:07:30.648Z    2019-03-05T01:27:20Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb  2019-02-02T09:04:30.771Z    2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault 2019-02-26T14:14:30.254Z    2019-03-05T01:29:13Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault 2019-02-01T14:05:40.456Z    2019-03-05T01:31:05Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod   2019-02-02T14:06:26.050Z    2019-03-05T01:27:02Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod     2019-02-01T14:12:05.286Z    2019-03-05T01:26:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage   2019-03-02T09:54:29.053Z    2019-03-05T01:29:11Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod   2019-02-02T22:09:00.673Z    2019-03-05T01:29:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod 2019-03-02T09:54:20.729Z    2019-03-05T01:30:21Z
"@


$myarray = $awsreturn | ConvertFrom-Csv -header "Prod","DateStart","DateEnd" -delimiter "`t"

$myarray | Format-Table

$myarray | gm

当我在我的环境中运行它时，它产生了以下结果：

Prod                                                                      DateStart                DateEnd             
----                                                                      ---------                -------             
arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod                 2019-03-03T09:54:29.402Z 2019-03-05T01:25:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf             2019-03-01T09:04:31.477Z 2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb          2019-02-01T09:07:30.648Z 2019-03-05T01:27:20Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb        2019-02-02T09:04:30.771Z 2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault       2019-02-26T14:14:30.254Z 2019-03-05T01:29:13Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault       2019-02-01T14:05:40.456Z 2019-03-05T01:31:05Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod 2019-02-02T14:06:26.050Z 2019-03-05T01:27:02Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod       2019-02-01T14:12:05.286Z 2019-03-05T01:26:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage     2019-03-02T09:54:29.053Z 2019-03-05T01:29:11Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod                 2019-02-02T22:09:00.673Z 2019-03-05T01:29:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod                   2019-03-02T09:54:20.729Z 2019-03-05T01:30:21Z




   TypeName: System.Management.Automation.PSCustomObject

Name        MemberType   Definition                                                           
----        ----------   ----------                                                           
Equals      Method       bool Equals(System.Object obj)                                       
GetHashCode Method       int GetHashCode()                                                    
GetType     Method       type GetType()                                                       
ToString    Method       string ToString()                                                    
DateEnd     NoteProperty string DateEnd=2019-03-05T01:25:53Z                                  
DateStart   NoteProperty string DateStart=2019-03-03T09:54:29.402Z                            
Prod        NoteProperty string Prod=arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod

【讨论】：

它可能是制表符分隔的，但我认为这无关紧要。 \s+ 正则表达式将匹配空格，而不是空格，因此包含制表符。另外，我同意 JSON 输出可能会更好，但前提是他使用的是 Powershell v6+。 Powershell v5.1 及更低版本使用 MS 的 JSON 解析器，该解析器具有许多特定于 MS 的语法，尤其是日期时间（这就是 Json.NET 在前 Core .Net 下如此普遍的原因）。在被烧毁之后，我不相信它会在 v6+ 之前序列化来自外部系统的数据。

【解决方案2】：

让我们假设返回的数据看起来像这个模型（在问题中它的格式很奇怪）：

$awsReturn = @"
arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod     2019-03-03T09:54:29.402Z        2019-03-05T01:25:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf    2019-03-01T09:04:31.477Z        2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb   2019-02-01T09:07:30.648Z        2019-03-05T01:27:20Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb    2019-02-02T09:04:30.771Z        2019-03-05T01:28:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault   2019-02-26T14:14:30.254Z        2019-03-05T01:29:13Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault   2019-02-01T14:05:40.456Z        2019-03-05T01:31:05Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod    2019-02-02T14:06:26.050Z        2019-03-05T01:27:02Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod  2019-02-01T14:12:05.286Z        2019-03-05T01:26:53Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage 2019-03-02T09:54:29.053Z        2019-03-05T01:29:11Z
arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod     2019-02-02T22:09:00.673Z        2019-03-05T01:29:40Z
arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod       2019-03-02T09:54:20.729Z        2019-03-05T01:30:21Z
"@

然后，您可以这样做：

# Since I don't know if that is one single string or a string array:
if ($awsReturn -isnot [array]) { $awsReturn = $awsReturn -split '\r?\n' }

# write it to csv file
$awsReturn -replace '\s+', ',' | Set-Content -Path 'WhereEver.csv' -PassThru  # PassThru also displays on screen

获取可以用作 CSV 的文件（尽管它没有标题或引用字段）

如果要使用Export-CSV获取带有标题和引用字段的csv文件，则需要拆分行并输出objects。

类似这样的：

# Since I don't know if that is one single string or a string array:
if ($awsReturn -isnot [array]) { $awsReturn = $awsReturn -split '\r?\n' }

# write it to csv file (without headers or quotes values)
$awsReturn | ForEach-Object {
    $data = $_ -split '\s+'  # in this case we know we have 3 fields
    [PsCustomObject]@{
        Prod      = $data[0]
        DateStart = $data[1]
        DateEnd   = $data[2]
    }
} | Export-Csv -Path 'WhereEver.csv' -NoTypeInformation

WhereEver.csv 文件将如下所示：

"Prod","DateStart","DateEnd"
"arn:aws:rds:ap-southwest-2:9711387875370:db:catflow--prod","2019-03-03T09:54:29.402Z","2019-03-05T01:25:53Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:xyz-prod-rds-golf","2019-03-01T09:04:31.477Z","2019-03-05T01:28:40Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-stardb","2019-02-01T09:07:30.648Z","2019-03-05T01:27:20Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-asm-prod-rds-domaindb","2019-02-02T09:04:30.771Z","2019-03-05T01:28:40Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-ctz-prod-rds-datavault","2019-02-26T14:14:30.254Z","2019-03-05T01:29:13Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-gcp-prod-rds-datavault","2019-02-01T14:05:40.456Z","2019-03-05T01:31:05Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:prod-conformed-datavault-prod","2019-02-02T14:06:26.050Z","2019-03-05T01:27:02Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:prod-dqm-datavault-prod","2019-02-01T14:12:05.286Z","2019-03-05T01:26:53Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:prod-prod-dgc-cde-lineage","2019-03-02T09:54:29.053Z","2019-03-05T01:29:11Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:prod-rec-prod","2019-02-02T22:09:00.673Z","2019-03-05T01:29:40Z"
"arn:aws:rds:ap-southwest-2:9711387875370:db:-serve-prod","2019-03-02T09:54:20.729Z","2019-03-05T01:30:21Z"

【讨论】：