通过 AIX awk、sed、ksh 组合 CSV 文件并将一些列连接成单列答案

【问题标题】：Combine CSV files and concatenate some columns into single column via AIX awk, sed, ksh通过 AIX awk、sed、ksh 组合 CSV 文件并将一些列连接成单列
【发布时间】：2015-12-06 00:46:05
【问题描述】：

我有多个这样的文件：

File_1.csv:
"Job Id", "Batch Id","Id","Success","Created","Error","Col1","Col2","Col3"
 aaabbb111,xxxyyy999,"false","false","Horrible_Error: Really Bad Error occured: yeah", "Val1", "Val2", "Val3"
 cccddd222,pppqqq888,"","false","Horrible_Error: Anoter Bad Error occured: ouch", "Val1", "Val2", "Val3"

File_2.csv:
"Job Id", "Batch Id","Id","Success","Created","Error","Col1","Col2","Col3","Col4", "Col5"
 aaabbb111,xxxyyy999,"false","false","Horrible_Error: Really Bad Error occured: oops","Val1","Val2","Val3","Val4","Val5"
 cccddd222,pppqqq888,"","false","Horrible_Error: Anoter Bad Error occured: oh-no", "Val1","Val1","Val2","Val3","Val4","Val5"

每个文件中的前 6 列始终具有相同的名称。剩余列的名称和数量各不相同，我想将它们捕获为单列，用双引号、方括号或大括号或任何表示这是相同数据的内容。

我需要能够将这些文件组合成一个看起来像这样的单个文件。标题是可选的，仅用于说明：

"File_Name"|"Job Id"|"Batch Id"|"Id"|"Success"|"Created"|"Error"|"Tran_Header"|"Tran_Record" 
File_1.csv|aaabbb111|xxxyyy999|"false"|"false"|"Horrible_Error: Really Bad Error occured: yeah"|["Col1","Col2","Col3"]|["Val1","Val2","Val3"]
File_1.csv|cccddd222|pppqqq888|""|"false"|"Horrible_Error: Anoter Bad Error occured: ouch"|["Col1","Col2","Col3"]|["Val1","Val2","Val3"]
File_2.csv|aaabbb111|xxxyyy999|"false"|"false"|"Horrible_Error: Really Bad Error occured: oops"|["Col1","Col2","Col3","Col4", "Col5"]|["Val1","Val2","Val3","Val4","Val5"]
File_2.csv|cccddd222|pppqqq888|""|"false"|"Horrible_Error: Anoter Bad Error occured: oh-no"|["Col1","Col2","Col3","Col4", "Col5"]|["Val1","Val1","Val2","Val3","Val4","Val5"]

我尝试了以下方法来合并文件，但此代码有时会因替换双引号而窒息，然后我的 ETL 工具又会因解析串联列集而窒息（而且我不知道如何将标题捕获到单独的列）：

outdirectory=/some/directory
outfilename=some_file_name.csv
for i in *.csv
do
    filename=$(echo "${i}")

    tail +2 "${i}" | sed -e 's/,/#|#/1' -e 's/,/#|#/1' -e 's/,/#|#/1' -e 's/,/#|#/1' -e 's/,/#|#/1' -e 's/,/#|#/1' -e s/\"//g -e "s/^/#${filename}/" -e s/$/#/ | sed s/#/\"/g >> "${outdirectory}/${outfilename}" 

    mv $i $srcdir/
done

非常感谢任何帮助或想法。我完全是 UNIX shell 脚本的新手。差点忘了，我在 AIX v6.2 上

【问题讨论】：

标签： bash awk sed ksh aix

【解决方案1】：

使用awk 的解决方案（我使用gnu-awk）

awk 'BEGIN{FS=",";OFS="|"}
{
  if(FNR==1){
    if(NR==1){
      print "\"File_Name\"",$1,$2,$3,$4,$5,$6,"\"Tran_Header\"","\"Tran_Record\"";
    }
    $1=$2=$3=$4=$5=$6="";
    gsub("[|]+",",",$0);
    gsub("^,","",$0);
    titleCol = $0;
  }else{
    temp = FILENAME OFS $1 OFS $2 OFS $3 OFS $4 OFS $5 OFS "["titleCol"]";
    $1=$2=$3=$4=$5="";
    gsub("[|]+",",",$0);
    gsub("^,","",$0);
    print temp OFS "["$0"]";
  }
}' *.csv

你得到：

【讨论】：

谢谢，但这并不能解决主要问题 - 因为第 6 列之后的列没有合并为一个。此外，我不希望对标题进行硬编码并在出现时将其打印出来。