计算字符串组合的频率

Question

我有一个数字标识符列表，后跟一个或多个字符串。我想为每个标识符计算关联字符串的最长组合，这些字符串也存在于另一个标识符中。然后还有与每个标识符关联的字符串数。

例如

1   AAA    BBBA    ACA    CCD    ABADA
2   AAA    ACA     CCD
3   AAB    BBAC    DDAD
4   AAA    ACA     DDAD   CCD
5   AAA    ACA     DDAD   CCD

会导致：

ID  Longest Combo  Number of strings  
1         3               5
2         3               3
3         1               3
4         4               4
5         4               4

为了清楚起见对结果的解释：

1 - AAA, ACA and CCD present in 2 so longest combo is 3.
2 - AAA, ACA and CCD present in 1 so longest combo is 3.
3 - DDAD present in 4 and 5 so longest combo is 1.
4 - AAA, ACA, DDAD and CCD present in 5 so longest combo is 4.
5 - AAA, ACA, DDAD and CCD present in 4 so longest combo is 4.

通常我可以尝试自己一起破解一些东西，但是用这个碰了壁 - 甚至不知道从哪里开始。达到这一点的分析是在 awk 中进行的，因此 bash 是理想的，但可能这是一项更适合 R 的工作？

我试图根据这个问题Frequency of each unique combination in data frame 格式化我的数据，但没有成功。

任何帮助将不胜感激。

大约有 3000 个标识符，每个标识符包含 1-15 个字符串。

Answer 1

这是一个原型awk 解决方案。请注意，由于对称性，您只需要 i<j 的 (i,j) 条目。

$ awk 'NR==FNR {for(i=2;i<=NF;i++) a[$1,$i]; size=$1; next} 
               {for(i=$1+1;i<=size;i++) 
                  {for(j=2;j<=NF;j++) 
                     if((i,$j) in a) {c[$1,i]++; list[$1,i]=list[$1,i] FS $j}
                   if(max[$1]<c[$1,i]) {max[$1]=c[$1,i]; maxM[$1]=i}} 
                   if(maxM[$1]) print $1,maxM[$1],"->",max[$1],list[$1,maxM[$1]]}' file{,}

1 2 -> 3  AAA ACA CCD
2 4 -> 3  AAA ACA CCD
3 4 -> 1  DDAD
4 5 -> 4  AAA ACA DDAD CCD

可以再简化一点，我猜匹配长度可以从列表大小计算出来。

说明 双通道算法，将每个记录 id 的所有元素存储在查找表中；还设置大小。在第二遍中，比较具有更高索引的所有行的当前行。求交集大小并记录最大值、对应行和匹配字段。

Answer 2

不懂 R，不喜欢 awk。 bash 解决方案怎么样？呵呵

已编辑以包含 cmets。做了一些小的改动，发现了一个可能很重要的警告。请注意，您有一个 AAA 和一个 AAAA，当前代码可能会给您一个错误匹配，因为 AAAA 匹配 *AAA*。如果是这样的话，换入新的（但仍然有注释的）行，它仍然可以工作，并且会处理这种可能性。

$: cat proc
#! /bin/env bash

# predeclare these as associative arrays - string-based dictionary lookups
declare -A cnt=() set=() lookup=() combos=() hitlst=() hitcnt=() hitid=()

while read -a lst # this declares lst as a normal array and loads each line in fields
do id=${lst[0]}                            # field 0 assigned as the id
   lst=( ${lst[@]:1} )                     # lst assigned itself MINUS the first field
   cnt[$id]=${#lst[@]}                     # count for this id assigned # of  remaining elements in lst
   set[$id]="${lst[@]}"                    # set for id assigned the space-delimited elements of lst
   for k in ${lst[@]}                      # this will iterate k as each item in lst
   do  lookup["$k"]="${lookup["$k"]} $id"  # additively assigns, space delimited, this id as having this string
   done
done<dataset                               # when done reading the file, each key has a list of which id's it's on

# now we analyze our accumulated data
for id1 in ${!cnt[@]}                      # for every id (cnt has how many strings for each, ${!cnt[@]} is all the ids)
do  for k in ${set["$id1"]}                # iterate k over each string id1 had
    do  for id2 in ${lookup["$k"]}         # iterate id2 over each id that had this string
        do  [[ "$id1" == "$id2" ]] && continue # skip when both id pointers are looking at the same record
            case "${set["$id2"]}" in       # look at the whole set of strings assigned to id2 as a space-delimited string
            ### ==>> CAVEAT: might needed to be "$k "*|*" $k "*|*" $k") so that AAA doesn't match AA
            ### "$k "*|*" $k "*|*" $k") hitlst["$id1 $id2"]="${hitlst["$id1 $id2"]} $k";
            *$k*) hitlst["$id1 $id2"]="${hitlst["$id1 $id2"]} $k"; # when the current key from $id1 is anywhere in it
                  (( hitcnt["$id1 $id2"]++ )) ;;                   # increment the matches per pair of ids
            esac
            if (( ${hitcnt["$id1 $id2"]}0 > ${hitcnt["$id1"]}0 ))  # if matches/pair > stored max for id1
            then hitcnt["$id1"]=${hitcnt["$id1 $id2"]}             # upgrade to the new max
                 hitid["$id1"]="$id2"                              # and remember to report which rec while we're at it
            fi
       done
    done
done

printf "%s\t%s\t%s\t%s\n" "ID" "Longest Combo" "#strings" "[List of strings] / [matches] / matching ID"
for id in ${!cnt[@]}   # again, for every id
do  id2=${hitid[$id]}  # and again, this is the rec that had the most matches in case you wanted it (not requested)
    # print the fields with some formatting to align them
    printf "%s\t%s\t%s\t%s\n" "$id" "${hitcnt[$id]}" "${cnt[$id]}" "[${set[$id]}] / [${hitlst["$id $id2"]# }] / $id2"
done

$: proc
 ID Longest Combo  #strings [List of strings] / [matches] / matching ID
  1             3         5 [AAA BBBA ACA CCD ABADA] / [AAA ACA CCD] / 2
  2             3         3 [AAA ACA CCD] / [AAA ACA CCD] / 1
  3             1         3 [AAB BBAC DDAD] / [DDAD] / 4
  4             4         4 [AAA ACA DDAD CCD] / [AAA ACA DDAD CCD] / 5
  5             4         4 [AAA ACA DDAD CCD] / [AAA ACA DDAD CCD] / 4

我添加了一些字段并调整了输出格式，但你明白了。只有 3k 记录，不应该花费 太 时间。