计算大型数据集中规则的支持度

Question

df = pd.Series([["26"], ["81", "15", "27"], ["50"], ["8"], ["81", "15"], 
["10"], ["81"]]).to_frame(name='itemsets')


       itemsets
0          [26]
1  [81, 15, 27]
2          [50]
3           [8]
4      [81, 15]
5          [10]
6          [81]

rule = [["81"],["15"]]

我试过这个，但它很慢，因为数据集有 900 万行，所以我寻找一个有效的解决方案来执行这个代码，这个解决方案需要超过 4 秒，这是一种将数据帧转换为 numpy 的方法数组或执行此代码的所有内容？

def support(rule):
    rule = rule[0] + rule[1]
    support = 0
    support = sum([set(rule)<=set(row) for row in df])
return support/SIZE

when i try it on the largest dataset which contains 9 million transactions the result is :

support(rule)
0.001039247773829178

这个想法是计算数据帧中非严格子集的出现次数

Answer 1

IIUC：

将rule 定义为

rule = ['81', '15']

df.itemsets.apply(set).le(set(rule)).mean()

0.2857142857142857

---

Numpy 替代方案

应该加快速度

def support(rule, series):  
    n = len(series)  
    i = np.arange(n).repeat(series.str.len())
    out = np.ones(n, bool)
    np.logical_and.at(out, i, np.in1d(np.concatenate(series), rule))
    return out.mean()

support(rule, df.itemsets)

0.2857142857142857