在窗口函数中计数不同答案

【问题标题】：Count distinct in window functions在窗口函数中计数不同
【发布时间】：2020-02-09 10:38:15
【问题描述】：

我试图计算每个 c 的唯一列 b，而不进行分组。我知道这可以通过加入来完成。如何在不诉诸加入的情况下对（按 c 分区）进行计数（不同的 b）。为什么窗口函数不支持计数不同。先感谢您。给定这个数据框：

val df= Seq(("a1","b1","c1"),
                ("a2","b2","c1"),
                ("a3","b3","c1"),
                ("a31",null,"c1"),
                ("a32",null,"c1"),
                ("a4","b4","c11"),
                ("a5","b5","c11"),
                ("a6","b6","c11"),
                ("a7","b1","c2"),
                ("a8","b1","c3"),
                ("a9","b1","c4"),
                ("a91","b1","c5"),
                ("a92","b1","c5"),
                ("a93","b1","c5"),
                ("a95","b2","c6"),
                ("a96","b2","c6"),
                ("a97","b1","c6"),
                ("a977",null,"c6"),
                ("a98",null,"c8"),
                ("a99",null,"c8"),
                ("a999",null,"c8")
                ).toDF("a","b","c");

【问题讨论】：

最后这个对我有用：```` df. .withColumn("count_distinct", expr(" dense_rank() over (partition by c order by b desc)+dense_rank() over (partition by c order by b asc)- max(case when b is null then 1 else 0 end ) over (partition by c)-1 ")) ```` 让我知道这个是否有错误或问题。它还从计数中排除 null
您是在寻找纯 SQL 解决方案还是 spark 中的东西？你标记了两者。

标签： sql apache-spark apache-spark-sql

【解决方案1】：

一些数据库确实支持count(distinct) 作为窗口函数。有两种选择。一个是密集秩的总和：

select (dense_rank() over (partition by c order by b asc) +
        dense_rank() over (partition by c order by b desc) -
        1
       ) as count_distinct
from t;

第二个使用子查询：

select sum(case when seqnum = 1 then 1 else 0 end) over (partition by c)
from (select t.*, row_number() over (partition by c order by b) as seqnum
      from t
     ) t;

【讨论】：

【解决方案2】：

每个 c 的唯一列 b 的计数，而不进行分组。

典型的 SQL 解决方法是使用选择不同元组的子查询，然后在外部查询中使用窗口计数：

SELECT c, COUNT(*) OVER(PARTITION BY c) cnt
FROM (SELECT DISTINCT b, c FROM mytable) x

【讨论】：