如何在不使用 DISTINCT 的情况下从存储过程中删除重复项

Question

已编写包含重复项的存储过程。 ROW_NUMBER 已尝试但没有用。 DISTINCT 已工作，但无法检索所需的大量记录（约 700,000 条）。还有另一种使用 RANK 或 GROUP BY 删除重复项的方法吗？

我使用了 DISTINCT，但它没有检索到足够的记录。我没有成功使用 GROUP BY。

我曾尝试使用 ROW NUMBER，但这也不起作用（您可以看到它被注释掉的位置）。

CREATE PROCEDURE [report].[get_foodDetails] 
    @foodgroup_id INT, 
    @shop_id INT = 0, 
    @product_id INT = 0, 
    @maxrows INT = 600, 
    @expiry INT = 1, 
    @productactive INT = 1, 
    @expiryPeriod DATETIME = '9999-12-31 23:59:59' 
AS 
    IF (@expiryPeriod >= '9999-12-31') 
    BEGIN 
        SET @expiryPeriod = GETDATE() 
    END 

    SELECT  
        -- dp.RowNumber 
        ISNULL([FoodType], '') AS [Foodtype],
        ISNULL([FoodColour], '') AS [FoodColour],
        ISNULL([FoodBarcode], '') AS [FoodBarcode],
        ISNULL([FoodArticleNum], 0) AS [FoodArticleNum],
        ISNULL([FoodShelfLife, '9999-21-31') AS [FoodShelfLIFe]
    INTO 
        #devfood 
    FROM 
        report.[GetOrderList] (@foodgroup_id, @product_id, @productactive, @expiry, @expiryPeriod, @shop_id, @maxrows ) dp 
    INNER JOIN 
        food_group fg ON fg.food_group_id = it.item_FK_item_group_id 

    SELECT TOP(@maxrows) * 
    FROM #devfood 
    ORDER BY [device_packet_created_date]  
 END 

检索到大约 700,000 条记录。尽管有重复，但目前已实现。使用 DISTINCT 时只检索到 20,000 个（但没有重复）。

Answer 1

下面的示例代码来自我用来演示 CTE 的演示文稿。这是删除重复项的常用机制，而且速度非常快。在这种情况下，直接从表中删除重复项。如果这不是您的目标，您可以使用临时表或先前的链式 CTE。请注意，重要的是您分区的列。如果在示例中仅按 [name] 进行分区，则不会同时看到红玫瑰和白玫瑰。

-------------------------------------------------
if object_id(N'[flower].[order]', N'U') is not null
  drop table [flower].[order];

go

create table [flower].[order]
  (
     [id]       int identity(1, 1) not null constraint [flower.order.id.clustered_primary_key] primary key clustered
     , [flower] nvarchar(128)
     , [color]  nvarchar(128)
     , [count]  int
  );

go

insert into [flower].[order]
            ([flower]
             , [color]
             , [count])
values      (N'rose',N'red',5),
            (N'rose',N'red',3),
            (N'rose',N'white',2),
            (N'rose',N'red',1),
            (N'rose',N'red',9),
            (N'marigold',N'yellow',2),
            (N'marigold',N'yellow',9),
            (N'marigold',N'yellow',4),
            (N'chamomile',N'amber',9),
            (N'chamomile',N'amber',4),
            (N'lily',N'white',12);

go

select [flower]
       , [color]
from   [flower].[order];

go

--
-------------------------------------------------
with [duplicate_finder]([name], [color], [sequence])
     as (select [flower]
                , [color]
                , row_number()
                    over (
                      partition by [flower], [color]
                      order by [flower] desc) as [sequence]
         from   [flower].[order])
delete from [duplicate_finder]
where  [sequence] > 1;

--
-- no duplicates
-------------------------------------------------
select [flower]
       , [color]
from   [flower].[order]; 

Answer 2

我知道您说过您尝试过ROW_NUMBER，但您是否尝试过这两种方法？

首先，CTE。这里的CTE 只是您现有的查询，但附加了ROW_NUMBER 窗口函数。对于记录的每个重复迭代，它将向RowNumber 添加一个。对于下一组唯一记录，RowNumber 重置为 1。

拉取后，只取RowNumber = 1 的记录。我一直使用它来从基础记录集中删除重复项，但它也可以很好地识别它们。

WITH NoDupes AS
  (
    SELECT
      ROW_NUMBER() OVER (PARTITION BY
                           ISNULL(FoodType, '')
                          ,ISNULL(FoodColour, '')
                          ,ISNULL(FoodBarcode, '')
                          ,ISNULL(FoodArticleNum, '')
                          ,ISNULL(FoodShelfLife, '9999-21-31')
                         ORDER BY
                           (
                             SELECT
                               0
                           )
                        ) AS RowNumber
     ,ISNULL(FoodType, '') AS Foodtype
     ,ISNULL(FoodColour, '') AS FoodColour
     ,ISNULL(FoodBarcode, '') AS FoodBarcode
     ,ISNULL(FoodArticleNum, 0) AS FoodArticleNum
     ,ISNULL(FoodShelfLife, '9999-21-31') AS FoodShelfLIFe
    FROM
      report.GetOrderList(@foodgroup_id, @product_id, @productactive, @expiry, @expiryPeriod, @shop_id, @maxrows) AS dp
    INNER JOIN
      food_group AS fg
        ON
        fg.food_group_id = it.item_FK_item_group_id
  )
SELECT
  nd.Foodtype
 ,nd.FoodColour
 ,nd.FoodBarcode
 ,nd.FoodArticleNum
 ,nd.FoodShelfLIFe
INTO
  #devfood
FROM
  NoDupes AS nd
WHERE
  NoDupes.RowNumber = 1;

或者（更短）您可以尝试SELECT TOP (1) WITH TIES，使用相同的ROW_NUMBER 函数对记录集进行排序。 TOP (1) WITH TIES 部分的功能与CTE 相同，只返回每组重复项的第一条记录。

SELECT
  TOP (1) WITH TIES
  ISNULL(FoodType, '') AS Foodtype
 ,ISNULL(FoodColour, '') AS FoodColour
 ,ISNULL(FoodBarcode, '') AS FoodBarcode
 ,ISNULL(FoodArticleNum, 0) AS FoodArticleNum
 ,ISNULL(FoodShelfLife, '9999-21-31') AS FoodShelfLIFe
INTO
  #devfood
FROM
  report.GetOrderList(@foodgroup_id, @product_id, @productactive, @expiry, @expiryPeriod, @shop_id, @maxrows) AS dp
INNER JOIN
  food_group AS fg
    ON
    fg.food_group_id = it.item_FK_item_group_id
ORDER BY
  ROW_NUMBER() OVER (PARTITION BY
                       ISNULL(FoodType, '')
                      ,ISNULL(FoodColour, '')
                      ,ISNULL(FoodBarcode, '')
                      ,ISNULL(FoodArticleNum, '')
                      ,ISNULL(FoodShelfLife, '9999-21-31')
                     ORDER BY
                       (
                         SELECT
                           0
                       )
                    );

CTE 对于下一个查看代码的人的意图可能更清楚一些，但TOP 的性能可能会更好一些。