【问题标题】:row_number and count very slow in query with dynamic sql and CTE, how to implement skip and take动态sql和CTE查询row_number和count很慢,如何实现skip和take
【发布时间】:2015-03-29 00:38:08
【问题描述】:

我有以下场景:一个最多包含 20000 个条目的表和另一个包含相应自定义字段的表。 我需要在所有列(包括自定义字段)上实现一个带有过滤机会的查询,并跳过并获取,并且我需要过滤后的总行数。 在动态 sql 的帮助下,我设法实现了一个查询,它将自定义字段作为列添加到第一个表中。 但是我真的很难实现运行非常快并且还返回总行数的skip and take功能。 由于并非我们所有的客户都在 SQL Server 2012 上运行,因此最好的方法是通过 row_number 上的 where 子句实现 take 和 skip,但我认为这也是最慢的选择。我喜欢 2012 案例的 OFFSET 和 FETCH 功能,但仍然获取行数似乎减慢了很多。

这是我从动态 sql 生成的查询,位于整个动态查询的下方,两者都有两个替代项作为 cmets

With tempTable AS 
(
SELECT *
 --1. ALTERNATIVE:
,ROW_NUMBER() Over (ORDER BY  Nachname, Vorname desc) As  ROW_NUMBER ,COUNT(1) OVER () as Total_Rows
FROM
(
    SELECT  [a].*, [DFSF].[Datenfeld_Name],[Datenfeld_Inhalt]
    FROM 
    [dbo].[Ansprechpartner] AS a left join [dbo].[Datenfeld] AS dfe
    ON [a].[Id] = [dfe].[Datenfeld_AnsprechpartnerID]
    left join
    [Datenfeld_Standardfelder] AS DFSF 
    ON dfe.[StandardfeldID] = [DFSF].[id] and datenfeld_kategorie = 'Ansprechpartner'   

) AS j
PIVOT
(
  max([Datenfeld_Inhalt]) FOR [j].[Datenfeld_Name]  IN ([Medium],[Kontaktthema],[Mediengattung],[Medienthema],[E-Mail],[Homepage],[Rolle])
) AS p
)
SELECT *
--2. ALTERNATIVE:
--,COUNT(1) OVER ()
FROM tempTable
 WHERE 1=1
 -- 1. ALTERNATIVE:
and Row_Number BETWEEN 0 AND 100
ORDER BY  Nachname, Vorname DESC
--2. ALTERNATIVE:
OFFSET 0 ROWS FETCH NEXT 100 ROWS ONLY
 ; 

并跟踪整个动态查询。 我实际上认为,其中有一个错误,因为这样我不会得到正确的行数,我可能不得不在没有 row_number 过滤器的情况下再次调用它才能正确...

DECLARE @filterExpression nvarchar(MAX)
DECLARE @showOnlyDoublets int
DECLARE @sortExpression nvarchar(MAX)
 DECLARE @skip AS int
 DECLARE @take AS [int]
 SELECT @skip = 0
 SELECT @take = 100
--SELECT @filterExpression = 'WHERE Vorname like ''%luc%'''
SELECT @filterExpression = ' WHERE 1=1'
SELECT @sortExpression = 'ORDER BY  Nachname, Vorname desc'
SELECT @showOnlyDoublets = 0

DECLARE @idList nvarchar(MAX)
select @idList =  COALESCE(@idList + '],[', '[') + [DFSF].[Datenfeld_Name] from
[Datenfeld_Standardfelder] AS DFSF 
where datenfeld_kategorie = 'Ansprechpartner'

SELECT @idList = @idList +']'
--SELECT @idList

DECLARE @sqlToRun nvarchar(max)
SET @sqlToRun = 
'With tempTable As 
(
SELECT *
, ROW_NUMBER() Over (' + @sortExpression + ') As  Row_Number 
FROM
(
    SELECT  [a].*, [DFSF].[Datenfeld_Name],[Datenfeld_Inhalt]--, CAST( ROW_NUMBER()  OVER(ORDER BY [DFSF].[Datenfeld_Name] DESC) AS varchar(20)) 
    FROM 
    [dbo].[Ansprechpartner] AS a left join [dbo].[Datenfeld] AS dfe
    ON [a].[Id] = [dfe].[Datenfeld_AnsprechpartnerID]
    left join
    [Datenfeld_Standardfelder] AS DFSF 
    ON dfe.[StandardfeldID] = [DFSF].[id] and datenfeld_kategorie = ''Ansprechpartner'' 

) AS j
PIVOT
(
  max([Datenfeld_Inhalt]) FOR [j].[Datenfeld_Name]  IN (' + @idList + ')
) AS p
)
SELECT *,  COUNT(*) OVER () as Total_Rows FROM tempTable
' + @filterExpression +  '
AND Row_Number BETWEEN ' + CAST ( @skip AS varchar ) + ' AND ' + CAST ( @take AS varchar ) + '
' + @sortExpression + '
--OFFSET ' + CAST ( @skip AS varchar ) + ' ROWS FETCH NEXT ' + CAST ( @take AS varchar ) + ' ROWS ONLY
 ;'


PRINT @sqlToRun
EXECUTE sp_executesql @sqlToRun

所以我的问题是:有没有办法改进这个查询(两种选择之一)?还是您有完全不同的想法,因为我认为,无论哪种方式,如果我正确调用计数,都会花费很多时间。

【问题讨论】:

    标签: sql-server performance count dynamic-sql skip-take


    【解决方案1】:

    弗里德,

    看看下面这个存储过程。我添加的 cmets 希望能有所帮助。

    它提供整个查询的总行数,无论每页返回多少条记录或用户在哪个页面上。可能干扰您的行数的是行号是代码中 where 子句的一部分。在此示例中查看如何返回子集,而无需在 where 子句中处理该逻辑。

    您的性能问题可能与索引和数据透视有关。我看到您正在通过任意文本输入过滤列。您需要将该列与所有其他返回的列一起编入索引。下面的查询中描述了一种可能加快所有速度的替代方法,在该查询中,您首先找到与您的谓词匹配的所有 ID(在搜索列上具有非聚集索引并包括主键 ID 列),然后加入实际查询到那张桌子。这将整个操作(包括枢轴)限制为仅匹配的行。

    CREATE PROC [api].[cc_pcp_member_search] 
    (
         @string VARCHAR(255)    
        ,@plan_long_id BIGINT 
        ,@num_rows BIGINT
        ,@page_num BIGINT
        ,@order_by VARCHAR(255) = 'last_name'
        ,@sort_order VARCHAR(10) = 'ASC'
    )
    
    AS 
    SET NOCOUNT ON
    
    DECLARE @MemberList AS TABLE (MemberId INT)
    INSERT INTO @MemberList
    
    /* What is happening in the EXEC statement below, is that I am running a text search against
    very specific indexes that only return the ID's(clustered) of the items I care about.
    This has proven to be far more performant than applying the text predicate to the entire
    set due to indexing reasons. The 'real' query further below that brings back all the needed fields
    uses this list of ID's to define the scope of the set. You might consider something similar if 
    you are querying aginst arbitrary text values.
    */
    EXEC dbo.uspGetMemberIdForMemberSearch @string 
    
    /* The table variable does slow things down and you might not need it at all.
    In this case, the developers were using Entity Framework and it was having 
    trouble determining the signature and data types of the stored procedure without
    a typed result set
    */
    DECLARE @ResultSet AS TABLE (   member_id BIGINT,
                    first_name VARCHAR(255),
                    last_name VARCHAR(255),
                    full_name VARCHAR(255),
                    hicn VARCHAR(255),
                    gender VARCHAR(255),
                    plan_id BIGINT,
                    plan_name VARCHAR(255),
                    phone_1 VARCHAR(255),
                    phone_2 VARCHAR(255),
                    dob DATETIME,
                    total_records INT)
    
    INSERT INTO @ResultSet
    SELECT
    CAST(M.MemberId AS BIGINT) AS member_id,
    M.FirstName AS first_name,
    M.LastName AS last_name,
    CONCAT(M.FirstName, ' ', M.LastName) AS full_name,
    M.HICN AS hicn,
    CASE WHEN M.Gender = 0 THEN 'F' ELSE 'M' END AS gender,
    CAST(P.VirtusPlanId AS BIGINT) AS plan_id,
    P.PlanName AS plan_name,
    M.PhoneNumber1 AS phone_1,
    M.PhoneNumber2 AS phone_2,
    M.DateOfBirth AS dob,
    CAST(0 AS INT) AS total_records --<<< This is the place holder for total count, see further below
    FROM
      Member M
      INNER JOIN [Plan] P
        ON M.PlanId = P.PlanId
      INNER JOIN @MemberList ML --<<< This is where the final filtering on ID's happens
        ON M.MemberId = ML.MemberId
    
    /* Heres the core of what you are probably dealing with. The proc allows the caller
    to specify sort order, query string, records per page, and current page in the set.
    The users will usually search by name, sort by last name, and the grid then pages
    through those results 10 rows (defaulted in app) at a time.
    */
    DECLARE @InputPageNumber int = @page_num
    DECLARE @RowsPerPage INT = @num_rows
    DECLARE @RealPageNumber INT = @InputPageNumber + 1 --<<< 0 based index adjustment
    DECLARE @OrderBy VARCHAR(255) = @order_by
    DECLARE @SortOrder VARCHAR(4) = @sort_order
    
    
      SELECT
         member_id
        ,first_name
        ,last_name
        ,full_name
        ,hicn
        ,gender
        ,plan_id
        ,plan_name
        ,phone_1
        ,phone_2
        ,dob
        /* Here is your total row count of the set regardless of how many
        rows are being paged at the moment. Because OVER() is counting the
        filtered set in the table variable only, it is fast. It would do the same
        with a CTE. You're already using the same thing.
        */
        ,CAST(COUNT(*) OVER() AS INT) AS total_records
      FROM @ResultSet
      WHERE plan_id = @plan_long_id
    
        /* In this usecase, the output of this query feeds a paged web grid. Below is the 
        logic used to determine the sort column at execution time. I'm not pretending it's 
        pretty but it works.
        */
        ORDER BY CASE 
            WHEN @OrderBy = 'first_name' AND @SortOrder = 'ASC'
              THEN first_name
            WHEN @OrderBy = 'last_name' AND @SortOrder = 'ASC'
              THEN last_name
            WHEN @OrderBy = 'full_name' AND @SortOrder = 'ASC'
              THEN full_name
            WHEN @OrderBy = 'hicn' AND @SortOrder = 'ASC'
              THEN hicn
            WHEN @OrderBy = 'plan_name' AND @SortOrder = 'ASC'
              THEN plan_name
            WHEN @OrderBy = 'gender' AND @SortOrder = 'ASC'
              THEN gender
            END ASC,
            CASE 
            WHEN @OrderBy = 'first_name' AND @SortOrder = 'DESC'
              THEN first_name
            WHEN @OrderBy = 'last_name' AND @SortOrder = 'DESC'
              THEN last_name
            WHEN @OrderBy = 'full_name' AND @SortOrder = 'DESC'
              THEN full_name
            WHEN @OrderBy = 'hicn' AND @SortOrder = 'DESC'
              THEN hicn
            WHEN @OrderBy = 'plan_name' AND @SortOrder = 'DESC'
              THEN plan_name
            WHEN @OrderBy = 'gender' AND @SortOrder = 'DESC'
              THEN gender
            END DESC
      OFFSET (@RealPageNumber - 1) * @RowsPerPage ROWS  -- dynamic offset arithmetic to convert requested page number to row offset
      FETCH NEXT @RowsPerPage ROWS ONLY;                -- dynamic number of rows to display per page 
    

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 2020-12-31
      • 2012-06-12
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2011-11-18
      • 2018-02-03
      • 1970-01-01
      相关资源
      最近更新 更多