Django - 复杂查询

Question

假设我有两个模型：

class Profile(models.Model):
    #some fields here

class Ratings(models.Model):
    profile = models.ForeignKey(profile)
    category = models.IntegerField()
    points = models.IntegerField()

假设以下 MySQL 表“评级”示例：

profile    |    category    |    points
   1                1               10
   1                1               4
   1                2               10
   1                3               0
   1                4               10
   1                4               10
   1                4               10
   1                5               0

我的 POST 数据和其他字段值中有以下值：

category_1_avg_val = 7
category_2_avg_val = 5
category_3_avg_val = 5
category_4_avg_val = 7
category_5_avg_val = 9

我想过滤平均评分高于或等于所需值的类别的配置文件。

一些过滤器最初应用为：

q1 = [('associated_with', search_for),
      ('profile_type__slug__exact', profile_type),
      ('gender__in', gender),
      ('rank__in', rank),
      ('styles__style__in', styles),
      ('age__gte', age_from),
      ('age__lte', age_to)]
q1_list = [Q(x) for x in q1 if x[1]]

q2 = [('user__first_name__icontains', search_term),
      ('user__last_name__icontains', search_term),
      ('profile_type__name__icontains', search_term),
      ('styles__style__icontains', search_term),
      ('rank__icontains', search_term)]
q2_list = [Q(x) for x in q2 if x[1]]

if q1_list:
    objects = Profile.objects.filter(
        reduce(operator.and_, q1_list))

if q2_list:
    if objects:
        objects = objects.filter(
            reduce(operator.or_, q2_list))
    else:
        objects = Profile.objects.filter(
            reduce(operator.or_, q2_list))

if order_by_ranking_level == 'desc':
    objects = objects.order_by('-ranking_level').distinct()
else:
    objects = objects.order_by('ranking_level').distinct()

现在我想过滤其（平均点数）（按类别分组）>=（帖子中类别的平均值）的配置文件

我试着一一做到这一点

objects = objects.filter(
    ratings__category=1) \
    .annotate(avg_points=Avg('ratings__points'))\
    .filter(avg_points__gte=category_1_avg_val)


objects = objects.filter(
    ratings__category=2) \
    .annotate(avg_points=Avg('ratings__points'))\
    .filter(avg_points__gte=category_2_avg_val)

但我认为这是错误的。请帮帮我。如果 return 是一个很好的查询集。

已编辑 使用hynekcer 发布的答案，我想出了稍微不同的解决方案，因为我已经查询了需要根据评级进行更多过滤的配置文件集。

def check_ratings_avg(pr, rtd):
    ok = True
    qr = Ratings.objects.filter(profile__id=pr.id) \
        .values('category')\
        .annotate(points_avg=Avg('points'))
    qr = {i['category']:i['points_avg'] for i in qr}

    for cat in rtd:
        val = rtd[cat]
        if qr[cat] >= val:
            pass
        else:
            ok = False
            break
    return ok


rtd = {1: category_1_avg_val, 2: category_2_avg_val, 3: category_3_avg_val,
       4: category_4_avg_val, 5: category_5_avg_val}
objects = [i for i in objects if check_ratings_avg(i, rtd)]

Answer 1

您可以向经理添加方法

# Untested code
class ProfileManager(models.Manager):
    def with_category_average(self, cat, avg):
        # Give each filter a unique annotation key
        key = 'avg_pts_' + str(cat)
        return self.filter(ratings__category=cat) \
                   .annotate(**{key: Avg('ratings__points')}) \
                   .filter(**{key + '__gte': avg})

    # Expects a dict of `cat: avg` pairs
    def filter_by_averages(self, avg_dict):
        qs = self.get_query_set()
        for key, val in avg_dict.items():
            qs &= self.with_category_average(key, val)
        return qs

Answer 2

您的复杂查询原则上需要一个子查询。可能的解决方案是：

• 由'extra' queryset method 或raw SQL 编写的子查询 查询。它不是 DRY，并且某些数据库后端不支持它，例如在某些版本的 MySQL 中，子查询从 Django 1.1 开始以某种有限的方式使用。
• 将中间结果保存到数据库中的临时表中。这在 Django 中并不好。
• 在 Python 中通过循环模拟外部查询。最好的通用解决方案。在 Python 中对由第一个查询聚合的数据库数据进行循环可以足够快地聚合和过滤数据。

A) Python 模拟的子查询

from django.db.models import Q, Avg
from itertools import groupby
from myapp.models import Profile, Ratings

def iterator_filtered_by_average(dictionary):
    qr = Ratings.objects.values('profile', 'category', 'points').order_by(
            'profile', 'category').annotate(points_avg=Avg('points'))
    f = Q()
    for k, v in dictionary.iteritems():
        f |= Q(category=k, points_avg__gte=v)
    for profile, grp in groupby(qr.filter(f).values('profile')):
        if len(list(grp)) == len(dictionary):
            yield profile

#example
FILTER_DATA = {1:category_1_avg_val, 2:category_2_avg_val, 3:category_3_avg_val,
               4:category_4_avg_val, 5:category_5_avg_val}
for row in iterator_filtered_by_average(FILTER_DATA):
    print row

这是原始问题的简单解决方案，无需后续额外要求。

B) 带有子查询的解决方案：
对于更详细的问题版本是必要的，因为如果初始过滤器基于 ManyToManyField 类型的某些字段，并且还因为它包含 distinct 子句：

# objects:  QuerySet that you get from your initial filters. Not yet executed.
if rtd:
    # Method `as_nested_sql` removes the `order_by` clase, unlike `as_sql`
    subquery3 = objects.values('id').query \
            .get_compiler(connection=connection).as_nested_sql()
    subquery2 = ("""SELECT profile_id, category, avg(points) AS points_avg
          FROM myapp_ratings
          WHERE profile_id in
          ( %s
          ) GROUP BY profile_id, category
            """ % subquery3[0], subquery3[1]
    )
    where_sql = ' OR '.join(
            'category = %d AND points_avg >= %%s' % cat for cat in rtd.keys()
    )
    subquery = (
        """SELECT profile_id
        FROM
        ( %s
        ) subquery2
        WHERE %s
        GROUP BY profile_id
        HAVING count(*) = %s
        """ % (subquery2[0], where_sql, len(rtd)),
        subquery2[1] + tuple(rtd.values())
    )
    assert order_by_ranking_level in ('asc', 'desc')
    mainquery = ("""SELECT myapp_profile.* FROM myapp_profile
      INNER JOIN
      ( %s
      ) subquery ON subquery.profile_id=myapp_profile.id
      ORDER BY ranking_level %s"""
        % (subquery[0], order_by_ranking_level), subquery[1]
    )
    objects = Profile.objects.raw(mainquery[0], params=mainquery[1])
return objects

请将所有字符串 myapp 替换为 name_of_your_application。

此代码生成的 SQL 示例

SELECT myapp_profile.* FROM myapp_profile
  INNER JOIN
  ( SELECT profile_id
    FROM
    ( SELECT profile_id, category, avg(points) AS points_avg
      FROM myapp_ratings
      WHERE profile_id IN
      ( SELECT U0.`id` FROM `myapp_profile` U0 WHERE U0.`ranking_level` >= 4
      ) GROUP BY profile_id, category
    ) subquery2
    WHERE category = 1 AND points_avg >= 7 OR category = 2 AND points_avg >= 5
       OR category = 3 AND points_avg >= 5 OR category = 4 AND points_avg >= 7
       OR category = 5 AND points_avg >= 9
    GROUP BY profile_id
    HAVING count(*) = 5
  ) subquery ON subquery.profile_id=myapp_profile.id
  ORDER BY ranking_level asc

（此 SQL 是为了提高可读性，手动解析字符串 %s 并替换为参数，但出于安全原因，数据库引擎接收未解析的参数。）

---

您的问题是由于对 Django 生成的子查询的支持很少。只有来自更复杂查询文档的示例才会创建子查询。 （例如，aggregate 在annotate 之后或count 在annotate 或aggregate 在distinct 之后，但在distinct 或annotate 之后没有annotate）复杂的嵌套聚合被简化为一个查询出乎意料。

对于生产环境中不鼓励为第一个查询过滤的每个对象执行新的单个 SQL 查询的所有其他解决方案，尽管它们对于测试任何更好解决方案的结果非常有用。