在 PostgreSQL 中查询大量 squid 访问请求

Question

大家好，我在 Squid Proxy 中使用日志守护进程 (https://github.com/paranormal/blooper) 将访问日志放入 PostreSQL，并创建了一个触发函数：

DECLARE
  newtime varchar := EXTRACT (MONTH FROM NEW."time")::varchar;
  newyear varchar := EXTRACT (YEAR FROM NEW."time")::varchar;
  user_name varchar := REPLACE (NEW.user_name, '.', '_');
  partname varchar := newtime || '_' ||  newyear;
  tablename varchar := user_name || '.accesses_' || partname;
BEGIN

  IF NEW.user_name IS NOT NULL THEN
    EXECUTE 'CREATE SCHEMA IF NOT EXISTS ' || user_name;

    EXECUTE 'CREATE TABLE IF NOT EXISTS '
    || tablename
    || '('
    || 'CHECK (user_name = ''' || NEW.user_name || ''' AND EXTRACT(MONTH FROM "time") = ' || newtime || ' AND EXTRACT (YEAR FROM "time") = ' || newyear || ')'
    || ') INHERITS (public.accesses)';

    EXECUTE 'CREATE INDEX IF NOT EXISTS access_index_' || partname || '_user_name ON ' || tablename || ' (user_name)';
    EXECUTE 'CREATE INDEX IF NOT EXISTS access_index_' || partname || '_time ON ' || tablename || ' ("time")';

    EXECUTE 'INSERT INTO ' || tablename || ' SELECT $1.*' USING NEW;
  END IF;

  RETURN NULL;
END;

它的主要功能是根据用户名和访问的月份-年份进行表分区，继承自一个主清理表：

CREATE TABLE public.accesses
(
  id integer NOT NULL DEFAULT nextval('accesses_id_seq'::regclass),
  "time" timestamp with time zone NOT NULL,
  time_response integer,
  mac_source macaddr,
  ip_source inet NOT NULL,
  ip_destination inet,
  user_name character varying(40),
  http_status_code numeric(3,0) NOT NULL,
  http_reply_size bigint NOT NULL,
  http_request_method character varying(15) NOT NULL,
  http_request_url character varying(4166) NOT NULL,
  http_content_type character varying(100),
  squid_hier_code character varying(20),
  squid_request_status character varying(50),
  user_id integer,
  CONSTRAINT accesses_http_request_method_fkey FOREIGN KEY (http_request_method)
  REFERENCES public.http_requests (method) MATCH SIMPLE
  ON UPDATE NO ACTION ON DELETE NO ACTION,
  CONSTRAINT accesses_http_status_code_fkey FOREIGN KEY (http_status_code)
  REFERENCES public.http_statuses (code) MATCH SIMPLE
  ON UPDATE NO ACTION ON DELETE NO ACTION,
  CONSTRAINT accesses_user_id_fkey FOREIGN KEY (user_id)
  REFERENCES public.users (id) MATCH SIMPLE
  ON UPDATE NO ACTION ON DELETE NO ACTION
)

主要问题是通过用户名和时间得到http_reply_size分组的总和，我的查询是：

SELECT
  "time",
  user_name,
  sum(http_reply_size)
FROM
  accesses
WHERE
  extract(epoch from "time") BETWEEN 1516975122 AND 1516996722
GROUP BY
  "time",
  user_name

但此查询在服务器中非常慢（目前仅 2 天内 3'237'976 行）。因此，PostgreSQL 可以根据需要优化查询，或者我需要使用另一个 SQL 或 NoSQL 系统。

Answer 1

尝试在每个分区上包含CHECK 条件，这样就不必扫描所有表。

在我的情况下是这样的：

CREATE TABLE IF NOT EXISTS ' || table_name || '( 
        CONSTRAINT ' || pk || ' PRIMARY KEY (avl_id),
        CHECK ( event_time >= ''' || begin_time || ''' AND event_time < ''' || end_time || ''' )
        ) INHERITS (avl_db.avl);

也不要使用extract(epoch from "time")，因为它需要计算每一行的值，并且不能使用你为"time"创建的索引

所以像这样使用来获得索引的优势。

WHERE "time" >= '2018-01-01'::timestamp with time zone
  and "time" <  '2018-02-01'::timestamp with time zone