在 NULL 处拆分 sqlite 表

Question

我有一个如下所示的数据库：

 DT                      OB          OA         OL           OP  
"2015-01-21 00:12:00"   "1.15444"   "1.15467"   "1.15449"   "1.99999999999978"
"2015-01-21 00:32:00"   "1.15516"   "1.15468"   "1.15449"   "4.80000000000036"
"NULL"  "NULL"  "NULL"  "NULL"  "NULL"
"2015-01-21 01:06:00"   "1.15573"   "1.15534"   "1.15521"   "4.10000000000021"
"2015-01-21 08:04:00"   "1.15471"   "1.15582"   "1.15521"   "10.9000000000004"
"2015-01-21 08:10:00"   "1.15602"   "1.15514"   "1.15521"   "8.99999999999901"
"NULL"  "NULL"  "NULL"  "NULL"  "NULL"
"2015-01-21 02:32:00"   "1.15646"   "1.15615"   "1.15619"   "3.40000000000007"
"2015-01-21 05:18:00"   "1.15611"   "1.15644"   "1.15619"   "2.89999999999901"
"2015-01-21 05:30:00"   "1.15692"   "1.15631"   "1.15619"   "6.39999999999974"
"2015-01-21 07:32:00"   "1.15609"   "1.15685"   "1.15619"   "7.19999999999832"
"2015-01-21 07:42:00"   "1.15712"   "1.15643"   "1.15619"   "6.59999999999883"
"2015-01-21 07:56:00"   "1.15599"   "1.1567"    "1.15619"   "7.39999999999963"
"2015-01-21 08:40:00"   "1.15742"   "1.15628"   "1.15619"   "11.2000000000001"
"NULL"  "NULL"  "NULL"  "NULL"  "NULL"
"2015-01-21 03:40:00"   "1.15731"   "1.15664"   "1.15663"   "7.1000000000021"
"2015-01-21 04:34:00"   "1.15653"   "1.15684"   "1.15663"   "3.20000000000098"

一段时间以来，我一直在尝试将表拆分为空值，然后对数据库的“块”进行计算。我一直在尝试破解一些东西，但有时我在连续的行中有 2 或 3 个 NULL 值，然后有时我在最后得到一个 NULL，或者在开始时得到一个或两个 NULL，这意味着使用列表和索引来完成所有操作是很郁闷。

即这条路线：

dt = '2015-01-21'
ar2 = list([k[0] for k in cursor.execute("SELECT* FROM "+qi(dt)).fetchall()])

iters   = ([i+1 for i,k in enumerate(ar2) if k == None])
iters.append(0)
sit = sorted(iters,reverse = False)

kp = []
for k in range(0, len(sit)):
    if k < len(sit)-1:
        if k>1:
            start = str(sit[k-1]+1)
            fin   = str(sit[k]-1)

            oo = cursor.execute("SELECT * FROM "+qi(dt)+" WHERE rowid >= "+qi(start)+" AND rowid <= "+qi(fin)).fetchall()

我不建议您尝试理解我在该代码中所做的事情，但也许有人可以就更好的方法提供一些建议吗？ Pandas 可以轻松地做这样的事情吗？我的耳朵是张开的。

谢谢。

*请注意，有数百个类似的 db，如果只是这个，我会破解它。

*编辑

CREATE TABLE "2015-01-02" 
pair TEXT, 
side TEXT, 
t_time TEXT, 
open_price REAL, 
stop_loss REAL, 
p_line REAL, 
pip_risk REAL, 
max_min REAL, 
hour INTEGER, EOP REAL;

Answer 1

我的方法是使用itertools.groupby 将非 NULL 行块组合在一起。出于测试目的，我创建了自己的表，其中空行是所有 NULL 值的行：

sqlite> SELECT * FROM data;
501,haiv
502,johnk
,
601,karenc
602,richardc
,
701,petery
702,paul
703,maryt

我的代码：

import sqlite3
from itertools import groupby

def is_empty_row(row):
    """ Return True if the row contains all None, False otherwise """
    return not any(row)


if __name__ == '__main__':
    db = sqlite3.connect('data.sqlite3')
    cursor = db.cursor()
    query = 'SELECT * FROM data'
    result = cursor.execute(query)
    for is_empty, rows_group in groupby(result, is_empty_row):
        if is_empty:
            # Skip the empty rows
            continue

        # At this point, the non empty rows are grouped together
        for row in rows_group:
            print(row)
        print('---')

输出：

(501, u'haiv')
(502, u'johnk')
---
(601, u'karenc')
(602, u'richardc')
---
(701, u'petery')
(702, u'paul')
(703, u'maryt')
---

在这种方法中，我创建了一个辅助函数 is_empty_row，如果该行为空，则返回 True。 groupby 函数将根据该分类对行序列进行分组。其余代码应该不难理解。