将 MultiIndex Pandas DataFrame 添加到 PyTables HDFStore 的问题

Question

我在一个充满栅格的列表上使用 for 循环。在每个栅格中，我提取一个数据数组，并且我想使用栅格的基本名称（日期）作为该数组的索引。为此，我使用 Pandas DataFrame 多索引。然后将包含新集合索引的数组附加到 HDFStore。接下来选择另一个日期的栅格

代码sn-ps：

root, ext = os.path.splitext(raster)
name = int(decimal.Decimal(os.path.basename(root)))

array = ma.MaskedArray.compressed(raster)
arr2df = pd.DataFrame(pd.Series(data = array), columns=['rastervalue'])
arr2df['timestamp'] = pd.Series(name,index=arr2df.index)
arr2df.set_index('timestamp')
store.append('rastervalue',arr2df)

DataFrame 似乎没问题（顺便说一句，我怎样才能检索 MultiIndex？）。

>>> arr2df
<class 'pandas.core.frame.DataFrame'>
  MultiIndex: 123901 entries, (0, 20060101) to (123900, 20060101)
  Data columns (total 1 columns):
  rastervalue    123901  non-null values
  dtypes:        int32(1)

但是在我检查 HDFStore 的那一刻，我的 Multi-Index 似乎消失了并变成了“values_block_1”

>>> store.root.rastervalue.table.read
<bound method Table.read of /rastervalue/table (Table(12626172,)) ''
  description := {
  "index": Int64Col(shape=(), dflt=0, pos=0),
  "values_block_0": Int32Col(shape=(1,), dflt=0, pos=1),
  "values_block_1": Int64Col(shape=(1,), dflt=0, pos=2)}
  byteorder := 'little'
  chunkshape := (3276,)
  autoIndex := True
  colindexes := {
    "index": Index(6, medium, shuffle, zlib(1)).is_CSI=False}>

>>> store.root.rastervalue.table.read(field="values_block_1")
array([[20060101],
       [20060101],
       [20060101],
       ...,
       [ 20060914],
       [ 20060914],
       [ 20060914]], dtype=int64)

通过阅读documentation，我无法弄清楚如何正确存储或更改 HDFStore 中的 MultiIndex。有什么建议？最终我想查询表为：

 store.select('rastervalue', [ pd.Term('index', '=', '20060101')])

Answer 1

这是一个工作示例。

In [43]: df = DataFrame(dict(ivalue = range(123901), date = 20060101, 
              value = Series([1]*123901,dtype='int32'))).set_index(['ivalue','date'])

In [44]: df
Out[44]: 
<class 'pandas.core.frame.DataFrame'>
MultiIndex: 123901 entries, (0, 20060101) to (123900, 20060101)
Data columns (total 1 columns):
value    123901  non-null values
dtypes: int32(1)

In [45]: df.head()
Out[45]: 
                 value
ivalue date           
0      20060101      1
1      20060101      1
2      20060101      1
3      20060101      1
4      20060101      1

In [46]: store = pd.HDFStore('test.h5',mode='w')

In [48]: store.append('df',df)

In [49]: store
Out[49]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df            frame_table  (typ->appendable_multi,nrows->123901,ncols->3,indexers->[index],dc->[date,ivalue])

In [50]: store.get_storer('df')
Out[50]: frame_table  (typ->appendable_multi,nrows->123901,ncols->3,indexers->[index],dc->[date,ivalue])
In [51]: store.get_storer('df').attrs
Out[51]: 
/df._v_attrs (AttributeSet), 14 attributes:
   [CLASS := 'GROUP',
    TITLE := '',
    VERSION := '1.0',
    data_columns := ['date', 'ivalue'],
    encoding := None,
    index_cols := [(0, 'index')],
    info := {'index': {}},
    levels := ['ivalue', 'date'],
    nan_rep := 'nan',
    non_index_axes := [(1, ['ivalue', 'date', 'value'])],
    pandas_type := u'frame_table',
    pandas_version := '0.10.1',
    table_type := u'appendable_multiframe',
    values_cols := ['values_block_0', 'date', 'ivalue']]

In [52]: store.get_storer('df').table
Out[52]: 
/df/table (Table(123901,)) ''
  description := {
  "index": Int64Col(shape=(), dflt=0, pos=0),
  "values_block_0": Int32Col(shape=(1,), dflt=0, pos=1),
  "date": Int64Col(shape=(), dflt=0, pos=2),
  "ivalue": Int64Col(shape=(), dflt=0, pos=3)}
  byteorder := 'little'
  chunkshape := (2340,)
  autoIndex := True
  colindexes := {
    "date": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "index": Index(6, medium, shuffle, zlib(1)).is_CSI=False,
    "ivalue": Index(6, medium, shuffle, zlib(1)).is_CSI=False}