【发布时间】：2016-06-03 17:48:58
【问题描述】：
搜索了一段时间后，我发现了许多与此问题相关的问题/答案，但没有真正解决我正在寻找的问题。基本上，我在 python 中实现代码，以便能够从星目录（特别是第谷 2 星目录）中查询信息。
这些数据存储在一个较大的（约 0.5 GB）文本文件中，其中每一行对应一个星号条目。
一些示例行是
0001 00008 1| | 2.31750494| 2.23184345| -16.3| -9.0| 68| 73| 1.7| 1.8|1958.89|1951.94| 4|1.0|1.0|0.9|1.0|12.146|0.158|12.146|0.223|999| | | 2.31754222| 2.23186444|1.67|1.54| 88.0|100.8| |-0.2
0001 00013 1| | 1.12558209| 2.26739400| 27.7| -0.5| 9| 12| 1.2| 1.2|1990.76|1989.25| 8|1.0|0.8|1.0|0.7|10.488|0.038| 8.670|0.015|999|T| | 1.12551889| 2.26739556|1.81|1.52| 9.3| 12.7| |-0.2
0001 00016 1| | 1.05686490| 1.89782870| -25.9| -44.4| 85| 99| 2.1| 2.4|1959.29|1945.16| 3|0.4|0.5|0.4|0.5|12.921|0.335|12.100|0.243|999| | | 1.05692417| 1.89793306|1.81|1.54|108.5|150.2| |-0.1
0001 00017 1|P| 0.05059802| 1.77144349| 32.1| -20.7| 21| 31| 1.6| 1.6|1989.29|1985.38| 5|1.4|0.6|1.4|0.6|11.318|0.070|10.521|0.051| 18|T| | 0.05086583| 1.77151389|1.78|1.55| 30.0| 45.6|D|-0.2
信息既是定界的，也是固定宽度的。每列包含有关该恒星的不同信息。现在，对于我的 python 实用程序，我希望能够快速搜索这些信息并检索与用户指定的一组标准匹配的星的条目。
例如，我希望能够找到所有亮度大于 5.5（col 18 或 19）、赤经在 0 到 30 度（col 3）之间、赤纬在 -45 到 -35 之间的恒星度（col 4）有效。现在，如果我可以将所有这些信息存储在内存中，那么将文件读入一个 numpy 结构化数组或 pandas 数据帧并使用逻辑索引检索我想要的星星就很容易了。不幸的是，我正在使用的机器没有足够的内存来执行此操作（在任何给定时间我只有大约 0.5 GB 的可用内存，而我正在使用的其余程序占用了大量内存）。
我当前的解决方案包括遍历文本文件的每一行、解释数据以及仅当条目与指定的条件匹配时才将条目存储在内存中。我必须这样做的方法是
def getallwithcriteria(self, min_vmag=1., max_vmag=17., min_bmag=1., max_bmag=17., min_ra=0., max_ra=360.,
min_dc=-90., max_dc=90., min_prox=3, search_center=None, search_radius=None):
"""
This method returns entire star records for each star that meets the specified criterion. The defaults for each
criteria specify the entire range of the catalogue. Do not call this without changing the defaults as this will
likely overflow memory and cause your system to drastically slow down or crash!
Note that all of the keyword argument do not need to be specified. For instance, we could run
import tychopy as tp
tyc = tp.Tycho('/path/to/catalogue')
star_records = tyc.getallwithcritera(min_vmag=3,max_vmag=4)
to return all stars that have a visual magnitude between 3 and 4.
This method returns a numpy structured array where each element contains the complete record for a star that
matches the criterion specified by the user. The output array has the following dtype:
[('starid', 'U12'),
('pflag', 'U1'),
('starBearing', [('rightAscension', float), ('declination', float)]),
('properMotion', [('rightAscension', float), ('declination', float)]),
('uncertainty', [('rightAscension', int), ('declination', int), ('pmRA', float), ('pmDc', float)]),
('meanEpoch', [('rightAscension', float), ('declination', float)]),
('numPos', int),
('fitGoodness', [('rightAscension', float), ('declination', float), ('pmRA', float), ('pmDc', float)]),
('magnitude', [('BT', [('mag', float), ('err', float)]), ('VT', [('mag', float), ('err', float)])]),
('starProximity', int),
('tycho1flag', 'U1'),
('hipparcosNumber', 'U9'),
('observedPos', [('rightAscension', float), ('declination', float)]),
('observedEpoch', [('rightAscension', float), ('declination', float)]),
('observedError', [('rightAscension', float), ('declination', float)]),
('solutionType', 'U1'),
('correlation', float)]
see the readme of the Tycho 2 catalogue for a more formal description of each field.
If no stars are found that match the specified input then an empty numpy array with the above dtype is returned.
Note that both a rectangular and a circular area can be specified. The rectangular search area is specified
using the min_ra/dc max_ra/dc keyword arguments while the circular search area is specified using the
search_center and search_radius keyword arguments where the search_center is a tuple, list, numpy array, or
other array like object which contains the center right ascension in element 0 and the center declination in
element 1. It is not recommended to specify both the circular and rectangular search areas. If the search
areas do not overlap then no stars will be returned.
:param min_vmag: the minimum (brightest) visual magnitude to return
:param max_vmag: the maximum (dimmest) visual magnitude to return
:param min_bmag: the minimum (brightest) blue magnitude to return
:param max_bmag: the maximum (dimmest) blue magnitude to return
:param min_ra: the minimum right ascension to return
:param max_ra: the maximum right ascension to return
:param min_dc: the minimum declination to return
:param max_dc: the maximum declination to return
:param min_prox: the closest proximity to a star to return
:param search_center: An array like object containing the center point from which to search radially for stars.
:param search_radius: A float specifying the radial search distance to use
:return: A numpy structure array containing the star records for stars that meet the specified criteria
"""
# form the dtype list that genfromtxt will use to interpret the star records
dform = [('starid', 'U12'),
('pflag', 'U1'),
('starBearing', [('rightAscension', float), ('declination', float)]),
('properMotion', [('rightAscension', float), ('declination', float)]),
('uncertainty', [('rightAscension', int), ('declination', int), ('pmRA', float), ('pmDc', float)]),
('meanEpoch', [('rightAscension', float), ('declination', float)]),
('numPos', int),
('fitGoodness', [('rightAscension', float), ('declination', float), ('pmRA', float), ('pmDc', float)]),
('magnitude', [('BT', [('mag', float), ('err', float)]), ('VT', [('mag', float), ('err', float)])]),
('starProximity', int),
('tycho1flag', 'U1'),
('hipparcosNumber', 'U9'),
('observedPos', [('rightAscension', float), ('declination', float)]),
('observedEpoch', [('rightAscension', float), ('declination', float)]),
('observedError', [('rightAscension', float), ('declination', float)]),
('solutionType', 'U1'),
('correlation', float)]
# initialize a list which will contain the star record strings for stars that match the input criteria
records = []
# loop through each record in the Tycho2 catlogue
for record in self._catalogueFile:
# interpret the record as simply as we can
split_record = record.split(sep="|")
# check that we are examining a good star, that it falls within the bearing bounds, and that it is far
# enough away from other stars
if ("X" not in split_record[1]) and min_ra <= float(split_record[2]) <= max_ra \
and min_dc <= float(split_record[3]) <= max_dc and int(split_record[21]) >= min_prox:
# perform the radial search if the user has specified a center and radius
if search_center is None or pow(pow(float(split_record[2])-search_center[0], 2) +
pow(float(split_record[3])-search_center[1], 2), 1/2.) < search_radius:
# Check to see if we have values for both blue and visual magnitudes, and check to see if these
# magnitudes fall within the specified magnitude bounds
# We need to split this up like this in order to make sure that either the bmag or the vmag exist
if bool(split_record[17].strip()) and bool(split_record[19].strip()) \
and min_bmag <= float(split_record[17]) <= max_bmag \
and min_vmag <= float(split_record[19]) <= max_vmag:
records.append(record+'\n')
# if only the visual magnitude exists then check its bounds - also check if the user has specified
# its bounds
elif not bool(split_record[17].strip()) and bool(split_record[19].strip()) \
and min_vmag <= float(split_record[19]) <= max_vmag and (max_vmag != 17. or min_vmag != 1.):
records.append(record+'\n')
# if only the blue magnitude exists the check its bounds - also check if the user has specified its
# bounds
elif not bool(split_record[19].strip()) and bool(split_record[17].strip()) \
and min_bmag <= float(split_record[17]) <= max_bmag and (max_bmag != 17. or min_bmag != 1.):
records.append(record+'\n')
# otherwise check to see if the use has changed the defaults. If they haven't then store the star
elif max_bmag == 17. and max_vmag == 17. and min_bmag == 1. and min_vmag == 1.:
records.append(record+'\n')
# check to see if any stars met the criteria. If they didn't then return an empty array. If they did then use
# genfromtxt to interpret the string of star records
if not bool(records):
nprecords = np.empty((1,), dtype=dform)
warnings.warn('No stars were found meeting your criteria. Please try again.')
else:
nprecords = np.genfromtxt(BytesIO("".join(records).encode()), dtype=dform, delimiter='|', converters={
0: lambda s: s.strip(),
1: lambda s: s.strip(),
22: lambda s: s.strip(),
23: lambda s: s.strip(),
30: lambda s: s.strip()})
if self._includeProperMotion:
applypropermotion(nprecords, self.newEpoch, copy=False)
# reset the catalogue back to the beginning for future searches
self._catalogueFile.seek(0, os.SEEK_SET)
return nprecords
这仍然很慢（尽管比用完所有内存并将其他所有内容都放入交换中要快）。作为比较，每次我需要检索星星大约需要 2-3 分钟，并且我需要在我正在编写的程序中从这 40 次左右（每次使用不同的标准）中检索星星。程序的其余部分总共需要大约 5 秒。
我现在的问题是，加快此过程的最佳方法是什么（除了获得具有更多内存的更好计算机之外）。我愿意接受任何建议，只要它们得到很好的解释并且不会花费我几个月的时间来实施。我什至愿意编写一个函数，将原始目录文件修改为更好的格式（按特定列排序的固定宽度二进制文件），以加快速度。
到目前为止，我已经考虑过对文件进行 memmap 处理，但决定反对它，因为我真的认为这对我需要做的事情没有帮助。我还考虑过从数据中创建一个数据库，然后使用 sqlalchemy 或类似的东西以这种方式查询数据；但是，我对数据库不是很熟悉，不知道这是否会带来任何真正的速度提升。
【问题讨论】：
-
您是否考虑过将数据文件转换为适合重复查询的内容，例如HDF5 store？第一次这样做会产生少量的计算和硬盘存储成本，但会显着加快查询速度。
-
我考虑过将文件重写为类似的东西，但不是专门针对 HDF5。我会调查它，看看它是否能满足我的需要
-
那个文件是 ASCII 码吗？可以分块吗？搜索查询是否从整个文件中获取结果，或者您是否可以期待本地化结果？ Numpy 二进制文件的加载速度非常快（np.load 和 np.save）。假设您可以将数据切成小块，您可以单独（或在线程中）运行每个位。
-
你能构建一个或多个索引，将最重要的列映射到行号和
seek位置吗？ -
@armatita 是的，文件是 ASCII。我曾考虑将其切割成 ra 和 dec 的块（即，将其拆分为较小的文件，其中每个较小的文件只考虑天空的一部分）。如果没有其他选项出现，那可能是我需要走的路。此外，文件的排序方式现在查询通常从整个文件中检索片段。一般来说，虽然大多数搜索将集中在位置上，所以如果文件以这种方式排序，那么结果可能会被分块。
标签： python database numpy pandas