Elasticsearch Spring boot findAll 结果窗口太大，from+size 必须小于等于：[10000] 但为[331576]答案

【问题标题】：Elasticsearch Spring boot findAll Result window is too large, from + size must be less than or equal to: [10000] but was [331576]Elasticsearch Spring boot findAll 结果窗口太大，from+size 必须小于等于：[10000] 但为[331576]
【发布时间】：2020-10-28 21:29:36
【问题描述】：

我是 elasticsearch 新手，我正在使用 spring data elasticsearch (https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#reference)。

这是我的 pom.xml

<properties>
        <spring-data-elasticsearch.version>3.2.6</spring-data-elasticsearch.version>

...
<!-- Elasticsearch -->
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-elasticsearch</artifactId>
            <version>${spring-data-elasticsearch.version}.RELEASE</version>
        </dependency>
    </properties>

我有密码

Iterable<Data> dataList = this.dataRepository.findAll();

我的 DataRepository 就是这样

public interface DataRepository extends ElasticsearchRepository<myData, String> {
}

我收到了错误

"Result window is too large, from + size must be less than or equal to: [10000] but was [331576]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"dfs","grouped":true,"failed_shards":[{"shard":0,"index":"hardwarezone_index","node":"Psv3GnjpQ52aNB52QfvWWw","reason":{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [331576]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}]},"status":500}

我需要检索所有数据以对其进行一些处理。我该怎么做？

【问题讨论】：

标签： spring-boot elasticsearch spring-data-jpa

【解决方案1】：

这是一个弹性搜索限制（并且有充分的理由）。您不能（也不应该）在单个搜索操作中加载庞大的数据集。 Elasticsearch scroll APIs 正是为此目的而设计的。

spring-data-elasticsearch 确实支持滚动 API 透明。您所要做的就是将存储库方法的返回类型从Iterable<Data> 更改为Stream<Data>，spring-data-elasticsearch 将开始在后台使用滚动 API。这记录在 spring-data-elasticsearch 文档的 11.2. Using Scroll For Big Result Set 部分中。

附带说明，虽然这会减少存储和持久层的负载，但如果您将此数据转换为集合而不是使用流直到您的控制器和spring 中的 HttpConverter 层。然而，这是一个单独的话题。如果您打算将其用于生产用例，请与您分享。

【讨论】：