无法在远程分区中执行并行执行答案

【问题标题】：unable to perform parallel execution in remote partition无法在远程分区中执行并行执行
【发布时间】：2019-06-13 14:58:34
【问题描述】：

要求：- 使用 Spring Batch 定义远程分区

看起来所有分区都没有执行并行执行。根据我的理解，例如如果 gridSize = 3 那么所有三个分区都应该与从属步骤（reader\process\writer）类并行执行。

    Configurations:-
        TaskExecutor:-
           corePoolSize = 3
           maxPoolSize  = 20
           queueCapacity = 100
       gridSize = 3
       chunkSize = 10

我已经定义了分区器类，它返回具有相同网格大小的 executionContext。 ExecutionContext={PART3= start=0, PART1=start=0, PART2= start=0}

ReaderClass:-

  public void beforeStep(StepExecution stepExecution) {
    log.info(stepExecution);
  }

  public Scope read(){
     --- some operations---
      return new Scope();
  }

  ProcessClass:-

   public Data process(Scope scope){
      -- some operations --
     return new Data(scope);
   } 

   WriterClass :-

    public void write(List<? extends Data> dataItemsSize){

      -- some operations --
     }

日志：-

            [INFO ] 2019-06-13 09:18:13.389 [Container-28] Reader - beforeStep() reader: stepExecution=StepExecution: id=52175, version=1, name=MatchStep:PART2, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=0, rollbackCount=0, exitDescription=
        [INFO ] 2019-06-13 09:18:13.452 [Container-49] Reader - beforeStep() reader: stepExecution=StepExecution: id=52173, version=1, name=MatchStep:PART3, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=0, rollbackCount=0, exitDescription=
        [INFO ] 2019-06-13 09:18:13.455 [Container-28] Reader - read() - END: scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2855, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.455 [Container-28] Reader - read() - END: scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2841, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.456 [Container-28] Reader - read() - END: scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2851, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.456 [Container-28] Reader - read() - END: scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2863, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.476 [Container-30] Reader - beforeStep() reader: stepExecution=StepExecution: id=52171, version=1, name=MatchStep:PART1, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=0, rollbackCount=0, exitDescription=
        [INFO ] 2019-06-13 09:18:13.477 [Container-49] Reader - read() - END: scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2845, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.477 [Container-49] Reader - read() - END: scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2847, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.477 [Container-49] Reader - read() - END: scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2861, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.477 [Container-49] Reader - read() - END: scope=null
        [INFO ] 2019-06-13 09:18:13.483 [Container-49] Process - process(): scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2845, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.483 [Container-28] Process - process(): scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2855, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2857, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2843, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2849, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2853, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2859, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2865, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=null
        [INFO ] 2019-06-13 09:18:13.507 [Container-30] Process - process(): scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2857, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.509 [Container-28] Process - process(): scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2841, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.509 [Container-49] Process - process(): scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2847, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.513 [Container-28] Process - process(): scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2851, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.513 [Container-49] Process - process(): scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2861, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.516 [Container-28] Process - process(): scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2863, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.593 [Container-30] Process - process(): scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2843, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.598 [Container-30] Process - process(): scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2849, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.608 [Container-28] Writer - write(): dataItemsSize=4
        [INFO ] 2019-06-13 09:18:13.608 [Container-49] Writer - write(): dataItemsSize=3

【问题讨论】：

标签： java spring-batch

【解决方案1】：

看起来您正在使用TaskExecutorPartitionHandler，根据您的日志，分区按预期由不同的线程（Container-28、Container-30、Container-49）并行执行。

要求：- 使用 Spring Batch 定义远程分区

如果要配置远程分区，需要使用MessageChannelPartitionHandler。你可以找到一个如何使用它的例子here。

【讨论】：

非常感谢您的回复。我已经在使用 PartitonHandler 接口进行处理。我应该更改为 MessageChannelPartitionHandler 类吗？
最后一点，每个容器将获得相同数量的执行线程。意味着 corePoolSize = 3，每个容器（28,30,49）将获得每个（3 min 和 max 10）线程执行？对吗？
没错。关于should i change to MessageChannelPartitionHandler class?，是的，这是用于远程分区的分区处理程序实现。