【问题标题】:How to clusterExport in a foreach loop (OS:Windows)如何在 foreach 循环中进行 clusterExport(操作系统:Windows)
【发布时间】:2018-02-04 15:51:17
【问题描述】:

我有以下问题。我需要全局环境中变量的时间索引,但是当我想在全局环境的并行处理期间将它们导出到集群中时,我收到以下消息:

{ 中的错误:任务 1 失败 - “找不到对象‘Szenario’”

我的原始代码的一个最小示例,它会产生错误:

    Historical <- structure(c(18.5501872568473, 24.3295787432955, 14.9342384460288, 
    13.0653757599636, 8.67294618896797, 13.4587662721594, 20.792126254714, 
    17.5162747884424, 28.8253151239752, 23.0568765432192), index = structure(c(-7305, 
    -7304, -7303, -7302, -7301, -7300, -7299, -7298, -7297, -7296
    ), class = "Date"), class = "zoo")

    Szenario <- structure(c(10.2800684124652, 14.5495871489361, 9.8565852930294, 
    21.1654540678686, 21.1936990312861, 12.4209005842752, 9.77473132000267, 
    17.1997402736739, 17.884107611858, 13.622588360244), index = structure(c(13149, 
    13150, 13151, 13152, 13153, 13154, 13155, 13156, 13157, 13158
    ), class = "Date"), class = "zoo")

    library(doParallel)   
    library(foreach)     
    library(raster)
    library(zoo)
    library(parallel)

    # Parallelisation Settings
    # Definition of how many cores you want to use
    UseCores <- detectCores() -2 # -1 at max because one core has to be used for other tasks 
    # Register CoreCluster
    cl <- makeCluster(UseCores)
    registerDoParallel(cl)

foreach(fn=1:1) %dopar% {

  library(raster) # needed libraries have to be loaded inside the loop, while parallel processing occurs
  library(zoo)
  library(base)
  library(parallel)

  #In my original script, I'm looping through Filenames, which are called like my variables in my Global environment (without .tif at the end of the filename), variables names are saved as characters
  file.referenz.name <- c("Historical")
  file.szenario.name <- c("Szenario")

  #Create timeindex für rasterstacks to subset later on with them (getZ, setZ)

  clusterExport(cl, varlist = c(file.szenario.name, file.referenz.name), envir = .GlobalEnv)
  time.index.szenario <- index(get(file.szenario.name))
  time.index.referenz <- index(get(file.referenz.name))

}

#end cluster
stopCluster(cl)

【问题讨论】:

    标签: r windows parallel-processing environment-variables


    【解决方案1】:

    试试这个

    foreach(fn=1:1, .export=c("Szenario"), 
                    .packages=c("raster", "zoo", "base", "parallel") %dopar%
    

    %dopar% {} 中的clusterExport 令人困惑。您可以在foreach 之前将clusterExport 到每个cl,或者在foreach 中简单地使用.export

    您可以删除%dopar% {} 中的library 语句。

    【讨论】:

    • 谢谢。我可以从全局环境中导出所有变量吗? .也许像这样:.export=environment()
    • 好的,它的.export=ls(.GlobalEnv)
    • 小心导出一切,因为导出许多和/或不需要导出的大型对象会增加大量开销。相反,请尝试仔细识别所需的全局变量集,例如通过反复试验。或者,您可以使用 doFuture 后端,参见。 stackoverflow.com/a/45898676/1072091
    • 感谢您的建议!我会注意这一点。
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2012-03-22
    • 1970-01-01
    • 2015-12-20
    • 2018-09-19
    • 2019-11-22
    相关资源
    最近更新 更多