高效的并行策略

Question

我正试图围绕并行策略展开思考。我想我理解每个组合器的作用，但每次我尝试将它们用于超过 1 个内核时，程序都会大大减慢。

例如，不久前，我尝试从大约 700 个文档中计算直方图（并从中计算出唯一的单词）。我认为使用文件级粒度就可以了。使用-N4，我得到 1.70 的工作余额。然而，-N1 的运行时间是-N4 的一半。我不确定问题到底是什么，但我想知道如何决定在何处/何时/如何进行并行化并对此有所了解。这将如何并行化，以使速度随着内核的增加而不是降低？

import Data.Map (Map)
import qualified Data.Map as M
import System.Directory
import Control.Applicative
import Data.Vector (Vector)
import qualified Data.Vector as V
import qualified Data.Text as T
import qualified Data.Text.IO as TI
import Data.Text (Text)
import System.FilePath ((</>))
import Control.Parallel.Strategies
import qualified Data.Set as S
import Data.Set (Set)
import GHC.Conc (pseq, numCapabilities)
import Data.List (foldl')

mapReduce stratm m stratr r xs = let
  mapped = parMap stratm m xs
  reduced = r mapped `using` stratr
  in mapped `pseq` reduced

type Histogram = Map Text Int

rootDir = "/home/masse/Documents/text_conversion/"

finnishStop = ["minä", "sinä", "hän", "kuitenkin", "jälkeen", "mukaanlukien", "koska", "mutta", "jos", "kuitenkin", "kun", "kunnes", "sanoo", "sanoi", "sanoa", "miksi", "vielä", "sinun"]
englishStop = ["a","able","about","across","after","all","almost","also","am","among","an","and","any","are","as","at","be","because","been","but","by","can","cannot","could","dear","did","do","does","either","else","ever","every","for","from","get","got","had","has","have","he","her","hers","him","his","how","however","i","if","in","into","is","it","its","just","least","let","like","likely","may","me","might","most","must","my","neither","no","nor","not","of","off","often","on","only","or","other","our","own","rather","said","say","says","she","should","since","so","some","than","that","the","their","them","then","there","these","they","this","tis","to","too","twas","us","wants","was","we","were","what","when","where","which","while","who","whom","why","will","with","would","yet","you","your"]
isStopWord :: Text -> Bool
isStopWord x = x `elem` (finnishStop ++ englishStop)

textFiles :: IO [FilePath]
textFiles = map (rootDir </>) . filter (not . meta) <$> getDirectoryContents rootDir
  where meta "." = True
        meta ".." = True
        meta _ = False

histogram :: Text -> Histogram
histogram = foldr (\k -> M.insertWith' (+) k 1) M.empty . filter (not . isStopWord) . T.words

wordList = do
  files <- mapM TI.readFile =<< textFiles
  return $ mapReduce rseq histogram rseq reduce files
  where
    reduce = M.unions

main = do
  list <- wordList
  print $ M.size list

至于文本文件，我正在使用转换为文本文件的 pdf，因此我无法提供它们，但出于此目的，几乎所有来自古腾堡项目的书籍/书籍都应该这样做。

编辑：向脚本添加导入

Answer 1

在实践中，让并行组合器很好地扩展可能很困难。 其他人提到让你的代码更严格，以确保你实际上是 并行执行工作，这绝对很重要。

真正影响性能的两件事是大量的内存遍历和 垃圾收集。即使你不生产很多垃圾，很多 内存遍历给 CPU 缓存带来了更大的压力，最终你的 内存总线成为瓶颈。您的 isStopWord 函数执行很多 字符串比较，并且必须遍历一个相当长的链表才能这样做。 您可以使用内置的 Set 类型节省大量工作，或者更好的是 HashSet 类型来自 unordered-containers 包（因为重复的字符串 比较可能会很昂贵，特别是如果它们共享公共前缀）。

import           Data.HashSet                (HashSet)
import qualified Data.HashSet                as S

...

finnishStop :: [Text]
finnishStop = ["minä", "sinä", "hän", "kuitenkin", "jälkeen", "mukaanlukien", "koska", "mutta", "jos", "kuitenkin", "kun", "kunnes", "sanoo", "sanoi", "sanoa", "miksi", "vielä", "sinun"]
englishStop :: [Text]
englishStop = ["a","able","about","across","after","all","almost","also","am","among","an","and","any","are","as","at","be","because","been","but","by","can","cannot","could","dear","did","do","does","either","else","ever","every","for","from","get","got","had","has","have","he","her","hers","him","his","how","however","i","if","in","into","is","it","its","just","least","let","like","likely","may","me","might","most","must","my","neither","no","nor","not","of","off","often","on","only","or","other","our","own","rather","said","say","says","she","should","since","so","some","than","that","the","their","them","then","there","these","they","this","tis","to","too","twas","us","wants","was","we","were","what","when","where","which","while","who","whom","why","will","with","would","yet","you","your"]

stopWord :: HashSet Text
stopWord = S.fromList (finnishStop ++ englishStop)

isStopWord :: Text -> Bool
isStopWord x = x `S.member` stopWord

用这个版本替换你的isStopWord 功能会更好 并且规模更好（尽管绝对不是1-1）。你也可以考虑 出于同样的原因，使用HashMap（来自同一个包）而不是Map， 但我这样做并没有得到明显的改变。

另一个选项是增加默认堆大小以占用一些 给 GC 施加压力，并为其提供更多移动空间。给予 编译后的代码默认堆大小为 1GB（-H1G 标志），我得到的 GC 余额为 在 4 个内核上大约 50%，而我只有 ~25% 没有（它也运行 ~30% 更快）。

通过这两个更改，四个内核上的平均运行时间（在我的机器上） 从 ~10.5s 下降到 ~3.5s。可以说，在基础上还有改进的余地 GC 统计数据（仍然只有 58% 的时间用于生产性工作）， 但要做得更好可能需要更剧烈的改变 你的算法。

Answer 2

我认为 Daniel 说得对——Data.Map 和列表是一种惰性数据结构；您应该同时使用 foldl' 和 insertWith' 来确保每个块的工作都急切地完成 --- 否则所有工作都会延迟到顺序部分（减少）。

此外，为每个文件制作火花是否是正确的粒度并不明显，尤其是在文件大小差异很大的情况下。如果是这种情况，最好将单词列表连接起来并分成偶数大小的块（参见 parListChunk 组合器）。

当您使用它时，我还会看看使用惰性 IO (readFile) 打开许多文件的一些缺陷（运行时系统可能会用完文件句柄，因为它持有它们的时间过长）。