使用 Armadillo 和 OpenBLAS 进行多线程处理时性能不一致

Question

使用Armadillo 我写了一个矩阵向量乘法和一个线性系统求解。 Armadillo 从源代码编译并使用OpenBLAS，也是从源代码编译的。不幸的是，单线程和多线程运行的结果不一致。矩阵向量乘法在单线程上运行得更快，而线性系统求解在多线程上运行得更快。我希望有人能给我一些关于我做错了什么的指示。

见下文：

• 源代码
• 编译并运行 bash 脚本
• 结果
• 系统信息

ma​​tmul_armadillo.cpp

#include <armadillo>

using namespace arma;

int main(int argc, char *argv[])
{
    const int n = atoi(argv[1]);

    mat A = randu<mat>(n, n);
    vec x = randu<vec>(n);

    A*x;

    return 0;
}

solve_armadillo.cpp

#include <armadillo>

using namespace arma;

int main(int argc, char *argv[])
{
    const int n = atoi(argv[1]);

    mat A = randu<mat>(n, n);
    vec b = randu<vec>(n);
    vec x;

    x = solve(A, b);

    return 0;
}

benchmark.sh

#!/bin/bash

g++ matmul_armadillo.cpp -o matmul_armadillo -O3 -march=native -std=c++11 -larmadillo
g++ solve_armadillo.cpp -o solve_armadillo -O3 -march=native -std=c++11 -larmadillo

N=7500

export OPENBLAS_NUM_THREADS=1
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=2
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=3
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=4
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=5
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=6
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=7
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=8
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N

结果

$ ./benchmark.sh 
Running matmul_armadillo on 1 threads

real    0m0.943s
user    0m0.628s
sys     0m0.159s

Running solve_armadillo on 1 threads

real    0m13.910s
user    0m13.553s
sys     0m0.300s

Running matmul_armadillo on 2 threads

real    0m1.528s
user    0m1.361s
sys     0m0.402s

Running solve_armadillo on 2 threads

real    0m15.815s
user    0m29.097s
sys     0m1.083s

Running matmul_armadillo on 3 threads

real    0m1.534s
user    0m1.480s
sys     0m0.533s

Running solve_armadillo on 3 threads

real    0m11.729s
user    0m31.022s
sys     0m1.290s

Running matmul_armadillo on 4 threads

real    0m1.543s
user    0m1.619s
sys     0m0.674s

Running solve_armadillo on 4 threads

real    0m10.013s
user    0m34.055s
sys     0m1.696s

Running matmul_armadillo on 5 threads

real    0m1.545s
user    0m1.620s
sys     0m0.664s

Running solve_armadillo on 5 threads

real    0m9.945s
user    0m33.803s
sys     0m1.669s

Running matmul_armadillo on 6 threads

real    0m1.543s
user    0m1.607s
sys     0m0.684s

Running solve_armadillo on 6 threads

real    0m10.069s
user    0m34.283s
sys     0m1.699s

Running matmul_armadillo on 7 threads

real    0m1.542s
user    0m1.622s
sys     0m0.661s

Running solve_armadillo on 7 threads

real    0m10.041s
user    0m34.154s
sys     0m1.704s

Running matmul_armadillo on 8 threads

real    0m1.546s
user    0m1.576s
sys     0m0.712s

Running solve_armadillo on 8 threads

real    0m10.123s
user    0m34.492s
sys     0m1.697s

系统信息

• openSUSE 13.1 64 位
• 犰狳 4.100.2（从源代码编译）
• OpenBLAS 0.2.8（从源代码编译）

Answer 1

我怀疑

A*x;

可能已经被优化掉了，因为你没有对结果做任何事情。 Armadillo 中乘法运算的延迟评估模板魔法很容易导致永远不会调用计算的 Lapack 例程。因此，如果您启用线程，您只需测量设置它的开销。因此，您的程序在禁用线程的情况下执行得更快。

有

x = solve(A, b);

这是不同的，因为它直接导致了相应的 Lapack 调用，这可能无法优化掉，因为编译器无法排除副作用并且您实际上将结果分配给了一个变量。 solve 调用受益于针对此类大型矩阵的多处理。

要修正你的基准，你应该做两件事：

• 利用计算结果阻止优化器做太多事情
• 重复计算几次以获得更好的统计数据并减少初始设置成本的影响

这是一个未经测试的例子：

#include <iostream>
#include <armadillo>

using namespace arma;

int main(int argc, char *argv[])
{
    const int n = atoi(argv[1]);

    mat A = randu<mat>(n, n);
    vec x = randu<vec>(n);

    for (int i = 0; i < 100; ++i) {
        x = A*x;
    }
    x.print(std::cout);

    return 0;
}

print 调用可能不是必需的。