4 GPU 中 1bit SGD 与普通 SGD 的 Python CNTK 速度比较答案

【问题标题】：Python CNTK speed comparation of 1bit SGD with normal SGD in 4 GPUs4 GPU 中 1bit SGD 与普通 SGD 的 Python CNTK 速度比较
【发布时间】：2017-05-17 10:43:17
【问题描述】：

我在带有 Ubuntu (python 3.4) 的 Azure NC24 GPU VM 中安装了 CNTK 的版本 2.0.beta7。该机器有 4 个 NVIDIA K80 GPU。构建信息：

            Build type: release
            Build target: GPU
            With 1bit-SGD: yes
            With ASGD: yes
            Math lib: mkl
            CUDA_PATH: /usr/local/cuda-8.0
            CUB_PATH: /usr/local/cub-1.4.1
            CUDNN_PATH: /usr/local
            Build Branch: HEAD
            Build SHA1: 8e8b5ff92eff4647be5d41a5a515956907567126
            Built by Source/CNTK/buildinfo.h$$0 on bbdadbf3455d
            Build Path: /home/philly/jenkins/workspace/CNTK-Build-Linux

我在分布式模式下运行 CIFAR 示例：

mpiexec -n 4 python TrainResNet_CIFAR10_Distributed.py -n resnet20 -q 32

Finished Epoch [1]: [Training] loss = 1.675002 * 50176, metric = 62.5% * 50176 112.019s (447.9 samples per second)
Finished Epoch [1]: [Training] loss = 1.675002 * 50176, metric = 62.5% * 50176 112.019s (447.9 samples per second)
Finished Epoch [1]: [Training] loss = 1.675002 * 50176, metric = 62.5% * 50176 112.018s (447.9 samples per second)
Finished Epoch [1]: [Training] loss = 1.675002 * 50176, metric = 62.5% * 50176 112.019s (447.9 samples per second)
Finished Epoch [2]: [Training] loss = 1.247423 * 50176, metric = 45.4% * 50176 8.210s (6111.3 samples per second)
Finished Epoch [2]: [Training] loss = 1.247423 * 50176, metric = 45.4% * 50176 8.210s (6111.4 samples per second)
Finished Epoch [2]: [Training] loss = 1.247423 * 50176, metric = 45.4% * 50176 8.210s (6111.8 samples per second)
Finished Epoch [2]: [Training] loss = 1.247423 * 50176, metric = 45.4% * 50176 8.210s (6111.6 samples per second)
...
...
Finished Epoch [160]: [Training] loss = 0.037745 * 49664, metric = 1.2% * 49664 7.883s (6300.4 samples per second)
Finished Epoch [160]: [Training] loss = 0.037745 * 49664, metric = 1.2% * 49664 7.883s (6299.7 samples per second)
Finished Epoch [160]: [Training] loss = 0.037745 * 49664, metric = 1.2% * 49664 7.884s (6299.7 samples per second)
Finished Epoch [160]: [Training] loss = 0.037745 * 49664, metric = 1.2% * 49664 7.884s (6299.2 samples per second)

但是，当我使用 1bit SGD 运行它时，我得到：

mpiexec -n 4 python TrainResNet_CIFAR10_Distributed.py -n resnet20 -q 1 -a 50000

...
Finished Epoch [160]: [Training] loss = 0.059290 * 49664, metric = 2.1% * 49664 10.055s (4939.1 samples per second)
Finished Epoch [160]: [Training] loss = 0.059290 * 49664, metric = 2.1% * 49664 10.056s (4938.9 samples per second)
Finished Epoch [160]: [Training] loss = 0.059290 * 49664, metric = 2.1% * 49664 10.056s (4938.9 samples per second)
Finished Epoch [160]: [Training] loss = 0.059290 * 49664, metric = 2.1% * 49664 10.056s (4938.9 samples per second)

正如here 解释的那样，1bit 应该比普通对应物更快。任何帮助表示赞赏。

【问题讨论】：

标签： python neural-network gpu deep-learning cntk

【解决方案1】：

当 GPU 之间的通信时间比小批量的计算时间长时，1 位 sgd 是一种有效的策略。

您的上述实验存在两个“问题”：您正在训练的模型的参数很少（计算量不大），并且 4 个 GPU 在同一台机器上（与通过网络）。 ~~此外，在机器内 CNTK 使用 NVIDIA nccl，它比 1 位使用的通用 MPI 实现优化得多。~~ 更新： 在发表此评论时 NCCL默认不使用。

【讨论】：

谢谢。因此，如果我处理更大的问题（比如 ImageNet 和 ResNet 152），我是否应该看到加速？
是的，这可能会有所帮助，尽管 ResNet 通常没有像 VGG 这样的旧网络那样多的参数，这些网络在末端采用完全连接的层。