python 多核并行计算-python并行框架

现在电脑的CPU一般都是多核的。 在Python中，可以使用multiprocessing包，更方便地将计算任务分配给多个核，使其并行计算，达到加速的效果。

常用的语法是

获取CPU核心数：

n_cpu = multiprocessing.cpu_count()

并行执行函数：

proc = multiprocessing.Process(target=single_run, args=(digits, "parallel"))
proc.start()
proc.join()

其中target属性是要并行执行的函数名，args是函数的参数python 多核并行计算，注意使用元组的形式。

下面用一个简单的例子来演示CPU并行的效果。 对10类MINST-digits数据进行t-SNE降维，比较并行运行和串行运行的时间差。

import numpy as np
import multiprocessing
from sklearn.manifold import TSNE
import time
path = "E:\\blog\\data\\MNIST50m\\"
def run_tsne(data):
    t_sne = TSNE(n_components=2, perplexity=30.0)
    Y = t_sne.fit_transform(data)
    return Y
def single_run(digits, fold="1by1"):

    for digit in digits:
        print(str(digit) + " starting...")
        X = np.loadtxt(path+str(digit)+".csv", dtype=np.float, delimiter=",")
        t_sne = TSNE(n_components=2, perplexity=30.0)
        Y = t_sne.fit_transform(X)
        np.savetxt(path+fold+"\\Y"+str(digit)+".csv", Y, fmt='%f', delimiter=",")
        print(str(digit) + " finished.")
def one_by_one():
    begin_time = time.time()
    digits = [1, 2, 3, 4, 5, 6, 7, 8, 9]
    # digits = [1, 2, 3, 4, 5, 6]
    single_run(digits, "1by1")
    end_time = time.time()
    print("one by one time: ", end_time-begin_time)
def parallel():
    begin_time = time.time()
    n = 10  # 10
    procs = []
    n_cpu = multiprocessing.cpu_count()
    chunk_size = int(n/n_cpu)

    for i in range(0, n_cpu):
        min_i = chunk_size * i
        if i < n_cpu-1:
            max_i = chunk_size * (i+1)
        else:
            max_i = n
        digits = []
        for digit in range(min_i, max_i):
            digits.append(digit)
        procs.append(multiprocessing.Process(target=single_run, args=(digits, "parallel")))
    for proc in procs:
        proc.start()
    for proc in procs:
        proc.join()
    end_time = time.time()
    print("parallel time: ", end_time-begin_time)
if __name__ == '__main__':
    # one_by_one()

    parallel()

串口输出如下，可以看出耗时500多秒。

1 starting...
1 finished.
2 starting...
2 finished.
3 starting...
3 finished.
4 starting...
4 finished.
5 starting...
5 finished.
6 starting...
6 finished.
7 starting...
7 finished.
8 starting...
8 finished.
9 starting...
9 finished.
one by one time:  538.7096929550171

我的六核i5-9400F上并行输出如下。 可以看到用了300多秒，快了一点，但是效果并不理想。

4 starting...
3 starting...

0 starting...
5 starting...
2 starting...
1 starting...
0 finished.
2 finished.
4 finished.
3 finished.
5 finished.
6 starting...
1 finished.
6 finished.
7 starting...
7 finished.
8 starting...
8 finished.
9 starting...
9 finished.
parallel time:  339.75568318367004

为了更好的体现CPU并行化和串行化的区别，我让他们对6位进行t-SNE降维，并行化的速度大约是串行化的4倍。

6位串口输出：

1 starting...
1 finished.
2 starting...

2 finished.
3 starting...
3 finished.
4 starting...
4 finished.
5 starting...
5 finished.
6 starting...
6 finished.
one by one time:  357.5319800376892

6位并行输出：

3 starting...
4 starting...
1 starting...
5 starting...
2 starting...
0 starting...
5 finished.
0 finished.
4 finished.
2 finished.
1 finished.
3 finished.
parallel time:  85.06037616729736

一般来说，对于一些计算需求python 多核并行计算，CPU多核并行可以在一定程度上提高计算速度，但提升能力有限。 比如6核i5处理器，速度提升不会超过6倍。 所以想要大大提高速度，还是得用GPU版的并行。

Python的GPU编程可以参考《Python并行编程Cookbook》一书。 这是一本开源的书，电子版应该在网上比较容易找到。 如果实在找不到（或者懒得找），也可以联系我拿。