一、进程（process）

在Python中，进程的主要模块为subprocess模块，此模块旨在代替老的os.system，os.spawn*，os.popen*，popen2，commands模块，进一步减少编程人员的工作量，实现import this中体现的宗旨。附上官方文档链接：https://docs.python.org/3/library/subprocess.html 多进程变成在Linux中相当于打开多个终端进行操作，即在不同终端中运行不同的命令，在Windows中相当于打开多个cmd（暂不论powershell） 。本文在Windows 10 cmd下进行介绍。

1、简单执行指令

先行介绍subprocess.Popen类的成员，这个类是最进程中重要的一个类，相当于进程的模板。其成员与下面将要介绍的subprocess.call()函数相同。 subprocess.Popen( args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0) 其中args为必填项，后面皆有默认值。其中stdin，stdout，stderr，shell等较为常用。 subprocess的最简单的方法为subprocess.call()函数，基本用法如下： import subprocess child = subprocess.call("ping cn.bing.com") 这样相当于在cmd中输入ping cn.bing.com命令，当然，官方更推荐使用args，即将命令写成多个单词组成的列表形式： import subprocess child = subprocess.call(["ping", "cn.bing.com"]) 这两种输入运行结果相同（若网络连接正常）。 当然，在输入与系统相关命令时，需要将shell设为True，例如： import subprocess child = subprocess.call(["dir", "/a"], shell=True) 若不设定shell的值，则运行时会报错。 FileNotFoundError: [WinError 2] 系统找不到指定的文件。 这种最简单的call函数属于非同步（asynchronously）执行，即只有当call执行完之后，控制权才会返回主进程。这种方式会造成许多不便。因此，Python还有另外一种实现方式：同步执行（synchronously）。 同步执行利用的是subprocess.Popen类，这个类可以用来构造进程。Popen的方法如下： Popen.poll() Check if child process has terminated. Set and returnreturncode attribute. Popen.wait(timeout=None) Wait for child process to terminate. Set and return returncode attribute. If the process does not terminate after timeout seconds, raise a TimeoutExpired exception. It is safe to catch this exception and retry the wait. Note This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that. Note The function is implemented using a busy loop (non-blocking call and shortsleeps). Use theasyncio module for an asynchronous wait: see asyncio.create_subprocess_exec. Changed in version 3.3: timeout was added. Deprecated since version 3.4: Do not use the endtime parameter. It is was unintentionally exposed in 3.3 but was left undocumented as it was intended to be private for internal use. Use timeout instead. Popen.communicate(input=None, timeout=None) Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait forprocess to terminate. The optional input argument should be data to be sent to the child process, orNone, if no data should be sent to the child. If streams were opened in text mode,input must be a string. Otherwise, it must be bytes. communicate() returns a tuple (stdout_data, stderr_data). The data will be strings if streams were opened in text mode; otherwise, bytes. Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too. If the process does not terminate after timeout seconds, a TimeoutExpired exception will be raised. Catching this exception and retrying communication will not lose any output. The child process is not killed if the timeout expires, so in order tocleanup properly a well-behaved application should kill the child process and finish communication: Popen.send_signal(signal) Sends the signal signal to the child. Note On Windows, SIGTERM is an alias for terminate(). CTRL_C_EVENT and CTRL_BREAK_EVENT can be sent to processes started with acreationflags parameter which includes CREATE_NEW_PROCESS_GROUP. Popen.terminate() Stop the child. On Posix OSs the method sends SIGTERM to the child. OnWindows the Win32 API function TerminateProcess() is called to stop the child. Popen.kill() Kills the child. On Posix OSs the function sends SIGKILL to the child. On Windows kill() is an alias for terminate(). The following attributes are also available: Popen.args The args argument as it was passed to Popen – a sequence of program arguments or else a single string. New in version 3.3. Popen.stdin If the stdin argument was PIPE, this attribute is a writeable stream object as returned by open(). If the encoding or errors arguments were specified or the universal_newlines argument was True, the stream is a text stream, otherwise it is a byte stream. If the stdin argument was not PIPE, this attribute is None. Popen.stdout If the stdout argument was PIPE, this attribute is a readable stream object as returned by open(). Reading from the stream provides output from the child process. If the encoding or errors arguments were specified or the universal_newlines argument was True, the stream is a text stream, otherwise it is a byte stream. If the stdout argument was not PIPE, this attribute is None. Popen.stderr If the stderr argument was PIPE, this attribute is a readable stream object as returned by open(). Reading from the stream provides error output from the child process. If the encoding or errors arguments were specified or the universal_newlines argument was True, the stream is a text stream, otherwise it is a byte stream. If the stderr argument was not PIPE, this attribute is None. Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process. Popen.pid The process ID of the child process. Note that if you set the shell argument to True, this is the process ID of the spawned shell. Popen.returncode The child return code, set by poll() and wait() (and indirectly by communicate()). A None value indicates that the process hasn’t terminated yet. A negative value -N indicates that the child was terminated by signal N (POSIX only).

以上摘自Python官方文档，链接开头已附上。

该类最简单的用法如下：

import subprocess child = subprocess.Popen(["dir", "/a"], shell=True) 这样就会让两个进程同时运行，即主进程不会等待子进程运行结束，下例可说明：

import subprocess child = subprocess.Popen(["echo", "HelloWorld"], shell=True) print("Main process ending.") 输出

Main process ending. Process finished with exit code 0 并没有输出HelloWorld，说明主进程没有等待继续运行，结束时便直接退出了。若想要输出HelloWorld，则需要改为

import subprocess child = subprocess.Popen(["echo", "HelloWorld"], shell=True) child.wait() print("Main process ending.") 此时输出

HelloWorld Main process ending. Process finished with exit code 0 其中child.wait()函数使主进程等待，直至child进程运行完毕。类似于非同步的call函数，以上代码等价于

import subprocess child = subprocess.call(["echo", "HelloWorld"], shell=True) print("Main process ending.")

2、获取输出

文本输入输出的控制由上面介绍的stdin，stdout，stderr三个成员所控制。这三个成员均默认为None，若不加以设置，则会默认无输入，并直接输出到控制台。如果需要将输出结果收集起来为其它进程所用，则应将stdout=None（输出到控制台）改为stdout=subprocess.PIPE（输出到管道缓冲区），stderr也相似。修改之后，控制台将不再输出命令执行结果，而结果会储存在缓冲区供用户调用。例如：

import subprocess child1 = subprocess.Popen(["dir", "/a"], shell=True, stdout=subprocess.PIPE) child2 = subprocess.Popen(["echo", "{0}".format(child1.stdout)], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) (out, err) = child2.communicate() print(out, "\n", err) 输出结果

b'"<_io.BufferedReader name=3>"\r\n' b'' Process finished with exit code 0

在将stdout设置为PIPE之后，便可以在任何地方使用Popen.stdout调用PIPE中对应的内容，即获取该进程的输出，error也是如此。这里的communicate函数用来取出输出和错误，其返回值为一个元组，形如（out，err），以供用户自行取用。

注意在向缓存区输出时，不可太大，或输出太多东西却不用communicate等函数或手动清理，否则可能会爆栈（死锁）。

3、控制输入

输入的控制与输出相似，不过一般来说，不会将stdin直接设为PIPE（下有反例），而是确定的指出是PIPE中的哪一个位置，例如child2.stdout（ 前提是child2.stdout要输出到了PIPE）。引用其他文章中一个常用与介绍输入的Linux例子： #!/usr/bin/env python import subprocess child1 = subprocess.Popen(["ls","-l"], stdout=subprocess.PIPE) child2 = subprocess.Popen(["wc"], stdin=child1.stdout,stdout=subprocess.PIPE) out = child2.communicate() print(out) 注：wc是word count命令。 其中，communicate也可以提供输入，仍然用一个常用的Linux例子： import subprocess child = subprocess.Popen(["cat"], stdin=subprocess.PIPE) child.communicate("vamei") 这是将stdin设为PIPE中所有的值，因为此时PIPE为空，直到communicate给出输入vamei，此时PIPE中只有vamei，于是被作为了cat后面的下一个arg，即为cat vamei。

4、其他有用的方法

可以通过修改executable成员来运行不同的程序，例如计算器，Python等，在Windows下，我们以powershell为例。

import subprocess child = subprocess.Popen(["ls"], executable="C:\Windows\System32\WindowsPowerShell\\v1.0\powershell.exe", stdin=subprocess.PIPE) (out, err) = child.communicate(b"cd ..") print(out, err) 这个与上面的例子有些类似，也可加深对subprocess的理解。运行结果为：

Windows PowerShell ��Ȩ���� (C) 2016 Microsoft Corporation����������Ȩ���� PS D:\Python Code\Study\Multiprocess> cd ..PS D:\Python Code\Study> None None Process finished with exit code 0 可见，其只运行了cd ..命令。这说明Popen创建child时并不执行ls，而会等待输入，直到communicate输入cd ..命令。

#这个部分还有待探究，欢迎各位博友探讨。

参考文章：

http://hackerxu.com/2014/10/09/subprocess.html

https://imsardine.wordpress.com/tech/shell-scripting-in-python/

http://www.cnblogs.com/vamei/archive/2012/09/23/2698014.html

二、线程（Thread）

1、基础知识

线程相当于在一个控制台下执行多个指令。线程的常用模块有thread和threading模块，在Python3.5中，thread模块已经被基本放弃，改为了私有的_thread，其中最常用的thread.start_new_thread()函数功能也不太简单，不建议使用。而是转而使用threading中的threading.Thread类。threading官方文档见链接：https://docs.python.org/2/library/threading.html threading.Thread类的构造器如下： class threading.Thread(group=None, target=None, name=None, args=(), kwargs={}) This constructor should always be called with keyword arguments. Arguments are: group should be None; reserved for future extension when a ThreadGroup class is implemented. target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called. name is the thread name. By default, a unique name is constructed of the form “Thread-N” where N is a small decimal number. args is the argument tuple for the target invocation. Defaults to (). kwargs is a dictionary of keyword arguments for the target invocation. Defaults to {}. threading.Thread类提供如下方法： start() Start the thread’s activity. It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control. This method will raise a RuntimeError if called more than once on the same thread object. run() Method representing the thread’s activity. You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively. join([timeout]) Wait until the thread terminates. This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception – or until the optional timeout occurs. When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call isAlive() after join() to decide whether a timeout happened – if the thread is still alive, the join() call timed out. When the timeout argument is not present or None, the operation will block until the thread terminates. A thread can be join()ed many times. join() raises a RuntimeError if an attempt is made to join the current thread as that would cause a deadlock. It is also an error to join() a thread before it has been started and attempts to do so raises the same exception. name A string used for identification purposes only. It has no semantics. Multiple threads may be given the same name. The initial name is set by the constructor. New in version 2.6. getName() setName() Pre-2.6 API for name. ident The ‘thread identifier’ of this thread or None if the thread has not been started. This is a nonzero integer. See the thread.get_ident() function. Thread identifiers may be recycled when a thread exits and another thread is created. The identifier is available even after the thread has exited. New in version 2.6. is_alive() isAlive() Return whether the thread is alive. This method returns True just before the run() method starts until just after the run() method terminates. The module function enumerate() returns a list of all alive threads. Changed in version 2.6: Added is_alive() spelling. daemon A boolean value indicating whether this thread is a daemon thread (True) or not (False). This must be set before start() is called, otherwise RuntimeError is raised. Its initial value is inherited from the creating thread; the main thread is not a daemon thread and therefore all threads created in the main thread default to daemon = False. The entire Python program exits when no alive non-daemon threads are left. New in version 2.6. isDaemon() setDaemon() 以上为Thread提供的用来控制线程的方法，下面介绍具体使用。 每个Python程序都至少有一个线程，即主线程。有两种方法创建新的线程，一为调用threading.Thread()，并向其传递一个可调用的对象；二为构建 threading.Thread()的子类，重写__init__()和__run__()方法。第二种方法较为灵活与直接，因此本文将重点介绍。 继承后模板大约如下： import threading class MyThread(threading.Thread): def __init__(self, threadID, data): super().__init__() self.threadID = threadID self.__data = data def run(self): print("Start thread {0}".format(self.threadID)) process(self.__data) #自定义处理函数 print("End thread {0}".format(self.threadID)) 之后，使用这个类创建线程时，直接构造即可。每次构造时，系统都会调用一次run函数，其中process中的代码就会起到作用，但是绝不要自己显式地调用run函数。

2、数据锁

当要有多个线程访问同一个（组）数据时，容易产生访问冲突。例如存在一个全部是0的列表（list），一个线程从后往前更改数据，另外一个线程从前往后打印数据，于是就会出现打印出来的结果非常有趣，一部分是0，另一部分不是0。因此，我们在更改数据时，会请求一个“锁”来锁住数据（注意，若仅读取数据无此需要），防止在使用的同时被其他线程所更改。这个锁在threading模块中也有提供，即threading.Lock类。其下有两个函数：acquire()和release()，分别表示锁住与释放。基本使用如下：

lock = threading.Lock() lock.acquire() change_data() #改动数据的函数 lock.release() 要注意的是每次使用完数据之后，一定要调用相应的release函数结束锁定，否则其他线程将无法使用此数据。

当有多个锁时，可能会出现更复杂的情况：线程1已经获得了数据A的锁，在尚未解锁A的情况下请求数据B并等待；同时，恰巧线程2已经获得了数据B的锁，在尚未解锁的情况下请求数据A的锁并等待。于是，尴尬的场面发生了，两个线程均会无限制的“厮守”下去，这就是线程“死锁”。要避免这种情况，当遇到多个锁时，要规定每一个锁请求的顺序，例如，只有先请求到数据A，才能去请求数据B。

3、并行处理数据

多线程的一个重要的作用就是并行编程（parallel）。目前的处理器均为多核处理器。当处理一个很复杂的问题时，有时使用多个线程同时处理，比单个线程处理要快得多。著名的work-span model中的不等式T<=W/P+S说明的就是这个问题。

以下的代码功能是算出一组数据的和。为体现并行与串行算法的区别，我将每次计算加法的时间设为了1s，来将简单的加法看成是一种较为复杂的运算。

#Copyright 2015 by JerryLife #All rights reserved #python 3.5.2 import threading import time def calc(x, y): time.sleep(1) #每次计算的时间约为1s return x + y class AddThread(threading.Thread): def __init__(self, x, y): super().__init__() self.__x = x self.__y = y self.__rst = 0 def run(self): self.__rst = calc(self.__x, self.__y) @property def rst(self): return self.__rst #并行算法求和 data_lock = threading.Lock() def reduce_add(data : list): #赋初始值 rst_tmp = [] threads = [] #彻底清空列表内存 rst_tmp[:] = [] threads[:] = [] if data is None: return None elif len(data) == 1: return data[0] else: data_lock.acquire() #锁定数据 pos = 0 #处理长度为奇数的情况 if len(data) % 2 != 0: rst_tmp.append(data[0]) pos = 1 #读取所有data并开始工作 while pos < len(data) - 1: thread_tmp = AddThread(data[pos], data[pos + 1]) thread_tmp.start() threads.append(thread_tmp) pos += 2 data_lock.release() #解锁数据 #等待所有线程工作完毕，并统计结果 for thr in threads: thr.join() rst_tmp.append(thr.rst) return reduce_add(rst_tmp) #串行算法求和 def iterate_add(data : list): rst = 0 for num in data: rst = calc(num, rst) return rst '''-----------开始测试速度--------------''' num = [3, 5, 1, 2, 8, 6, 9, 10, 14] #测试数据 start = time.clock() rst = reduce_add(num) end = time.clock() print("[Parallel] Result is {0}, costs {1} seconds.".format(rst, end - start)) start = time.clock() rst = iterate_add(num) end = time.clock() print(" [Linear] Result is {0}, costs {1} seconds.".format(rst, end - start)) 输出结果为

[Parallel] Result is 58, costs 4.005013394140106 seconds. [Linear] Result is 58, costs 9.00151731829814 seconds. Process finished with exit code 0显然，并行算法与串行算法的结果一样，但耗时却差了许多。根据理论分析，当处理器数量足够多的时候，并行耗时为O(logn)，而串行耗时为O(n)，可见，合理地运用并行算法可以大大提高效率。

Tips：多线程编程当出现电脑慢慢变卡的情况，一定要立即停止运行，不然CPU全满连安全模式也进不了的时候，就只能拔电源了。