multithreading - Reading and writing from a single serial port simultaneously from multiple threads -

i have serial port gives me lot of different data different pieces of hardware. need send different commands serial port receive different kinds of data it. so, need write , read data simultaneously port in different functions. sometimes, might need read , write simultaneously serial port in 10 different threads. best way of writing code in situation simultaneously reading/writing data single port? threads, sub processes, etc.

start straightforward code works in case. if script starts multiple threads add reader/writer threads communicate controller -- other threads send/receive data these threads: send_queue consumed writer thread used send commands controller , each worker thread has own receive_queue populated reader thread:

#!/usr/bin/env python2 """communicate dummy controller 10 worker threads.  - start controller subprocess, worker threads - send requests , check responses in worker threads concurrently - wait until responses received """ import json import logging import sys queue import queue subprocess import popen, pipe threading import thread  log = logging.getlogger(__name__).debug  def worker(tid, send_queue, receive_queue):     """send dummy requests, responses."""     log('starting worker tid=%s', tid)     in range(5):         log('sending request key=%s, data=%d', tid, i)         send_queue.put(dict(key=tid, data=i)) # send request         response = receive_queue.get() # response         # check response         log('got response %s, key=%s, data=%d', response, tid, i)         assert response['key'] == tid , response['data'] == i**2  def reader(pipe, thread_queues):     """read responses controller (*pipe*).         dispatch them corresponding threads.     """     log('starting reader')     pipe:         line in iter(pipe.readline, b''): # read response controller             log('got line %r', line)             response = json.loads(line)             # dispatch corresponding thread             thread_queues[response['key']].put(response)  def writer(pipe, send_queue):     """write requests controller (*pipe*) *send_queue*."""     log('starting writer')     pipe:         request in iter(send_queue.get, none):             log('sending request %s', request)             print >>pipe, json.dumps(request)  def start_daemon_thread(*args, **kwargs):     """start , return daemonic thread."""     t = thread(*args, **kwargs)     t.daemon = true # die program     t.start()     return t  def main():     nthreads = 10     timeout = 1 # seconds, ctrl+c     logging.basicconfig(format="%(asctime)-15s %(threadname)s %(message)s",                         datefmt='%f %t', level=logging.debug)     # start controller     controller = popen([sys.executable, "-u", "dummy_controller.py"],                        stdin=pipe, stdout=pipe, bufsize=1)     # start threads     send_queue = queue(maxsize=50) # limit number of queued requests (backlog)     thread_queues = {tid: queue() tid in range(1, nthreads + 1)}     threads = [start_daemon_thread(name='worker-' + str(tid),                                    target=worker, args=[tid, send_queue, q])                tid, q in thread_queues.items()]      # read/write until there requests     start_daemon_thread(name='writer', target=writer,                         args=[controller.stdin, send_queue])     log('about start reader')     reader_thread = start_daemon_thread(name='reader', target=reader,                                         args=[controller.stdout, thread_queues])     # wait until workers done     while threads:         t in threads:             t.join(timeout) # workaround, enable ctrl+c on python 2.7             if not t.is_alive():                 threads.remove(t)                 break     log('no worker threads')     send_queue.put(none) # no more requests     # wait until responses received     while reader_thread.is_alive():         reader_thread.join(timeout) # workaround, enable ctrl+c on python 2.7     controller.wait() # wait subprocess exit (to avoid zombies)     log('done')  if __name__ == "__main__":     main() 

where dummy_controller.py simple script computes x**2 each input x:

#!/usr/bin/env python2 """dummy controller.""" import json import sys  line in iter(sys.stdin.readline, b''):     request = json.loads(line)     # square data field in request , send     print json.dumps(dict(request, data=int(request['data'])**2)) 

your code don't need subprocess, should call appropriate read/write methods serial port instead of starting subprocess , using pipes.

the code verbose , not optimized speed. if worker threads cpu-bound should find way release gil (e.g., on cpython implementation): c extensions or offload work process pool; otherwise 1 cpu core used python implementation gil (cpython, pypy (default)).

example output

2015-09-27 20:02:38 worker-1 starting worker tid=1 2015-09-27 20:02:38 worker-1 sending request key=1, data=0 2015-09-27 20:02:38 worker-2 starting worker tid=2 2015-09-27 20:02:38 worker-3 starting worker tid=3 2015-09-27 20:02:38 worker-3 sending request key=3, data=0 2015-09-27 20:02:38 worker-2 sending request key=2, data=0 2015-09-27 20:02:38 worker-5 starting worker tid=5 2015-09-27 20:02:38 worker-4 starting worker tid=4 2015-09-27 20:02:38 worker-6 starting worker tid=6 2015-09-27 20:02:38 worker-7 starting worker tid=7 2015-09-27 20:02:38 worker-5 sending request key=5, data=0 2015-09-27 20:02:38 worker-6 sending request key=6, data=0 2015-09-27 20:02:38 worker-4 sending request key=4, data=0 2015-09-27 20:02:38 worker-8 starting worker tid=8 2015-09-27 20:02:38 worker-7 sending request key=7, data=0 2015-09-27 20:02:38 worker-9 starting worker tid=9 2015-09-27 20:02:38 worker-8 sending request key=8, data=0 2015-09-27 20:02:38 worker-9 sending request key=9, data=0 2015-09-27 20:02:38 worker-10 starting worker tid=10 2015-09-27 20:02:38 worker-10 sending request key=10, data=0 2015-09-27 20:02:38 mainthread start reader 2015-09-27 20:02:38 writer starting writer 2015-09-27 20:02:38 reader starting reader 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 1} 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 3} 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 2} 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 5} 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 6} 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 4} 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 7} 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 8} 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 9} 2015-09-27 20:02:38 writer sending request {'data': 0, 'key': 10} 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 1}\n' 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 3}\n' 2015-09-27 20:02:38 worker-1 got response {u'data': 0, u'key': 1}, key=1, data=0 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 2}\n' 2015-09-27 20:02:38 worker-3 got response {u'data': 0, u'key': 3}, key=3, data=0 2015-09-27 20:02:38 worker-1 sending request key=1, data=1 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 5}\n' 2015-09-27 20:02:38 worker-2 got response {u'data': 0, u'key': 2}, key=2, data=0 2015-09-27 20:02:38 worker-3 sending request key=3, data=1 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 1} 2015-09-27 20:02:38 worker-5 got response {u'data': 0, u'key': 5}, key=5, data=0 2015-09-27 20:02:38 worker-2 sending request key=2, data=1 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 6}\n' 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 3} 2015-09-27 20:02:38 worker-5 sending request key=5, data=1 2015-09-27 20:02:38 worker-6 got response {u'data': 0, u'key': 6}, key=6, data=0 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 4}\n' 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 2} 2015-09-27 20:02:38 worker-6 sending request key=6, data=1 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 7}\n' 2015-09-27 20:02:38 worker-4 got response {u'data': 0, u'key': 4}, key=4, data=0 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 5} 2015-09-27 20:02:38 worker-4 sending request key=4, data=1 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 8}\n' 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 6} 2015-09-27 20:02:38 worker-7 got response {u'data': 0, u'key': 7}, key=7, data=0 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 9}\n' 2015-09-27 20:02:38 worker-8 got response {u'data': 0, u'key': 8}, key=8, data=0 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 4} 2015-09-27 20:02:38 worker-7 sending request key=7, data=1 2015-09-27 20:02:38 reader got line '{"data": 0, "key": 10}\n' 2015-09-27 20:02:38 worker-9 got response {u'data': 0, u'key': 9}, key=9, data=0 2015-09-27 20:02:38 worker-8 sending request key=8, data=1 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 1}\n' 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 7} 2015-09-27 20:02:38 worker-10 got response {u'data': 0, u'key': 10}, key=10, data=0 2015-09-27 20:02:38 worker-9 sending request key=9, data=1 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 3}\n' 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 8} 2015-09-27 20:02:38 worker-1 got response {u'data': 1, u'key': 1}, key=1, data=1 2015-09-27 20:02:38 worker-10 sending request key=10, data=1 2015-09-27 20:02:38 worker-3 got response {u'data': 1, u'key': 3}, key=3, data=1 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 2}\n' 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 9} 2015-09-27 20:02:38 worker-1 sending request key=1, data=2 2015-09-27 20:02:38 worker-3 sending request key=3, data=2 2015-09-27 20:02:38 worker-2 got response {u'data': 1, u'key': 2}, key=2, data=1 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 5}\n' 2015-09-27 20:02:38 writer sending request {'data': 1, 'key': 10} 2015-09-27 20:02:38 worker-2 sending request key=2, data=2 2015-09-27 20:02:38 worker-5 got response {u'data': 1, u'key': 5}, key=5, data=1 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 6}\n' 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 1} 2015-09-27 20:02:38 worker-5 sending request key=5, data=2 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 4}\n' 2015-09-27 20:02:38 worker-6 got response {u'data': 1, u'key': 6}, key=6, data=1 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 3} 2015-09-27 20:02:38 worker-4 got response {u'data': 1, u'key': 4}, key=4, data=1 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 7}\n' 2015-09-27 20:02:38 worker-6 sending request key=6, data=2 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 2} 2015-09-27 20:02:38 worker-4 sending request key=4, data=2 2015-09-27 20:02:38 worker-7 got response {u'data': 1, u'key': 7}, key=7, data=1 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 8}\n' 2015-09-27 20:02:38 worker-7 sending request key=7, data=2 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 5} 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 9}\n' 2015-09-27 20:02:38 worker-8 got response {u'data': 1, u'key': 8}, key=8, data=1 2015-09-27 20:02:38 reader got line '{"data": 1, "key": 10}\n' 2015-09-27 20:02:38 worker-8 sending request key=8, data=2 2015-09-27 20:02:38 worker-9 got response {u'data': 1, u'key': 9}, key=9, data=1 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 6} 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 1}\n' 2015-09-27 20:02:38 worker-10 got response {u'data': 1, u'key': 10}, key=10, data=1 2015-09-27 20:02:38 worker-9 sending request key=9, data=2 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 4} 2015-09-27 20:02:38 worker-1 got response {u'data': 4, u'key': 1}, key=1, data=2 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 3}\n' 2015-09-27 20:02:38 worker-10 sending request key=10, data=2 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 7} 2015-09-27 20:02:38 worker-1 sending request key=1, data=3 2015-09-27 20:02:38 worker-3 got response {u'data': 4, u'key': 3}, key=3, data=2 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 2}\n' 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 8} 2015-09-27 20:02:38 worker-3 sending request key=3, data=3 2015-09-27 20:02:38 worker-2 got response {u'data': 4, u'key': 2}, key=2, data=2 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 5}\n' 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 9} 2015-09-27 20:02:38 worker-2 sending request key=2, data=3 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 6}\n' 2015-09-27 20:02:38 writer sending request {'data': 2, 'key': 10} 2015-09-27 20:02:38 worker-5 got response {u'data': 4, u'key': 5}, key=5, data=2 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 4}\n' 2015-09-27 20:02:38 worker-6 got response {u'data': 4, u'key': 6}, key=6, data=2 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 1} 2015-09-27 20:02:38 worker-5 sending request key=5, data=3 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 7}\n' 2015-09-27 20:02:38 worker-4 got response {u'data': 4, u'key': 4}, key=4, data=2 2015-09-27 20:02:38 worker-6 sending request key=6, data=3 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 3} 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 8}\n' 2015-09-27 20:02:38 worker-7 got response {u'data': 4, u'key': 7}, key=7, data=2 2015-09-27 20:02:38 worker-4 sending request key=4, data=3 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 2} 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 9}\n' 2015-09-27 20:02:38 worker-7 sending request key=7, data=3 2015-09-27 20:02:38 worker-8 got response {u'data': 4, u'key': 8}, key=8, data=2 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 5} 2015-09-27 20:02:38 worker-9 got response {u'data': 4, u'key': 9}, key=9, data=2 2015-09-27 20:02:38 reader got line '{"data": 4, "key": 10}\n' 2015-09-27 20:02:38 worker-8 sending request key=8, data=3 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 6} 2015-09-27 20:02:38 worker-9 sending request key=9, data=3 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 1}\n' 2015-09-27 20:02:38 worker-10 got response {u'data': 4, u'key': 10}, key=10, data=2 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 4} 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 3}\n' 2015-09-27 20:02:38 worker-10 sending request key=10, data=3 2015-09-27 20:02:38 worker-1 got response {u'data': 9, u'key': 1}, key=1, data=3 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 7} 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 2}\n' 2015-09-27 20:02:38 worker-3 got response {u'data': 9, u'key': 3}, key=3, data=3 2015-09-27 20:02:38 worker-3 sending request key=3, data=4 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 8} 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 5}\n' 2015-09-27 20:02:38 worker-2 got response {u'data': 9, u'key': 2}, key=2, data=3 2015-09-27 20:02:38 worker-2 sending request key=2, data=4 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 9} 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 6}\n' 2015-09-27 20:02:38 worker-5 got response {u'data': 9, u'key': 5}, key=5, data=3 2015-09-27 20:02:38 worker-1 sending request key=1, data=4 2015-09-27 20:02:38 writer sending request {'data': 3, 'key': 10} 2015-09-27 20:02:38 worker-6 got response {u'data': 9, u'key': 6}, key=6, data=3 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 4}\n' 2015-09-27 20:02:38 worker-5 sending request key=5, data=4 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 3} 2015-09-27 20:02:38 worker-6 sending request key=6, data=4 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 7}\n' 2015-09-27 20:02:38 worker-4 got response {u'data': 9, u'key': 4}, key=4, data=3 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 2} 2015-09-27 20:02:38 worker-7 got response {u'data': 9, u'key': 7}, key=7, data=3 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 8}\n' 2015-09-27 20:02:38 worker-4 sending request key=4, data=4 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 1} 2015-09-27 20:02:38 worker-7 sending request key=7, data=4 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 9}\n' 2015-09-27 20:02:38 worker-8 got response {u'data': 9, u'key': 8}, key=8, data=3 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 5} 2015-09-27 20:02:38 reader got line '{"data": 9, "key": 10}\n' 2015-09-27 20:02:38 worker-9 got response {u'data': 9, u'key': 9}, key=9, data=3 2015-09-27 20:02:38 worker-8 sending request key=8, data=4 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 6} 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 3}\n' 2015-09-27 20:02:38 worker-10 got response {u'data': 9, u'key': 10}, key=10, data=3 2015-09-27 20:02:38 worker-9 sending request key=9, data=4 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 4} 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 2}\n' 2015-09-27 20:02:38 worker-3 got response {u'data': 16, u'key': 3}, key=3, data=4 2015-09-27 20:02:38 worker-10 sending request key=10, data=4 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 7} 2015-09-27 20:02:38 worker-2 got response {u'data': 16, u'key': 2}, key=2, data=4 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 1}\n' 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 8} 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 5}\n' 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 9} 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 6}\n' 2015-09-27 20:02:38 worker-5 got response {u'data': 16, u'key': 5}, key=5, data=4 2015-09-27 20:02:38 worker-1 got response {u'data': 16, u'key': 1}, key=1, data=4 2015-09-27 20:02:38 writer sending request {'data': 4, 'key': 10} 2015-09-27 20:02:38 worker-6 got response {u'data': 16, u'key': 6}, key=6, data=4 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 4}\n' 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 7}\n' 2015-09-27 20:02:38 worker-4 got response {u'data': 16, u'key': 4}, key=4, data=4 2015-09-27 20:02:38 worker-7 got response {u'data': 16, u'key': 7}, key=7, data=4 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 8}\n' 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 9}\n' 2015-09-27 20:02:38 worker-8 got response {u'data': 16, u'key': 8}, key=8, data=4 2015-09-27 20:02:38 reader got line '{"data": 16, "key": 10}\n' 2015-09-27 20:02:38 worker-9 got response {u'data': 16, u'key': 9}, key=9, data=4 2015-09-27 20:02:38 worker-10 got response {u'data': 16, u'key': 10}, key=10, data=4 2015-09-27 20:02:38 mainthread no worker threads 2015-09-27 20:02:38 mainthread done