Downloaded 97 times
Uploaded byAndrii Mishkovskyi
Python concurrency: libraries overview
The document discusses concurrency solutions in Python, including built-in threading and multiprocessing modules, and third-party libraries like Twisted, Stackless Python, Kamaelia, and cogen. It provides examples of using each solution and notes that while the Global Interpreter Lock limits parallelism, these third-party frameworks provide cleaner approaches to concurrency than the built-in solutions alone.
More Related Content
Similar to Python concurrency: libraries overview
![UiPath Automation Suite Installation (Hands-On) [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/automationsuitecommunitysession2-251015095633-a6d862f1-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Python concurrency: libraries overview
- 1. Built-in solutions 3-rd Party Solutions Summary Concurrency in Python Overview of current solutions Andrii V. Mishkovskyi December 28, 2009 Andrii V. Mishkovskyi Concurrency in Python
- 2. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines threading module Kernel threads Similar to JVM’s threading module Lots of synchronization primitives Andrii V. Mishkovskyi Concurrency in Python
- 3. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines Global Interpreter Lock Simple locking mechanism Doesn’t allow execution of more than 1 thread at a time Unless it’s IO-bound thread CPU-bound threads don’t benefit from parallel execution Andrii V. Mishkovskyi Concurrency in Python
- 4. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines multiprocessing module API similar to multithreading Operates on process level Nice way of avoiding GIL issues Andrii V. Mishkovskyi Concurrency in Python
- 5. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines PEP 342 Simple support through “enhanced generators” yield expression No built-in library or framework Andrii V. Mishkovskyi Concurrency in Python
- 6. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Twisted Emacs of Python concurrency Tons of unit tests Years of active development Supports almost every protocol there is Andrii V. Mishkovskyi Concurrency in Python
- 7. Twisted example fromtwisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
- 8. Twisted example fromtwisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
- 9. Twisted example fromtwisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
- 10. Twisted example fromtwisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
- 11. Twisted example fromtwisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
- 12. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Stackless Python Patch to CPython to not use C stack Ability to monkeypatch existing codebase with “stackless sockets” 100% compatible with CPython Andrii V. Mishkovskyi Concurrency in Python
- 13. Stackless Example, partI import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
- 14. Stackless Example, partI import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
- 15. Stackless Example, partI import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
- 16. Stackless Example, partII def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
- 17. Stackless Example, partII def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
- 18. Stackless Example, partII def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
- 19. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Kamaelia Developed by BBC Research Follows UNIX principles Components all the way round Andrii V. Mishkovskyi Concurrency in Python
- 20. Kamaelia Example importAxon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
- 21. Kamaelia Example importAxon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
- 22. Kamaelia Example importAxon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
- 23. Kamaelia Example importAxon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
- 24. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others cogen Coroutine-based Extensible scheduler Quite low-level (if compared to Kamaelia) Andrii V. Mishkovskyi Concurrency in Python
- 25. cogen example, partI from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
- 26. cogen example, partI from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
- 27. cogen example, partI from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
- 28. cogen example, partI from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
- 29. cogen example, partII @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
- 30. cogen example, partII @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
- 31. cogen example, partII @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
- 32. cogen example, partII @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
- 33. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Others Concurrence – nice asynchrounous framework with async MySQL driver eventlet – based on coroutines, developed and used by “Second Life” py.execnet – distribute tasks over network Andrii V. Mishkovskyi Concurrency in Python
- 34. Built-in solutions 3-rd Party Solutions Summary GIL doesn’t harm 90% of the existing code But Twisted, Kamaelia and others are often cleaner Python is not Erlang, it’s better ;) Andrii V. Mishkovskyi Concurrency in Python