Python Async IO Horizon

1.
19 Dec 2013 @lukmdo PythonAsynchronous I/O asynchronous landscape refresher Photo by Jake Brown used under CC BY 2.0/“Mono+Balance”

2.
Agenda ⏰10 minutes ! • Background(can skip) • Problem statement • “Current state” • Python 3 async horizon (by examples)

3.
Warmup ! • CPU IO (CPUbound) • Pure IO (IO bound) • C10k • Concurrency vs. parallelism

4.
Warmup ! • Lazy schedule atask for later • Avoid waiting for IO (i.e. 2x DB queries) • Solve embarrassingly parallel problems • Control and manage status of task

5.
PEP index brief PEP Name Status Doc PEP255 Simple Generators 2.2 ✔ »» PEP 342 Coroutines via Enhanced Generators 2.5 ✔ » PEP 380 Syntax for Delegating to a Subgenerator 3.3 ✔ » PEP 3148 Futures - execute computations asynchronously 3.2 ✔ » PEP 3153 Asynchronous IO support 3.3 ✔ » PEP 3156 Asynchronous IO Support Rebooted: the "asyncio" Module 3.4 ✔ »

6.
PEP index brief PEP Name PEP219 Stackless Python ❙❙ PEP 220 Coroutines, Generators, Continuations ✗ PEP 319 Python Synchronize/Asynchronize Block ✗ PEP 3145 PEP 3152 Asynchronous I/O For subprocess.Popen Cofunctions Status ❙❙ ❙❙

7.
stackless In the Class concurrent.futures threading tornado greenlet subpocess eventlet celery twisted pyuv gevent shedasyncio asyncore

8.
In the Class asyncio

9.
stackless In the Class concurrent.futures threading tornado greenlet subpocess eventlet celery twisted pyuv gevent shedasyncio asyncore

10.
In the Class concurrent.futures threading tornado subpocess celery twisted asyncio

11.
Slang* • future ~ resultwrapper • coro != generator function (but close) • task = coro wrapped in Future • result = yield from future/coro (to wait for result)

12.
new* old styleconcurrency def update_foo(data): data1 = run_query1(data) data2 = run_query2(data) ! data3 = process(data1, data2) run_update_query(data3) run_update_cache(data3) ! return ["OK"]

13.
new* old styleconcurrency def update_foo(data): data1 = run_query1(data) data2 = run_query2(data) ! data3 = process(data1, data2) run_update_query(data3) run_update_cache(data3) ! return ["OK"]

14.
new* old styleconcurrency

15.

16.
new* old styleconcurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]

17.

18.

19.

20.

21.

22.
new* old styleconcurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]

23.

24.

25.

26.

27.

28.
new* old styleconcurrency def api_calls(): data1 = call_api1() data2 = call_api2() data3 = call_api3() ! return [data1, data2, data3]

29.
new* old styleconcurrency def api_calls(): data1 = call_api1() data3 = call_api2() data3 = call_api3() ! return [data1, data2, data3]

30.

31.

32.
new* old styleconcurrency @asyncio.coroutine def api_calls(): future1 = asyncio.Task(call_api1()) future2 = asyncio.Task(call_api2()) future3 = asyncio.Task(call_api3()) ! return ( yield from asyncio.gather( future1, future2, future3) )

33.
new* old styleconcurrency @asyncio.coroutine def api_calls(): future1 = asyncio.Task(call_api1()) future2 = asyncio.Task(call_api2()) future3 = asyncio.Task(call_api3()) ! return ( yield from asyncio.gather( future1, future2, future3) )

34.

35.

36.

37.
new* old styleconcurrency from aiohttp import request ! @asyncio.coroutine def call_api(): url = "URL" response = yield from request('GET', url) return (yield from response.read())

38.
Takeaways • concurrency patterns • asyncio bootstrap • earlyadopters

39.
Links • Video by BDFLTulip: Async I/O for Python 3 • Video by BDFL PyCon US 2013 Keynote • Code by Nikolay Kim aiohttp • Code by google appengine-pipeline • Code by google ndb tasklets Photo by Jake Brown used under CC BY 2.0/“Mono+Tile”

Python Async IO Horizon

More Related Content

What's hot

Viewers also liked

Similar to Python Async IO Horizon

Recently uploaded

Python Async IO Horizon