Async Web and Python

Async Web and Python

• 1.
  Jasim Muhammed ASYNC WEBAND PYTHON
• 2.
  ME? • Senior SoftwareArchitect, Fetchr, Dubai • Hacking around PostgreSQL, NoSQL, Python, Go, C
• 3.
  SINGLE DELIVERY GUYSHOP Pic source: https://www.nginx.com/blog/thread-pools-boost-performance-9x/
• 4.
  ASYNC DELIVERY ATSHOP Pic source: https://www.nginx.com/blog/thread-pools-boost-performance-9x/
• 5.
  HOW REQUEST ISPROCESSED? Browser
• 6.
  APACHE REQUEST MODES •Prefork • Worker • Event
• 7.
  PYTHON CONCURRENCY OPTIONS •Multiprocessing • Threading • Async processes
• 8.
  PYTHON CONCURRENCY OPTIONS TheoryReality Source: https://www.reddit.com/r/aww/comments/2oagj8/multithreaded_programming_theory_and_practice
• 9.
  ISSUES WITH MULTIPROCESSING •Time to create process • High processor usage • Private memory space for processes
• 10.
  ISSUES WITHTHREADING • Pythonthreads are run in serial due to GIL • High processor usage
• 11.
  ASYNC ADVANTAGES • Processrequest when an IO bound operation happens • No high processor usage • Moderate operations
• 12.
  ASYNC LIBRARIES • Gevent(Python 2.7) • Asyncio(Python 3)
• 13.
  DISTRIBUTED JOB SERVER CELERY •Offload heavy processing • Distribute processing to multiple servers • Cron or maintenance jobs • Easier to implement Webhooks
• 14.
  HOW CELERY HELPSIN CONCURRENT REQUESTS? • Request processing is done very fast • Offloads heavy computation or third party request services to job server workers. • Releases the application server workers for further requests
• 15.
  CELERYTASK EXAMPLE • RegularAsync works without any returns • Mail sending • SMS Sending • In caseTask computation result is important • Report generation • Cron/Maintenance jobs like temporary files cleanup
• 16.
  BEST PRACTISES • Separatequeue for separate tasks • Run workers to multiple nodes • Task retry is there, but be careful. In the case of task failed, you can push to new queue • Don’t restart workers with ‘KILL -9’, use ‘KILL -15’ or SIGTERM • Pass only json serialisable data from to celery. • Use RabbitMQ for broker and Redis for result backend • AsyncDelay and JobID
• 17.
  hello@jasimmk.ninja http://jasimmk.ninja ! jasimmk THANKS