Download as PDF, PPTX
![md5(‘select... a’) [addon 3615]
mozilla](https://image.slidesharecdn.com/anatomy-of-a-large-django-site-110328122427-phpapp01/85/Anatomy-of-a-large-Django-site-25-320.jpg)
![md5(‘select... a’) [addon 3615]
addon 3615 md5(‘select... a’)
mozilla](https://image.slidesharecdn.com/anatomy-of-a-large-django-site-110328122427-phpapp01/85/Anatomy-of-a-large-Django-site-26-320.jpg)
![md5(‘select... a’) [addon 3615]
md5(‘select... b’) [addon 3615, addon 1685]
md5(‘select... a’)
addon 3615
md5(‘select... b’)
mozilla](https://image.slidesharecdn.com/anatomy-of-a-large-django-site-110328122427-phpapp01/85/Anatomy-of-a-large-Django-site-27-320.jpg)
![@staticmethod
def transformer(addons):
addon_dict = dict((a.id, a) for a in addons)
vs = filter(None, (a.current_version_id for a in addons)
versions = list(Version.objects.filter(id__in=vs))
for version in versions:
addon_dict[version.addon_id].current_version = version
slightly outdated
example
mozilla](https://image.slidesharecdn.com/anatomy-of-a-large-django-site-110328122427-phpapp01/85/Anatomy-of-a-large-Django-site-34-320.jpg)
![md5(‘select... a’) [addon 3615]
mozilla](https://image.slidesharecdn.com/anatomy-of-a-large-django-site-110328122427-phpapp01/75/Anatomy-of-a-large-Django-site-25-2048.jpg)
![md5(‘select... a’) [addon 3615]
addon 3615 md5(‘select... a’)
mozilla](https://image.slidesharecdn.com/anatomy-of-a-large-django-site-110328122427-phpapp01/75/Anatomy-of-a-large-Django-site-26-2048.jpg)
![md5(‘select... a’) [addon 3615]
md5(‘select... b’) [addon 3615, addon 1685]
md5(‘select... a’)
addon 3615
md5(‘select... b’)
mozilla](https://image.slidesharecdn.com/anatomy-of-a-large-django-site-110328122427-phpapp01/75/Anatomy-of-a-large-Django-site-27-2048.jpg)
![@staticmethod
def transformer(addons):
addon_dict = dict((a.id, a) for a in addons)
vs = filter(None, (a.current_version_id for a in addons)
versions = list(Version.objects.filter(id__in=vs))
for version in versions:
addon_dict[version.addon_id].current_version = version
slightly outdated
example
mozilla](https://image.slidesharecdn.com/anatomy-of-a-large-django-site-110328122427-phpapp01/75/Anatomy-of-a-large-Django-site-34-2048.jpg)
Uploaded byConFoo
Anatomy of a large Django site
This document discusses the anatomy of a large Django site used by Mozilla to manage add-ons. It covers performance optimizations made, including caching, reducing database queries, and using Celery to push tasks asynchronously. Localization of content in over 40 languages is supported through translation of database content and templates. Reusable packages like Bleach and django-qunit are used for sanitizing HTML and JavaScript testing.
More Related Content
Similar to Anatomy of a large Django site
Anatomy of a large Django site
- 1. Anatomy of alarge Django site Andy McKay Mozilla mozilla
- 2. Vancouver mozilla
- 3. Python Zope and Plone ...nowat Mozilla mozilla
- 4. Using Django http://www.djangoproject.com Credit: http://www.flickr.com/photos/abiavati/3110357974/ mozilla
- 5. 1. About thesite 2. Performance 3. Localisation 4. Reuse mozilla
- 6. 1. About thesite mozilla
- 7.
- 8. All code isopen: https://github.com/jbalogh/zamboni mozilla
- 9. All* bugs areopen: https://bugzilla.mozilla.org mozilla
- 10. Convert from CakePHP(remora) to Django (zamboni) mozilla
- 11.
- 12. Changing one URLat a time from CakePHP to Django General trend to move away from PHP and do more Python and Django mozilla
- 13. How large? 250k+ addons 150 mn views month 500 mn API hits day mozilla
- 14. Lines of code PHP 40k Python 18k mozilla
- 15. Lines of code PHP 40k Python 18k Unit tests 15k mozilla
- 16. running both phpand python side by side. a few issues on that No pages go out until they are faster mozilla
- 17. 3 zeus loadbalancers 24 django (and php) 1 mysql + 4 slaves 3 memcached 3 sphinx 1 rabbitmq + 2 celeryd 1 redis master + 1 slave Credit: http://www.flickr.com/photos/tbridge/15300843/ mozilla
- 18. 2. Performance mozilla
- 19. As usual, databasebottleneck mozilla
- 20. Cache machine http://bit.ly/cache-machine Credit: http://www.flickr.com/photos/mwichary/4063534688/ mozilla
- 21. from django.db importmodels import caching.base class Addon(caching.base.CachingMixin, models.Model): ... status = models.IntegerField() objects = caching.base.CachingManager() available as a mixin need to addin the custom manager mozilla
- 22.
- 23. >>> Addon.objects.filter(status=public) >>> len(connection.queries) 13 >>> Addon.objects.filter(status=public) >>> len(connection.queries) 13 mozilla
- 24. Invalidation mozilla
- 25. md5(‘select... a’) [addon 3615] mozilla
- 26. md5(‘select... a’) [addon 3615] addon 3615 md5(‘select... a’) mozilla
- 27. md5(‘select... a’) [addon 3615] md5(‘select... b’) [addon 3615, addon 1685] md5(‘select... a’) addon 3615 md5(‘select... b’) mozilla
- 28.
- 29.
- 30. Home page 20+ addons 400+ sql queries mozilla
- 31. add-on version version version files standard answer in django is select-related files files mozilla
- 32.
- 33.
- 34. @staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version slightly outdated example mozilla
- 35. big SQL statements...:( 14313 character Home page 20+ addons ~14 sql queries mozilla
- 36. Update Called on startup about:config extensions.update.url mozilla
- 37. Incoming 8,000 req/sec Uncached1,600 req/sec Im used to Plone in the bad old days mozilla
- 38. Incoming 8,000 req/sec Uncached1,600 req/sec PHP 550 req/sec Im used to Plone in the bad old days mozilla
- 39. v1 Plain Django PHP Python mozilla
- 40. v2 Min. SQL queries PHP Python mozilla
- 41. oh god mozilla
- 42. v3 Django and raw SQL max-requests 200, actually we hit 210 PHP Python mozilla
- 43. v4 WSGI no Django PHP Python mozilla
- 44. v5 Pooling, optimised queries Thats 700 req/sec PHP Python which translates into mozilla
- 45. Reducing the SQLqueries... doesn’t always help mozilla
- 46. mySQL query cacheis fast mozilla
- 47. Celery http://celeryproject.org Credit: http://www.flickr.com/photos/chiotsrun/3843988392/ mozilla
- 48. Push things async email image processing add-on validation specifically fixing data changes bet ween php and python mozilla
- 49. from celeryutils importtask @task def update_tag(tag, **kw): tag.update_stat() mozilla
- 50. from celeryutils importtask @task(rate_limit='60/h') def update_tag(tag, **kw): tag.update_stat() mozilla
- 51. from tasks importupdate_tag update_tag.delay(tag) mozilla
- 52. Measurement mozilla
- 53. Timing Middleware http://bit.ly/timing-ware Credit: http://www.flickr.com/photos/wwarby/3297205226/ mozilla
- 54.
- 55. 3. Localization mozilla
- 56. show site inarabic? 40+ languages including rtl mozilla
- 57. content translated and templates Database strings mozilla
- 58. class Addon(caching.base.CachingMixin, models.Model): ... name = models.ForeignKey(Translation) mozilla
- 59. addon.name = 'name' addon.save() mozilla
- 60. addon.name = 'name' addon.save() addon.name= {'fr': 'la nomme'} addon.save() mozilla
- 61. Templates mozilla
- 62. Django {% blocktrans withapp=request.APP %} Add-ons for {{ app }} {% endblocktrans %} mozilla
- 63. Jinja2 http://jinja.pocoo.org/ {{ _('Add-ons for {0}')|f(request.APP) }} mozilla
- 64. Python Unicode hell UnicodeDecodeError:'ascii' codec can't decode byte 0xd0 in position 16: ordinal not in range(128) mozilla
- 65. 4. Reuse mozilla
- 66.
- 67.
- 68. >>> bleach.linkify('an http:// ex.comurl') 'an <a href="http://ex.com" rel="nofollow">http://ex.com</ a> url' mozilla
- 69. Javascript tests mozilla
- 70. django-qunit http://bit.ly/django-qunit kumar mozilla
- 71. test('English', function() { z.refreshL10n('en-us'); equals($('textarea:visible', this.sandbox).text().trim(), 'Firebug integrates with Firefox to put ' + 'a wealth of development tools...'); }); mozilla
- 72. test('Japanese', function() { z.refreshL10n('ja'); equals($('textarea:visible', this.sandbox).text().trim(), 'Firebug Web ' + ' Firefox' + ' '); }); mozilla
- 73. So we usehudson for CI, but haven’t got the automated tests in yet Hoping to do this via jstestnet Use HudsonJenkins http://bit.ly/jstestnet mozilla
- 74. pep 8 py flakes MacCabe Flake8 http://bit.ly/flake8 Credit: http://www.flickr.com/photos/nebarnix/357779131/ mozilla
- 75. ~/sandboxes/zamboni(632719) $ flake8apps/editors/tasks.py apps/editors/tasks.py:1: 'datetime' imported but unused apps/editors/tasks.py:3: 'stat' imported but unused mozilla
- 76. Playdoh http://bit.ly/mozilla-playdoh fred wenzel Credit: http://www.flickr.com/photos/ahmee/97960570/ mozilla
- 77. Basis for newprojects mozilla
- 78. Celery support Jinja2 support Simplemigrations By default: SHA-512 password hashing X-Frame-Options: DENY secure and httponly flags on cookies fred wenzel mozilla
- 79. Take inspiration from... butnot the best for you for example jinja2 which makes integration with lots of django addons possible, but a bit harder mozilla
- 80. Questions? @andymckay andym@mozilla.com andym on irc.freenode.net, irc.mozilla.org mozilla