├── .gitignore ├── Efficient-Django-QuerySet-Use.key ├── Efficient-Django-QuerySet-Use.pdf ├── LICENSE ├── README.md ├── demo-optimized ├── demo │ ├── __init__.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py ├── django_pg_utils │ └── __init__.py ├── example │ ├── __init__.py │ ├── models.py │ ├── tests.py │ ├── utils.py │ └── views.py └── manage.py ├── demo ├── demo │ ├── __init__.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py ├── django_db_utils │ └── __init__.py ├── example │ ├── .models.py.swp │ ├── __init__.py │ ├── models.py │ ├── tests.py │ ├── utils.py │ └── views.py └── manage.py └── if-you-only-see-one-slide.jpg /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[cod] 2 | 3 | # C extensions 4 | *.so 5 | 6 | # Packages 7 | *.egg 8 | *.egg-info 9 | dist 10 | build 11 | eggs 12 | parts 13 | bin 14 | var 15 | sdist 16 | develop-eggs 17 | .installed.cfg 18 | lib 19 | lib64 20 | 21 | # Installer logs 22 | pip-log.txt 23 | 24 | # Unit test / coverage reports 25 | .coverage 26 | .tox 27 | nosetests.xml 28 | 29 | # Translations 30 | *.mo 31 | 32 | # Mr Developer 33 | .mr.developer.cfg 34 | .project 35 | .pydevproject 36 | -------------------------------------------------------------------------------- /Efficient-Django-QuerySet-Use.key: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jfalkner/Efficient-Django-QuerySet-Use/c9b692fe2e5e389e272d586f81f11718a53d7136/Efficient-Django-QuerySet-Use.key -------------------------------------------------------------------------------- /Efficient-Django-QuerySet-Use.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jfalkner/Efficient-Django-QuerySet-Use/c9b692fe2e5e389e272d586f81f11718a53d7136/Efficient-Django-QuerySet-Use.pdf -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2013 Jayson Falkner 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so, 10 | subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Efficient Django QuerySet Use 2 | ============================= 3 | 4 | Code and slides from [Jayson Falkner's](http://bitapocalypse.com/jayson/about/2013/07/17/jayson-falkner/) "Efficient Django QuerySet Use". 5 | 6 | This talk is about advanced use of Django's QuerySet API based on lessons learned at [Counsyl](https://www.counsyl.com/jobs/). You'll learn how to debug, optimize and use the QuerySet API efficiently. The main examples focus on demystifying how QuerySet maps to SQL and how seemingly simple Python code, presumably fast O(1) queries, can result in unacceptably long runtime and often O(N) or worse queries. Solutions are provided for keeping your Python code simple and RDMS use performant. 7 | 8 | Here is one of the key slides to pique your interest. It is the same query tuned via different strategies to get optimal performance. 9 | 10 | ![If you only see one slide](if-you-only-see-one-slide.jpg) 11 | 12 | ### Speaker Bio 13 | 14 | Jayson Falker is a software developer for Counsyl. He has an EEN/CS degree and a PhD in Bioinformatics. He is a long-time open-source Linux/Java/Web developer, book author, and presenter. He has focused on proteomics and genomics research since the mid-2000's, including working for a few years at startups and for Dow AgroSciences. Jayson is quick to get his geek on and also loves to build ridiculous side-projects, including a 6' tall Jenga set that weighs 300lbs and mobile apps that give you awesome virtual mustaches. 15 | 16 | ### Presentations 17 | 18 | This material comes from work experience at Counsyl and has been presented at the following events: 19 | 20 | - Oct 10th, 2013 - [The San Francisco Django Meetup Group](http://www.meetup.com/The-San-Francisco-Django-Meetup-Group/events/141505312/) @ [Counsyl](http://maps.google.com/maps?q=180+Kimball+Way%2C+South+San+Francisco%2C+CA) 21 | - Oct 23rd, 2013 - [Petri Dish (Counsyl Blog)](http://blog.counsyl.com/2013/10/23/efficient-django-queryset-use/). Includes [slideshare + audio](http://www.slideshare.net/Counsyl/efficient-djangoquery-setuse). 22 | - Nov 12th 2013 - [Portland Python Users Group](http://www.meetup.com/pdxpython/events/139924722/) @ [Urban Airship](http://maps.google.com/maps?q=1417+NW+Everett+St%2C+Portland%2C+OR) 23 | 24 | If you think this talk would be good for your user group or conference, contact jayson@counsyl.com. 25 | 26 | ### Reuse and Other Acknowledgement 27 | 28 | You are encouraged to take this material and reuse it according to the included [MIT license](LICENSE). In short, you can use it however you like but the work is provided AS IS and you need to include the license to recognize the original authors and Counsyl. 29 | 30 | Thanks to Counsyl co-workers Kevin Haas for original authorship of the `pg_bulk_update()` code and Constantin Berzan for guidance regarding bulk update strategies using Postgres. 31 | 32 | Thanks to Justine Lam, Imran Haque, and Dave Peticolas for discussion and feedback on the presentation, slides and related code. 33 | 34 | ### Examples 35 | 36 | Below are the scripts to run the examples from the slides. All scripts assume: 37 | 38 | - You are running Python 2.7.2 (newer versions may work fine too) 39 | - You have Django 1.5 installed 40 | - You are in the base directory. The one with this file. 41 | - You have `pip install sqlparse` 42 | 43 | #### Example Set #1: QuerySet With Loop (no optimization and prefetch_related()) 44 | 45 | This set includes doing the often intuitive yet worst possible QuerySet use. The optimization method of using `prefetch_related()` demonstrates a 10x speed increase. Examples are done with 1,000 rows in the `Sample` table. 46 | 47 | 48 | These examples require use of the `demo` Django app included in this repository. From the base directory that this README.md file is in do the following. 49 | 50 | 51 | ``` 52 | # Setup the demo's data models. 53 | cd demo 54 | psql -c "CREATE DATABASE demo;" 55 | python manage.py syncdb 56 | 57 | # Run the demo. 58 | python manage.py shell 59 | ``` 60 | 61 | The rest of the commands can be copy-and-pasted in to Django's shell. For the `track_sql()` and `print_sql()` timing metrics to be helpful you must paste each complete `%cpaste` through `--` section all at once. 62 | 63 | ``` 64 | # Make up 1,000 samples to get sub-second queries for looping. 65 | from example.utils import make_fake_data 66 | make_fake_data() 67 | 68 | # Import Models. 69 | from example.models import Sample, SampleStatus 70 | import counsyl.db 71 | from django.db.models import Max 72 | 73 | 74 | # Run the worst performance, looping query. 75 | %cpaste 76 | counsyl.db.track_sql() 77 | results = [] 78 | samples = Sample.objects.filter( 79 | production=True, 80 | statuses__status_code=SampleStatus.LAB) 81 | for sample in samples: 82 | results.append((sample.barcode, sample.status().created)) 83 | counsyl.db.print_sql() 84 | counsyl.db.print_sql(show_queries=False) 85 | -- 86 | 87 | 88 | # Run the prefetch_related(), looping query. 89 | %cpaste 90 | counsyl.db.track_sql() 91 | results = [] 92 | samples = Sample.objects.filter( 93 | production=True, 94 | statuses__status_code=SampleStatus.LAB) 95 | samples = samples.prefetch_related('statuses') 96 | for sample in samples: 97 | results.append((sample.barcode, sample.status().created)) 98 | counsyl.db.print_sql() 99 | counsyl.db.print_sql(show_queries=False) 100 | -- 101 | ``` 102 | 103 | #### Example Set #2: QuerySet using SQL JOIN (select_related() or annotate()) 104 | 105 | This example set relies on an SQL JOIN to 20-50x more efficiently perform the original query. Examples are using 10,000 rows in the `Sample` table. 106 | 107 | You do not need to reset the Django `demo` app for these examples. The same code base is used for the JOIN examples. 108 | 109 | ``` 110 | # Make up 10,000 samples to get second(ish) queries for JOIN use. 111 | from example.utils import make_fake_data 112 | make_fake_data(samples_to_make=10000) 113 | 114 | 115 | # Run the JOIN without select_related() 116 | %cpaste 117 | counsyl.db.track_sql() 118 | vals = list(Sample.objects 119 | .annotate(latest_status_code=Max('statuses__status_code')) 120 | .filter(production=True, 121 | latest_status_code=SampleStatus.LAB) 122 | .values_list('barcode', 'statuses__created')) 123 | counsyl.db.print_sql() 124 | -- 125 | 126 | 127 | # Run the select_related(), looping query. Denormalize fail. 128 | %cpaste 129 | counsyl.db.track_sql() 130 | results = [] 131 | samples = Sample.objects.filter( 132 | production=True, 133 | statuses__status_code=SampleStatus.LAB) 134 | samples = samples.select_related('latest_status') 135 | for sample in samples: 136 | results.append((sample.barcode, sample.latest_status.created)) 137 | counsyl.db.print_sql() 138 | -- 139 | 140 | 141 | 142 | ``` 143 | 144 | #### Example #3: Denormalization and Multi-column INDEX 145 | 146 | This set includes the optimal known use of QuerySet using Postgres including denormalizing the fields used for the query and a multicolumn index. This is roughly 100,000x improvement over the original query and 1,000x over the SQL JOIN based strategies. 147 | 148 | You will have to `reset demo`, swith to the `demo-optimized` Django app and `syncdb` in order to run these examples. The denormalized data model is slightly different than that of the original examples. 149 | 150 | ``` 151 | # Reset the data models and load a denormalized view. 152 | cd demo-optimized 153 | psql -c "DROP DATABASE demo;" 154 | psql -c "CREATE DATABASE demo;" 155 | python manage.py syncdb 156 | 157 | 158 | # Launch the shell. 159 | python manage.py shell 160 | 161 | # Make up 1,000,000 fake samples. Won't take long. 162 | from example.utils import make_fake_data 163 | make_fake_data(samples_to_make=1000000, batch_threshold=100000, make_statuses=False) 164 | 165 | # Import models. 166 | from example.models import Sample, SampleStatus 167 | import counsyl.db 168 | 169 | # Query without a multicolumn index. 170 | %cpaste 171 | counsyl.db.track_sql() 172 | vals = list(Sample.objects 173 | .filter(production=True, 174 | status_code = SampleStatus.LAB) 175 | .values_list('barcode', 'status_created')) 176 | counsyl.db.print_sql() 177 | -- 178 | 179 | 180 | # Create the multicolumn index. 181 | from counsyl.db import pg_multicolumn_index 182 | pg_multicolumn_index(Sample, ['production', 'status_code']) 183 | 184 | 185 | # Re-do the same query above but this time the index is available. 186 | %cpaste 187 | counsyl.db.track_sql() 188 | vals = list(Sample.objects 189 | .filter(production=True, 190 | status_code = SampleStatus.LAB) 191 | .values_list('barcode', 'status_created')) 192 | counsyl.db.print_sql() 193 | -- 194 | 195 | 196 | # Make 100,000,000 rows. This takes awhile! 197 | from example.utils import make_fake_data 198 | make_fake_data(samples_to_make=100000000, batch_threshold=1000000, make_statuses=False) 199 | 200 | 201 | # Optimal query including multicolumn index. 100,000,000 in sub second! 202 | %cpaste 203 | counsyl.db.track_sql() 204 | vals = list(Sample.objects 205 | .filter(production=True, 206 | status_code = SampleStatus.LAB) 207 | .values_list('barcode', 'status_created')) 208 | counsyl.db.print_sql() 209 | -- 210 | ``` 211 | 212 | #### Django 1.4 and below multi-column INDEX 213 | 214 | Django's `index_together` option was added in 1.5. If you are running an older version of 215 | Django you can still manually add an SQL index and obtain the same performance boost. Use 216 | the `pg_multicolumn_index` method. 217 | 218 | For example, to replicate the index on this presentation's `Sample` class. 219 | 220 | ``` 221 | # Launch a shell. 222 | ./manage.py shell 223 | 224 | # Add the multi-column index using the helper method. 225 | from django_pg_utils import pg_multicolumn_index 226 | pg_multicolumn_index(Sample, ['production', 'status_code']) 227 | ``` 228 | 229 | You can confirm that the index exists by using `psql` (if using Postgres) or equivalent to 230 | browse the schema for the `example_sample` table. 231 | 232 | ``` 233 | # Launch psql to directly interact with Postgres. 234 | psql -h 127.0.0.1 demo 235 | 236 | # Display the table and constraints. 237 | demo=# \d example_sample Table "public.example_sample" 238 | Column | Type | Modifiers 239 | ----------------+--------------------------+------------------------------------------------------------- 240 | id | integer | not null default nextval('example_sample_id_seq'::regclass) 241 | barcode | character varying(10) | not null 242 | production | boolean | not null 243 | created | timestamp with time zone | not null 244 | status_code | smallint | not null 245 | status_created | timestamp with time zone | not null 246 | Indexes: 247 | "example_sample_pkey" PRIMARY KEY, btree (id) 248 | "example_sample_barcode_key" UNIQUE CONSTRAINT, btree (barcode) 249 | "example_sample_6d7e6b4a" btree (production, status_code) 250 | "example_sample_barcode_like" btree (barcode varchar_pattern_ops) 251 | Check constraints: 252 | "example_sample_status_code_check" CHECK (status_code >= 0) 253 | Referenced by: 254 | TABLE "example_samplestatus" CONSTRAINT "example_samplestatus_sample_id_fkey" FOREIGN KEY (sample_id) REFERENCES example_sample(id) DEFERRABLE INITIALLY DEFERRED 255 | ``` 256 | 257 | The specific index you are looking for is `"example_sample_6d7e6b4a" btree (production, status_code)`. The 258 | particular name doesn't matter as long as `(production, status_code)` are grouped. 259 | -------------------------------------------------------------------------------- /demo-optimized/demo/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jfalkner/Efficient-Django-QuerySet-Use/c9b692fe2e5e389e272d586f81f11718a53d7136/demo-optimized/demo/__init__.py -------------------------------------------------------------------------------- /demo-optimized/demo/settings.py: -------------------------------------------------------------------------------- 1 | # Django settings for demo project. 2 | 3 | DEBUG = True 4 | TEMPLATE_DEBUG = DEBUG 5 | 6 | ADMINS = ( 7 | # ('Your Name', 'your_email@example.com'), 8 | ) 9 | 10 | MANAGERS = ADMINS 11 | 12 | DATABASES = { 13 | 'default': { 14 | 'NAME': 'demo', 15 | 'ENGINE': 'django.db.backends.postgresql_psycopg2', 16 | 'USER': 'django', 17 | 'PASSWORD': 'secret', 18 | } 19 | } 20 | 21 | # Local time zone for this installation. Choices can be found here: 22 | # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name 23 | # although not all choices may be available on all operating systems. 24 | # In a Windows environment this must be set to your system time zone. 25 | TIME_ZONE = 'America/Chicago' 26 | 27 | # Language code for this installation. All choices can be found here: 28 | # http://www.i18nguy.com/unicode/language-identifiers.html 29 | LANGUAGE_CODE = 'en-us' 30 | 31 | SITE_ID = 1 32 | 33 | # If you set this to False, Django will make some optimizations so as not 34 | # to load the internationalization machinery. 35 | USE_I18N = True 36 | 37 | # If you set this to False, Django will not format dates, numbers and 38 | # calendars according to the current locale. 39 | USE_L10N = True 40 | 41 | # If you set this to False, Django will not use timezone-aware datetimes. 42 | USE_TZ = True 43 | 44 | # Absolute filesystem path to the directory that will hold user-uploaded files. 45 | # Example: "/home/media/media.lawrence.com/media/" 46 | MEDIA_ROOT = '' 47 | 48 | # URL that handles the media served from MEDIA_ROOT. Make sure to use a 49 | # trailing slash. 50 | # Examples: "http://media.lawrence.com/media/", "http://example.com/media/" 51 | MEDIA_URL = '' 52 | 53 | # Absolute path to the directory static files should be collected to. 54 | # Don't put anything in this directory yourself; store your static files 55 | # in apps' "static/" subdirectories and in STATICFILES_DIRS. 56 | # Example: "/home/media/media.lawrence.com/static/" 57 | STATIC_ROOT = '' 58 | 59 | # URL prefix for static files. 60 | # Example: "http://media.lawrence.com/static/" 61 | STATIC_URL = '/static/' 62 | 63 | # Additional locations of static files 64 | STATICFILES_DIRS = ( 65 | # Put strings here, like "/home/html/static" or "C:/www/django/static". 66 | # Always use forward slashes, even on Windows. 67 | # Don't forget to use absolute paths, not relative paths. 68 | ) 69 | 70 | # List of finder classes that know how to find static files in 71 | # various locations. 72 | STATICFILES_FINDERS = ( 73 | 'django.contrib.staticfiles.finders.FileSystemFinder', 74 | 'django.contrib.staticfiles.finders.AppDirectoriesFinder', 75 | # 'django.contrib.staticfiles.finders.DefaultStorageFinder', 76 | ) 77 | 78 | # Make this unique, and don't share it with anybody. 79 | SECRET_KEY = 'ihtizgay%njctey_9pdhgf+dyttb82!m3+(y@_w378lr9kl=ef' 80 | 81 | # List of callables that know how to import templates from various sources. 82 | TEMPLATE_LOADERS = ( 83 | 'django.template.loaders.filesystem.Loader', 84 | 'django.template.loaders.app_directories.Loader', 85 | # 'django.template.loaders.eggs.Loader', 86 | ) 87 | 88 | MIDDLEWARE_CLASSES = ( 89 | 'django.middleware.common.CommonMiddleware', 90 | 'django.contrib.sessions.middleware.SessionMiddleware', 91 | 'django.middleware.csrf.CsrfViewMiddleware', 92 | 'django.contrib.auth.middleware.AuthenticationMiddleware', 93 | 'django.contrib.messages.middleware.MessageMiddleware', 94 | # Uncomment the next line for simple clickjacking protection: 95 | # 'django.middleware.clickjacking.XFrameOptionsMiddleware', 96 | ) 97 | 98 | ROOT_URLCONF = 'demo.urls' 99 | 100 | # Python dotted path to the WSGI application used by Django's runserver. 101 | WSGI_APPLICATION = 'demo.wsgi.application' 102 | 103 | TEMPLATE_DIRS = ( 104 | # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates". 105 | # Always use forward slashes, even on Windows. 106 | # Don't forget to use absolute paths, not relative paths. 107 | ) 108 | 109 | INSTALLED_APPS = ( 110 | # 'django.contrib.auth', 111 | 'django.contrib.contenttypes', 112 | 'django.contrib.sessions', 113 | 'django.contrib.sites', 114 | 'django.contrib.messages', 115 | 'django.contrib.staticfiles', 116 | 'example' 117 | # Uncomment the next line to enable the admin: 118 | # 'django.contrib.admin', 119 | # Uncomment the next line to enable admin documentation: 120 | # 'django.contrib.admindocs', 121 | ) 122 | 123 | # A sample logging configuration. The only tangible logging 124 | # performed by this configuration is to send an email to 125 | # the site admins on every HTTP 500 error when DEBUG=False. 126 | # See http://docs.djangoproject.com/en/dev/topics/logging for 127 | # more details on how to customize your logging configuration. 128 | LOGGING = { 129 | 'version': 1, 130 | 'disable_existing_loggers': False, 131 | 'filters': { 132 | 'require_debug_false': { 133 | '()': 'django.utils.log.RequireDebugFalse' 134 | } 135 | }, 136 | 'handlers': { 137 | 'mail_admins': { 138 | 'level': 'ERROR', 139 | 'filters': ['require_debug_false'], 140 | 'class': 'django.utils.log.AdminEmailHandler' 141 | } 142 | }, 143 | 'loggers': { 144 | 'django.request': { 145 | 'handlers': ['mail_admins'], 146 | 'level': 'ERROR', 147 | 'propagate': True, 148 | }, 149 | } 150 | } 151 | -------------------------------------------------------------------------------- /demo-optimized/demo/urls.py: -------------------------------------------------------------------------------- 1 | from django.conf.urls import patterns, include, url 2 | 3 | # Uncomment the next two lines to enable the admin: 4 | # from django.contrib import admin 5 | # admin.autodiscover() 6 | 7 | urlpatterns = patterns('', 8 | # Examples: 9 | # url(r'^$', 'demo.views.home', name='home'), 10 | # url(r'^demo/', include('demo.foo.urls')), 11 | 12 | # Uncomment the admin/doc line below to enable admin documentation: 13 | # url(r'^admin/doc/', include('django.contrib.admindocs.urls')), 14 | 15 | # Uncomment the next line to enable the admin: 16 | # url(r'^admin/', include(admin.site.urls)), 17 | ) 18 | -------------------------------------------------------------------------------- /demo-optimized/demo/wsgi.py: -------------------------------------------------------------------------------- 1 | """ 2 | WSGI config for demo project. 3 | 4 | This module contains the WSGI application used by Django's development server 5 | and any production WSGI deployments. It should expose a module-level variable 6 | named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover 7 | this application via the ``WSGI_APPLICATION`` setting. 8 | 9 | Usually you will have the standard Django WSGI application here, but it also 10 | might make sense to replace the whole Django WSGI application with a custom one 11 | that later delegates to the Django one. For example, you could introduce WSGI 12 | middleware here, or combine a Django application with an application of another 13 | framework. 14 | 15 | """ 16 | import os 17 | 18 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "demo.settings") 19 | 20 | # This application object is used by any WSGI server configured to use this 21 | # file. This includes Django's development server, if the WSGI_APPLICATION 22 | # setting points here. 23 | from django.core.wsgi import get_wsgi_application 24 | application = get_wsgi_application() 25 | 26 | # Apply WSGI middleware here. 27 | # from helloworld.wsgi import HelloWorldApplication 28 | # application = HelloWorldApplication(application) 29 | -------------------------------------------------------------------------------- /demo-optimized/django_pg_utils/__init__.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | from django.db import connection as con 3 | import sqlparse 4 | 5 | 6 | def track_sql(): 7 | global start_time 8 | global query_count 9 | start_time = datetime.now() 10 | query_count = len(con.queries) 11 | 12 | def print_sql(show_queries=True): 13 | print "Total Run Time: %s"%(datetime.now()-start_time) 14 | print "Total Postgres Time: %s"%sum([float(query['time']) for query in con.queries[query_count:]]) 15 | print "Queries: %s"%(len(con.queries)-query_count) 16 | 17 | if show_queries: 18 | for query in con.queries[query_count:]: 19 | print "\nPostgres Time: %s"%query['time'] 20 | print sqlparse.format(query['sql'], reindent=True, keyword_case='upper') 21 | 22 | 23 | def pg_multicolumn_index(model, column_names, cursor=None): 24 | """CREATE INDEX db_sample_prod_lab ON db_sample (status_code, production)""" 25 | cursor = cursor or con.cursor() 26 | # Get table name and column name as stored in database. 27 | db_table = model._meta.db_table 28 | db_column_names = [model._meta.get_field(column_name).column for column_name in column_names] 29 | db_index = "%s-%s"%(db_table, '_'.join(db_column_names)) 30 | # Create the index. 31 | cursor.execute( 32 | "CREATE INDEX \"" + db_index + "\"" + 33 | " ON \"" + db_table + "\" (%s);"%(','.join(db_column_names))) 34 | cursor.execute("COMMIT;") 35 | 36 | 37 | def pg_bulk_update(model, filter_name, update_name, 38 | filter_column_data, update_column_data, cursor=None): 39 | """ 40 | Postgres database utility to quickly update an entire column in a table 41 | with the values provided in update_column_data matched against 42 | filter_column_data. 43 | 44 | Model is the Django model to be updated, filter_name is the name of the 45 | field to match rows to be updated, update_name is the field to be updated 46 | with data, filter_column_data is the data to test for match with row in 47 | table (typically primary keys of model) and update_column_data is the 48 | matching list of values with which to update the table (primary key ids 49 | if the field is a ForeignKey). 50 | """ 51 | cursor = cursor or con.cursor() 52 | # Get table name and column name for filter and update attributes as 53 | # stored in database. 54 | db_table = model._meta.db_table 55 | model_filter = model._meta.get_field(filter_name).column 56 | model_update = model._meta.get_field(update_name).column 57 | # Auto-convert tuples to lists. 58 | if type(filter_column_data) is tuple: 59 | filter_column_data = list(filter_column_data) 60 | if type(update_column_data) is tuple: 61 | update_column_data = list(update_column_data) 62 | # Input data as Django sanitized parameters, 63 | cursor.execute( 64 | "UPDATE " + db_table + 65 | " SET " + model_update + " = input.update" + 66 | " FROM (SELECT unnest(%s), unnest(%s)) AS input (filter, update)" 67 | " WHERE " + model_filter + " = input.filter;", [filter_column_data, 68 | update_column_data]) 69 | cursor.execute("COMMIT;") 70 | -------------------------------------------------------------------------------- /demo-optimized/example/__init__.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | from django.db import connection as con 3 | import sqlparse 4 | 5 | 6 | def start(): 7 | global start_time 8 | global query_count 9 | start_time = datetime.now() 10 | query_count = len(con.queries) 11 | 12 | def finish(): 13 | print "Time: %s"%(datetime.now()-start_time) 14 | print "Queries: %s"%(len(con.queries)-query_count) 15 | 16 | for query in con.queries[query_count:]: 17 | print sqlparse.format(query['sql'], reindent=True, keyword_case='upper') 18 | -------------------------------------------------------------------------------- /demo-optimized/example/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | 3 | 4 | class Sample(models.Model): 5 | barcode = models.CharField(max_length=10, unique=True) 6 | production = models.BooleanField() 7 | created = models.DateTimeField() 8 | status_code = models.PositiveSmallIntegerField() 9 | status_created = models.DateTimeField() 10 | 11 | class Meta: 12 | index_together = [["production", "status_code"],] 13 | 14 | 15 | class SampleStatus(models.Model): 16 | sample = models.ForeignKey(Sample, related_name='statuses') 17 | status_code = models.PositiveSmallIntegerField() 18 | created = models.DateTimeField() 19 | 20 | RECEIVED = 1 21 | LAB = 2 22 | COMPLETE = 3 23 | ERROR = 4 24 | -------------------------------------------------------------------------------- /demo-optimized/example/tests.py: -------------------------------------------------------------------------------- 1 | """ 2 | This file demonstrates writing tests using the unittest module. These will pass 3 | when you run "manage.py test". 4 | 5 | Replace this with more appropriate tests for your application. 6 | """ 7 | 8 | from django.test import TestCase 9 | 10 | 11 | class SimpleTest(TestCase): 12 | def test_basic_addition(self): 13 | """ 14 | Tests that 1 + 1 always equals 2. 15 | """ 16 | self.assertEqual(1 + 1, 2) 17 | -------------------------------------------------------------------------------- /demo-optimized/example/utils.py: -------------------------------------------------------------------------------- 1 | from django.utils.timezone import utc 2 | 3 | from django_db_utils import pg_bulk_update 4 | 5 | from example.models import Sample, SampleStatus 6 | 7 | 8 | def now(): 9 | from datetime import datetime 10 | return datetime.utcnow().replace(tzinfo=utc) 11 | 12 | 13 | def make_fake_data(samples_to_make=100000, batch_threshold=100000, delete_existing=True, make_statuses=True, years=5): 14 | """Makes mock data for testing performance. Optionally, resets db. 15 | """ 16 | if delete_existing: 17 | Sample.objects.all().delete() 18 | print "Deleted existing" 19 | 20 | # Make up a set of 21 | offset = samples_to_make - samples_to_make/52/years 22 | 23 | # Create all the samples. 24 | samples = [] 25 | barcodes = range(samples_to_make) 26 | for barcode in barcodes: 27 | sample = Sample() 28 | sample.barcode = str(barcode) 29 | sample.created = now() 30 | sample.status_created = sample.created 31 | if barcode < offset: 32 | sample.status_code = SampleStatus.COMPLETE 33 | else: 34 | sample.status_code = SampleStatus.LAB 35 | sample.production = True 36 | samples.append(sample) 37 | if len(samples) >= batch_threshold: 38 | Sample.objects.bulk_create(samples) 39 | del samples[:] 40 | print "Made %s samples." % Sample.objects.count() 41 | if samples: 42 | Sample.objects.bulk_create(samples) 43 | print "Finished making %s samples." % Sample.objects.count() 44 | 45 | if not make_statuses: 46 | return 47 | 48 | # Pull all ids for samples. 49 | sample_ids = Sample.objects.values_list('id', flat=True) 50 | 51 | # Create all the statuses. 52 | offset = len(sample_ids)-len(sample_ids)/52/years 53 | statuses = [] 54 | for sample in sample_ids[:offset]: 55 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.RECEIVED, created=now())) 56 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.LAB, created=now())) 57 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.COMPLETE, created=now())) 58 | if len(statuses) >= batch_threshold: 59 | SampleStatus.objects.bulk_create(statuses) 60 | del statuses[:] 61 | for sample in sample_ids[offset:]: 62 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.RECEIVED, created=now())) 63 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.LAB, created=now())) 64 | if len(statuses) >= batch_threshold: 65 | SampleStatus.objects.bulk_create(statuses) 66 | del statuses[:] 67 | print "Made %s statuses."%SampleStatus.objects.count() 68 | if statuses: 69 | SampleStatus.objects.bulk_create(statuses) 70 | print "Finished making %s statuses."%SampleStatus.objects.count() 71 | 72 | # Make all the denormalized status_code vars match. 73 | sync_status(limit=batch_threshold) 74 | print "Statuses synchronized" 75 | 76 | 77 | def sync_status(limit=100000): 78 | # Stream through all samples. 79 | sample_count = Sample.objects.count() 80 | for index in range(0, sample_count, limit): 81 | vals = Sample.objects.order_by('id', '-statuses__status_code').distinct('id').values_list('id', 'status_code', 'statuses__id', 'statuses__status_code')[index:index+limit] 82 | # Pull all mismatching values. 83 | ids = [] 84 | status_codes = [] 85 | # status_ids = [] 86 | for sample_id, status_code, status_id, latest_status_code in vals: 87 | if status_code != latest_status_code: 88 | ids.append(sample_id) 89 | status_codes.append(latest_status_code) 90 | # status_ids.append(status_id) 91 | # Sync using a bulk update. 92 | if ids: 93 | pg_bulk_update(Sample, 'id', 'status_code', list(ids), list(status_codes)) 94 | # pg_bulk_update(Sample, 'id', 'status_id', list(ids), list(status_ids)) 95 | print 'Synced %s out of %s samples at %s'%(len(ids), limit, index) 96 | -------------------------------------------------------------------------------- /demo-optimized/example/views.py: -------------------------------------------------------------------------------- 1 | # Create your views here. 2 | -------------------------------------------------------------------------------- /demo-optimized/manage.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os 3 | import sys 4 | 5 | if __name__ == "__main__": 6 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "demo.settings") 7 | 8 | from django.core.management import execute_from_command_line 9 | 10 | execute_from_command_line(sys.argv) 11 | -------------------------------------------------------------------------------- /demo/demo/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jfalkner/Efficient-Django-QuerySet-Use/c9b692fe2e5e389e272d586f81f11718a53d7136/demo/demo/__init__.py -------------------------------------------------------------------------------- /demo/demo/settings.py: -------------------------------------------------------------------------------- 1 | # Django settings for demo project. 2 | 3 | DEBUG = True 4 | TEMPLATE_DEBUG = DEBUG 5 | 6 | ADMINS = ( 7 | # ('Your Name', 'your_email@example.com'), 8 | ) 9 | 10 | MANAGERS = ADMINS 11 | 12 | DATABASES = { 13 | 'default': { 14 | 'NAME': 'demo', 15 | 'ENGINE': 'django.db.backends.postgresql_psycopg2', 16 | 'USER': 'django', 17 | 'PASSWORD': 'secret', 18 | } 19 | } 20 | 21 | # Local time zone for this installation. Choices can be found here: 22 | # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name 23 | # although not all choices may be available on all operating systems. 24 | # In a Windows environment this must be set to your system time zone. 25 | TIME_ZONE = 'America/Chicago' 26 | 27 | # Language code for this installation. All choices can be found here: 28 | # http://www.i18nguy.com/unicode/language-identifiers.html 29 | LANGUAGE_CODE = 'en-us' 30 | 31 | SITE_ID = 1 32 | 33 | # If you set this to False, Django will make some optimizations so as not 34 | # to load the internationalization machinery. 35 | USE_I18N = True 36 | 37 | # If you set this to False, Django will not format dates, numbers and 38 | # calendars according to the current locale. 39 | USE_L10N = True 40 | 41 | # If you set this to False, Django will not use timezone-aware datetimes. 42 | USE_TZ = True 43 | 44 | # Absolute filesystem path to the directory that will hold user-uploaded files. 45 | # Example: "/home/media/media.lawrence.com/media/" 46 | MEDIA_ROOT = '' 47 | 48 | # URL that handles the media served from MEDIA_ROOT. Make sure to use a 49 | # trailing slash. 50 | # Examples: "http://media.lawrence.com/media/", "http://example.com/media/" 51 | MEDIA_URL = '' 52 | 53 | # Absolute path to the directory static files should be collected to. 54 | # Don't put anything in this directory yourself; store your static files 55 | # in apps' "static/" subdirectories and in STATICFILES_DIRS. 56 | # Example: "/home/media/media.lawrence.com/static/" 57 | STATIC_ROOT = '' 58 | 59 | # URL prefix for static files. 60 | # Example: "http://media.lawrence.com/static/" 61 | STATIC_URL = '/static/' 62 | 63 | # Additional locations of static files 64 | STATICFILES_DIRS = ( 65 | # Put strings here, like "/home/html/static" or "C:/www/django/static". 66 | # Always use forward slashes, even on Windows. 67 | # Don't forget to use absolute paths, not relative paths. 68 | ) 69 | 70 | # List of finder classes that know how to find static files in 71 | # various locations. 72 | STATICFILES_FINDERS = ( 73 | 'django.contrib.staticfiles.finders.FileSystemFinder', 74 | 'django.contrib.staticfiles.finders.AppDirectoriesFinder', 75 | # 'django.contrib.staticfiles.finders.DefaultStorageFinder', 76 | ) 77 | 78 | # Make this unique, and don't share it with anybody. 79 | SECRET_KEY = 'ihtizgay%njctey_9pdhgf+dyttb82!m3+(y@_w378lr9kl=ef' 80 | 81 | # List of callables that know how to import templates from various sources. 82 | TEMPLATE_LOADERS = ( 83 | 'django.template.loaders.filesystem.Loader', 84 | 'django.template.loaders.app_directories.Loader', 85 | # 'django.template.loaders.eggs.Loader', 86 | ) 87 | 88 | MIDDLEWARE_CLASSES = ( 89 | 'django.middleware.common.CommonMiddleware', 90 | 'django.contrib.sessions.middleware.SessionMiddleware', 91 | 'django.middleware.csrf.CsrfViewMiddleware', 92 | 'django.contrib.auth.middleware.AuthenticationMiddleware', 93 | 'django.contrib.messages.middleware.MessageMiddleware', 94 | # Uncomment the next line for simple clickjacking protection: 95 | # 'django.middleware.clickjacking.XFrameOptionsMiddleware', 96 | ) 97 | 98 | ROOT_URLCONF = 'demo.urls' 99 | 100 | # Python dotted path to the WSGI application used by Django's runserver. 101 | WSGI_APPLICATION = 'demo.wsgi.application' 102 | 103 | TEMPLATE_DIRS = ( 104 | # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates". 105 | # Always use forward slashes, even on Windows. 106 | # Don't forget to use absolute paths, not relative paths. 107 | ) 108 | 109 | INSTALLED_APPS = ( 110 | # 'django.contrib.auth', 111 | 'django.contrib.contenttypes', 112 | 'django.contrib.sessions', 113 | 'django.contrib.sites', 114 | 'django.contrib.messages', 115 | 'django.contrib.staticfiles', 116 | 'example' 117 | # Uncomment the next line to enable the admin: 118 | # 'django.contrib.admin', 119 | # Uncomment the next line to enable admin documentation: 120 | # 'django.contrib.admindocs', 121 | ) 122 | 123 | # A sample logging configuration. The only tangible logging 124 | # performed by this configuration is to send an email to 125 | # the site admins on every HTTP 500 error when DEBUG=False. 126 | # See http://docs.djangoproject.com/en/dev/topics/logging for 127 | # more details on how to customize your logging configuration. 128 | LOGGING = { 129 | 'version': 1, 130 | 'disable_existing_loggers': False, 131 | 'filters': { 132 | 'require_debug_false': { 133 | '()': 'django.utils.log.RequireDebugFalse' 134 | } 135 | }, 136 | 'handlers': { 137 | 'mail_admins': { 138 | 'level': 'ERROR', 139 | 'filters': ['require_debug_false'], 140 | 'class': 'django.utils.log.AdminEmailHandler' 141 | } 142 | }, 143 | 'loggers': { 144 | 'django.request': { 145 | 'handlers': ['mail_admins'], 146 | 'level': 'ERROR', 147 | 'propagate': True, 148 | }, 149 | } 150 | } 151 | -------------------------------------------------------------------------------- /demo/demo/urls.py: -------------------------------------------------------------------------------- 1 | from django.conf.urls import patterns, include, url 2 | 3 | # Uncomment the next two lines to enable the admin: 4 | # from django.contrib import admin 5 | # admin.autodiscover() 6 | 7 | urlpatterns = patterns('', 8 | # Examples: 9 | # url(r'^$', 'demo.views.home', name='home'), 10 | # url(r'^demo/', include('demo.foo.urls')), 11 | 12 | # Uncomment the admin/doc line below to enable admin documentation: 13 | # url(r'^admin/doc/', include('django.contrib.admindocs.urls')), 14 | 15 | # Uncomment the next line to enable the admin: 16 | # url(r'^admin/', include(admin.site.urls)), 17 | ) 18 | -------------------------------------------------------------------------------- /demo/demo/wsgi.py: -------------------------------------------------------------------------------- 1 | """ 2 | WSGI config for demo project. 3 | 4 | This module contains the WSGI application used by Django's development server 5 | and any production WSGI deployments. It should expose a module-level variable 6 | named ``application``. Django's ``runserver`` and ``runfcgi`` commands discover 7 | this application via the ``WSGI_APPLICATION`` setting. 8 | 9 | Usually you will have the standard Django WSGI application here, but it also 10 | might make sense to replace the whole Django WSGI application with a custom one 11 | that later delegates to the Django one. For example, you could introduce WSGI 12 | middleware here, or combine a Django application with an application of another 13 | framework. 14 | 15 | """ 16 | import os 17 | 18 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "demo.settings") 19 | 20 | # This application object is used by any WSGI server configured to use this 21 | # file. This includes Django's development server, if the WSGI_APPLICATION 22 | # setting points here. 23 | from django.core.wsgi import get_wsgi_application 24 | application = get_wsgi_application() 25 | 26 | # Apply WSGI middleware here. 27 | # from helloworld.wsgi import HelloWorldApplication 28 | # application = HelloWorldApplication(application) 29 | -------------------------------------------------------------------------------- /demo/django_db_utils/__init__.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | from django.db import connection as con 3 | import sqlparse 4 | 5 | 6 | def track_sql(): 7 | global start_time 8 | global query_count 9 | start_time = datetime.now() 10 | query_count = len(con.queries) 11 | 12 | def print_sql(show_queries=True): 13 | print "Total Run Time: %s"%(datetime.now()-start_time) 14 | print "Total Postgres Time: %s"%sum([float(query['time']) for query in con.queries[query_count:]]) 15 | print "Queries: %s"%(len(con.queries)-query_count) 16 | 17 | if show_queries: 18 | for query in con.queries[query_count:]: 19 | print "\nPostgres Time: %s"%query['time'] 20 | print sqlparse.format(query['sql'], reindent=True, keyword_case='upper') 21 | 22 | 23 | def pg_bulk_update(model, filter_name, update_name, 24 | filter_column_data, update_column_data, cursor=None): 25 | """ 26 | Postgres database utility to quickly update an entire column in a table 27 | with the values provided in update_column_data matched against 28 | filter_column_data. 29 | 30 | Model is the Django model to be updated, filter_name is the name of the 31 | field to match rows to be updated, update_name is the field to be updated 32 | with data, filter_column_data is the data to test for match with row in 33 | table (typically primary keys of model) and update_column_data is the 34 | matching list of values with which to update the table (primary key ids 35 | if the field is a ForeignKey). 36 | """ 37 | cursor = cursor or con.cursor() 38 | # Get table name and column name for filter and update attributes as 39 | # stored in database. 40 | db_table = model._meta.db_table 41 | model_filter = model._meta.get_field(filter_name).column 42 | model_update = model._meta.get_field(update_name).column 43 | # Input data as Django sanitized parameters, 44 | cursor.execute( 45 | "UPDATE " + db_table + 46 | " SET " + model_update + " = input.update" + 47 | " FROM (SELECT unnest(%s), unnest(%s)) AS input (filter, update)" 48 | " WHERE " + model_filter + " = input.filter;", [filter_column_data, 49 | update_column_data]) 50 | -------------------------------------------------------------------------------- /demo/example/.models.py.swp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jfalkner/Efficient-Django-QuerySet-Use/c9b692fe2e5e389e272d586f81f11718a53d7136/demo/example/.models.py.swp -------------------------------------------------------------------------------- /demo/example/__init__.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | from django.db import connection as con 3 | import sqlparse 4 | 5 | 6 | def start(): 7 | global start_time 8 | global query_count 9 | start_time = datetime.now() 10 | query_count = len(con.queries) 11 | 12 | def finish(): 13 | print "Time: %s"%(datetime.now()-start_time) 14 | print "Queries: %s"%(len(con.queries)-query_count) 15 | 16 | for query in con.queries[query_count:]: 17 | print sqlparse.format(query['sql'], reindent=True, keyword_case='upper') 18 | -------------------------------------------------------------------------------- /demo/example/models.py: -------------------------------------------------------------------------------- 1 | from django.db import models 2 | 3 | 4 | class Sample(models.Model): 5 | barcode = models.CharField(max_length=10, unique=True) 6 | production = models.BooleanField() 7 | created = models.DateTimeField() 8 | latest_status = models.ForeignKey('SampleStatus', related_name='+', null=True) 9 | 10 | def status(self): 11 | return self.statuses.all()[0] 12 | 13 | 14 | class SampleStatus(models.Model): 15 | sample = models.ForeignKey(Sample, related_name='statuses') 16 | status = models.CharField(max_length=20) 17 | status_code = models.PositiveSmallIntegerField() 18 | created = models.DateTimeField() 19 | 20 | RECEIVED = 1 21 | LAB = 2 22 | COMPLETE = 3 23 | ERROR = 4 24 | -------------------------------------------------------------------------------- /demo/example/tests.py: -------------------------------------------------------------------------------- 1 | """ 2 | This file demonstrates writing tests using the unittest module. These will pass 3 | when you run "manage.py test". 4 | 5 | Replace this with more appropriate tests for your application. 6 | """ 7 | 8 | from django.test import TestCase 9 | 10 | 11 | class SimpleTest(TestCase): 12 | def test_basic_addition(self): 13 | """ 14 | Tests that 1 + 1 always equals 2. 15 | """ 16 | self.assertEqual(1 + 1, 2) 17 | -------------------------------------------------------------------------------- /demo/example/utils.py: -------------------------------------------------------------------------------- 1 | from django.utils.timezone import utc 2 | 3 | from django_db_utils import pg_bulk_update 4 | 5 | from example.models import Sample, SampleStatus 6 | 7 | 8 | def now(): 9 | from datetime import datetime 10 | return datetime.utcnow().replace(tzinfo=utc) 11 | 12 | 13 | def make_fake_data(samples_to_make=1000, batch_threshold=1000, delete_existing=True, make_statuses=True, years=1): 14 | """Makes mock data for testing performance. Optionally, resets db. 15 | """ 16 | if delete_existing: 17 | Sample.objects.all().delete() 18 | print "Deleted existing" 19 | 20 | # Make up a set of 21 | offset = samples_to_make-samples_to_make/52/years 22 | 23 | # Create all the samples. 24 | samples = [] 25 | for barcode in range(samples_to_make): 26 | sample = Sample() 27 | sample.barcode = str(barcode) 28 | sample.created = now() 29 | sample.production = True 30 | samples.append(sample) 31 | if len(samples) >= batch_threshold: 32 | Sample.objects.bulk_create(samples) 33 | del samples[:] 34 | print "Made %s samples."%Sample.objects.count() 35 | if samples: 36 | Sample.objects.bulk_create(samples) 37 | print "Finished making %s samples."%Sample.objects.count() 38 | 39 | if not make_statuses: 40 | return 41 | 42 | # Pull all ids for samples. 43 | sample_ids = Sample.objects.values_list('id', flat=True) 44 | 45 | # Create all the statuses. 46 | offset = len(sample_ids)-len(sample_ids)/52/years 47 | statuses = [] 48 | for sample in sample_ids[:offset]: 49 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.RECEIVED, created=now())) 50 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.LAB, created=now())) 51 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.COMPLETE, created=now())) 52 | if len(statuses) >= batch_threshold: 53 | SampleStatus.objects.bulk_create(statuses) 54 | del statuses[:] 55 | for sample in sample_ids[offset:]: 56 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.RECEIVED, created=now())) 57 | statuses.append(SampleStatus(sample_id=sample, status_code=SampleStatus.LAB, created=now())) 58 | if len(statuses) >= batch_threshold: 59 | SampleStatus.objects.bulk_create(statuses) 60 | del statuses[:] 61 | print "Made %s statuses."%SampleStatus.objects.count() 62 | if statuses: 63 | SampleStatus.objects.bulk_create(statuses) 64 | print "Finished making %s statuses."%SampleStatus.objects.count() 65 | 66 | # Set all the statuses to lab. 67 | vals = (Sample.objects 68 | .filter(statuses__status_code=SampleStatus.LAB) 69 | .values_list('id', 'statuses__id')) 70 | sample_ids, sample_status_ids = zip(*vals) 71 | pg_bulk_update(Sample, 'id', 'latest_status', list(sample_ids), list(sample_status_ids)) 72 | 73 | # Set all the statuses to completed. 74 | vals = (Sample.objects 75 | .filter(statuses__status_code=SampleStatus.COMPLETE) 76 | .values_list('id', 'statuses__id')) 77 | sample_ids, sample_status_ids = zip(*vals) 78 | pg_bulk_update(Sample, 'id', 'latest_status', list(sample_ids), list(sample_status_ids)) 79 | -------------------------------------------------------------------------------- /demo/example/views.py: -------------------------------------------------------------------------------- 1 | # Create your views here. 2 | -------------------------------------------------------------------------------- /demo/manage.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os 3 | import sys 4 | 5 | if __name__ == "__main__": 6 | os.environ.setdefault("DJANGO_SETTINGS_MODULE", "demo.settings") 7 | 8 | from django.core.management import execute_from_command_line 9 | 10 | execute_from_command_line(sys.argv) 11 | -------------------------------------------------------------------------------- /if-you-only-see-one-slide.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jfalkner/Efficient-Django-QuerySet-Use/c9b692fe2e5e389e272d586f81f11718a53d7136/if-you-only-see-one-slide.jpg --------------------------------------------------------------------------------