django memory leaks, part II

On my previous post I talked about django memory management, the little-known maxrequests parameter in particular, and how it can help ‘pop’ some balloons, i.e. kill and restart some django processes in order to release some memory. On this post I’m going to cover some of the things to do or avoid in order to keep memory usage low from within your code. In addition, I am going to show at least one method to monitor (and act automatically!) when memory usage shoots through the roof.

Efficient code

Django makes a lot of things very easy. One of its most prominent features is its ORM (Object Relational Mapper). The django ORM really makes it easy to write complex database queries without a single line of SQL. This however might come at a price of sub-optimal queries, which can take a very long time to process, and also consume more memory. If you hit django memory issues, it’s very likely related to some heavy data being loaded from the database, consuming lots and lots of memory. Luckily, there are a few rules that if followed can hugely improve both the queries and memory usage. Most of those are documented clearly on the Database access optimization page. I won’t repeat everything that’s being documented there, but rather focus on a few key points.

Where do you start?

First of all, you might already see crashes on your django, get reports from users about pages loading slowly, and you’ve hopefully checked your server (e.g. using top/htop or free commands) and noticed high memory utilization. The recommendations on part I should give a little stability until you manage to figure out where the problem lies, but it is by no means a sufficient solution.

Monitor log files

The easiest place to start is your log files. If you’re not logging requests already, you seriously should. One helpful method that doesn’t add much overhead to django logging is adding the time that the request took to the logs. This can be achieved with a simple logging middleware. The time each request takes usually gives a fair indication of what’s going on, and from my experience there’s usually a close link between execution time and memory footprint. The same logging middleware can be used with DEBUG=True to view how many SQL queries are executed for each request. This can give another indication of ‘hot-spots’ to look for.

Profiling

The next step is to profile django and see which requests consume high amount of memory. I would suggest following the instructions on Using Guppy to debug Django memory leaks. This should really help pin-pointing areas of code that consume large amounts of memory. Once those are identified, it is easier to try to optimize the code there. Don’t try to do everything at once, but from optimizing one area of code you would learn a lot and could easily apply the same methods across the entire codebase.

Quick-n-dirty improvements

There are a few recommendations that generally help with reducing memory footprint. Be aware not to use them blindly though. They can have a knock-on effect on performance in other areas. So exercise some judgment. The most useful pointers are:

Make sure DEBUG=False
.iterator() – In many cases, there’s simply a need to retrieve a list of objects from the database, and return those in a certain format. If you go over the list more-or-less sequentially and use each object only once, it makes a lot of sense to use iterator() – this will eliminate django internal caching, which can save on memory. If, however, you might go back to the same object – this caching could really speed things up, so be aware of what you’re doing
Using .update()/.delete() – can be a huge saver if you perform a simple update of many objects, or delete a bunch of them. Rather than walking through objects one by one, you can perform this operation with one query
Using .values() – As above, if all you need is a bunch of values from the database, you can save a lot of memory by fetching them as values instead of as complex queryset objects
Using .only() – can also save a lot of memory, if you only use a portion of data from each object.

These are just a few pointers to the same page on the django documentation. Please spend time reading through the entire page, as it contains many very useful techniques and other information to help you better understand how the ORM works and how to use it more effectively.

External Monitoring

Even if you manage to optimize your code, tweak the maxrequests parameter to ensure memory is cleared regularly, and brush your teeth morning and night, there are still cases when it simply won’t be enough. One user will make too many requests, the data-set will just be too big, or some query will stay sub-optimal, and you’d still end up with bloated processes eating through your memory. If that happens, you’re almost back to where we started. This is where some external tools can work wonders. They can:

Let you know when this happens, perhaps even before memory is completely depleted.
Take (automatic) action, and more gracefully restart django, to avoid a full-scale crash.

Monit

My weapon of choice in this case (and many others) is Monit. It’s lightweight, powerful and has a very easy and intuitive configuration syntax, which makes it a snap to use. There are so many uses of this little devil, but in this case I will focus on monitoring process memory. It only takes a few lines to let monit watch over django, and make sure things are running smoothly:

check process your-django-process with pidfile /path/to/your/django.pid
    start program "/etc/init.d/your-django reload"
    stop program "/etc/init.d/your-django stop"
    if totalmem > 70% then exec "/usr/local/bin/highmem" # more about this highmem later...
    if totalmem > 85% for 2 cycles then restart

Lets go over this monit config snippet quickly. What it does is very simple:

Monit checks our django process. Make sure you specify the correct pid file (which we set when invoking django using manage.py)
start and stop program directives should point to your django daemon script
If the total memory of the django process (including children) exceeds 70% of available memory, then it executes a custom ‘highmem’ command. The highmem command will get django to clear some memory by restarting its internal processes. We will cover this command shortly. This will also automatically send an email to alert you (make sure you configure your monit alert settings correctly)
The second check is a “safety-belt”, to restart django if memory stays high for too long despite all our efforts

highmem

The highmem command is a very simple 1 line bash script:

#!/bin/bash
kill -SIGUSR1 `cat /path/to/your/django.pid`

All it does is send a SIGUSR1 to our django process. So what does SIGUSR1 do? When django runs in fastcgi, it uses flup for process handling. Since version 1.0.3, flup uses the SIGUSR1 signal to safely respawn those django processes. Popping those balloons. If a request is in-progress, it will wait until it finishes, which is a very nice feature. Just hope that this process waiting to complete won’t take out ALL memory left… You can read more about it on How to Gracefully Restart Django Running FastCGI (look for the comments section in particular). Please note that you’d need flup version 1.0.3. If you’re using an older version, it might just kill your django processes instead of respawning them safely. It’s easy to get flup, simply use sudo pip install flup (or sudo easy_install flup), and you’re done.

The last resort

The last thing on our arsenal of tools, the last resort, is if some process is eating our memory so fast, that even the highmem command didn’t manage to release it, and we simply have no choice but to restart django. However, even then, we’d probably want to do it as gracefully as possible. I’m not going to repeat it, but the page on How to Gracefully Restart Django Running FastCGI covers what you need to do to modify your manage.py in order to give django just a few seconds before it’s restarted. Then just make sure that the scripts starting/restarting django send a KILL -HUP to your django process to restart it nicely.