A sneak peek into python memory manager

Python's memory manager keeps it's own pool of memory(Private heap) for python objects and this pool of memory was claimed from the OS via raw malloc call at low level. Python's memory manager abstract the malloc call and provides its own special malloc method specifically for different types of python objects. Python won't release the memory back to the OS when an object goes out of scope in python program. Instead it keeps this memory for future reuse.

The python garbage collector does reclaim memory back into OS if it meet certain criteria - Please read this article for more information about python's memory allocation policies.

We won't come across this problem if our system has enough RAM and peak memory usage of the program comes under the RAM size. We usually start worrying about the python's memory problems once the OOM Killer starts terminating the process. Immediate steps to solve this problem are,

1. Optimize the python program by using generators where every possible.
2. Learn best python programming paradigms. Usually newbies get into this problem because of the bad coding practices.
3. Know how much memory needs for your application in peak time and add that much RAM or Increase the SWAP. Use of swap space may not be very good because it will reduce the performance of the program if program constantly swapping data from RAM to Disk.

I have encountered this problem while working with long running python programs with it's peak memory usage is very higher than the average memory usage.

With Celery

Celery is a very famous distributed job scheduler in python. Most of them are using it for their asynchronous task executions and other complex scenarios. Those who using celery for simple tasks wouldn't have faced the memory issue. But in our case -- we use celery very heavily.

RabbitMQ is the AMQP Message queue; backbone to the celery.

In our case what happened was that, we call celery tasks with kwargs - which holds very large objects with no defined structure. Celery passes this data through rabbitmq's queue and deliver to the actual celery's job runner which runs the task with given kwargs. Since the size of the kwargs was very high and the structure varies each time the celery's main process keep on increase its memory foot print. Obviously if we have enough RAM then we can handle the peak memory requirement but this minimum RAM size wasn't affordable to us, so we have to find solution to prevent celery's main process from acquiring lot of memory.

The solution for this problem is, use kwargs (or message) with fixed structure and small in size. Only pass ids or indexes through the message so that the celery's main process won't acquire more memory. And celery has option to kill its worker processes and respawn after some time, you can make use of this also so that the work processes also release all its memory after some time and starts fresh.

RabbitMQ also recommends to use message with smaller size so that we get better performance out of the system. This doesn't mean rabbitmq can't handle large messages. The smaller message size reduce the IO overload required for the RabbitMQ.

I hope this brief explanation will help you to tackle similar problems. If you have any questions or suggestion please put it on the comment section bellow.

References

1. https://deeplearning.net/software/theano/tutorial/python-memory-management.html
2. https://www.evanjones.ca/memoryallocator/
3. https://arctrix.com/nas/python/gc/
4. https://docs.python.org/2/c-api/memory.html