Monday, April 9, 2012

Interactive Python Eats CPU!

For the last few months I have stopped using the interactive Python shell to try things out while developing my django apps. It was limiting but I could get around the inconvenience...

The reason I stopped using the interactive shell was simple - it used all my system CPU and my laptop ground to a halt; now that was a major inconvenience!

At first I assumed it was a broken installation of Python, or perhaps a memory issue when using Eclipse (ie Java) and Python at the same time, even though that has been my usual working environment for some time.

At last I have found the problem. I had enabled a history file for the interactive shell (so I could avoid re-typing some of the long djano database queries), and that history file was now 388Mb in size. Python was thrashing away reading a history of everything I had typed since late 2011.

The first solution was to removed the file, and the shell became snappy and responsive again.

The second solution is to use logrotate to manage the file.
So this is the simple configuration file, stored as $HOME/.logrotate.conf
Update: oops, forgot to actually trim the file afetr making the backup...

# Global 
options 
compress 
weekly 
rotate=4 missingok 
# log files to rotate 
/home/james/.pythonhistory {
  postrotate
    tail -50 /home/james/.pythonhistory >/tmp/pythonhistory.$$
    cp /tmp/pythonhistory.$$ /home/james/.pythonhistory
    rm /tmp/pythonhistory.$$
  endscript


and a simple addition to my personal crontab to rotate personal files at 1am

# run logrotate for personal files 0 11 * * * /usr/sbin/logrotate -s /home/james/.logrotate.status /home/james/.logrotate.conf

Problem solved!