Interactive Python Eats CPU!

For the last few months I have stopped using the interactive Python shell to try things out while developing my django apps. It was limiting but I could get around the inconvenience...

The reason I stopped using the interactive shell was simple - it used all my system CPU and my laptop ground to a halt; now that was a major inconvenience!

At first I assumed it was a broken installation of Python, or perhaps a memory issue when using Eclipse (ie Java) and Python at the same time, even though that has been my usual working environment for some time.

At last I have found the problem. I had enabled a history file for the interactive shell (so I could avoid re-typing some of the long djano database queries), and that history file was now 388Mb in size. Python was thrashing away reading a history of everything I had typed since late 2011.

The first solution was to removed the file, and the shell became snappy and responsive again.

The second solution is to use logrotate to manage the file.
So this is the simple configuration file, stored as $HOME/.logrotate.conf
Update: oops, forgot to actually trim the file afetr making the backup...

# Global 
options 
compress 
weekly 
rotate=4 missingok 
# log files to rotate 
/home/james/.pythonhistory {
  postrotate
    tail -50 /home/james/.pythonhistory >/tmp/pythonhistory.$$
    cp /tmp/pythonhistory.$$ /home/james/.pythonhistory
    rm /tmp/pythonhistory.$$
  endscript
} 

and a simple addition to my personal crontab to rotate personal files at 1am

# run logrotate for personal files 0 11 * * * /usr/sbin/logrotate -s /home/james/.logrotate.status /home/james/.logrotate.conf

Problem solved!

Being "nice" on Linux

Many thanks to a post from Colin Waters for showing me a better way to run background jobs on Linux without slowing down other processes.

Previously I would have just prefixed the command with "nice" and be done with it, and that was pretty good.

Now I use "chrt --idle 0 ionice -t -c3". Actually I wrap that into an environment variable and put $NEWNICE before the command. The two commands set the kernel and IO scheduling policies for the command to IDLE.

This has been making a huge difference on my video conversion scripts, and last night it meant that I could take a 9 hour migration, consisting of image conversions that wanted 100% CPU, and run it on a production webserver without losing service.