Interactive Python Eats CPU!

For the last few months I have stopped using the interactive Python shell to try things out while developing my django apps. It was limiting but I could get around the inconvenience...

The reason I stopped using the interactive shell was simple - it used all my system CPU and my laptop ground to a halt; now that was a major inconvenience!

At first I assumed it was a broken installation of Python, or perhaps a memory issue when using Eclipse (ie Java) and Python at the same time, even though that has been my usual working environment for some time.

At last I have found the problem. I had enabled a history file for the interactive shell (so I could avoid re-typing some of the long djano database queries), and that history file was now 388Mb in size. Python was thrashing away reading a history of everything I had typed since late 2011.

The first solution was to removed the file, and the shell became snappy and responsive again.

The second solution is to use logrotate to manage the file.
So this is the simple configuration file, stored as $HOME/.logrotate.conf
Update: oops, forgot to actually trim the file afetr making the backup...

# Global 
options 
compress 
weekly 
rotate=4 missingok 
# log files to rotate 
/home/james/.pythonhistory {
  postrotate
    tail -50 /home/james/.pythonhistory >/tmp/pythonhistory.$$
    cp /tmp/pythonhistory.$$ /home/james/.pythonhistory
    rm /tmp/pythonhistory.$$
  endscript
} 

and a simple addition to my personal crontab to rotate personal files at 1am

# run logrotate for personal files 0 11 * * * /usr/sbin/logrotate -s /home/james/.logrotate.status /home/james/.logrotate.conf

Problem solved!

Submission to MMP Review

Just finished making my 5-minute post to the MMP Review.

The quick post they have is limited to 750 characters, so you don't get to say very much, but I only really wanted to say that party list MPs who leave their party no longer have a right to be in Parliment, I mean they left the party so they aren't on the list anymore right?

If you have an opinion one way or the other, head over and make your submission.

Here is mine.

There is no problem with party list candidates also standing for electorate seats.

A party list is clearly a tool for the party to keep a pool of experience available in Parliment. A candidate standing in an electorate seat  offers to represent the voters of the electorate; while those on the party list represent the party.

However I see a very clear requirement that an MP who enters Parliment through the party list, should only stay in Parliment for as long as they represent that party. A party list MP who leaves their party has no mandate to be in Parliment, and should immediately leave in favour of the next available person on the list.

An MP from an electorate is free to leave and join any parties, as they are reponsible to voters.

plan2ics

This is a quick note about a small Python script I wrote to translate my home calendar into the iCalendar format.

I use an "old" calendar system called plan (sometimes "netplan" which is the server program, and easier to find on Google). It is fast, simple, and I have been using it since 1997, so I don't see any reason to change.

Then I bought an Android phone... which loves to sync with the Google Calendar, and can be convinced to sync with most iCal sources, but has no support for my system.

So what I decided to not do was to write an Android content provider to connect to my netplan server and get the calendar directly. This is the most elegant solution, but I didn't have the time to learn how to do it. (But when I do have time, I should look at the aCal app to see what needs doing...)

Instead I wrote a conversion script. plan2ics is on Launchpad.net, and it takes the netplan file as input and creates an iCal output. I allow the iCal file to be served by a webserver, then tell Google Calendar to import the calendar from an external URL. Once Google has it, the calendar gets synced down to my Android phone.
Oh, and I can also use the calendar URL for email clients like Evolution or Thunderbird.

So it works, but are there any problems?

Well yes, thanks for asking.

First, it is a READ-ONLY solution. So no updating the entries on the web or phone and expecting them to appear in netplan - for this I need to make my own client (see above).

Second, I'm not handling the UID for events properly. At the moment I have a fudge that will break when events are moved around, then I get duplicates in the calendar. To fix this I need to borrow a trick from some old tools to sync netplan with Palm Pilots, and that is to write the UID back into the netplan file as a comment, then reuse the UID in the next export.

Being "nice" on Linux

Many thanks to a post from Colin Waters for showing me a better way to run background jobs on Linux without slowing down other processes.

Previously I would have just prefixed the command with "nice" and be done with it, and that was pretty good.

Now I use "chrt --idle 0 ionice -t -c3". Actually I wrap that into an environment variable and put $NEWNICE before the command. The two commands set the kernel and IO scheduling policies for the command to IDLE.

This has been making a huge difference on my video conversion scripts, and last night it meant that I could take a 9 hour migration, consisting of image conversions that wanted 100% CPU, and run it on a production webserver without losing service.

When you have nothing to say...

When you have nothing to say, you can always redesign your layout!

... and suddenly a year has gone by

I look away, and suddenly a whole year has gone by.

Father's Day 2008 was last Sunday. I fed the little girl (now quite the mobile toddler) her breakfast, and the adorable little boy prepared bananas, Weetbix, and yoghurt for me. Actually his first instinct was chopped bananas sprinkled with Milo, but that felt more desert so I asked for the Weetbix instead.

We went swimming at the Tepid Baths, waved the lawnmower over the onion weed in the back and did a little reading. The day before we visited the French Market for the weekly crepe, juice, and whitebait fritters - so a pleasant weekend all round.

The little girl is walking, and has been since around May-June. She looks forward to getting out of the house, and will often sit by the front door clutching her shoes... and if that hint fails, she has been known to deliver adult shoes to a grownup, who should obviously put them on and take her out.

The boy is now pretty big... he has taken to riding his first proper bicycle, which impressed me a lot.
His rugby season has just finished, and again he did very well. Since last year, he has basically got the hang of what he is supposed to be doing, and seemed to enjoy himself a lot more - scoring tries each week, and even making a tackle most weeks.

Meanwhile I have recently been indulging my media addiction, and very much enjoying The Middleman, Season 3 of Outrageous Fortune, Wanted, Pushing Daisies, and Life. I loved the gem that is Time Crimes and thought that the new Clone Wars movie was not as bad as it might have been...

Finally, watch this space for the complete run-down of "books I am reading while not finishing Proust...".

Fathers Day

So today was Fathers Day... and very nice too.

It started at around 2am, then again at 4am, 5:30am, and finally 6am - as the little girl, unable to contain her obvious excitement at Fathers Day, decided to wake and demand a cuddle.

The little boy was very understanding and he (and his Mum) allowed me to snooze until about 9am. It was also very sweet of him, since he has spent 3 days plotting and hiding a Fathers Day card for me. When he delivered them to me, I was amazed to see they were actually three cards. One he made at school, and one he made at home for each day he was waiting before the Day.

At noon we went over to the parent-in-laws for a big lunch with all three (or should that be four?) families.