Unkillable zombie blog

sorting using keys in python.

I’ve used comparator functions while sorting in python quite often. I had a complex object today that i had to sort on using non-standard comparator functions.

My object was a dictionary that contained a list of 2 dictionaries that in turn containd lists of objects. I wanted to sort the first dictinary based on the len() of the objects of the lists at the other end of the hierarchy.

using a cmp function i had to do something like:

def comparator(x,y):
   #4 lines do stuff to get the len() of data inside x


   #4 lines do stuff to get the len() of data inside y

   if len_x > len_y:
       return 1
   elif len_x == len_y:
       return 0
   else:
       return -1

However i came upon the sorting howto on wiki.python.org and a way to do sorting that i did not know. Using the key funciton.

Instead of sorted(iterable,cmp=my_cmp_func) you can do sorted(iterable,key=my_key_func).

my_key_func is called with each element of the iterable as argument. It must returns a value that is used in sorting (I assume using standard cmp).

This meant that instead of my cmp function i could do only:

def my_key_func(x):
   #4 lines do stuff to get the len() of data inside x
   return len_x

It makes more sense because basically we are not sorting x and y based on their own value, but we are sorting x and y based on some remote data connected to x and y.

Certainly nicer.

Too many unused .gr domain names?

Recently i have been developing a few small web sites. Some of them are only hobby projects.  Naturally i looked for a good domain name for them.

My search for good domain names indicated that most are already taken. Not very surprising as such, ofcourse ,but when i look up a domain name and it turns up unavailable i have the strange impulse to see what its being used for. Well, as it turns out for 80% (number out of my ass) of the domain names i look for they were not being used. Some of them have been registered as far as  seven years ago and are still “comming soon”. Others resolve to nothing.

So, we have a strange phenomenon here. While the greek web space is very small, the domain name address space is running out for no useful reason. That bothers me. I don’t mind it so much if i can’t find a good domain name because it’s being used. But so many domain names being unavailable and completely useless for many years is really infuriating.

This reminded me reading a few years back in EETT’s policies (the Greek national body regulating domain names), that a domain name that was unsused for more than 2 (or 3?) years would be expire. Maybe i should turn evil mode on and send a little email to EETT :p

Site thumbnail generation using khtml2png

For a friend’s project, i investigated various ways to generate a site thumbnail.

The obvious one is to simply go to the desired website using your favorite browser, take a snapshot using your favorite program. Guaranteed to work :).

    For obvious reasons that never became a real option so my googling revealed various services that do this. Some do it for free but in a limited manner , other are pay services but i was not satisfied with their options (or having to pay). The most famous of them is probably Alexa Site Thumbnails (AST) that is an Amazon web service. I didn’t like it either, especially since according to their terms you can’t store the image , but are only allowed to cache it up to 24 hours. Not good enough for me. (It’s dirt cheap though).

    After a little more searching i found out about the fantastic khtml2png utility. What this does is use khtml libs (from kde3.x) to render a site into a png (and a lot of other image types).

    The catch is that it requires an X server, but on their project page they have instructions to run it on a headless machine using Xvfb (an X server running on a virtual frame buffer).

It’s really easy:

On my machine i did:

Xvfb :1 -screen 0 1600x1200x24&
export DISPLAY="localhost:1.0"
khtml2png2 --width 1280 --height 1024 http://slashdot.org slashdot.png

And that’s it! I can now turn a headless Pentium III i have running various services (mail etc) to my own private , highly configurable web snapshot service.

Resolver is a great idea.

I was reading about Resolver quite often, mostly on planet python. I had made a mental memo to try it out for my self sometime but always seemed to be delaying it. I tried it out yesterday i was really impressed by the concept. 

With Resolver you have a spreadsheet with tightly integrated python programmability. The window has 3 panes. The ‘spreadsheet’ pane, the Python editor, and the ‘output’ pane. You can define any python function you like and make it available for use in your spreadsheet. This means you can do database connectivity using python, or advanced data manipulation that is either no possible with Excel, or very difficult.  I can’t tell you how many times i wished i could do that easily with Excel. In fact, i very often needed to pull data from the database of a Django project and do some manipulation using Excel , and present it to some users. But i wanted to use the Django ORM instead of raw SQL. With Resolver it should be very easy.

At least for me, it was often the case that i needed to solve a problem that was too complex for Excel, and too simple for writing a full-blown application. I needed something that would combine the best of both worlds.

Resolver is written in IronPython and uses Python for programming.  It might be possible to run it using mono but i haven’t tried anything of the sort. Ironpython supposedly runs on linux , but mono has an incomplete Forms implmementation.

Best of all, it free for non-commercial use, so you need to try it out.

IBAN validating code

Recently i needed to validate some IBAN numbers (International Bank Account Number). I quick search revealed the wikipedia article on the subject.

Though i had initially hacked up a Greek-only IBAN validator i though it might be of use to other people as well and modified it to validate  every possible IBAN (as long as the wikipedia article is complete).

So here it is:

#!/usr/bin/python
# -*- coding: utf-8

# iban.py
# http://kill9.eu/
#
# Copyright (C) 2008 Konstantinos Metaxas 
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published
# by the Free Software Foundation; version 2 only.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

import string

class IBAN(object):

    def __init__(self,iban):
        """Represents an IBAN number.
        iban must be supplied as a string"""
        self.iban = iban.strip().upper()
        #List retrieved from http://en.wikipedia.org/wiki/IBAN
        self.error_list = []
        self.country_length_table = {
                'AD':('Andora',24),
                'AT':('Austria',20),
                'BE':('Belgium',16),
                'BA':('Bosnia and Hetzegovina',20),
                'BG':('Bulgaria',22),
                'HR':('Croatia',21),
                'CY':('Cyprus',28),
                'CZ':('Czech Republic',24),
                'DK':('Denmark',18),
                'EE':('Estonia',20),
                'FO':('Faroe Islands',18),
                'FI':('Finland',18),
                'FR':('France',27),
                'DE':('Germany',22),
                'GI':('Gibraltar',23),
                'GR':('Greece',27),
                'GL':('Greenland',18),
                'HU':('Hungary',28),
                'IS':('Iceland',26),
                'IE':('Republic of Ireland',22),
                'IL':('Israel',23),
                'IT':('Italy',27),
                'LV':('Latvia',21),
                'LI':('Liechtenstein',21),
                'LT':('Lithuania',20),
                'LU':('Luxembourg',20),
                'MK':('FYROM',19),
                'MT':('Malta',31),
                'MC':('Monaco',27),
                'ME':('Montenegro',22),
                'MA':('Morocco',24),
                'NL':('Netherlands',18),
                'NO':('Norway',15),
                'PL':('Poland',28),
                'PT':('Portugal',25),
                'RO':('Romania',24),
                'SM':('San Marino',27),
                'RS':('Serbia',22),
                'SK':('Slovakia',24),
                'SI':('Slovenia',19),
                'ES':('Spain',24),
                'SE':('Sweden',24),
                'CH':('Switzerland',21),
                'TR':('Turkey',26),
                'TN':('Tunisia',24),
                'GB':('United Kingdom',22),

                }

    def is_valid(self):
        """checks if it is a valid IBAN"""

        if not self._is_valid_for_countrycode():
            return False
        if not self._swift_check():
            return False
        return True

    def _is_valid_for_countrycode(self):
        code = self.iban[:2]
        try:
            max_length = self.country_length_table[code][1]
        except KeyError:
            return False
        if len(self.iban) != max_length:
            return False
        else:
            return True

    def _swift_check(self):
        """
        http://en.wikipedia.org/wiki/IBAN
        """
        head_to_tail_iban = self.iban[4:] + self.iban[:4]
        numerical_ascii_map = dict( zip(string.ascii_uppercase,range(10,36)) )

        translated_iban = ""
        for c in head_to_tail_iban:
            if c in numerical_ascii_map:
                translated_iban += str(numerical_ascii_map[c])
            else:
                translated_iban += c

        try:
            integer_iban = int(translated_iban)
        except ValueError:
            return False

        modulo = integer_iban % 97

        if modulo == 1:
            return True
        else:
            return False

Using it as as simple as:

Python 2.5.1 (r251:54863, Oct 14 2007, 19:24:49)
[GCC 4.2.2 (Gentoo 4.2.2 p1.0)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import iban
>>> myIBAN = iban.IBAN('GRXXXXXXXXXXXXXXXXXX') # Actual IBAN omitted for privacy reasons
>>> myIBAN.is_valid()
True

It should be easy to make it into a custom Django Form Field , and i might post the relevant code later.

XML feed με το πρόγραμμα των ελληνικών καναλιών

Γιατί τα ελληνικά κανάλια στα site τους δεν παρέχουν XML feeds με το πρόγραμμά τους?

Δεν είναι κρυφή πληροφορία, και θα ήταν πολύ χρήσιμο σε projects όπως το mythtv, αλλα και σε όσους θα ήθελαν να αυτοματοποιήσουν κάποιο σύστημα βασισμένοι στην πληροφορία αυτή.

Είναι μια απλή παράλειψη ή υπάρχει κάποιος άλλη εξήγηση?

Displaying a message after a POST/Redirect/GET

I use the POST/Redirect/GET (RPG) pattern to avoid multiple submission problems. Sometimes however i want the GET part to display a message related to the POST part. Eg, “Your change has been done. Thank you” without creating special view just for this.
I found the solution to this problem in the www.djangosnippets.org site. It’s snippet no 319 titled “Flash Message Template Tag”.
What it does is use sessions and a special template tag to display a message in the template and immediately delete it from the session. After you set it up, all you have to do is something like this:

request.session['flash_msg'] = 'Your FOO has been BARed'
request.session['flash_params'] = {'type': 'success'}

Even my sister can hack Altec telecoms

A few days ago, my sister’s boyfriend, who is currently in the army in some far away island, wanted to change the DNS settings on his domain, because his hosting provider changed DNS servers. Not being very knowledgable in these things he had given the registrar a contact email on his domain, which meant he couldn’t get the verification-code he needed to login and do the changes.

He called my sister and told her to fax them , telling them to change the contact email address on his domain. My sister popped-up MS Word and wrote a letter without contact details, just the company name, and asked them to do the changes. Sure enough they sent an email with the verification-code …. for a different company. Apparently anyone can send any piece of paper, no matter how awful, asking them for the verification code of any domain, and the automatons over at Altec Telecoms will happily oblige.

Besides the obvious security issue, i must comment on their service level. My sister would keep contacting them for two days, being given from representative to representative to technical staff, to sales staff, to the janitor and ‘whoever-would-be-walking-besides-a-ringing-telephone’. At some point she got mad and asked the lady she was speaking with about her name. The lady, which identified heslef as ‘Papadopoulou’ told my sister to ‘fuck off’ while hunging up. Probably though my sister had not head me. Upon hearing this i tried to call them and complain to some supervisor, but i was put on hold for more than five minutes (their way of telling me…. ‘fuck off, too’ i guess) and i hang up.

I wish i could say that Altec Telecoms is alone in being such a horrible pain to deal with, but my own ISP, Forthnet, has been just a evil…

Captcha widgets in Django

This blog was getting stormed by comment spam so i decided to use some captcha protection. I chose pycaptcha for this and had to integrate it nicely with Django. I wanted to make it an easy widget that does it’s own validation and does not require view code for validating the captcha. So here is what i came up with:

EDIT: Updated for SVN trunk version of Django ( value_from_datadict has an extra parameter)

First, the custom form field:

class CaptchaField(CharField):
    widget = CaptchaWidget
    def __init__(self, *args, **kwargs):
        super(CaptchaField, self).__init__(*args, **kwargs)


    def clean(self,values):
        value,captchaid = values
        #Check if the value matches that in the session
        factory = PersistentFactory('/tmp/captcha_persistence')
        test = factory.get(str(captchaid))
        factory.storedInstances.close()
        if (not test) or (not test.valid):
            raise ValidationError(u"""Captcha does not exist or is
            invalid. Please  refresh to get new captcha""")
        if not test.testSolutions([value.strip()]):
            raise ValidationError(u"""You did not type the correct
            captcha text""")
        return ""

The Widget used by this form field:

class CaptchaWidget(Input):
    input_type = "text"
    def render(self,name,value,attrs=None):
        #Always generate a new captcha on rendering
        factory = PersistentFactory('/tmp/captcha_persistence')
        test = factory.new(Tests.PseudoGimpy)
        factory.storedInstances.close()
        captchaid = test.id
        superclass_output = super(CaptchaWidget,self).render(name,"",attrs)
        output = """<img src="/captcha/%s" border="0"/><br/>
                    <input type="hidden" name="captchaid" value="%s" />
        %s
        """ %(captchaid,captchaid,superclass_output)
        return output

    def value_from_datadict(self,data,files,name):
        return [ data[name],data['captchaid'] ]

We also need a view to deliver the image for the captcha so here it is:

def captcha_image(request,id):

    factory = PersistentFactory('/tmp/captcha_persistence')
    test = factory.get(str(id))
    factory.storedInstances.close()
    if not test:
        return HttpResponseNotFound()

    response = HttpResponse(mimetype='image/jpeg')
    test.render().save(response,'JPEG')
    return response

Dont forget to add a urls.py entry for our view. something like:

(r'^captcha/(?P<id>\w+)','blogon.blog.views.captcha_image'),

should be enough (change the view to your view. If you change the url then be sure to change it in the html generated by the widget.

. Notice that in our code we use a file in /tmp for writing the persistence of pycaptcha. This may be a security problem depending on our environment. So be careful, or us a trusted location.
I should use a template for the HTML generated by the widget so as to make it more easily customizable but i just got bored after i got it working.

What people do to get their revenge…