Unkillable zombie blog

Gafkalo is going strong

Announcing Gafkalo

Annoucing the release of Gafkalo. A tool to manage a Confluent Kafka platform.

While there are a few tools that manage Kafka resources with any current solution, and as they says goes, i scratched a personal itch.

topics:
  - name: SKATA.VROMIA.POLY
    partitions: 6
    replication_factor: 1
    # Any topic configs can be added to this key
    configs:
      cleanup.policy: delete
      min.insync.replicas: 1
      retention.ms: 10000000
    key:
      schema: "schema-key.json"
      compatibility: BACKWARD
    value:
      schema: "schema.json"
      compatibility: NONE
  - name: SKATA.VROMIA.LIGO
    partitions: 6
    replication_factor: 3
    configs:
      cleanup.policy: delete
      min.insync.replicas: 1
    key:
      schema: "schema-key.json"
  - name: SKATA1
    partitions: 1
    replication_factor: 1

clients:
  # principals must be in the form User:name or Group:name
  # For each principal you can have a consumer_for, producer_for or resourceowner_for
  # and the topics for each of these categories
  - principal: User:poutanaola
    consumer_for:
      # By default we will use PREFIXED.
      # set prefixed: false to set it to LITERAL
      - topic: TOPIC1.
      - topic: TOPIC2.
        prefixed: false
    producer_for:
      - topic: TOPIC1.
    resourceowner_for:
      - topic: TOPIC4.
  - principal: Group:malakes
    consumer_for:
      - topic: TOPIC1.
      - topic: TOPIC2.
    producer_for:
      - topic: TOPIC1.
        strict: false
    groups:
      - name: consumer-produser-
        # if not specified, roles is [DeveloperRead]
        # roles: ["ResourceOwner"]
        # prefixed is true by default but can be disabled like below
        refixed: false

gafkalo plan --config myconfig.yaml

gafkalo apply --config myconfig.yaml

K8S upgrade

This shitty, unmaintained blog that is made with a static site generator, is now running in Kubernetes.

I took the time to setup a private cluster to serve mostly internal tools for my home (Identity management etc) available over my VPN, but also moved some external facing stuff.

So that meant that this blog,that i have not written to in years, gets to run in Kubernetes. :)

In fact it even has its own Helm chart, and certificates with cert-manager now :D

More than one OR in Odoo ORM domains

I’ve been using Odoo for some time and written some apps for the printing industry. I’ve actually tried it since it was called TinyERP then OpenERP and now Odoo. In the last couple of years, its documentation went from abysmal to somewhat decent. It is still far from complete but at least one can have a hope of making progress at Odoo development.

There is a ton of undocumented stuff that i’ve found over the years and wanted to blog but had no time (Twitter is easy yo!).

Much of the ‘documentation is basically looking into the source code itself.

So my problem was simple. In the search field of Products (sales/inventory) i wanted to have Odoo also search by default the product code associated with a product (since some products have more than one seller and often people will request them by the vendor’s product code)

In my custom app in edited my myapp_view.xml to override the default search field for name with this:

<record model="ir.ui.view" id="metaxas_product_template_search">
  <field name="name">metaxas.product.template.search</field>
  <field name="model">product.template</field>
  <field name="inherit_id" ref="product.product_template_search_view"/>
  <field name="arch" type="xml">
    <xpath expr="//search/field[@name='name']" position="replace">
        <field name="name" string="Product" filter_domain="['|','|',('default_code','ilike',self),('name','ilike',self),('seller_ids.product_code','ilike',self)]"/>
      </xpath>
  </field>

</record>

Notice that the default Odoo search field is defined thus:

<field name="name" string="Product" filter_domain="['|',('default_code','ilike',self),('name','ilike',self)]"/>

According to Odoo docs , domains default to AND and if you want to use OR (or NOT) you prefix using ‘!’,’|’ etc. But what is not documented is how this prefix is applied. Adding a new third domain expresssion (‘field’,’operand’,’value) would default it to AND and basically screw your results. So i had to look into the source code with a GREP (ack rocks) and find out how Odoo does it and lo and behold, addonns/membership/membership_view.xml had the answer:

<field name="name"  filter_domain="['|','|',('name','ilike',self),('parent_id','ilike',self),('ref','=',self)]"/>

You need to prefix twice (or ,i assume, as many prefixes you require) Odoo docs need a lot of love still, despite significant progress.

Dynamic choices in wtforms SelectField.

I’ve built a Django app that is used internally at my job. I’m migrating it to Odoo (formerly OpenERP).

One of the things i had to do was move from django forms to wtforms (since Odoo does not supply a forms system)

Now the official wtforms documentation states that in order to provide dynamic forms to a SelectField you have to supply the assign the choices argument at instantiation time (instead of the class definition). This simply would not do for my setup because it is a complicated ‘plugin’-like system. The class that instantiates forms knows knothing about the form definition or where it came from.

A quick Google and Stackoverflow search did not turn up a solution so i dug into the wtforms sourcecode and found out that choices is treated as an iterable. No type checks or anything.

So i simply created a class for my data that implemented the __iter__ method:

from wtforms import Form , SelectField
class RandomChoicesIterable(object):
    def __iter__(self):
        from random import randint
        for i in xrange(0,5):
            pair = (randint(1,100),randint(1,100))
            yield pair

class MyForm(Form):
        randomchoices = SelectField(
            choices = RandomChoicesIterable(),
            label = "Random choices every time!"
            )

Now, you would get your database stuff in the __iter__ method instead of giving random data but this is only an example.

For example:

>>> form = MyForm()
>>> for field in form:
...    print field
...
<select id="randomchoices" name="randomchoices"><option value="40">9</option><option value="41">15</option><option value="11">6</option><option value="1">95</option><option value="39">33</option></select>
>>> for field in form: print field
...
<select id="randomchoices" name="randomchoices"><option value="82">81</option><option value="43">7</option><option value="97">87</option><option value="100">48</option><option value="85">59</option></select>
>>> for field in form: print field
...
<select id="randomchoices" name="randomchoices"><option value="29">89</option><option value="10">37</option><option value="58">37</option><option value="14">97</option><option value="10">22</option></select>
>>> for field in form: print field
...
<select id="randomchoices" name="randomchoices"><option value="22">18</option><option value="96">92</option><option value="27">81</option><option value="59">94</option><option value="28">42</option></select>
>>> for field in form: print field
...
<select id="randomchoices" name="randomchoices"><option value="17">5</option><option value="73">31</option><option value="21">68</option><option value="82">11</option><option value="81">61</option></select>
>>> for field in form: print field
...
<select id="randomchoices" name="randomchoices"><option value="37">53</option><option value="13">18</option><option value="69">73</option><option value="16">51</option><option value="86">14</option></select>
>>> for field in form: print field
...
<select id="randomchoices" name="randomchoices"><option value="81">42</option><option value="86">50</option><option value="29">71</option><option value="45">74</option><option value="43">35</option></select>

Τι έμαθα απο εναν επιστήμονα.

Μια ωραία Κυριακή, εκεί που απολαμβάναμε το ουζάκι μας σε παραλιακή ψαροταβέρνα, η παρέα μας μεγάλωσε αναπάντεχα (οπως συμβαίνει μερικές φορές). Πάντα ευπρόσδεκτες οι νέες γνωριμίες και μαζί με αυτές οι γνώση απο τις συζητήσεις.

Υπήρξα ιδιαίτερα τυχερός όταν στην παρέα ήταν ένας λαμπρός νέος επιστήμων. Χημικός απο Ελληνικό πανεπιστήμιο.

Σας παραθέτω μερικά τμήματα απο την επιστημονική γνώση απο τόσο μεγαλόχερα σκόρπισε αυτό το λαμπρό πνεύμα.

Αλλαγή blog engine

To byteflow που χρησιμοποιούσα στο kill9.eu είναι unmaintained εδώ και χρόνια και χρειαζόταν συνέχεια προσοχή και σκάλισμα.

Αφού έψαξα διάφορα blog engines σε Django ανακάλυψα οτι καμία δεν μου έκανε (!).

Ηθελα:

• Απλό. Χωρίς πολλά dependencies και configuration
• Syntax highlighting.
• Maintained χωρίς φόβο οτι σε 1 χρόνο θα ψάχνω νέο engine
• Να μπορώ να μεταφέρω έυκολα τα υπάρχοντα post που είναι σε HTML / RST και MARKDOWN
• Να είναι cool :)

Τελικά αποφάσισα να δοκιμάσω το pelican static blog generator.

Not dead yet.

This blog is not dead yet. Not yet, Kameraden!

Tools for generating PDFs from django.

For a Django project i’m working on i needed to generate PDF reports. For example a “job ticket” that displays the client data, the material list and a list of “processing” actions to be performed, or an invoice (classic use case).

What i wanted was a way to generate these PDF files from a template using Django’s templating language. I had previously done something similar using PISA but i was not satisfied. There documentation was rather bad, generating HTML/CSS is not my strong point and automatic tools produce unreadable code. It does work though.

After some heavy and masterful googling i came up with several options.

1. Reportlab. Very well known, very powerful, but the good parts are not free. I did not want to write the layout using Python code. I’d rather use some high level abstraction on top of Reportlab. Their RML framework (which does exactly that) seems great but as far as i can see it is not free. There are open source alternatives though, which i’ll talk about later.
2. Pod Uses OpenOffice in server mode to generate documents in PDF. I’ll be running my software in a memory limited VPS and running OpenOffice (even in server mode) seems very scary. I did not investigate the exact memory requirements though because it seemed like a waste of time.
3. rst2pdf. RST is very awesome, Sphinx uses it but from my reading of rst it seems that generating accurate and custom layouts seems somewhat difficult (for a report that needs specific elements in specific locations).
4. trml2pdf. This is a tiny and free implementation of RML. This seemed to be a better solution, but what i wanted was a way to generate RML documents using a GUI tool (i’m a lazy bastard). The solution is tiny_sxw2rml which produces an RML document from an OpenOffice .odt file.
5. JasperReports. This is a very impressive Java package and GUI designer that can generate beautiful reports from various data sources. This was a serious contender (** especially** with their very nice GUI designer) but i chose to avoid it because i could not use the Django ORM to do templating (i can use raw SQL but that is severely suboptimal in templating , especially on complex templates). I could use the Django ORM to produce an XML file with all the information but the prospect of using an XML file (using XPath ) seemed dreadful (but it would work nicely). It is a Java package with Python bindings available for it and a web interface that can be used to access it using a standard interface (HTTP)

The fourth solution (trml2pdf) is the one i chose. These tools are developed by the openerp framework for their reporting needs.

However, it is far from a perfect solution. There is no official repository for this package with various versions floating around the web with different features. More specifically, i was only able to use barcode generation using the version from openerp server source code, which i had to patch to make it work (because it relied on other modules of OpenERP).

Additionally, i had to patch it to make it work with unicode fonts as per the instructions from the OpenErp Docs. Not something you would want to depend on.

Nevertheless i was able to make it all work. I could use the tool tiny_sxw2rml to produce an rml template that would first be processed by the Django templating system and then fed into trml2pdf to produce a PDF. It works.

All in all i would have to say that JasperReports is the best solution from a technical standpoint since it has a great GUI designer that produces beautiful documents with perfect layout, has a lot of features, a large community, can be used as an independent server that produces PDF documents and much much more. The main problem is that you lose that advantage of integrating your workflow into the philosophy of Django (ORM/templates etc) and have to walk the extra mile.

I’ve already integrated trml2pdf but i’m keeping my eye open for a way to switch to JasperReports.

I’d like to hear what solutions others may have used/considered for this.

Deployment with Fabric

Τί είναι:

To fabric είναι ένα εργαλείο για αυτοματοποίηση εργασιών μέσω SSH που διευκολύνει το deployment ή την διαχείρηση συστημάτων.

Συγκεκριμένα, πρόκειται για ένα command line εργαλείο και μια βιβλιοθήκη (API) γραμμένη σε Python που επιτρέπει την εκτέλεση προγραμμάτων command line ή και Python κώδικα, σε τοπικό η και απομακρυσμένο server(s).

Παράδειγμα:

Έστω οτι έχω ένα Django site που το αναπτύσσουμε στον υπολογιστή μας, και τρέχουμε ένα production server σε ένα VPS.

Οταν έχω μια νέα έκδοση του site μου, τρέχω μερικές εντολές στο τοπικό PC (πχ το test framework, κάνω push τις αλλαγές κλπ), μετά κάνω login στο VPS μου μέσω SSH και κάνω τις αναβαθμίσεις. Αυτά τα βήματα μπορούν, αφού καταγραφούν, και μετά να αυτοματοποιηθούν κάνοντας χρήση του fabric.

Εγκατάσταση.

Η εγκατάσταση είναι πολύ έυκολη. easy_install fabric ή απο το package manager της διανομής μας

Το fabfile

Το πρώτο βήμα για την χρήση του fabric είναι η δημιουργία του αρχείου στο οποίο βάζουμε τις “εντολές” μας. Το αρχείο αυτό το ονομάζουμε fabfile.py γιατί με αυτό το όνομα το βρίσκει αυτόματα το εργαλείο fab (το command line εργαλείο του fabric)

Στο fabfile αρχείο καταγράφουμε τις διάφορες “λειτουργίες” που θα εκτελεί το Fabric.

Παράδειγμα:

def hello():
    print "Hello world"

Τώρα μπορούμε να τρέξουμε την “hello” τρέχοντας:

arcanum@localhost ~ $ fab hello

Το fabric αυτόματα θα ψάξει για ένα αρχείο με όνομα fabfile.py και μέσα του για ένα callable με όνομα hello και θα το εκτελέσει.

Φυσικά αυτό είναι υπερβολικά απλό και όχι ιδιαίτερα χρήσιμο απο μόνο του. Η χρησιμότητα του Fabric έρχεται απο το API του.

Παράδειγμα με το API του Fabric

Όπως είπαμε θέλουμε να τρέξουμε μερικές εντολές στο τοπικό PC και μερικές στο απομακρυσμένο. Το fabric μας παρέχει ένα API για αυτή την δουλειά Αυτό είναι ένα (τροποποιημένο για πολλούς λόγους) script που χρησιμοποιώ σε ένα project μου.

def import datetime
from fabric.api import local, run, cd, env

# The app directory of metaxas.gr
PROJECT_DIR='/home/arcanum/django_projects/myproject'

env.hosts = ['arcanum@metaxas.gr:2000']

def deploy():
    backup_database()
    sync_remote_repo()
    run_migrations()
    restart_wsgi()

def push_updates():
    """
    Push updates from local repo to remote repo
    Does *NOT* commit uncommited modifications
    """
    # TODO check for uncommitted modifications and give warning
    local("hg push")

def sync_remote_repo():
    """
    Sync the remote stable repository on the production server
    """
    with cd(PROJECT_DIR):
        run('hg pull')
        run('hg up')

def run_migrations():
    """
    Run any migration on the remote server
    """
    with cd(PROJECT_DIR):
        run('./manage.py migrate')


def restart_wsgi():
    """
    Restart the WSGI app by 'touch'ing the wsgi file.
    Thi means updating the last modification date of that file, which is
    monitored by the WSGI app server
    """
    with cd(PROJECT_DIR):
        run('touch wsgi/django.wsgi')


def backup_database():
    """
    Make a backup of the database.
    """
    with cd(PROJECT_DIR):
        now = datetime.datetime.now()
        now_str = now.strftime("%d-%m-%Y")
        BACK_CMD = 'pg_dump --format=t mydatabase |bzip2 > mydatabase-%s.sql.bz2'%(now_str, )

        run(BACK_CMD)

Θεωρώ οτι είναι αρκετά απλό παράδειγμα και κάποιος με εμπειρία στην Python θα καταλάβει αμέσως τι γίνεται όμως θα ανεφέρω μερικά πράγματα:

1. Τρέχοντας fab deploy το fabric θα βρεί την function με όνομα deploy και θα την εκτελέσει. Αυτή με την σειρά της καλεί διάφορες άλλες functions μέσα στο fabfile.

2. Απο το API του fabric χρησιμοποιούμε μερικές τυπικές λειτουργίες όπως local , run, cd κλπ.

Η local μας δίνει την δυνατότητα να τρέξουμε μια εντολή (σαν να είμασταν στο shell) στο τοπικό μηχάνημα.

Η run τρέχει μια εντολή (πάλι shell) στο remote μηχάνημα. Αυτό γίνεται transparently μέσω SSH. Το remote host τον βρίσκει απο το την env.hosts λίστα. Θα μπορούσαμε να συνδεόμαστε σε πολλούς hosts (αν για παράδειγμα είχαμε πολλούς web server σε ένα scalable site).

Το cd κάνει αυτό που φαντάζεσται. Η ανάγκη του προέρχεται απο την έλλειψη state της των shell-less SSH συνδέσεων. Το cd έχει ένα context manager (with statement ) οπότε οτι τρέχει μέσα στο context του with γίνεται prepend το path.

Γενικά το fabric λύνει πολλά προβλήματα έυκολα και γρήγορα. Απο οτι διαβάζω υπάρχουν πιο ευέλικτες αλλά πιό πολύπλοκες λύσεις αλλά δεν έχω ασχοληθεί γιατί το fabric είναι ικανότατο.

Το API του fabric είναι, προφανώς, πολύ μεγαλύτερο με δυνατότητες για prompts, file transfers, κλπ.

Για περισσότερα ανατρέξτε στο fabric documentation