Category Archives: Knowledge base

Mounting external usb drives automatically to its label

I am maintaining a file server running Ubuntu 12.04 in my company and was faced with the problem how to backup my system easily and on a low cost. As we have a huge amount of data and a rather slow internet connection I have made the decision to backup onto external hard drives which we will rotate frequently. In the end USB drives are still the cheapest option to backup your data… ;).

This blog post is though not about backing up your data but rather how to mount a USB drive automatically when connected to a server and access it with its label. This helped me to have a specific mount point for backup hard discs but won’t interfere if anyone else needs to connect another USB drive to the server. I am sure you might have different use cases for such feature as well. I have tested following configuration on a Ubuntu Server 12.04 but it should run on other Debian based operating systems as well.

OK this said let’s get started with the configuration then.

While I was looking for a tool to automatically mount my usb drives I found usbmount which was very simple and pretty much worked out of the box.

You can install it with the following command:
apt-get install usbmount

After usbmount is installed you can simply connect your usb drive to your server and usbmount will automatically mount it to /media/usb0. When an additional drive is connected it will be mounted to /media/usb1 and so on.

That’s a good start… but there is a draw back that every drive connected to the server will be mounted to /media/usb0 so we cannot really distinguish the different drives from each other. But usbmount has a feature for this that it creates a symlink for each drive in the folder /var/run/usbmount.

Such symlink can look like the following:
lrwxrwxrwx 1 root root 11 Apr 11 10:53 SAMSUNG_HM160HI_1 -> /media/usb0

The name of the symlink is built by using the vendor, model and partition number. So accessing the hard disc you simply access it using this symlink. This is great when you want to identify a specific hard disc. There is still one problem with it though when I want a group of hard disc to be accessed the same way I cannot really use this symlink.

So my idea was why not using the volume label and when I format hard drives I simply give all those drives the same volume label.

This can be done for instance with following format command using ext4 as an example (of course you can change ext4 to whatever file system you need):
mkfs.ext4 -L VOLUMENAME /dev/sdb1

Unfortunately usbmount doesn’t create any symlink of drives by its volume label. I therefore created following script to do this:

#!/bin/sh
# This script creates the volume label symlink in /var/run/usbmount.
# Copyright (C) 2014 Oliver Sauder
#
# This file is free software; the copyright holder gives unlimited
# permission to copy and/or distribute it, with or without
# modifications, as long as this notice is preserved.
#
# This file is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY, to the extent permitted by law; without
# even the implied warranty of MERCHANTABILITY or FITNESS FOR A
# PARTICULAR PURPOSE.
#
set -e

# Exit if device or mountpoint is empty.
test -z "$UM_DEVICE" && test -z "$UM_MOUNTPOINT" && exit 0

# get volume label name
label=`blkid -s LABEL -o value $UM_DEVICE`

# If the symlink does not yet exist, create it.
test -z $label || test -e "/var/run/usbmount/$label" || ln -sf "$UM_MOUNTPOINT" "/var/run/usbmount/$label"

exit 0

This script you have to save to /etc/usbmount/mount.d/01_create_label_symlink and do not forget to add executable rights to it (chmod +x /etc/usbmount/mount.d/01_create_label_symlink). usbmount will run this script whenever a hard disc is connected.

Beside creating this symlink we also have to make sure that the symlink will be removed when the hard disc is disconnected. usbmount already has a script which removes symlink from /var/run/usbmount but we have to adjust it as it only removes the first link and not all. We can simply open the file /etc/usbmount/umount.d/00_remove_model_symlink and remove the break statement in line 19.

The script will then look like the following:

#!/bin/sh
# This script removes the model name symlink in /var/run/usbmount.
# Copyright (C) 2005 Martin Dickopp
#
# This file is free software; the copyright holder gives unlimited
# permission to copy and/or distribute it, with or without
# modifications, as long as this notice is preserved.
#
# This file is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY, to the extent permitted by law; without
# even the implied warranty of MERCHANTABILITY or FITNESS FOR A
# PARTICULAR PURPOSE.
#
set -e

ls /var/run/usbmount | while read name; do
    if test "`readlink "/var/run/usbmount/$name" || :`" = "$UM_MOUNTPOINT"; then
        rm -f "/var/run/usbmount/$name"
    fi
done

exit 0

So now when we connected a hard disc to the server there will be a symlink created /var/run/usbmount/VOLUMENAME and such path we can then use to point to for backups or whatever other use case you have.

Hope this was helpful. While working with usbmount I have found some more articles which might be helpful for you e.g. when you have issues with permissions or want to use ntfs as a file system.

Writing tests for Vala

There has been a long outstanding issue to get a unit test environment up and running for Diodon which is written in Vala. Finally, I have found a suitable solution. In this blog post I want to talk about my experience. Hope this helps you to get your Vala code tested as well. Comments are welcome.

When I started my research on different vala unit test framework I found quite a few. Amongst them were vala-gtest-module and valadate.

Those frameworks looked all quite promising. Unfortunately, all do not seem to be maintained anymore and are not available as official package in any distribution repository (Note: It seems that Yorba has adopted valadate. So there is still hope… ;)).
Those issues make it very difficult to use these frameworks, as it would mean to run tests, the test framework library needs to be built from source as well. And as the code base of such is quite old, it might happen that it does not even build with the current vala version.

So I had to look for another solution. Doing so I came across GTest which is included in glib. This has the great advantage that I would not even need another dependency to get my vala code tested. And luckily the developers behind libgee created a very useful TestCase class based on GTest so to use the GTest api is not as cumbersome.

Let’s take a look at this class and see how we can use it (note that the class is licensed under LGPL and written by Julien Peeters, see original source code):

public abstract class Gee.TestCase : Object {

	private GLib.TestSuite suite;
	private Adaptor[] adaptors = new Adaptor[0];

	public delegate void TestMethod ();

	public TestCase (string name) {
		this.suite = new GLib.TestSuite (name);
	}

	public void add_test (string name, owned TestMethod test) {
		var adaptor = new Adaptor (name, (owned)test, this);
		this.adaptors += adaptor;

		this.suite.add (new GLib.TestCase (adaptor.name,
		                                   adaptor.set_up,
		                                   adaptor.run,
		                                   adaptor.tear_down ));
	}

	public virtual void set_up () {
	}

	public virtual void tear_down () {
	}

	public GLib.TestSuite get_suite () {
		return this.suite;
	}

	private class Adaptor {

		public string name { get; private set; }
		private TestMethod test;
		private TestCase test_case;

		public Adaptor (string name,
		                owned TestMethod test,
		                TestCase test_case) {
			this.name = name;
			this.test = (owned)test;
			this.test_case = test_case;
		}

		public void set_up (void* fixture) {
			this.test_case.set_up ();
		}

		public void run (void* fixture) {
			this.test ();
		}

		public void tear_down (void* fixture) {
			this.test_case.tear_down ();
		}
	}
}

What we can do now is to implement a test case class like the following:

class TestExample : TestCase {

  public TestExample() {
    // assign a name for this class
    base("TestExample");
    // add test methods
    add_test("test_example", test_example);
   }

   public override void set_up () {
     // setup your test
   }

   public void test_example() {
     // add your expressions
     // assert(expression);
   }

   public override void tear_down () {
     // tear down your test
   }
}

There are no naming convention here, how a test class and its test methods are called. However I prefer to name a test class the same as the class its testing + Test as prefix. For example the class above TestExample tests the functionality of the class Example. I do the same for test methods. So this means that the method test_example above tests the method example on the class Example.

This just as a note. Let’s go back to setting up our testing environment. So to be able to run the test we have just created, we need to compile all test classes into one executable with the following main method:

public static int main(string[] args) {
  Test.init (ref args);
  // add any of your test cases here
  TestSuite.get_root().add_suite(new TestExample().get_suite());
  return Test.run ();
}

Of course, you can create this executable by using your preferred build tool. In this post I just wanted to show you how this can be done with waf which I am currently using to build diodon.

So a wscript_build of a tests folder where all test cases are located could look like this:

prog = bld.new_task_gen (
  features = 'c cprogram test',
  target = 'example-test',
  source = bld.path.ant_glob (incl='**/*.vala'))

You probably noted that there is a feature test configured. This is a feature to run tests while building the executable with waf. I will come back to this later.

Let’s just make the assumption that this is not configured. This way we would now have an executable example-test which we could simply run in a terminal.

The output could look like this:

/TestExample/test_example: OK

Running a test executable would now always run all test cases. However this might take quite a while considering you wrote thousands of tests ;). For this reason, there is a tool called gtester included in glib which has some nice features such as running only a specific test case.

So let’s say we only want to run the test TestExample, we could use the following command:

gtester --verbose -p /TestExample example-test

gtester has many more options. Best take a look at its documentation.

It’s all great that we can now run our written tests manually to see whether our changes broke anything. However, it would be even better if such would be run every time we built our source code. And when a test fails that the build will fail as well (of course there should always be an option to disable running tests). This has the advantage when someone builds or packages your vala code on a different system with different dependencies, that errors the user might have with your code, already occur while building respectively running the tests.

So to do this with waf I want to come back to the test feature I have mentioned above. Beside configuring this feature we have to adjust the main wscript which could look like the following:

...
def options(opt):
  ...
  opt.add_option('--skiptests', action='store_true', default=False,
    dest='skiptests', help='Skip unit tests')
  ...

def build(ctx):
  ...
  if not Options.options.skiptests:
        ctx.add_subdirs('tests')
  ...
  ctx.add_post_fun(post)

  # to execute all tests:
  # $ waf --alltests
  # to set this behaviour permanenly:    
  ctx.options.all_tests = True

def post(ctx):
  waf_unit_test.summary(ctx)

  # Tests have to pass
  lst = getattr(ctx, 'utest_results', [])
  if lst:
    tfail = len([x for x in lst if x[1]])
    if tfail:
      ctx.fatal("Some test failed.")

When ./waf build is now called, the tests will be run as well. It is also possible to skip tests by simply using ./waf build –skiptests.

You can find a full fledge sample in the diodon source code.

If you use automake I would recommend you take a look at the libgee source code which have implemented a similar approach with automake.

This is it. Hope this was helpful. And do not forget: Test more!!! ;).

Global hotkeys with vala

The issue

Whilst working on Diodon I needed to implement a global hotkey. After browsing the web and some source code I came up with the following solution.

Get the code

Here is a sample class to easily bind a delegate with a global hotkey.

/**
 * This class is in charge to grab keybindings on the X11 display
 * and filter X11-events and passing on such events to the registed
 * handler methods.
 * 
 * @author Oliver Sauder 
 */
public class KeybindingManager : GLib.Object
{
    /**
     * list of binded keybindings
     */
    private Gee.List bindings = new Gee.ArrayList();
    
    /**
     * locked modifiers used to grab all keys whatever lock key
     * is pressed.
     */
    private static uint[] lock_modifiers = {
        0,
        Gdk.ModifierType.MOD2_MASK, // NUM_LOCK
        Gdk.ModifierType.LOCK_MASK, // CAPS_LOCK
        Gdk.ModifierType.MOD5_MASK, // SCROLL_LOCK
        Gdk.ModifierType.MOD2_MASK|Gdk.ModifierType.LOCK_MASK,
        Gdk.ModifierType.MOD2_MASK|Gdk.ModifierType.MOD5_MASK,
        Gdk.ModifierType.LOCK_MASK|Gdk.ModifierType.MOD5_MASK,
        Gdk.ModifierType.MOD2_MASK|Gdk.ModifierType.LOCK_MASK|Gdk.ModifierType.MOD5_MASK
    };
    
    /**
     * Helper class to store keybinding
     */
    private class Keybinding
    {
        public Keybinding(string accelerator, int keycode,
            Gdk.ModifierType modifiers, KeybindingHandlerFunc handler)
        {
            this.accelerator = accelerator;
            this.keycode = keycode;
            this.modifiers = modifiers;
            this.handler = handler;
        }
    
        public string accelerator { get; set; }
        public int keycode { get; set; }
        public Gdk.ModifierType modifiers { get; set; }
        public KeybindingHandlerFunc handler { get; set; }
    }
    
    /**
     * Keybinding func needed to bind key to handler
     * 
     * @param event passing on gdk event
     */
    public delegate void KeybindingHandlerFunc(Gdk.Event event);

    public KeybindingManager()
    {
        // init filter to retrieve X.Events
        Gdk.Window rootwin = Gdk.get_default_root_window();
        if(rootwin != null) {
            rootwin.add_filter(event_filter);
        }
    }
    
    /**
     * Bind accelerator to given handler
     *
     * @param accelerator accelerator parsable by Gtk.accelerator_parse
     * @param handler handler called when given accelerator is pressed
     */
    public void bind(string accelerator, KeybindingHandlerFunc handler)
    {
        debug("Binding key " + accelerator);
        
        // convert accelerator
        uint keysym;
        Gdk.ModifierType modifiers;
        Gtk.accelerator_parse(accelerator, out keysym, out modifiers);

        Gdk.Window rootwin = Gdk.get_default_root_window();     
        X.Display display = Gdk.x11_drawable_get_xdisplay(rootwin);
        X.ID xid = Gdk.x11_drawable_get_xid(rootwin);
        int keycode = display.keysym_to_keycode(keysym);            
                 
        if(keycode != 0) {
            // trap XErrors to avoid closing of application
            // even when grabing of key fails
            Gdk.error_trap_push();

            // grab key finally
            // also grab all keys which are combined with a lock key such NumLock
            foreach(uint lock_modifier in lock_modifiers) {     
                display.grab_key(keycode, modifiers|lock_modifier, xid, false,
                    X.GrabMode.Async, X.GrabMode.Async);
            }
            
            // wait until all X request have been processed
            Gdk.flush();
            
            // store binding
            Keybinding binding = new Keybinding(accelerator, keycode, modifiers, handler);
            bindings.add(binding);
            
            debug("Successfully binded key " + accelerator);
        }
    }
    
    /**
     * Unbind given accelerator.
     *
     * @param accelerator accelerator parsable by Gtk.accelerator_parse
     */
    public void unbind(string accelerator)
    {
        debug("Unbinding key " + accelerator);
        
        Gdk.Window rootwin = Gdk.get_default_root_window();     
        X.Display display = Gdk.x11_drawable_get_xdisplay(rootwin);
        X.ID xid = Gdk.x11_drawable_get_xid(rootwin);
        
        // unbind all keys with given accelerator
        Gee.List remove_bindings = new Gee.ArrayList();
        foreach(Keybinding binding in bindings) {
            if(str_equal(accelerator, binding.accelerator)) {
                foreach(uint lock_modifier in lock_modifiers) {
                    display.ungrab_key(binding.keycode, binding.modifiers, xid);
                }
                remove_bindings.add(binding);                    
            }
        }
        
        // remove unbinded keys
        bindings.remove_all(remove_bindings);
    }
    
    /**
     * Event filter method needed to fetch X.Events
     */
    public Gdk.FilterReturn event_filter(Gdk.XEvent gdk_xevent, Gdk.Event gdk_event)
    {
        Gdk.FilterReturn filter_return = Gdk.FilterReturn.CONTINUE;
                   
        void* pointer = &gdk_xevent;
        X.Event* xevent = (X.Event*) pointer;
         
         if(xevent->type == X.EventType.KeyPress) {
            foreach(Keybinding binding in bindings) {
                // remove NumLock, CapsLock and ScrollLock from key state
                uint event_mods = xevent.xkey.state & ~ (lock_modifiers[7]);
                if(xevent->xkey.keycode == binding.keycode && event_mods == binding.modifiers) {
                    // call all handlers with pressed key and modifiers
                    binding.handler(gdk_event);
                }
            }
         }
         
        return filter_return;
    }
}

How to use it

As some bindings needed to be adjusted you need at least version 0.11.3 of vala.

Furthermore are following packages needed:

  • gtk+-2.0
  • x11
  • gdk-x11-2.0
  • gee-1.0

Here is a sample how to compile this class:

valac --pkg gtk+-2.0 --pkg x11 --pkg gdk-x11-2.0 --pkg gee-1.0 keybinding-manager.vala

So and finally here some sample code how to bind a key:

public static int main (string[] args)
{
    Gtk.init (ref args);
    
    KeybindingManager manager = new KeybindingManager();
    manager.bind("V", test);

    Gtk.main ();
    return 0;
}
    
private static void test()
{
    debug("hotkey pressed");
}

Hope this sample class will be helpful for you. Happy coding ;).

Easen use of svn switch with bash prompt

The issue

svn switch can safe a lot of time when working in different branches as not all source code has to be checked out again.
But well you know it: How fast does it happen that a file gets committed in the wrong branch?
To prevent you from always have to call svn info before committing, here comes the script which includes the current branch in the bash prompt so you may never commit code in the wrong branch again ;).

Get the script

#!/usr/bin/perl

use Cwd;
my $dir = getcwd;
unless( -r $dir."/.svn" )
{
  exit;
}
$ret="svn";
$dir=`svn info | grep URL`;
$dir =~ s/n//;
if($dir =~ m//trunk/)
{
  $ret="trunk";
}
if($dir =~ m/branches/|tags//)
{
  $dir=$dir."/";
  $dir =~ s/.*(b|t)(ranches|ags)/(.*?)/.*/$1:$3/;
  $ret=$dir;
}
print " [$ret]";

What needs to be installed

Subversion of course and perl to run the script, which is usually installed in most distributions.

How to use it

Let’s store the script to svn-ps1.pl somewhere let’s say $HOME/bin.
Now open the bashrc file with vi ~/.bashrc and adjust the export of the PS1 variable:

PS1='${debian_chroot:+($debian_chroot)}[33[01;32m]u@h[33[00m]:[33[01;34m]w[33[00m]$ '

to the following:

PS1='${debian_chroot:+($debian_chroot)}[33[01;32m]u@h[33[00m]:[33[01;34m]w[33[00m]$(~/bin/svn-ps1.pl)$ '

Now simply change to some source code checked out with subversion and you will see that the prompt will look like this:
user@laptop:~/src/svn/some-code [b:branch-name]$

Cheers!

Indent json strings on the bash

The issue

While I was playing around with CouchDB and cURL I pretty quickly missed a feature to indent the json output string. So after looking around in the Internet for a simple solution I hit simplejson which provides methods to indent a json string.

Get the script

With simplejson I’ve created the following python script:

#!/usr/bin/env python

import sys
import simplejson
import string

content = string.join(sys.stdin.readlines())
json = simplejson.loads(content)
print simplejson.dumps(json, sort_keys=True, indent=2)

What needs to be installed

To be able to run the script simplejson needs to be installed. On a debian based system you can easily run following command to assure simplejson is installed:
sudo apt-get install python-simplejson

How to use it

Let’s store the script to jsonindent.py in a executable path directory such as $HOME/bin (run echo $PATH to get more possible directories). To indent a json script returned for instance from a CouchDB you can easily use following command:
curl -sX GET http://127.0.0.1:5984/example/hello-world | jsonindent.py

The output will look like this:

{
  "_id": "hello-world",
  "_rev": "1-97dd85b06c25328a300f3f4041def370",
  "body": "Well hello and welcome to my new blog...",
  "date": "2009/01/15 15:52:20",
  "title": "Hello World"
}

Looks pretty neat, doesn’t it ;)?