Negate a part of regular expression

Sometimes it is useful to negate a part of a regular expression. Not in all cases negating a group with the ^ character gets the job done.

Often negating a particular string is required. Technically what is usually needed then is called a negative lookahead.

When for example we want to match the string "AB" not followed by "CD" then the regular expression should be:

AB(?!CD)

The above will match "ABDC" but will not match "ABCD".

Make builds nice

Nice is an often forgotten but very useful part of coreutils. It adjusts the priority of the given command which in GNU/Linux (and UNIX) is called niceness. This name is quite convenient as it avoids the unnecessary confusion as to increase the priority the value has to be decreased. So a process with a high niceness value is "nice" to other processes.

The good thing about niceness is that it is inherited to child processes. So you just need to run your build script or make with nice:

nice -n 11 make -j8 all

All spawned processes (sub-make, compiler, linker etc.) get the same niceness value.

This can make a busy CI system responsive even under high load.

python's shortcuts

Let's try to optimize some simple python code like this:

a = ['eggs', 'bacon']

no_spam = True
for x in a:
    if x == 'spam':
        no_spam = False
if no_spam:
    print('there is no spam in it')

This might seem pretty obvious but an often forgotten fact is that a python's for loop may have an optional else clause.

So to make it slightly shorter and avoid using an additional variable:

a = ['eggs', 'bacon']

for x in a:
    if x == 'spam':
        break
else:
    print('there is no spam in it')

If you want to make if even shorter you can use the built-in functions any and map instead of the for loop:

a = ['eggs', 'bacon']

if not any(map(lambda x: True if x == 'spam' else False, a)):
    print('there is no spam in it')

Howto deploy Gerrit and Jenkins on Tomat

First get Tomcat, Gerrit and Jenkins. The preferred way is to use your distribution's package manager. If you don't have access to the root account or require a newer version you can grab the software from the respective project's websites:

wget http://www.nic.funet.fi/pub/mirrors/apache.org/tomcat/tomcat-7/v7.0.39/bin/apache-tomcat-7.0.39.tar.gz
wget http://gerrit.googlecode.com/files/gerrit-2.6-rc0.war
wget http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war

Unpack tomcat and place the war files in webapps dir:

tar -xf apache-tomcat-7.0.39.tar.gz
cp jenkins.war apache-tomcat-7.0.39/webapps/
cp gerrit-2.6-rc0.war apache-tomcat-7.0.39/webapps/gerrit.war

Setup the gerrit site:

java -jar gerrit-2.6-rc0.war init -d /install_dir/gerrit

*** Gerrit Code Review 2.6-rc0
***

Create '/install_dir/gerrit' [Y/n]?

*** Git Repositories
***

Location of Git repositories   [git]:

*** SQL Database
***

Database server type           [h2]:

*** User Authentication
***

Authentication method          [OPENID/?]: ?
       Supported options are:
         openid
         openid_sso
         http
         http_ldap
         client_ssl_cert_ldap
         ldap
         ldap_bind
         custom_extension
         development_become_any_account
Authentication method          [OPENID/?]: ldap
LDAP server                    [ldap://localhost]: ldap://ldap.server.net
LDAP username                  :
Account BaseDN                 [DC=ldap,DC=server,DC=net]: o=XXX
Group BaseDN                   [o=XXX]:

*** Email Delivery
***

SMTP server hostname           [localhost]:
SMTP server port               [(default)]:
SMTP encryption                [NONE/?]:
SMTP username                  :

*** Container Process
***

Run as                         [ute]:
Java runtime                   [/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre]:
Copy gerrit.war to /install_dir/gerrit/bin/gerrit.war [Y/n]? n

*** SSH Daemon
***

Listen on address              [*]:
Listen on port                 [29418]:

Gerrit Code Review is not shipped with Bouncy Castle Crypto v144
  If available, Gerrit can take advantage of features
  in the library, but will also function without it.
Download and install it now [Y/n]? Y
Downloading http://www.bouncycastle.org/download/bcprov-jdk16-144.jar ... OK
Checksum bcprov-jdk16-144.jar OK
Generating SSH host key ... rsa... dsa... done

*** HTTP Daemon
***

Behind reverse proxy           [y/N]?
Use SSL (https://)             [y/N]?
Listen on address              [*]:
Listen on port                 [8080]: 7003
Canonical URL                  [http://your.host.domain.net:7003/]: http://your.host.domain.net:4000/gerrit/

*** Plugins
***

Prompt to install core plugins [y/N]?

Initialized /install_dir/gerrit

Stop the built-in Gerrit server as tomcat will be used instead:

/install_dir/gerrit/bin/gerrit.sh stop

Now copy the jar files required for gerrit to start on tomcat:

cp gerrit/lib/bcprov-jdk16-144.jar apache-tomcat-7.0.39/lib/
java -jar apache-tomcat-7.0.39/webapps/gerrit.war cat lib/h2-1.3.168.jar >apache-tomcat-7.0.39/lib/h2-1.3.168.jar

Optionally change the ports:

sed -i 's/port="8080"/port="4000"/' apache-tomcat-7.0.39/conf/server.xml
sed -i 's/port="8443"/port="4443"/' apache-tomcat-7.0.39/conf/server.xml
sed -i 's/port="8009"/port="4009"/' apache-tomcat-7.0.39/conf/server.xml
sed -i 's/port="8005"/port="4005"/' apache-tomcat-7.0.39/conf/server.xml

To enable gerrit to access it's H2 database edit context.xml:

vi apache-tomcat-7.0.39/conf/context.xml

<Resource
    name="jdbc/ReviewDb"
    type="javax.sql.DataSource"
    username=""
    driverClassName="org.h2.Driver"
    password=""
    url="jdbc:h2:file:/install_dir/gerrit/db/ReviewDB"
    maxActive="100"
    maxIdle="20"/>
</Context>

Now create a starter script:

vi start.sh

#!/bin/sh
export CATALINA_HOME="/install_dir/apache-tomcat-7.0.39"
export CATALINA_PID="$CATALINA_HOME/tomcat.pid"
export CATALINA_OPTS="-DGERRIT_SITE=/install_dir/gerrit/ -DJENKINS_HOME=/install_dir/jenkins/ -Xmx1024m"
export JAVA_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:PermSize=256m -XX:MaxPermSize=512m"
cd $CATALINA_HOME/bin && ./startup.sh

Finally execute start.sh:

chmod +x start.sh
./start.sh

Now configure Jenkins using it's administration UI:

http://your.host.domain.net:4000/jenkins/

Setup gerrit permissions by logging in:

http://your.host.domain.net:4000/gerrit/

Optionally configure manager access for Tomcat. That's it!

OpenWRT and USB mounting

OpenWrt is a very nice open source router firmware. It runs Linux and is more like a distro than a typical router firmware. I have a Linksys WRT160NL with a USB port so I wanted to hook up a disk to have a SMB share. OpenWRT supports ext4 and this is the recommended filesystem.

To start install the packages required to access the USB and mount the filesystem. You can use uci command or LuCI web interface to configure the mount point.

Here comes the tricky part. The default priority order of init scripts starts usb after mounting fstab. This is a problem since mounting will fail. The results is hdd not mounting at startup but if you restart the fstab init script the mount works.

To overcome this problem simply change the priority of fstab init script:

cd /etc/init.d/
vi fstab

Change top line containing the START variable to:

START=59

Now just run:

./fstab disable
./fstab enable

This will setup correct symlinks in:

/etc/rc.d

Now the mounting works correctly on startup.

How to grep a binary file

Usually grep is used while working with text files. It can also be used with binary files. However a line or line number is probably note very useful in this case.

To get the byte offset you can use:

grep -boa "$PATTERN"

How to make a makefile

Sometimes a part of makefile needs to be dynamically generated. Instead of doing it prior to running make you can do a cool trick with GNU Make. You can just usually write the rule how to generate dynamic makefile and define dependencies and the recipe.

dynamic.mk: generate.sh config_file.xml
$< > $@

include dynamic.mk

A bit of explanation $< is the first prerequisite $@ is the target being made and the > is the normal bash redirection of standard output to file. So the second line is equivalent to:

generate.sh > dynamic.mk

When make encounters an include directive of a makefile it tires to look up the rules to update it as a target. It will remake any that are out of date or don't exist (issuing a warning for the non-existent ones).

You can define dependencies when the dynamic part of the makefile needs to be remade and when it is up to date. This way make itself ensures that makefiles are up to date.