Match http requests with responses on tshark

To do a tshark live capture for performance monitoring of http traffic it is nice to match http requests and responses by tcp streams. The command:

tshark -i lo -f 'tcp port 8000' -d 'tcp.port==8000,http' -Y http -a 'duration:600' -T fields -e frame.time_relative -e tcp.stream -e http.request.method -e http.request.full_uri -e http.time -e http.response.code -e http.content_length

The result is a nice view:

0.001100000 0 GET http://127.0.0.1:8000/admin/
0.097707000 0 0.096607000 200 3609
0.119656000 1 GET http://127.0.0.1:8000/static/grappelli/jquery/ui/jquery-ui.min.css
0.122288000 2 GET http://127.0.0.1:8000/static/grappelli/stylesheets/screen.css
0.123417000 3 GET http://127.0.0.1:8000/static/grappelli/stylesheets/mueller/grid/output.css
0.124670000 4 GET http://127.0.0.1:8000/static/grappelli/jquery/jquery.min.js
0.128406000 5 GET http://127.0.0.1:8000/static/grappelli/jquery/ui/jquery-ui.min.js
0.129479000 6 GET http://127.0.0.1:8000/static/grappelli/js/grappelli.js
0.136152000 1 0.016496000 304 0
0.136446000 7 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_collapsible.js
0.138702000 2 0.016414000 304 0
0.140253000 8 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_collapsible_group.js
0.143099000 3 0.019682000 304 0
0.143653000 9 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_timepicker.js
0.146036000 4 0.021366000 304 0
0.147956000 10 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_related_fk.js
0.150551000 5 0.022145000 304 0
0.150753000 11 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_related_m2m.js
0.153745000 6 0.024266000 304 0
0.153980000 12 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_related_generic.js
0.157603000 7 0.021157000 304 0
0.157806000 13 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_autocomplete_fk.js
0.161513000 8 0.021260000 304 0
0.163133000 14 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_autocomplete_m2m.js
0.164413000 9 0.020760000 304 0
0.164656000 15 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_autocomplete_generic.js
0.176289000 10 0.028333000 304 0
0.176572000 16 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_inline.js
0.177160000 11 0.026407000 304 0
0.180085000 12 0.026105000 304 0
0.185553000 13 0.027747000 304 0
0.187315000 14 0.024182000 304 0
0.189586000 15 0.024930000 304 0
0.193586000 16 0.017014000 304 0
0.289933000 17 GET http://127.0.0.1:8000/static/grappelli/images/icons-small-scbbb475e49.png
0.294903000 17 0.004970000 304 0

The second number is a tcp stream number that allows us to match a request with the response. Additionally for responses we can see the time it took, the status code and length of the content. If you want to add more fields look at Wireshark documentation. You can also look directly at http field reference.

Using tshark it is easy to monitor http requests. To run a 10 minute http monitoring packet capture:

tshark -i lo -f 'tcp port 8000' -d 'tcp.port==8000,http' -Y http -a 'duration:600'

Just replace lo with the network interface you need and 8000 with the port you need. This will look like this example:

  4   0.010996    127.0.0.1 -> 127.0.0.1    HTTP 801 GET /execution/create HTTP/1.1
 14   0.388598    127.0.0.1 -> 127.0.0.1    HTTP 11039 HTTP/1.0 200 OK  (text/html)
 22   0.553410    127.0.0.1 -> 127.0.0.1    HTTP 849 GET /static/cloud_template/css/bootstrap.min.css HTTP/1.1
 27   0.566527    127.0.0.1 -> 127.0.0.1    HTTP 855 GET /static/cloud_template/css/metro-bootstrap.min.css HTTP/1.1
 35   0.569113    127.0.0.1 -> 127.0.0.1    HTTP 87 HTTP/1.0 304 Not Modified
 45   0.571528    127.0.0.1 -> 127.0.0.1    HTTP 840 GET /static/cloud_template/css/main.css HTTP/1.1
 53   0.576831    127.0.0.1 -> 127.0.0.1    HTTP 87 HTTP/1.0 304 Not Modified
 62   0.579262    127.0.0.1 -> 127.0.0.1    HTTP 87 HTTP/1.0 304 Not Modified

bind mount works on files

I thought bind mounts only worked on directories. Yesterday I learned that you can bind mount a regular file. Small detail but a precious gem in some situations. My situation was read-only root filesystem (that you can't remount rw without a reboot) and you really need to edit /etc/hosts.

Python and LDAP authentication

Authenticating to LDAP from Python using python-ldap often fails on authentication. It took me awhile to realize that using bind_s the first parameter is often not just the username but the whole ldap path to the user.

user = 'CN=surname name,OU=UsrAccounts,DC=company-intranet,DC=net'
pw = '***'

uri = 'ldap://server.company-intranet.net:389'

lc = ldap.initialize(uri, trace_level=0)
lc.set_option(ldap.OPT_REFERRALS, 0)
lc.set_option(ldap.OPT_PROTOCOL_VERSION, 3)
lc.bind_s(user, pw)

Negate a part of regular expression

Sometimes it is useful to negate a part of a regular expression. Not in all cases negating a group with the ^ character gets the job done.

Often negating a particular string is required. Technically what is usually needed then is called a negative lookahead.

When for example we want to match the string "AB" not followed by "CD" then the regular expression should be:

AB(?!CD)

The above will match "ABDC" but will not match "ABCD".

Make builds nice

Nice is an often forgotten but very useful part of coreutils. It adjusts the priority of the given command which in GNU/Linux (and UNIX) is called niceness. This name is quite convenient as it avoids the unnecessary confusion as to increase the priority the value has to be decreased. So a process with a high niceness value is "nice" to other processes.

The good thing about niceness is that it is inherited to child processes. So you just need to run your build script or make with nice:

nice -n 11 make -j8 all

All spawned processes (sub-make, compiler, linker etc.) get the same niceness value.

This can make a busy CI system responsive even under high load.

python's shortcuts

Let's try to optimize some simple python code like this:

a = ['eggs', 'bacon']

no_spam = True
for x in a:
    if x == 'spam':
        no_spam = False
if no_spam:
    print('there is no spam in it')

This might seem pretty obvious but an often forgotten fact is that a python's for loop may have an optional else clause.

So to make it slightly shorter and avoid using an additional variable:

a = ['eggs', 'bacon']

for x in a:
    if x == 'spam':
        break
else:
    print('there is no spam in it')

If you want to make if even shorter you can use the built-in functions any and map instead of the for loop:

a = ['eggs', 'bacon']

if not any(map(lambda x: True if x == 'spam' else False, a)):
    print('there is no spam in it')

Howto deploy Gerrit and Jenkins on Tomat

First get Tomcat, Gerrit and Jenkins. The preferred way is to use your distribution's package manager. If you don't have access to the root account or require a newer version you can grab the software from the respective project's websites:

wget http://www.nic.funet.fi/pub/mirrors/apache.org/tomcat/tomcat-7/v7.0.39/bin/apache-tomcat-7.0.39.tar.gz
wget http://gerrit.googlecode.com/files/gerrit-2.6-rc0.war
wget http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war

Unpack tomcat and place the war files in webapps dir:

tar -xf apache-tomcat-7.0.39.tar.gz
cp jenkins.war apache-tomcat-7.0.39/webapps/
cp gerrit-2.6-rc0.war apache-tomcat-7.0.39/webapps/gerrit.war

Setup the gerrit site:

java -jar gerrit-2.6-rc0.war init -d /install_dir/gerrit

*** Gerrit Code Review 2.6-rc0
***

Create '/install_dir/gerrit' [Y/n]?

*** Git Repositories
***

Location of Git repositories   [git]:

*** SQL Database
***

Database server type           [h2]:

*** User Authentication
***

Authentication method          [OPENID/?]: ?
       Supported options are:
         openid
         openid_sso
         http
         http_ldap
         client_ssl_cert_ldap
         ldap
         ldap_bind
         custom_extension
         development_become_any_account
Authentication method          [OPENID/?]: ldap
LDAP server                    [ldap://localhost]: ldap://ldap.server.net
LDAP username                  :
Account BaseDN                 [DC=ldap,DC=server,DC=net]: o=XXX
Group BaseDN                   [o=XXX]:

*** Email Delivery
***

SMTP server hostname           [localhost]:
SMTP server port               [(default)]:
SMTP encryption                [NONE/?]:
SMTP username                  :

*** Container Process
***

Run as                         [ute]:
Java runtime                   [/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre]:
Copy gerrit.war to /install_dir/gerrit/bin/gerrit.war [Y/n]? n

*** SSH Daemon
***

Listen on address              [*]:
Listen on port                 [29418]:

Gerrit Code Review is not shipped with Bouncy Castle Crypto v144
  If available, Gerrit can take advantage of features
  in the library, but will also function without it.
Download and install it now [Y/n]? Y
Downloading http://www.bouncycastle.org/download/bcprov-jdk16-144.jar ... OK
Checksum bcprov-jdk16-144.jar OK
Generating SSH host key ... rsa... dsa... done

*** HTTP Daemon
***

Behind reverse proxy           [y/N]?
Use SSL (https://)             [y/N]?
Listen on address              [*]:
Listen on port                 [8080]: 7003
Canonical URL                  [http://your.host.domain.net:7003/]: http://your.host.domain.net:4000/gerrit/

*** Plugins
***

Prompt to install core plugins [y/N]?

Initialized /install_dir/gerrit

Stop the built-in Gerrit server as tomcat will be used instead:

/install_dir/gerrit/bin/gerrit.sh stop

Now copy the jar files required for gerrit to start on tomcat:

cp gerrit/lib/bcprov-jdk16-144.jar apache-tomcat-7.0.39/lib/
java -jar apache-tomcat-7.0.39/webapps/gerrit.war cat lib/h2-1.3.168.jar >apache-tomcat-7.0.39/lib/h2-1.3.168.jar

Optionally change the ports:

sed -i 's/port="8080"/port="4000"/' apache-tomcat-7.0.39/conf/server.xml
sed -i 's/port="8443"/port="4443"/' apache-tomcat-7.0.39/conf/server.xml
sed -i 's/port="8009"/port="4009"/' apache-tomcat-7.0.39/conf/server.xml
sed -i 's/port="8005"/port="4005"/' apache-tomcat-7.0.39/conf/server.xml

To enable gerrit to access it's H2 database edit context.xml:

vi apache-tomcat-7.0.39/conf/context.xml

<Resource
    name="jdbc/ReviewDb"
    type="javax.sql.DataSource"
    username=""
    driverClassName="org.h2.Driver"
    password=""
    url="jdbc:h2:file:/install_dir/gerrit/db/ReviewDB"
    maxActive="100"
    maxIdle="20"/>
</Context>

Now create a starter script:

vi start.sh

#!/bin/sh
export CATALINA_HOME="/install_dir/apache-tomcat-7.0.39"
export CATALINA_PID="$CATALINA_HOME/tomcat.pid"
export CATALINA_OPTS="-DGERRIT_SITE=/install_dir/gerrit/ -DJENKINS_HOME=/install_dir/jenkins/ -Xmx1024m"
export JAVA_OPTS="-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:PermSize=256m -XX:MaxPermSize=512m"
cd $CATALINA_HOME/bin && ./startup.sh

Finally execute start.sh:

chmod +x start.sh
./start.sh

Now configure Jenkins using it's administration UI:

http://your.host.domain.net:4000/jenkins/

Setup gerrit permissions by logging in:

http://your.host.domain.net:4000/gerrit/

Optionally configure manager access for Tomcat. That's it!

OpenWRT and USB mounting

OpenWrt is a very nice open source router firmware. It runs Linux and is more like a distro than a typical router firmware. I have a Linksys WRT160NL with a USB port so I wanted to hook up a disk to have a SMB share. OpenWRT supports ext4 and this is the recommended filesystem.

To start install the packages required to access the USB and mount the filesystem. You can use uci command or LuCI web interface to configure the mount point.

Here comes the tricky part. The default priority order of init scripts starts usb after mounting fstab. This is a problem since mounting will fail. The results is hdd not mounting at startup but if you restart the fstab init script the mount works.

To overcome this problem simply change the priority of fstab init script:

cd /etc/init.d/
vi fstab

Change top line containing the START variable to:

START=59

Now just run:

./fstab disable
./fstab enable

This will setup correct symlinks in:

/etc/rc.d

Now the mounting works correctly on startup.

How to grep a binary file

Usually grep is used while working with text files. It can also be used with binary files. However a line or line number is probably note very useful in this case.

To get the byte offset you can use:

grep -boa "$PATTERN"

How to make a makefile

Sometimes a part of makefile needs to be dynamically generated. Instead of doing it prior to running make you can do a cool trick with GNU Make. You can just usually write the rule how to generate dynamic makefile and define dependencies and the recipe.

dynamic.mk: generate.sh config_file.xml
$< > $@

include dynamic.mk

A bit of explanation $< is the first prerequisite $@ is the target being made and the > is the normal bash redirection of standard output to file. So the second line is equivalent to:

generate.sh > dynamic.mk

When make encounters an include directive of a makefile it tires to look up the rules to update it as a target. It will remake any that are out of date or don't exist (issuing a warning for the non-existent ones).

You can define dependencies when the dynamic part of the makefile needs to be remade and when it is up to date. This way make itself ensures that makefiles are up to date.

How to mount an UBI image

UBI and UBIFS do not work on top of block devices. So simple loop-mount will not work.

UBI is an abstraction layer that works on top of MTD raw flash devices. It's worth to note that raw flash is not the typical pen-drive, memory card or SSD but it is a flash chip without a FTL. UBI is usually it's used in embedded devices.

UBI is usually used as a supporting layer for the UBIFS filesystem. It's a successor to JJFS2 which does not use a supporting layer like UBI but works directly on MTD devices. UBIFS mounting works across these layers:

/mnt/fs (mount point) => /dev/ubi0_0 (UBIFS) => /dev/ubi0 (UBI) => /dev/mtd0 (MTD raw flash device)

TODO: make a simple diagram here

UBIFS images are created by mkfs.ubifs. UBI images are created by the ubinize utility taking UBIFS images as input. Note that an UBI image may contain multiple UBIFS filesystems.

For further information about UBI and UBIFS check the documentation and FAQ.

To mount an UBI image an MTD device is needed. If you have it on your system you can use it to flash the UBI image to the device. If you want to do this on a regular computer you need to emulate it. One way to do it is to use the mtdram linux kernel module. You need root to do it like this:

modprobe mtdram total_size=X

X is the size in KB of the emulated MTD device. Make sure it is larger than the image you wish to mount. To check the status of MTD devices you can use:

cat /proc/mtd

You should see a mtd0 device. Now you need to prepare it and flash the UBI image into it:

flash_erase /dev/mtd0 0 0

ubiformat /dev/mtd0 -f image.ubi

Now the MTD device contains the contents of the UBI image. Now you need to tell the UBI module to use it:

modprobe ubi

ubiattach -p /dev/mtd0

This will create UBI device (/dev/ubi0) and  the UBIFS (/dev/ubi0_0) partitions. Now just mount it:

mount -t ubifs /dev/ubi0_0 /mnt/ubifs

You could also use nandsim but then you would have to know the bytes of READ ID response of the particular flash chip and pass these during modprobe.

Build a module on Fedora

If you need to build a kernel module on Fedora you need the kernel-devel rpm. Don't forget you also need gcc :)