Match http requests with responses on tshark

To do a tshark live capture for performance monitoring of http traffic it is nice to match http requests and responses by tcp streams. The command:

tshark -i lo -f 'tcp port 8000' -d 'tcp.port==8000,http' -Y http -a 'duration:600' -T fields -e frame.time_relative -e tcp.stream -e http.request.method -e http.request.full_uri -e http.time -e http.response.code -e http.content_length

The result is a nice view:

0.001100000 0 GET http://127.0.0.1:8000/admin/
0.097707000 0 0.096607000 200 3609
0.119656000 1 GET http://127.0.0.1:8000/static/grappelli/jquery/ui/jquery-ui.min.css
0.122288000 2 GET http://127.0.0.1:8000/static/grappelli/stylesheets/screen.css
0.123417000 3 GET http://127.0.0.1:8000/static/grappelli/stylesheets/mueller/grid/output.css
0.124670000 4 GET http://127.0.0.1:8000/static/grappelli/jquery/jquery.min.js
0.128406000 5 GET http://127.0.0.1:8000/static/grappelli/jquery/ui/jquery-ui.min.js
0.129479000 6 GET http://127.0.0.1:8000/static/grappelli/js/grappelli.js
0.136152000 1 0.016496000 304 0
0.136446000 7 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_collapsible.js
0.138702000 2 0.016414000 304 0
0.140253000 8 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_collapsible_group.js
0.143099000 3 0.019682000 304 0
0.143653000 9 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_timepicker.js
0.146036000 4 0.021366000 304 0
0.147956000 10 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_related_fk.js
0.150551000 5 0.022145000 304 0
0.150753000 11 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_related_m2m.js
0.153745000 6 0.024266000 304 0
0.153980000 12 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_related_generic.js
0.157603000 7 0.021157000 304 0
0.157806000 13 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_autocomplete_fk.js
0.161513000 8 0.021260000 304 0
0.163133000 14 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_autocomplete_m2m.js
0.164413000 9 0.020760000 304 0
0.164656000 15 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_autocomplete_generic.js
0.176289000 10 0.028333000 304 0
0.176572000 16 GET http://127.0.0.1:8000/static/grappelli/js/jquery.grp_inline.js
0.177160000 11 0.026407000 304 0
0.180085000 12 0.026105000 304 0
0.185553000 13 0.027747000 304 0
0.187315000 14 0.024182000 304 0
0.189586000 15 0.024930000 304 0
0.193586000 16 0.017014000 304 0
0.289933000 17 GET http://127.0.0.1:8000/static/grappelli/images/icons-small-scbbb475e49.png
0.294903000 17 0.004970000 304 0

The second number is a tcp stream number that allows us to match a request with the response. Additionally for responses we can see the time it took, the status code and length of the content. If you want to add more fields look at Wireshark documentation. You can also look directly at http field reference.

Using tshark it is easy to monitor http requests. To run a 10 minute http monitoring packet capture:

tshark -i lo -f 'tcp port 8000' -d 'tcp.port==8000,http' -Y http -a 'duration:600'

Just replace lo with the network interface you need and 8000 with the port you need. This will look like this example:

  4   0.010996    127.0.0.1 -> 127.0.0.1    HTTP 801 GET /execution/create HTTP/1.1
 14   0.388598    127.0.0.1 -> 127.0.0.1    HTTP 11039 HTTP/1.0 200 OK  (text/html)
 22   0.553410    127.0.0.1 -> 127.0.0.1    HTTP 849 GET /static/cloud_template/css/bootstrap.min.css HTTP/1.1
 27   0.566527    127.0.0.1 -> 127.0.0.1    HTTP 855 GET /static/cloud_template/css/metro-bootstrap.min.css HTTP/1.1
 35   0.569113    127.0.0.1 -> 127.0.0.1    HTTP 87 HTTP/1.0 304 Not Modified
 45   0.571528    127.0.0.1 -> 127.0.0.1    HTTP 840 GET /static/cloud_template/css/main.css HTTP/1.1
 53   0.576831    127.0.0.1 -> 127.0.0.1    HTTP 87 HTTP/1.0 304 Not Modified
 62   0.579262    127.0.0.1 -> 127.0.0.1    HTTP 87 HTTP/1.0 304 Not Modified

bind mount works on files

I thought bind mounts only worked on directories. Yesterday I learned that you can bind mount a regular file. Small detail but a precious gem in some situations. My situation was read-only root filesystem (that you can't remount rw without a reboot) and you really need to edit /etc/hosts.

Python and LDAP authentication

Authenticating to LDAP from Python using python-ldap often fails on authentication. It took me awhile to realize that using bind_s the first parameter is often not just the username but the whole ldap path to the user.

user = 'CN=surname name,OU=UsrAccounts,DC=company-intranet,DC=net'
pw = '***'

uri = 'ldap://server.company-intranet.net:389'

lc = ldap.initialize(uri, trace_level=0)
lc.set_option(ldap.OPT_REFERRALS, 0)
lc.set_option(ldap.OPT_PROTOCOL_VERSION, 3)
lc.bind_s(user, pw)

Negate a part of regular expression

Sometimes it is useful to negate a part of a regular expression. Not in all cases negating a group with the ^ character gets the job done.

Often negating a particular string is required. Technically what is usually needed then is called a negative lookahead.

When for example we want to match the string "AB" not followed by "CD" then the regular expression should be:

AB(?!CD)

The above will match "ABDC" but will not match "ABCD".

Make builds nice

Nice is an often forgotten but very useful part of coreutils. It adjusts the priority of the given command which in GNU/Linux (and UNIX) is called niceness. This name is quite convenient as it avoids the unnecessary confusion as to increase the priority the value has to be decreased. So a process with a high niceness value is "nice" to other processes.

The good thing about niceness is that it is inherited to child processes. So you just need to run your build script or make with nice:

nice -n 11 make -j8 all

All spawned processes (sub-make, compiler, linker etc.) get the same niceness value.

This can make a busy CI system responsive even under high load.

python's shortcuts

Let's try to optimize some simple python code like this:

a = ['eggs', 'bacon']

no_spam = True
for x in a:
    if x == 'spam':
        no_spam = False
if no_spam:
    print('there is no spam in it')

This might seem pretty obvious but an often forgotten fact is that a python's for loop may have an optional else clause.

So to make it slightly shorter and avoid using an additional variable:

a = ['eggs', 'bacon']

for x in a:
    if x == 'spam':
        break
else:
    print('there is no spam in it')

If you want to make if even shorter you can use the built-in functions any and map instead of the for loop:

a = ['eggs', 'bacon']

if not any(map(lambda x: True if x == 'spam' else False, a)):
    print('there is no spam in it')