TracingHTTPRequests

Tracing HTTP Requests can done using tcpflow.

It can also be done with tcpdump and awk.

Once upon a time, I wrote a short opinionated guide to awk , which showcased real-world applications of awk, of which TCP flow review is just one.

Since tcpdump -X will output the text, it just needs assembly via awk, eg. tcpdump -qln -nn -X -i em1 port 514 | awk -v re=some.string -f tcp-grep.awk Where tcp-grep.awk is the following:

#!/usr/bin/awk -f
BEGIN { if ( ! re ) { re = "." } }

/^\t/ {
    for (i = 2; i < NF ; i++ ) {
        bin = bin " " $i
    }
    txt = txt $NF;
    next;
}

/^[^\t]/ {
      if ( txt ~ re ) {
          print hdr, txt, binary ? bin : "";
      }
      hdr = $0;
      txt = "";
      bin = "";
} 

And following a tcp flow can be done with the following:

#!/usr/bin/awk -f
# add tcp follow-transaction to tcpdump
# tcpdump -t -nn -l -i eth0 -s 0 -x tcp port 25 |
# OFFSET is 52 for IPv4 + Timestamp Option + TCP
BEGIN {
  if (! TARGET) { TARGET = "10.0.2.26.25"; }
  if (! REGEXP) { REGEXP = "552 5.0.0 Headers too large"; }
  if (! DELIM ) { DELIM  = "##### TRANSACTION #####"; }
  if (! OFFSET) { OFFSET = 52; }
  FLAGS = "[S";
}

function unhex(hex,    c) {
  c = strtonum("0x" hex);
  return ( c == 0x0a || (c >= 0x20 && c < 0x7f) ) ? sprintf("%c", c) : " ";
}

function process(hold,    i, n) {
  if ( substr(FLAGS, 2, 1) == "S" ) { return; }
  if ( substr(FLAGS, 2, 1) == "R" ) { return; }
  if ( substr(FLAGS, 2, 1) == "F" ) {
    if ( ! (KEY in pkts) ) { return; }
    if ( match( pkts[KEY], REGEXP ) ) {
      printf("%s\n%s\n", DELIM, pkts[KEY]);
    }
    delete pkts[KEY];
    return;
  }
  gsub(/[a-f0-9][a-f0-9]/, "& ", hold);
  n = split(hold, fields);
  for (i=OFFSET+1; i <= n; i++) {
    pkts[KEY] = pkts[KEY] unhex(fields[i]);
  }
}

/^IP/ {
  process(hold); hold = "";
  gsub(/:/, "", $0);
  src = $2 ; dst = $4; FLAGS = $6;
  KEY = (src == TARGET) ? dst : src;
}
$1 ~ /0x/ {
  $1 = "";
  hold = hold " " $0;
}