flstats.1

.\" for help on these macros, the mdoc package, see:
.\" http://mdocml.bsd.lv/man/mdoc.7.html
.Dd March 15, 1996
.Dt FLSTATS 1
.Os BSD 4.3
.Sh NAME
.Nm flstats
.Nd extract flow statistics from a tcpdump -w file
.Sh SYNOPSIS
.Nm flstats
.Op Fl cdfis
.Op Fl HILT
.Op Fl -binsecs Ar num
.Op Fl -evaluate Ar tclcommands
.Op Fl -flowtypes Ar flowspecifier[s]
.Op Fl -oexcl Ar {class|flow|ri} fieldnames
.Op Fl -omodify Ar {class|flow|ri} outputspecifier
.Op Fl -ospec Ar {class|flow|ri} outputspecifier
.Op Fl -scriptfile Ar filename
.Op Fl -sep Ar separator
.Op Fl -timebase Ar {T|t}{T|t|r}
.Op Ar filename
.Sh DESCRIPTION
.Nm Flstats
examines a binary
file created by
.Xr tcpdump  1
(using tcpdump's
.Fl w
option),
and reports statistics on "flows" (for various definitions
of flows) contained in the file.
.Pp
.Bl -tag -width Ic
.It Fl c
Report statistics of flow "classes" (aggregates of flows), rather than
about (or, in addition to) individual flows themselves.  The
difference might be that different individual flows have different
source and destination IP addresses, whereas the class for all these
flows is the class of flows which looks at (only) source and
destination IP addresses.  (This would be the case if
.Fl -flowtypes Ar dst/src
were specified on the command line -- there would be one class that
held its own statistics for all packets with source and destination
[IP] addresses in them.)
.It Fl d
Run in a mode for debugging flstats itself (normally not interesting
for users of
.Nm flstats
).
.It Fl f
Report statistics about individual flows.
.It Fl i
Run in "interactive" mode, in which the user is prompted for Tcl
commands to execute.  See usage note under

Usage note: only *one* of
.Fl c ,
.Fl f ,
and
.Fl i
can be specified in any given run.
.Fl c .
.It Fl s
Set the process to react to USR1 signals.  After the reception of a USR1
signal the current reporting interval will be terminated and a report
for that interval printed out.  The process will then resume counting
in a new reporting interval.
.It Fl H
Print out a header with the names of the fields that will be printed
in each "column".
.It Fl I
Print reporting interval statistics on a line by themselves before
printing out the class or flow statistics (selected by the
.FL c
or
.FL f
flags).  The class or flow statistics will be prefixed by some number
of blank spaces.
.It Fl L
Causes
.Nm flstats
to print out lines containing information about the input file and the
environment of the execution (for repeatability).  these lines are
prefixed with a '#' (hash mark).
.Fl H
flag) that describe the environment in which the trace is being run.
(This is done to help with repeatability.)
.It Fl T
When printing summary statistics (at the end of a reporting interval),
print the field name before printing the value.
.It Fl -binsecs Ar num
How many seconds (in the time sequence of the input file) should each
reporting interval last.  The default is zero (0), which causes the
statistics to be reported only once, at the end of the input file
(unless USR1 signals are caught by use of the
.Fl s
flag).
.It Fl -evaluate Ar tclcommands
.Ar Tclcommands
is a list of Tcl commands to be executed while starting
.Nm flstats .
.It Fl -flowtypes Ar flowspecifier[s]
A list of one or more flow specifiers.  While the format of a flow
specifier can, in general, be fairly complicated (see the
.Nm flstats
user's manual which, sadly, does not exist), for the purpose of this
man page a flow specifier is a list of one or more IP header field
names (plus the port fields from TCP and/or UDP headers) separated by
slashes ("/").  The header fields supported are named: ihv, ihl, tos,
len, id, df, mf, foff, ttl, prot, sum, src, dst, sport, dport.
.It Fl -oexcl Ar {class|flow|ri} fieldnames
This allows the deletion of fields from the summary statistics reported
at the end of each reporting interval.  This is not as general a
mechanism as is
.Fl -ospec .
.Ar class ,
.Ar flow ,
and
.Ar ri
specify which of the sets of statistics the
.Ar fieldnames
parameters apply.
.Ar Ri
applies to those statistics related to the specific reporting
interval.  This option can be repeated multiple times (for each of
.Ar class ,
.Ar flow ,
and
.Ar ri ,
say).  The fieldnames can be separated by a space (" ") or, to keep
from having to type quotation marks, by commas (",")

There is a default list of fields that are excluded (currently just
from the reporting interval --
.Ar ri
-- set of statistics).  To see *all* values, '--oexcl class "" --oexcl
flow "" --oexcl ri ""' can be specified
.It Fl -omodify Ar {class|flow|ri} outputspecifier
This output format option doesn't change the order in which data is
presented, nor does it allow any of the data to be deleted (you need
to specify the
.Fl -oexcl
option for these).  However, the
.Fl -omodify
*does* allow one to change the label printed for a given statistic, as
well as allowing "::int" (see the .Fl -oexcl option) to be added to
the print directive for one or more statistics that are to be printed.
See the
.Fl -oexcl
for more details on the contents of
.Ar outputspecifier .
In the case of
.Fl -omodify
only the elements of
.Ar outputspecifier
that match a tag of the statistics of the specified type to be printed
make a change in the printed output.
.It Fl -ospec Ar {class|flow|ri} outputspecifier
This allows finer control of what is printed out than do the
.Fl -oexcl
and
.Fl -omodify
options.  For a given set of statistics (specified, as in the
.Fl -oexcl
option, by
.Ar class ,
.Ar flow ,
and
.Ar ri )
the
.Ar outputspecifier
is a set of triples separated by colon (":"):
.Ar [fieldname][:[label][:int]
where
.Ar fieldname
is an output fieldname from the selected group of statistics;
.Ar label
is the label to be printed instead of
.Ar fieldname
(if missing, the default is
.Ar fieldname );
and
.Ar int
is the literal string "int".  The action of the
.Fl -ospec
is to cause only the listed fields of the specific group of statistics
(again, specified by the
.Ar class ,
.Ar flow ,
and
.Ar ri
parameter) to be printed.  Each such field is printed; if
.Ar int
for a field, it means the field (presumably a floating point number)
should be rounded (to the nearest integer) before being printed.

A special case is when no
.Ar fieldname
is specified (so, the spec looks like
.Ar :label[:int] ),
in which case the literal string of the
.Ar label
parameter is printed out at that point on the output line.  In this
special case, if
.Ar :int
is printed in this case, no separator is printed between this label
and the next printed field.

As with the
.Fl -oexcl
option, the specifiers can be separated from each other by a space or
by a comma.

(Note that whereas
.Fl -oexcl
does *not* change the *order* of fields printed, only excludes some of
those fields, the
.Fl -ospec
parameter *does* change to order of fields printed to the order in
.Ar outputspecifier .)
.It Fl -scriptfile Ar filename
The name of a file to be passed to
.Nm flstats'
internal Tcl interpreter during startup.
.It Fl -sep Ar separator
Specify the
.Ar separator
used to separate values in the summary statistics.  The
.Ar separator
is *not* printed before the first, or after that last, statistic on
the line.
.It Fl -timebase Ar {T|t}{T|t|r}
The default (which can be specified by specifying
.Ar TT )
is to print all times in the summary statistics as
secs.usecs from the Unix epoch (01.01.1970).  The
.Fl -timebase
option allows re-basing the printed times.  The time of the beginning
of reporting intervals, as well as the time the last packet in a given
class was received, can be printed relative to the beginning of the
trace by specifying a
.Ar t
as the first character of this option.

The rest of the times printed in the trace can also be specified to be
relative to the start of the trace by specifying
.Ar t
as the second character of this option.  Alternatively, the rest of
the times can be printed relative to the start of the
currently-being-reported reporting interval by specfying
.Ar r
as the second character of this option.
.Sh EXAMPLES
To see how many source/destination pairs there were in a trace, along
with the number of packets, bytes, and other information about the data
flowing between them:
.Bd -literal -offset indent
% flstats -T -c --flowtypes src/dst --timebase tr file.tcpd
binno 0 ri_start 0.000000 ri_end 15.762258 ri_firstpkt 0.002680 ri_lastpkt 15.559384 ri_pkts 507 ri_bytes 355444 ri_tsipg 0.105973 ri_isipg 0.105973 ignorepkts 24 ignorebytes 1132 unclpkts 0 unclbytes 0 type src/dst created 18 deleted 0 added 0 removed 0 active 18 pkts 507 bytes 355444 sipg 0.105973 lastrecv 15.559384
.Ed
.Pp
The above says that in bin number 0 (the only bin) there were 18
unique source/destination pairs which together exchanged 507 packets
totalling 355,444 bytes.  The smoothed interpacket arrival time was
105,973 microseconds.  The "ignored" packets were non-IP packets
(often, but not always, ARP packets).
.Pp
To actually list out the source/destination pairs in the trace:
.Bd -literal -offset indent
% flstats -TI -f --flowtypes src/dst --timebase tr file.tcpd
binno 0 ri_start 0.000000 ri_end 15.762258 ri_firstpkt 0.002680 ri_lastpkt 15.559384 ri_pkts 507 ri_bytes 355444 ri_tsipg 0.105973 ri_isipg 0.105973 ignorepkts 24 ignorebytes 1132 unclpkts 0 unclbytes 0
    class 1 type src/dst id 192.168.2.184/99.19.69.197 pkts 4 bytes 256 sipg 0.455354 created 0.002680 last 4.004762
    class 1 type src/dst id 192.168.2.176/239.255.255.250 pkts 10 bytes 3342 sipg 0.768854 created 2.730328 last 13.231953
    class 1 type src/dst id 192.168.2.176/255.255.255.255 pkts 10 bytes 3342 sipg 0.768861 created 2.730601 last 13.232308
    class 1 type src/dst id 192.168.2.184/192.168.2.255 pkts 2 bytes 88 sipg 0.887533 created 3.038121 last 10.138392
    class 1 type src/dst id 192.168.2.184/224.0.0.1 pkts 2 bytes 88 sipg 0.887542 created 3.038132 last 10.138474
    class 1 type src/dst id 192.168.2.83/192.168.2.184 pkts 2 bytes 112 sipg 0.889530 created 3.043029 last 10.159276
    class 1 type src/dst id 192.168.2.184/208.80.154.224 pkts 48 bytes 7228 sipg 0.280332 created 5.719181 last 15.017931
    class 1 type src/dst id 208.80.154.224/192.168.2.184 pkts 52 bytes 43405 sipg 0.313053 created 5.883181 last 15.245128
    class 1 type src/dst id 192.168.2.184/208.80.154.240 pkts 143 bytes 13047 sipg 0.003487 created 6.030878 last 13.286928
    class 1 type src/dst id 192.168.2.83/239.255.255.250 pkts 26 bytes 8878 sipg 0.308499 created 6.163833 last 15.559384
    class 1 type src/dst id 208.80.154.240/192.168.2.184 pkts 197 bytes 274037 sipg 0.016148 created 6.194215 last 13.398893
    class 1 type src/dst id 192.168.2.184/192.168.2.1 pkts 1 bytes 64 sipg 0.000000 created 6.454218 last 6.454218
    class 1 type src/dst id 192.168.2.1/192.168.2.184 pkts 1 bytes 182 sipg 0.000000 created 6.464751 last 6.464751
    class 1 type src/dst id 192.168.2.184/216.17.8.52 pkts 2 bytes 112 sipg 0.000000 created 7.218210 last 7.218213
    class 1 type src/dst id 216.17.8.52/192.168.2.184 pkts 2 bytes 92 sipg 0.004998 created 7.385571 last 7.425555
    class 1 type src/dst id 192.168.2.94/224.0.0.251 pkts 1 bytes 180 sipg 0.000000 created 9.009267 last 9.009267
    class 1 type src/dst id 108.160.163.109/192.168.2.184 pkts 2 bytes 429 sipg 0.027229 created 10.857595 last 11.075427
    class 1 type src/dst id 192.168.2.184/108.160.163.109 pkts 2 bytes 562 sipg 0.000313 created 10.857668 last 10.860177
.Ed
.Pp
Here, we've specified
.Ar I
in order to make the output a bit easier for we humans to understand.
The above output tells us that source 192.168.2.184 sent 143 packets,
totalling 13047 bytes, to destination 208.80.154.240, as well as
information about 17 other sources (including information about source
208.80.154.240 sending to destination 192.168.2.184).
.Pp
To see how many different TOS values there are in the trace:
.Bd -literal -offset indent
% flstats -HI -c --flowtypes tos --timebase tr file.tcpd
binno ri_start ri_end ri_firstpkt ri_lastpkt ri_pkts ri_bytes ri_tsipg ri_isipg ignorepkts ignorebytes unclpkts unclbytes
    type created deleted added removed active pkts bytes sipg lastrecv
0 0.000000 15.762258 0.002680 15.559384 507 355444 0.105973 0.105973 24 1132 0 0
    tos 1 0 0 0 1 507 355444 0.105973 15.559384
.Ed
.Pp
Here, we've changed the
.Fl T
to
.Fl H
to print out the column labels once, at the beginning of the trace.
This is a bit harder to read, but more compact.  The above output
tells us same tos field was set in all the processed packet.  To see
which value this was:
.Bd -literal -offset indent
% flstats -HI -f --flowtypes tos --ospec flow id --timebase tr file.tcpd
binno ri_start ri_end ri_firstpkt ri_lastpkt ri_pkts ri_bytes ri_tsipg ri_isipg ignorepkts ignorebytes unclpkts unclbytes
    id
0 0.000000 15.762258 0.002680 15.559384 507 355444 0.105973 0.105973 24 1132 0 0
    0
.Ed
.Pp
Here we've used the
.Fl -ospec
option to change the reporting for
.Ar flow
statistics to *only* print out the
.Ar id
field.  We see that the tos value used in all the packets in this
trace was the value 0.
.Pp
What about packet size distribution in the trace file?  First, to see
how many different packet sizes are in the file:
.Bd -literal -offset indent
% flstats -HI -c --ospec class active --flowtypes len --timebase tr file.tcpd
binno ri_start ri_end ri_firstpkt ri_lastpkt ri_pkts ri_bytes ri_tsipg ri_isipg ignorepkts ignorebytes unclpkts unclbytes
    active
0 0.000000 15.762258 0.002680 15.559384 507 355444 0.105973 0.105973 24 1132 0 0
    67
.Ed
.Pp
So, there are 67 distinct lengths in the file.  To see the actual
lengths:
.Bd -literal -offset indent
% flstats -HI -f --flowtypes len --timebase tr file.tcpd
binno ri_start ri_end ri_firstpkt ri_lastpkt ri_pkts ri_bytes ri_tsipg ri_isipg ignorepkts ignorebytes unclpkts unclbytes
    class type id pkts bytes sipg created last
0 0.000000 15.762258 0.002680 15.559384 507 355444 0.105973 0.105973 24 1132 0 0
    1 len 64 11 704 1.059213 0.002680 15.245128
    1 len 407 4 1628 1.066227 2.730328 12.478685
    1 len 460 4 1840 1.066230 2.981189 12.729570
    1 len 44 4 176 0.776602 3.038121 10.138474
    1 len 56 4 224 0.767211 3.043029 10.159276
    1 len 410 4 1640 1.066216 3.232132 12.980405
    1 len 208 4 832 1.066237 3.482898 13.231335
    1 len 186 4 744 1.066270 3.483570 13.232308
    1 len 1005 1 1005 0.000000 5.719181 5.719181
    1 len 52 178 9256 0.017057 5.883181 13.398893
    1 len 1381 12 16572 0.293116 6.013681 12.711069
    1 len 1492 188 280496 0.002739 6.016352 13.286427
    1 len 947 1 947 0.000000 6.019279 6.019279
    1 len 620 1 620 0.000000 6.030878 6.030878
    1 len 1403 2 2806 0.833673 6.038051 12.707436
    1 len 676 1 676 0.000000 6.040259 6.040259
    1 len 296 4 1184 0.933236 6.163833 14.632869
    1 len 774 1 774 0.000000 6.196387 6.196387
    1 len 1281 1 1281 0.000000 6.207290 6.207290
    1 len 233 1 233 0.000000 6.261779 6.261779
    1 len 182 1 182 0.000000 6.464751 6.464751
    1 len 593 1 593 0.000000 6.481737 6.481737
    1 len 60 1 60 0.000000 6.629571 6.629571
    1 len 276 1 276 0.000000 6.629852 6.629852
    1 len 242 1 242 0.000000 6.656202 6.656202
 ...
.Ed
.Pp

This also shows the interpacket arrival time (smoothed) between
packets of the same length.  To see only the lengths, packet counts
and byte counts, we could specify
.Fl -ospec Ar flow id,pkts,bytes
.
.Sh BUGS
.Nm flstats
does not support IPv6.

Specifying output formats for
.Nm flstats
is a bit tricky.
.Sh SEE ALSO
.Xr tclsh 1 ,
.Xr tcpdump 1 ,
.Xr pcap 3 ,
.Sh HISTORY
The
.Nm flstats
command was written at Ipsilon Networks in 1996.