Home

Finding the top bandwidth-users on an LTSP server

Tue, 2008.09.02 - 16:14 — müzso
To cut it short, use the nethogs utility. You'll see an output like this:
NetHogs version 0.6.0
  PID USER     PROGRAM                      DEV        SENT      RECEIVED       
19276 user1    rdesktop                     eth0       6.673      71.440 KB/sec
31777 root     sshd: user3 [priv]           eth0    1644.454      57.780 KB/sec
16792 root     sshd: user1 [priv]           eth0    1330.156      52.727 KB/sec
25846 root     sshd: user2 [priv]           eth0     767.100      32.171 KB/sec
(...)

  TOTAL                                             3751.003     215.619 KB/sec
Nethogs is not the perfect tool despite of its ease of use. Read more for some further tips.

Nethogs is a bit too simple. It lacks any sort of control. The version I saw in Ubuntu Hardy had fixed sorting: the top processes were selected based on incoming traffic. On an LTSP server the top network usage will come from the sshd processes that forward the X protocol to the thin clients. This is an outgoing traffic (at least from the perspective of the LTSP server), thus nethogs might not even show our top sshd process (if there're many processes using high incoming bandwidth). You might think that running nethogs on the loopback interface (lo) might still give you the expected result, but you're wrong. Unfortunately nethogs shows zero incoming traffic for all processes in this case. Sad

A more suitable approach is to use a combination of tools. You can use tcptrack to find the real bandwidth-killer. Unfortunately tcptrack has no ascii output (only curses), so it'd be a pain in the ass to get the top dog out of it. But your eyes are just made for this kind of a job. Wink So start tcptrack, get the IP+port of the top bandwidth user connection and search for it in netstat's output. Eg.
$ netstat -tnp | fgrep '192.168.0.45:45834'
tcp6       0      0 192.168.0.101:22        192.168.0.45:35263      ESTABLISHED 31777/sshd: user3 [p

This tells you that the sshd (with PID=31777) of user3 is the problem, so probably you should ask him to do something about it. Smile

If you're interested in the specific application that sends that much data to the client, then start tcptrack for the loopback interface (tcptrack -i lo) and look for that IP+port combo in netstat's output.

You can use iftop too (instead of tcptrack) and this one gets you a more precise answer since tcptrack shows only TCP connections, while iftop's output is based on all IP packets (at least by default). Try this:
iftop -nNPBi eth0
Hit the "t" key 3 times to switch the display to the output-only mode. Now you can see the IP-connections generating the most (outgoing) traffic. Looking up the process with netstat gives you the user and the app. A simple ps uwwp <pid> will tell you the full command line that was used to invoke the given process.