Flash Media Server (FMS) monitoring with Munin

Munin is a great open source monitoring program available for a number of platforms. Flash Media Server (FMS) is a streaming server from Adobe aiming mainly at video playback in Flash applications (video players, video conference solutions, etc.). There was already a Munin plugin for FMS (monitoring the number of active connections) at MuninExchange, but the config parameters were hard-coded (in the plugin code) and it lacked documentation on usage and did not match the structure of standard Munin plugins. I've rewritten the plugin to come up for all these shortcomings. The new version of the plugin is available both attached to this post and at MuninExchange.

However after looking at the graphs of the plugin it became quite obvious that the numbers returned by the plugin do not reflect what I (and probably many others) was looking for. I want to see the number of active streaming connections and the "connected" value returned by the getServerStats() API call is just not that (if anybody knows what that "connected" value reflects, please, share with me). I've looked at the various get*Stats() calls and found that getAppStats() and getVHostStats() return the real "connected" value that I was after. I've created two new plugins based on the results of getAppStats():
  • fms_apps: this plugin shows the number of active streaming connections grouped by applications (ie. the graph shows a line for each application on the server)
  • fms_apps_rate: this plugin shows the number of new connections per minute grouped by applications.
Update! (Feb 16, 2009)
I've created yet another plugin (fms_playing) in the hope that this time it'll display the "real thing". Smiling It's based on the getActiveInstances() and getNetStreamStats() calls of the FMS Admin Server API.


Update! (July 10, 2009)
After months of testing I came to the conlusion that the stats in FMS 3.0.x are completely flawed. However they work OK in FMS 3.5.x. So I strongly advise to use these Munin plugins with FMS 3.5.x or later.


P.S.: I can count the number of my Perl scripts on one hand so don't be too hard on me with the critics. Eye-wink Writing these plugins gave me a first impression on why people love Perl that much. Smiling Eg. the embedded regexp usage is a killer. I wonder why did not all other scripting languages take over this syntax? Shocked

P.S.2: the scripts are in "pure" Perl and the extra modules I used are available for most platforms ... so I don't see any reason for the plugin not to work on any platform that FMS was ever released for. But I only tested on a Debian server so everything is possible.

AttachmentSize
fms.11.37 KB
fms_apps.9.3 KB
fms_apps_rate.9.39 KB
fms_playing.10.41 KB

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

wow, thank you so much, it

wow, thank you so much, it worked perfectly on my Gentoo servers with FMS 3.0
Until today i used the old one which gave pretty weird results as you said and was just thinking at doing a very basic plugin based on lsof but kind of failed and tried to search and see if anyone did a better job since last time i checked few months ago and i'm glad i found you Smiling

keep up the good work.

Re: wow, thank you so much, it

I'm happy to see that it works for you. Smiling

Re: wow, thank you so much, it

Btw. you could have used the output of netstat too. Using lsof here is kind of shooting a sparrow with a cannon. Smiling

smee again It looks like

smee again Smiling
It looks like something borked over night because i got some discrepancies between what netstat/lsof and the plugin tells me.

flash1 plugins # netstat -an | grep "1935" | grep ESTAB | wc -l
151
flash1 plugins # lsof -i -n | grep fms | grep ESTAB | wc -l
146
flash1 plugins # ./fms_apps
fms_app_live.value 0
fms_app_vod.value 441

Got any ideea?

Re: smee again It looks like

You're right, at least partly. FMS might not just listen on the 1935 TCP port. Eg. in my setup it listens on 443 and 80 too. There's an RTMPT protocol that stands for "RTMP over HTTP tunnel". If your player SWF implements this, it is possible that it first tries to connect to port 1935 via RTMP and if it does not succeed within a certain timeperiod, it tries port 80 via RTMPT. An even better solution is (in respect of client wait time) if the player initiates a connection to port 1935 and if it did not succeed in eg. 2 seconds, it automatically initiates a connection to 80 port via RTMPT. And the player uses the connection that first successfully connects to the server and drops/closes the other one. This is what we use in our player.

On the other hand you're right: there's a significant discrepancy between what I see from netstat output and what I get from the plugin (which gets the stats from the FMS Admin server's getAppStats() call.

Initially I thought I could explain this. The FMS Admin server shows a number with _active_ stream connections, while netstat shows any kind of a TCP connection. The latter can be even a connection that is not yet fully estabilished or that has been already ended and is waiting for being cleaned up. Take a look at the manpage of netstat. You'll find in the "State" section the list of various states of a TCP connection:
  • ESTABLISHED: The socket has an established connection.
  • SYN_SENT: The socket is actively attempting to establish a connection.
  • SYN_RECV: A connection request has been received from the network.
  • FIN_WAIT1: The socket is closed, and the connection is shutting down.
  • FIN_WAIT2: Connection is closed, and the socket is waiting for a shutdown from the remote end.
  • TIME_WAIT: The socket is waiting after close to handle packets still in the network.
  • CLOSE: The socket is not being used.
  • CLOSE_WAIT: The remote end has shut down, waiting for the socket to close.
  • LAST_ACK: The remote end has shut down, and the socket is closed. Waiting for acknowledgement.
  • LISTEN: The socket is listening for incoming connections. Such sockets are not included in the output unless you specify the --listening (-l) or --all (-a) option.
  • CLOSING: Both sockets are shut down but we still don’t have all our datasent.
  • UNKNOWN: The state of the socket is unknown.
The FMS Admin server could report only connections that are in ESTABLISHED state and that are "active", meaning that the connection had traffic in the last period of the FMS Admin Server statistics updater module. But unfortuantely this is not true. Sad I just saw the FMS Admin server returning the number 60 for the "vod" application and netstat reports only 39 connections (btw. all of them are ESTABLISHED, no connections between the FMS server and clients were in any other state). Sad

It seems FMS Admin server sucks in every way. Sad I'll modify the basic fms Munin plugin (that does not split connections by applications) to work based on netstat. It seems there's no real stat in FMS Admin Server's output that could give us the _real_ number of actual streams. It'd be nice if Adobe published some "real" documentation on how these stats in FMS Admin Server work. The description in the "FMS Administration API Reference" is crap and explains nothing. Or at least it does not give a plausible explanation on how to reproduce the numbers that FMS Admin Server reports. And certainly those numbers are not what FMS admins are interested in. Smiling

I understand what you're

I understand what you're saying, that's why i grepped for ESTAB and also my FMS listens only on 1935, im sure of it Smiling

Anyway another weird thing i noticed today:
- because of some special circumstances the fms_apps plugin was saying i have 5500 connections to vod app
- lsof/netstat usually shows with few hundred less and it was reporting almost 5000 connections
- same time the traffic graph for that server showed 235mbps upload
- all videos served are encoded at 500kbps

Now, it might be just me but i smell something fishy here so let's try to do a simple calculation hoping i don't make a fool out of myself Smiling
As i see it should be something like:

total upload speed / bitrate = number of users that are simultaneously viewing a video

..which means with the peak numbers i had today

235mbps / 500kbps = ~470 simultaneous users viewing video

When i got this number i didn't knew what to believe and how to explain what FMS / netstat / lsof were reporting 5000 connections
Even if let's say some reports "miscalculate" few hundred connections for various reasons there is still a waaaay too bigger gap to fill.

Is my formula so flawed? Did i badly miscalculated something?
Because if not i really don't know how we can really get the real number of users that are viewing a video as all the tools don't seem even close to being accurate.

p.s. do you have by any chance any tips&tricks to optimize the FMS3 (might upgrade to 3.5) load? i hit 25% cpu load today with the numbers above on a dual quad-core xeon at 2ghz and i basically just serve 500kbps h264 videos using the default app and i also intend to transmit a live feed at 360kbps

thanks!

Well it seems getAppsStats

Well it seems getAppsStats doesn't help much either Sad

flash1 plugins # ./fms_apps ; lsof -i -n | grep fms | grep ESTAB | wc -l ; netstat -an | grep "1935" | grep ESTAB | wc -l
fms_app_live.value 0
fms_app_vod.value 4257
1443
1502

..i'm pretty sure netstat/lsof doesn't really show me the exact number of users that view videos but it's closer then what FMS shows :|

I was thinking at something else that could give us better results, can't we use the logging facility provided by FMS? i don't really have any ideea on how to do it atm but i'll think about it so feel free to intervene if you have any clue Smiling

Re: Well it seems getAppsStats


..i'm pretty sure netstat/lsof doesn't really show me the exact number of users that view videos but it's closer then what FMS shows :|

Why are you so sure? Shocked I'm pretty sure that netstat shows all network (socket) connections between the FMS server and various clients. Of course some of these connections are most probably not "active" (I mean the client is not really streaming video) at that moment. I'd say that the "real" number of streaming connections is equal or less than the number we can get by parsing the output of netstat.

I was thinking at something else that could give us better results, can't we use the logging facility provided by FMS?

After all the trouble I had with the stats from FMS Admin server (getServerStats, getAppStats) I'm quite sceptical about accepting anything as truth from FMS. You can give it a shot, but I've decided it's not worth the effort (at least for me).

I've now modified the fms plugin to be able to use netstat's output and the number I get from it is just fine for me. Smiling

FMS Admin server vs. netstat

I wonder what is the relation between the getServerStats()/getAppStats() results and the netstat output. I'll keep monitoring the output of both the fms plugin and the fms_apps plugin. On one server I see 38 connections from getAppStats and 33 connections from netstat. On another server I see 76 connections from getAppStats and 12 connections from netstat. It seems that sometimes the results are close, sometimes quite far. I don't know ... probably an FMS bug. Shocked

Well i was comparing to the

Well i was comparing to the upload speed also and using the formula i wrote earlier and it didn't really add up.

here's another sample from me Smiling

flash1 plugins # ./fms_apps ; lsof -i -n | grep fms | grep ESTAB | wc -l ; netstat -an | grep "1935" | grep ESTAB | wc -l
fms_app_live.value 0
fms_app_vod.value 3985
715
736

..and upload speed was ~65mbps

Anyway which one you modified to use netstat?

Syndicate content