Munin is a great open source monitoring program available for a number of platforms.
Flash Media Server (FMS) is a streaming server from Adobe aiming mainly at video playback in Flash applications (video players, video conference solutions, etc.). There was already a Munin plugin for FMS (monitoring the number of active connections) at
MuninExchange, but the config parameters were hard-coded (in the plugin code) and it lacked documentation on usage and did not match the structure of standard Munin plugins. I've rewritten the plugin to come up for all these shortcomings. The new version of the plugin is available both attached to this post and at
MuninExchange.
However after looking at the graphs of the plugin it became quite obvious that the numbers returned by the plugin do not reflect what I (and probably many others) was looking for. I want to see the number of active streaming connections and the "connected" value returned by the
getServerStats() API call is just not that (if anybody knows what that "connected" value reflects, please, share with me). I've looked at the various
get*Stats() calls and found that
getAppStats() and
getVHostStats() return the
real "connected" value that I was after. I've created two new plugins based on the results of
getAppStats():
fms_apps: this plugin shows the number of active streaming connections grouped by applications (ie. the graph shows a line for each application on the server)
fms_apps_rate: this plugin shows the number of new connections per minute grouped by applications.
Update! (Feb 16, 2009)
I've created yet another plugin (fms_playing) in the hope that this time it'll display the "real thing".
It's based on the getActiveInstances() and getNetStreamStats() calls of the FMS Admin Server API.
Update! (July 10, 2009)
After months of testing I came to the conlusion that the stats in FMS 3.0.x are completely flawed. However they work OK in FMS 3.5.x. So I strongly advise to use these Munin plugins with FMS 3.5.x or later.
P.S.: I can count the number of my Perl scripts on one hand so don't be too hard on me with the critics.

Writing these plugins gave me a first impression on why people love Perl that much.

Eg. the embedded regexp usage is a killer. I wonder why did not all other scripting languages take over this syntax?
P.S.2: the scripts are in "pure" Perl and the extra modules I used are available for most platforms ... so I don't see any reason for the plugin not to work on any platform that FMS was ever released for. But I only tested on a Debian server so everything is possible.
Comments
wow, thank you so much, it
Until today i used the old one which gave pretty weird results as you said and was just thinking at doing a very basic plugin based on lsof but kind of failed and tried to search and see if anyone did a better job since last time i checked few months ago and i'm glad i found you
keep up the good work.
Re: wow, thank you so much, it
Re: wow, thank you so much, it
netstattoo. Usinglsofhere is kind of shooting a sparrow with a cannon.smee again It looks like
It looks like something borked over night because i got some discrepancies between what netstat/lsof and the plugin tells me.
flash1 plugins # netstat -an | grep "1935" | grep ESTAB | wc -l
151
flash1 plugins # lsof -i -n | grep fms | grep ESTAB | wc -l
146
flash1 plugins # ./fms_apps
fms_app_live.value 0
fms_app_vod.value 441
Got any ideea?
Re: smee again It looks like
On the other hand you're right: there's a significant discrepancy between what I see from
netstatoutput and what I get from the plugin (which gets the stats from the FMS Admin server'sgetAppStats()call.Initially I thought I could explain this. The FMS Admin server shows a number with _active_ stream connections, while
netstatshows any kind of a TCP connection. The latter can be even a connection that is not yet fully estabilished or that has been already ended and is waiting for being cleaned up. Take a look at the manpage ofnetstat. You'll find in the "State" section the list of various states of a TCP connection:- ESTABLISHED: The socket has an established connection.
- SYN_SENT: The socket is actively attempting to establish a connection.
- SYN_RECV: A connection request has been received from the network.
- FIN_WAIT1: The socket is closed, and the connection is shutting down.
- FIN_WAIT2: Connection is closed, and the socket is waiting for a shutdown from the remote end.
- TIME_WAIT: The socket is waiting after close to handle packets still in the network.
- CLOSE: The socket is not being used.
- CLOSE_WAIT: The remote end has shut down, waiting for the socket to close.
- LAST_ACK: The remote end has shut down, and the socket is closed. Waiting for acknowledgement.
- LISTEN: The socket is listening for incoming connections. Such sockets are not included in the output unless you specify the --listening (-l) or --all (-a) option.
- CLOSING: Both sockets are shut down but we still don’t have all our datasent.
- UNKNOWN: The state of the socket is unknown.
The FMS Admin server could report only connections that are in ESTABLISHED state and that are "active", meaning that the connection had traffic in the last period of the FMS Admin Server statistics updater module. But unfortuantely this is not true.It seems FMS Admin server sucks in every way.
fmsMunin plugin (that does not split connections by applications) to work based onnetstat. It seems there's no real stat in FMS Admin Server's output that could give us the _real_ number of actual streams. It'd be nice if Adobe published some "real" documentation on how these stats in FMS Admin Server work. The description in the "FMS Administration API Reference" is crap and explains nothing. Or at least it does not give a plausible explanation on how to reproduce the numbers that FMS Admin Server reports. And certainly those numbers are not what FMS admins are interested in.I understand what you're
Anyway another weird thing i noticed today:
- because of some special circumstances the fms_apps plugin was saying i have 5500 connections to vod app
- lsof/netstat usually shows with few hundred less and it was reporting almost 5000 connections
- same time the traffic graph for that server showed 235mbps upload
- all videos served are encoded at 500kbps
Now, it might be just me but i smell something fishy here so let's try to do a simple calculation hoping i don't make a fool out of myself
As i see it should be something like:
total upload speed / bitrate = number of users that are simultaneously viewing a video
..which means with the peak numbers i had today
235mbps / 500kbps = ~470 simultaneous users viewing video
When i got this number i didn't knew what to believe and how to explain what FMS / netstat / lsof were reporting 5000 connections
Even if let's say some reports "miscalculate" few hundred connections for various reasons there is still a waaaay too bigger gap to fill.
Is my formula so flawed? Did i badly miscalculated something?
Because if not i really don't know how we can really get the real number of users that are viewing a video as all the tools don't seem even close to being accurate.
p.s. do you have by any chance any tips&tricks to optimize the FMS3 (might upgrade to 3.5) load? i hit 25% cpu load today with the numbers above on a dual quad-core xeon at 2ghz and i basically just serve 500kbps h264 videos using the default app and i also intend to transmit a live feed at 360kbps
thanks!
Well it seems getAppsStats
flash1 plugins # ./fms_apps ; lsof -i -n | grep fms | grep ESTAB | wc -l ; netstat -an | grep "1935" | grep ESTAB | wc -l
fms_app_live.value 0
fms_app_vod.value 4257
1443
1502
..i'm pretty sure netstat/lsof doesn't really show me the exact number of users that view videos but it's closer then what FMS shows :|
I was thinking at something else that could give us better results, can't we use the logging facility provided by FMS? i don't really have any ideea on how to do it atm but i'll think about it so feel free to intervene if you have any clue
Re: Well it seems getAppsStats
Why are you so sure?
netstatshows all network (socket) connections between the FMS server and various clients. Of course some of these connections are most probably not "active" (I mean the client is not really streaming video) at that moment. I'd say that the "real" number of streaming connections is equal or less than the number we can get by parsing the output ofnetstat.After all the trouble I had with the stats from FMS Admin server (getServerStats, getAppStats) I'm quite sceptical about accepting anything as truth from FMS. You can give it a shot, but I've decided it's not worth the effort (at least for me).
I've now modified the fms plugin to be able to use
netstat's output and the number I get from it is just fine for me.FMS Admin server vs. netstat
Well i was comparing to the
here's another sample from me
flash1 plugins # ./fms_apps ; lsof -i -n | grep fms | grep ESTAB | wc -l ; netstat -an | grep "1935" | grep ESTAB | wc -l
fms_app_live.value 0
fms_app_vod.value 3985
715
736
..and upload speed was ~65mbps
Anyway which one you modified to use netstat?