How to capture an RTSP real audio stream and convert it into mp3

Ubuntu (and probably most other distributions) have all the tools that you'll need: Mplayer and Lame. Use the former to capture the stream into a RM (=Real Media) file:
mplayer "rtsp://server.example.com:554/sometrack.rm" -dumpstream -dumpfile "captured.rm"
Now convert the Real Media file to a WAV:
mplayer -ao pcm:waveheader:file="captured.wav" "captured.rm" -vo null -vc dummy
And finally convert the WAV into an MP3:
lame -h "captured.wav" "captured.mp3"

Of course you can do all this in a dozen other ways (using other tools or other options). I suggest you take your time and read the manpage of at least lame, because the result will depend mostly on what source you used (I mean the contents of the stream) and what options you used to create the mp3 file.

P.S.: you can use the same procedure on a Mac. Mplayer is available in binary form for the Mac too and LAME compiles on a Mac (at least on Leopard) without any problems (and of course you can find binary LAME packages too). To use Mplayer in a shell (aka. command line) you've to invoke it with the correct path (the binary is buried inside the application's dir). Eg. on an Intel-based Mac use the /Applications/MPlayer OSX.app/Contents/Resources/External_Binaries/mplayer_intel.app/Contents/MacOS/mplayer path to the binary.