13/11/2013
New traffic record for GRNET NOC streaming service
Around a year ago I wrote a blog post about how me and @zmousm scaled our streaming infrastructure at GRNET NOC so that we could cope with a sudden demand on the streaming service that we provide to the Greek Parliament. That setup was re-used again in January 2013 (Lagarde-list discussion) where we managed to surpass our previous record of 1.66Gbps reaching 1.79Gbps. We knew that the previous solution could definitely be improved though. Wowza does not seem to scale very well in our environment(*cough* java *cough*), so we modified our setup quite a bit.
What we did was take the original Wowza streamer, and ‘hide’ it behind two different categories of ‘proxy-servers’ that clients communicate with. The first category is made of three varnish proxies sitting at two different datacenters. All clients that fetch HTTP streams communicate only with the varnish proxies and not with the original streamer. Varnish uses very few resources and scales wonderfully. Then we added an nginx-rtmp server to offload RTMP clients from the original streamer. Now all RTMP clients communicate with nginx-rtmp only. We’ve also notified website owners that prefer using our RTMP stream to serve it through their own (flash) applications to switch to the nginx-rtmp endpoint. This means that the original Wowza streamer now mainly serves the three varnish proxies and the nginx-rtmp server as ‘clients’, and since the VM now has far less load, the stream it provides to the ‘proxy-servers’ doesn’t get ‘chopped’ from time to time, as it did previously when it served hundreds of clients.
While each wowza streamer previously needed 6Gb of RAM to serve around 500-600Mbit of traffic, varnish needs <1Gb and can easily serve 900Mbit. Our nginx-rtmp server also uses <1Gb of RAM. So we’re actually using fewer resources to serve more (happier) clients!
This setup gives us a lot of flexibility and extensibility. We can easily scale it horizontally when we want to just by adding more varnish or nginx-rtmp servers.
With this setup we were able to achieve 3.55Gbps and serve more than 6000 clients last Sunday (10/11/2013), that’s double of our previous record!
Here are the graphs:
Filed by kargig at 10:10 under Greek,Internet,Networking
Tags: Greek, grnet, HTTP, nginx, nginx-rtmp, parliament, rtmp, scaling, streaming, varnish, wowza
12 Comments | 14,532 views
Εύγε!
Χαχα, νομίζω ότι όσο φεύγει κόσμος έξω, τόσο θα μεγαλώνει και το demand…
Πάντως να σου πω την αλήθεια, σε εμένα κατά τις 12 είχε αρκετά glitches, δεν ξέρω αν ήταν στο application ή networking… πάντως θα είχε ενδιαφέρον να δοκιμάσεις και το http://erlyvideo.org/doc/description
And how the load(clients) is balanced among the varnish proxies? How do the proxies know where to connect me in order the load to be balanced?
Thank you
@john: RTMP ή HTTP stream χρησιμοποιούσες; οι varnish είχαν φτάσει 900+Mbit ο καθένας εκείνη την ώρα, και βγαίνουν από hardware nodes που έχουν 2 x 1gbit συνδέσεις σε bond με active+backup configuration. Οπότε ήταν κοντά στο όριο του physical link. Στο μέλλον αυτό θα το αλλάξουμε ώστε να εκμεταλλευόμαστε καλύτερα το bandwidth των 2x1gbit.
@Spiros: The flash application serves a specific HTTP endpoint hostname to the clients and for that hostname we use DNS RR with very low TTL. So it’s quite easy to add/remove servers to the DNS RR if needed. There’s currently no need for LVS or a more advanced varnish configuration, eg weight/load based balancing, but we could do that as well in the future.
Thank you for your answer. I think every day there, will be very interesting and challenging.
I would like such a job as nothing else 🙂
@john: Το erlyvideo.org μετά από μια περιπετειώδη πορεία φοβάμαι ότι τελείωσε, έμεινε μόνο το flussonic.com πλέον. Σε κάθε περίπτωση είχε διαφορετικό προσανατολισμό και δε βόλευε καθόλου σε σχέση με το μηχανισμό εισαγωγής του stream της Βουλής στο streaming server.
@kargig Να σου πω την αλήθεια δεν έχω ιδέα, ότι χρησιμοποιεί ο browser / flowviewer?, νομίζω το έβλεπα μέσα από ένα link του naftemporiki.gr, φαντάζομαι http.
@zmousm: κρίμα, νόμιζα ότι ήταν κάτι σαν αλλαγή του brand name.
Hi there
Do you use your varnishes to cache the HLS-Segments? If so, are you using adaptive bitrate? Because i try to do the same but get different filenames for each user connecting to the stream. I can just guess it’s caused by adaptive-streaming.
Have a nice day
Chris
Hi,
Interesting use case. I recently started playing with wowza, varnish and nginx-rtmp, and the setup is very cool. But I am having problems getting the same stream delivered to both rtmp and hls/hds.
If we are going to use varnish, we need to configure wowza as http origin, so that it stops generating unique urls for each visitor, and to set the correct max-age caching headers, so that varnish actually caches anything. That works fine.
But when we enable http origin mode, wowza disables rtmp support, as they say “it is not possible to proxy rtmp”. How have you overcome this limitation in wowza?
Håkon: http://www.wowza.com/forums/showthread.php?20876-Article-How-to-configure-a-Wowza-server-as-an-HTTP-caching-origin&p=141173#post141173
Chris: File names should not be different, however getting the segments to use properly aligned timecode can be tricky. All segmenters have various options for achieving this. One approach is to have the encoder do the segmentation.
Thanks, I’ll try that immediatly!
About timecode, I seem to have no problem getting segments to be aligned in regards of hls adaptive streaming. The transcoder is aligning the source with the transcoded parts automatically, (by adding even more delay though)
My encoders is giving keyframe every second, so for example if the segments are set to be 10 seconds long, it has no problem cutting it at exactly 10 seconds. I have set it down to 4 seconds to have less video delay, and quicker adaptive bitrate change though. But increased the number of segments in the playlist. Giving me a little bit less delay from the live stream.
Do you have good experience with the nginx-rtmp plugin in regards of stability in this use case?
Hou do you get those stats graphs?
I am looking into Munin to get some… any other options?
Also… nginx-rtmp, does it works with LIVE streaming?
Thanks