Embrace Open Source Monitoring With Me

Chris Tate - 14th September 2016

So after at least a couple of decades (my how time flies!) of learning, implementing, developing and maintaining monolithic enterprise solutions, the scales might have finally fallen from my eyes and I’m about to embrace open source monitoring principles.  I’m old school, not one of the cool kids, not part of the hipster gang (I can't grow a beard and I don’t like artisan pickles!), just an ordinary IT bod with a head full of passwords, regular expressions and the constant worry about which new software, system, language, standard and buzzword (they reel you in eventually!!!) I am going to have to learn.  So what has brought this head rush of lentil and craft beer infused delirium - in one word “Monitorama”.  This is the small conference dedicated all things monitoring and open source, with healthy doses of organisational and cognitive psychology and sociology (they have a code of conduct), statistics (yay! Zzzzzzzzzzz!), cute pigs in Oregon zoo and time series metric collections.  I know, the delirium is about to consume you too with those headliners, but stick with me…

So Chris, what’s the deal with open source stuff?

I’m glad you asked…

  1. It’s free
  2. It follows reductionist system principles
  3. It infinitely customisable
  4. There is a really great community to support you
  5. It puts you in the driving seat rather than the vendor
  6. Because it is component based, it is open and transparent

The commercial systems we know and love are like this:

Lovely exciting shiny systems that we can spend years honing and perfecting, only after time-served acceptance into the inner sanctum of vendor protected technical support and development labs do we start to see under the covers into exactly how it all works and usually this is on the back of fixing problems.  Don’t misunderstand me, this is a good thing, it creates work and generates revenue which fuels economies and drives development within those organisations.  So it’s a polished product with support and maintenance structures.  It’s what we have all bought into, and we have had successes and failures, things have been challenging in some respects not least because the bits that do stuff that we want are encapsulated inside the commercial shell, out of reach of the naughty IPR thieves.

But what if the cat gets out of the bag, what if we remove that shell, we might see something like this:

(Reproduced from “The Art of Monitoring” by James Turnbull © 2016)

Suddenly we can see what the moving parts are, we have a name for them, and we can see how they interact.  We have reduced the system to constituent parts, that are transparent to us, and we are back in the driving seat.  Ok, so now we have lots of things to support and maintain, and we don’t have the comfort of a nice expensive support and maintenance contract to help us sleep at night.  That is true, the open source monitoring is about rolling your own solution and supporting as a community for free, scary I know, but what if the fact that the components only did one job in the system, that’s easier to deal with perhaps.  And if you had written, and more importantly instrumented, that component yourself, then you'd know it like you know a chatty character you have written.  What about if you have disentangled the metric display system and you can use a dedicated rich palette to produce meaningful data display to your business leaders (see Grafana - http://grafana.org/), maybe we want to define schema and storage / retrieval for our data in our way (see ElasticSearch - https://www.elastic.co/).  What about how and what we collect, surely we could define that for ourselves (see Collectd - https://collectd.org/) and what about processing all that event information very fast in a structured coherent way that makes sense to our business (see Riemann - http://riemann.io/).

We can start by simulating our transactional space to understand where best to place our collection methods (see Adrian Cockcroft’s SpiGo and the Simian Army - https://github.com/adrianco/spigo), we can start making on-call guys wear health monitors and include metrics on the effects of broken sleep patterns (no really, we can and people are!!).  With the open source component based paradigm we are limited by our imagining alone (and how much R&D time our employers will give us!!)

The list is endless, the combinations countless, yes we need to become part of the community to extract the most from it, but the level of engagement is on our terms and that is fine.  And yes there is an overhead to producing really great monitoring for our systems, but that is true with COTS as well.  And hey, because its free, you can build your open source solution alongside your COTS products to see if it is a benefit, you lose nothing but gain everything.  Go on give it a go!!!!

Monitorama 2016 videos can be found here.