Metrics

Metrics are a means for workers to publish statistics about their operations for real-time plotting and later analysis. Vumi provides built-in support for publishing metric values to Carbon (the storage engine used by Graphite).

Using metrics from a worker

The set of metrics a worker wishes to plublish are managed via a MetricManager instance. The manager acts both as a container for the set of metrics and the publisher that pushes metric values out via AMQP.

Example:

class MyWorker(Worker):

    def startWorker(self, config):
        self.metrics = yield self.start_publisher(MetricManager,
                                                  "myworker.")
        self.metrics.register(Metric("a.value"))
        self.metrics.register(Count("a.count"))

In the example above a MetricManager publisher is started. All its metric names with be prefixed with myworker.. Two metrics are registered – a.value whose values will be averaged and a.count whose values will be summed. Later, the worker may set the metric values like so:

self.metrics["a.value"].set(1.23)
self.metrics["a.count"].inc()
class vumi.blinkenlights.metrics.MetricManager(prefix, publish_interval=5, on_publish=None)

Utility for creating and monitoring a set of metrics.

Parameters:
  • prefix (str) – Prefix for the name of all metrics registered with this manager.
  • publish_interval (int in seconds) – How often to publish the set of metrics.
  • on_publish (f(metric_manager)) – Function to call immediately after metrics after published.
register(metric)

Register a new metric object to be managed by this metric set.

A metric can be registered with only one metric set.

Parameters:metric (Metric) – Metric object to register. Will have the manager’s prefix added to its name.
Return type:For convenience, returns the metric passed in.
start(channel)

Start publishing metrics in a loop.

stop()

Stop publishing metrics.

Metrics

A Metric object publishes floating point values under a metric name. The name is created by combining the prefix from a metric manager with the suffix provided when the metric is constructed. A metric may only be registered with a single MetricManager.

When a metric value is set the value is stored in an internal list until the MetricManager polls the metric for values and publishes them.

A metric includes a list of aggregation functions to request that the metric aggregation workers apply (see later sections). Each metric class has a default list of aggregators but this may be overridden when a metric is created.

class vumi.blinkenlights.metrics.Metric(suffix, aggregators=None)

Simple metric.

Values set are collected and polled periodically by the metric manager.

Parameters:
  • suffix (str) – Suffix to append to the MetricManager prefix to create the metric name.
  • aggregators (list of aggregators, optional) – List of aggregation functions to request eventually be applied to this metric. The default is to average the value.

Examples:

>>> mm = MetricManager('vumi.worker0.')
>>> my_val = mm.register(Metric('my.value'))
>>> my_val.set(1.5)
>>> my_val.name
'vumi.worker0.my.value'
DEFAULT_AGGREGATORS = [<vumi.blinkenlights.metrics.Aggregator object at 0x41e8290>]

Default aggregators are [AVG]

manage(prefix)

Called by MetricManager when this metric is registered.

poll()

Called periodically by the MetricManager.

set(value)

Append a value for later polling.

class vumi.blinkenlights.metrics.Count(suffix, aggregators=None)

Bases: vumi.blinkenlights.metrics.Metric

A simple counter.

Examples:

>>> mm = MetricManager('vumi.worker0.')
>>> my_count = mm.register(Count('my.count'))
>>> my_count.inc()
DEFAULT_AGGREGATORS = [<vumi.blinkenlights.metrics.Aggregator object at 0x41e8250>]

Default aggregators are [SUM]

inc()

Increment the count by 1.

class vumi.blinkenlights.metrics.Timer(*args, **kws)

Bases: vumi.blinkenlights.metrics.Metric

A metric that records time spent on operations.

Examples:

>>> mm = MetricManager('vumi.worker0.')
>>> my_timer = mm.register(Timer('hard.work'))

Using the timer as a context manager:

>>> with my_timer:
>>>     process_data()

Or equivalently using .start() and stop() directly:

>>> my_timer.start()
>>> try:
>>>     process_other_data()
>>> finally:
>>>     my_timer.stop()
DEFAULT_AGGREGATORS = [<vumi.blinkenlights.metrics.Aggregator object at 0x41e8290>]

Default aggregators are [AVG]

Aggregation functions

Metrics declare which aggregation functions they wish to have applied but the actual aggregation is performed by aggregation workers. All values sent during an aggregation interval are aggregated into a single new value.

Aggregation fulfils two primary purposes:

  • To combine metrics from multiple workers into a single aggregated value (e.g. to determine the average time taken or total number of requests processed across multiple works).
  • To produce metric values at fixed time intervals (as is commonly required by metric storage backends such as Graphite and RRD Tool).

The aggregation functions currently available are:

  • SUM – returns the sum of the supplied values.
  • AVG – returns the arithmetic mean of the supplied values.
  • MIN – returns the minimum value.
  • MAX – returns the maximum value.

All aggregation functions return the value 0.0 if there are no values to aggregate.

New aggregators may be created by instantiating the Aggregator class.

Note

The aggregator must be instantiated in both the process that generates the metric (usually a worker) and the process that performs the aggregation (usually an aggregation worker).

class vumi.blinkenlights.metrics.Aggregator(name, func)

Registry of aggregate functions for metrics.

Parameters:
  • name (str) – Short name for the aggregator.
  • func (f(list of values) -> float) – The aggregation function. Should return a default value if the list of values is empty (usually this default is 0.0).

Metrics aggregation system

The metric aggregation system consists of MetricTimeBucket and MetricAggregator workers.

The MetricTimeBucket workers pull metrics messages from the vumi.metrics exchange and publish them on the vumi.metrics.buckets exchange under a routing key specific to the MetricAggregator which should process them. Once sufficient time has passed for all metrics for a specific time period (a.k.a. time bucket) to have arrived at the aggregator, the requested aggregation functions are applied and the resulting aggregated metrics are published to the vumi.metrics.aggregates exchange.

A typical metric aggregation setup might consist of the following workers: * 2 MetricTimeBucket workers * 3 MetricAggregator workers * a final metric collector, e.g. GraphiteMetricsCollector.

A shell script to start-up such a setup might be:

#!/bin/bash
BUCKET_OPTS="--worker_class=vumi.workers.blinkenlights.metrics.\
MetricTimeBucket --set-option=buckets:3 --set-option=bucket_size:5"

AGGREGATOR_OPTS="--worker_class=vumi.workers.blinkenlights.metrics.\
MetricAggregator --set-option=bucket_size:5"

GRAPHITE_OPTS="--worker_class=vumi.workers.blinkenlights.metrics.\
GraphiteMetricsCollector"

twistd -n start_worker $BUCKET_OPTS &
twistd -n start_worker $BUCKET_OPTS &

twistd -n start_worker $AGGREGATOR_OPTS --set-option=bucket:0 &
twistd -n start_worker $AGGREGATOR_OPTS --set-option=bucket:1 &
twistd -n start_worker $AGGREGATOR_OPTS --set-option=bucket:2 &

twistd -n start_worker $GRAPHITE_OPTS &

Publishing to Graphite

The GraphiteMetricsCollector collects aggregate metrics (produced by the metrics aggregators) and publishes them to Carbon (Graphite’s metric collection package) over AMQP.

You can read about installing a configuring Graphite at graphite.wikidot.com but at the very least you will have to enable AMQP support by setting:

[cache]
ENABLE_AMQP = True
AMQP_METRIC_NAME_IN_BODY = False

in Carbon’s configuration file.

If you have the metric aggregation system configured as in the section above you can start Carbon cache using:

carbon-cache.py --config <config file> --debug start

Project Versions

Table Of Contents

Previous topic

Implementing your own middleware

Next topic

Vumi Roadmap

This Page