We are using Graphite for one part of our monitoring in the company where I’m currently working at and unfortunately our sources are not able to tag the different metrics. So we have to somehow declare groups with hosts and other parameters, to be able to nicely differ/choose between the different hosts/groups in a dashboard.

Also it’s not really possible to do this nicely via Grafana supplied tools in a quick and easy way. So while searching for a solution I stumbled upon this github issue from 2014 https://github.com/grafana/grafana/issues/1032 which is still very active ;). There someone mentioned he is using a SimpleJSON source for doing that. So here is what I did and maybe this is useful for someone else out there.

Prerequisites

We are running Grafana on CentOS

For supplying the JSON I decided to use some simple flask-app running in it’s own python virtualenv

Setup

# install prerequisites for the python modules installed via pip
yum install make gcc python3-devel

# install virtualenv
pip3 install virtualenv

# add a simple user
adduser flaskapp

# work as user
su - flaskapp

# create the folder where the grafana-json-source is running and change into it
mkdir app-grafana-json-source
cd app-grafana-json-source

# create the python environment
virtualenv -p python3 env
source env/bin/activate

# check if really the correct pip3 is used
which pip3

# install flask and uwsgi
pip3 install flask uwsgi

The Flask App

create as user flaskapp the following files

~/app-grafana-json-source/grafana-json-source.py

from flask import Flask
from flask import jsonify
from flask import request
from flask import abort
from flask import json

app = Flask(__name__)
methods = ('GET', 'POST')


metric_finders= {}
metric_readers = {}
annotation_readers = {}
panel_readers = {}

# some parts used from https://gist.github.com/linar-jether/95ff412f9d19fdf5e51293eb0c09b850 
def add_reader(name, reader):
    metric_readers[name] = reader


def add_finder(name, finder):
    metric_finders[name] = finder


def add_annotation_reader(name, reader):
    annotation_readers[name] = reader


def add_panel_reader(name, reader):
    panel_readers[name] = reader


@app.route('/')
def root():
    return 'Simple JSON Source for our Grafana'

@app.route('/search', methods=methods)
def find_metrics():
    # print(request.headers, request.get_json())
    req = request.get_json()

    # print("type ", type(request))

    target = req.get('target', '*')
 
    # in groups you define which groups should be available
    # each group than can have multiple values
    # for example we have 2 groups - host-group and redis-group
    # host-group holds only some hostnames
    # redis-group is a little bit different
    #   * the first element is a port
    #   * the second and third are hostnames
    #   * the fourth is a REDIS instance-name we want to use for example in the Grafana legend 
    #     of corresponding graphs for easy identification
    data = {
         "groups": [
             "host-group",
             "redis-group",
         ],
         "host-group": [
             "host_a_suffix1",
             "host_b_suffix1",
             "host_c_suffix2",
             "host_d_suffix2"
         ],
         "redis-group": [
             "6400",
             "host-redis-001_suffix1",
             "host-redis-002_suffix1",
             "redis-instance-name-a"
         ],
    }
 
    add_finder('get_data', lambda q: data.get(q, data.keys()) if q != '*' else sum(data.values(), []))
 
    if ':' in target:
        finder, target = target.split(':', 1)
    else:
        finder = target
 
    if not target or finder not in metric_finders:
        metrics = []
        if target == '*':
            metrics += metric_finders.keys() + metric_readers.keys()
        else:
            metrics.append(target)
 
        return jsonify(metrics)
    else:
        return jsonify(list(metric_finders[finder](target)))
 
 
if __name__ == '__main__':
    app.run()

~/app-grafana-json-source/wsgi.py

#!/usr/bin/env python
from grafana_json_source import app as application


if __name__ == "__main__":
    application.run()

~/app-grafana-json-source/app-grafana-json-source.ini

[uwsgi]
module = wsgi

master = true
processes = 2

# use socket if reverse proxy in between
#socket = 127.0.0.1:4000
# else for simple serving
http = 127.0.0.1:4000
chmod-socket = 660
vacuum = true

die-on-idle = false
die-on-term = true

Systemd-Unit File

must be created as root of course 😉

/etc/systemd/system/grafana-json-source.service

[Unit]
Description=uWSGI server for grafana-json-source
After=network.target

[Service]
User=flaskapp
Group=flaskapp
Restart=always
WorkingDirectory=/home/flaskapp/app-grafana-json-source
Environment="PATH=/home/flaskapp/app-grafana-json-source/env/bin"
ExecStart=/home/flaskapp/app-grafana-json-source/env/bin/uwsgi --ini app-grafana-json-source.ini

[Install]
WantedBy=multi-user.target

Reload systemd and start the json-source and after that add it as datasource in Grafana

# reload systemctl so it can use the unit-file
systemctl daemon-reload
# start your small grafana-json-source for "groups"
systemctl start grafana-json-source

How to use this now in Grafana

Prerequisite installed json-datasource plugin (SimpleJson) and json-datasource 😉 – see: https://grafana.com/grafana/plugins/grafana-simple-json-datasource

• Create a dashboard
• Create a query variable called for example redis_group using the JSON datasource – for example:
  
  In the query field just use get_data:groups and filter for your wanted group by regex (e.g. redis-.*) in the next field. This should then find all groups starting with redis- in their name. In our example only the “redis-group” should then be listed
• You now can use those “groups” in further queries. Create a query variable called e.g. redis_host also using the JSON datasource and enable “Include All”-Option (depends on dashboard you want to build)
  
  In the query field now use get_data:$redis_group In the regex-field we would for this example type /host-redis-.*/. This would then list us those defined two hosts.
• The port could for example be extracted with a regex /\d+/ looking for values only containing numbers.

In your panel metric queries you now should be able to use the defined variables.