We are using Graphite for one part of our monitoring in the company where I'm currently working at and unfortunately our sources are not able to tag the different metrics. So we have to somehow declare groups with hosts and other parameters, to be able to nicely differ/choose between the different hosts/groups in a dashboard.
Also it's not really possible to do this nicely via Grafana supplied tools in a quick and easy way. So while searching for a solution I stumbled upon this github issue from 2014 https://github.com/grafana/grafana/issues/1032 which is still very active ;). There someone mentioned he is using a SimpleJSON source for doing that. So here is what I did and maybe this is useful for someone else out there.
Prerequisites
We are running Grafana on CentOS
For supplying the JSON I decided to use some simple flask-app running in it's own python virtualenv
Setup
# install prerequisites for the python modules installed via pip yum install make gcc python3-devel # install virtualenv pip3 install virtualenv # add a simple user adduser flaskapp # work as user su - flaskapp # create the folder where the grafana-json-source is running and change into it mkdir app-grafana-json-source cd app-grafana-json-source # create the python environment virtualenv -p python3 env source env/bin/activate # check if really the correct pip3 is used which pip3 # install flask and uwsgi pip3 install flask uwsgi
The Flask App
create as user flaskapp the following files
~/app-grafana-json-source/grafana-json-source.py
from flask import Flask from flask import jsonify from flask import request from flask import abort from flask import json app = Flask(__name__) methods = ('GET', 'POST') metric_finders= {} metric_readers = {} annotation_readers = {} panel_readers = {} # some parts used from https://gist.github.com/linar-jether/95ff412f9d19fdf5e51293eb0c09b850 def add_reader(name, reader): metric_readers[name] = reader def add_finder(name, finder): metric_finders[name] = finder def add_annotation_reader(name, reader): annotation_readers[name] = reader def add_panel_reader(name, reader): panel_readers[name] = reader @app.route('/') def root(): return 'Simple JSON Source for our Grafana' @app.route('/search', methods=methods) def find_metrics(): # print(request.headers, request.get_json()) req = request.get_json() # print("type ", type(request)) target = req.get('target', '*') # in groups you define which groups should be available # each group than can have multiple values # for example we have 2 groups - host-group and redis-group # host-group holds only some hostnames # redis-group is a little bit different # * the first element is a port # * the second and third are hostnames # * the fourth is a REDIS instance-name we want to use for example in the Grafana legend # of corresponding graphs for easy identification data = { "groups": [ "host-group", "redis-group", ], "host-group": [ "host_a_suffix1", "host_b_suffix1", "host_c_suffix2", "host_d_suffix2" ], "redis-group": [ "6400", "host-redis-001_suffix1", "host-redis-002_suffix1", "redis-instance-name-a" ], } add_finder('get_data', lambda q: data.get(q, data.keys()) if q != '*' else sum(data.values(), [])) if ':' in target: finder, target = target.split(':', 1) else: finder = target if not target or finder not in metric_finders: metrics = [] if target == '*': metrics += metric_finders.keys() + metric_readers.keys() else: metrics.append(target) return jsonify(metrics) else: return jsonify(list(metric_finders[finder](target))) if __name__ == '__main__': app.run()
~/app-grafana-json-source/wsgi.py
#!/usr/bin/env python from grafana_json_source import app as application if __name__ == "__main__": application.run()
~/app-grafana-json-source/app-grafana-json-source.ini
[uwsgi] module = wsgi master = true processes = 2 # use socket if reverse proxy in between #socket = 127.0.0.1:4000 # else for simple serving http = 127.0.0.1:4000 chmod-socket = 660 vacuum = true die-on-idle = false die-on-term = true
Systemd-Unit File
must be created as root of course 😉
/etc/systemd/system/grafana-json-source.service
[Unit] Description=uWSGI server for grafana-json-source After=network.target [Service] User=flaskapp Group=flaskapp Restart=always WorkingDirectory=/home/flaskapp/app-grafana-json-source Environment="PATH=/home/flaskapp/app-grafana-json-source/env/bin" ExecStart=/home/flaskapp/app-grafana-json-source/env/bin/uwsgi --ini app-grafana-json-source.ini [Install] WantedBy=multi-user.target
Reload systemd and start the json-source and after that add it as datasource in Grafana
# reload systemctl so it can use the unit-file systemctl daemon-reload # start your small grafana-json-source for "groups" systemctl start grafana-json-source
How to use this now in Grafana
Prerequisite installed json-datasource plugin (SimpleJson) and json-datasource 😉 - see: https://grafana.com/grafana/plugins/grafana-simple-json-datasource
- Create a dashboard
- Create a query variable called for example redis_group using the JSON datasource - for example:
In the query field just useget_data:groups
and filter for your wanted group by regex (e.g.redis-.*
) in the next field. This should then find all groups starting withredis-
in their name. In our example only the "redis-group" should then be listed - You now can use those "groups" in further queries. Create a query variable called e.g. redis_host also using the JSON datasource and enable "Include All"-Option (depends on dashboard you want to build)
In the query field now useget_data:$redis_group
In the regex-field we would for this example type/host-redis-.*/
. This would then list us those defined two hosts. - The port could for example be extracted with a regex
/\d+/
looking for values only containing numbers.
In your panel metric queries you now should be able to use the defined variables.
Kommentare