Prometheus and Grafana
Follow the steps in the
antidote_statsmodule to start the required monitoring services:
docker-compose up -d
This starts a prometheus server and a grafana database. If the containers started with no problems, open
http://localhost:9090/targetsin a web-browser. You will see the targets prometheus is configured to monitor. To change or add additional Antidote nodes, follow the necessesary steps in the README to add targets to the prometheus configuration.
To shutdown containers:
To view the monitoring data of antidote, open the Grafana Dashboard. With default the configuration open the URL
http://localhost:3000and log in with user
The monitoring system provides information for the following metrics: Antidote API, Antidote Ring Structure, Antidote Log Size (disk usage), Memory, I/O, CPU, and Erlang Internals.
Antidote has client, internal, and inter-dc operations which are monitored separately. Addionally, transactions are tracked.
riak_coreas a main dependency to provide sharding and means of distributing the log. The current state of the ring can be seen in the
Ring Statesection and should be checked when setting up the nodes for the first time or look for outages of nodes in a DC.
To know how much disk space Antidote is using, log sizes are monitored.
To check for unusually active vnodes, the log sizes of every vnode is checked, too.
Additional host metrics are provided. These include memory usage, network I/O, CPU usage, and Erlang internal metrics.
To add a metric collection call to the Antidote source code, use the macro
termis the term to send to the stat collector process. The actual Prometheus processing is done in the
antidote_statsmodule and every term should be implemented there. Unknown terms which are not yet supported are logged during runtime, but will not crash the system.