Netdata

Server monitoring/dashboard solution.

Installing

Prerequisites and dependencies

apt-get install -y zlib1g-dev uuid-dev libmnl-dev gcc make \ 
    autoconf autoconf-archive autogen automake pkg-config curl

apt-get install -y python python-yaml python-mysqldb \
    python-psycopg2 nodejs lm-sensors netcat

Installing Netdata on Linux

Here's a quick one-liner to avoid checking out the github repository.

bash <(curl -Ss https://my-netdata.io/kickstart.sh)

What It Do

The script does the following:

• Create users/groups as needed
• Get logrotate
• Set permissions
• Download default netdata configuration

Output from Install Process

Here is the output from the installation process, which contains some useful information:

--- Restore user edited netdata configuration files --- --- Fix generated files permissions --- [/usr/src/netdata.git]# find ./system/ -type f -a \! -name \*.in -a \! -name Makefile\* -a \! -name \*.conf -a \! -name \*.service -a \! -name \*.logrotate -exec chmod 755 \{\} \; OK --- Add user netdata to required user groups --- Adding netdata user group ... [/usr/src/netdata.git]# groupadd -r netdata OK Adding netdata user account ... [/usr/src/netdata.git]# useradd -r -g netdata -c netdata -s /usr/sbin/nologin -d / netdata OK Group 'docker' does not exist. Group 'nginx' does not exist. Group 'varnish' does not exist. Group 'haproxy' does not exist. Adding netdata user to the adm group ... [/usr/src/netdata.git]# usermod -a -G adm netdata OK Group 'nsd' does not exist. Adding netdata user to the proxy group ... [/usr/src/netdata.git]# usermod -a -G proxy netdata OK Group 'squid' does not exist. --- Install logrotate configuration for netdata --- [/usr/src/netdata.git]# cp system/netdata.logrotate /etc/logrotate.d/netdata OK [/usr/src/netdata.git]# chmod 644 /etc/logrotate.d/netdata OK --- Read installation options from netdata.conf --- --- Fix permissions of netdata directories (using user 'netdata') --- [/usr/src/netdata.git]# chown -R root:netdata /etc/netdata OK [/usr/src/netdata.git]# find /etc/netdata -type f -exec chmod 0640 \{\} \; OK [/usr/src/netdata.git]# find /etc/netdata -type d -exec chmod 0755 \{\} \; OK [/usr/src/netdata.git]# chown -R netdata:netdata /usr/share/netdata/web OK [/usr/src/netdata.git]# find /usr/share/netdata/web -type f -exec chmod 0664 \{\} \; OK [/usr/src/netdata.git]# find /usr/share/netdata/web -type d -exec chmod 0775 \{\} \; OK [/usr/src/netdata.git]# chown -R netdata:netdata /var/lib/netdata OK [/usr/src/netdata.git]# chown -R netdata:netdata /var/cache/netdata OK [/usr/src/netdata.git]# chown -R netdata:netdata /var/log/netdata OK [/usr/src/netdata.git]# chmod 755 /var/log/netdata OK [/usr/src/netdata.git]# chown netdata:root /var/log/netdata OK [/usr/src/netdata.git]# chown -R root /usr/libexec/netdata OK [/usr/src/netdata.git]# find /usr/libexec/netdata -type d -exec chmod 0755 \{\} \; OK [/usr/src/netdata.git]# find /usr/libexec/netdata -type f -exec chmod 0644 \{\} \; OK [/usr/src/netdata.git]# find /usr/libexec/netdata -type f -a -name \*.plugin -exec chmod 0755 \{\} \; OK [/usr/src/netdata.git]# find /usr/libexec/netdata -type f -a -name \*.sh -exec chmod 0755 \{\} \; OK [/usr/src/netdata.git]# chown root:netdata /usr/libexec/netdata/plugins.d/apps.plugin OK [/usr/src/netdata.git]# chmod 0750 /usr/libexec/netdata/plugins.d/apps.plugin OK [/usr/src/netdata.git]# setcap cap_dac_read_search\,cap_sys_ptrace+ep /usr/libexec/netdata/plugins.d/apps.plugin OK [/usr/src/netdata.git]# chown root:netdata /usr/libexec/netdata/plugins.d/cgroup-network OK [/usr/src/netdata.git]# chmod 4750 /usr/libexec/netdata/plugins.d/cgroup-network OK [/usr/src/netdata.git]# chown root /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh OK [/usr/src/netdata.git]# chmod 0550 /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh OK [/usr/src/netdata.git]# chmod a+rX /usr/libexec OK [/usr/src/netdata.git]# chmod a+rX /usr/share/netdata OK --- Install netdata at system init --- Installing systemd service... [/usr/src/netdata.git]# cp system/netdata.service /etc/systemd/system/netdata.service OK [/usr/src/netdata.git]# systemctl daemon-reload OK [/usr/src/netdata.git]# systemctl enable netdata Created symlink /etc/systemd/system/multi-user.target.wants/netdata.service -> /etc/systemd/system/netdata.service. OK --- Start netdata --- [/usr/src/netdata.git]# /bin/systemctl stop netdata OK [/usr/src/netdata.git]# /bin/systemctl restart netdata OK OK. NetData Started! ------------------------------------------------------------------------------- Downloading default configuration from netdata... [/usr/src/netdata.git]# curl -s -o /etc/netdata/netdata.conf.new http://localhost:19999/netdata.conf OK [/usr/src/netdata.git]# mv /etc/netdata/netdata.conf.new /etc/netdata/netdata.conf OK OK New configuration saved for you to edit at /etc/netdata/netdata.conf [/usr/src/netdata.git]# chown netdata /etc/netdata/netdata.conf OK [/usr/src/netdata.git]# chmod 0664 /etc/netdata/netdata.conf OK --- Check KSM (kernel memory deduper) --- Memory de-duplication instructions You have kernel memory de-duper (called Kernel Same-page Merging, or KSM) available, but it is not currently enabled. To enable it run: echo 1 >/sys/kernel/mm/ksm/run echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs If you enable it, you will save 40-60% of netdata memory. --- Check version.txt --- --- Check apps.plugin --- --- Generate netdata-uninstaller.sh --- --- Basic netdata instructions --- netdata by default listens on all IPs on port 19999, so you can access it with: http://this.machine.ip:19999/ To stop netdata run: systemctl stop netdata To start netdata run: systemctl start netdata Uninstall script generated: ./netdata-uninstaller.sh Update script generated  : ./netdata-updater.sh netdata-updater.sh can work from cron. It will trigger an email from cron only if it fails (it does not print anything when it can update netdata). --- Installing netdata-updater at cron --- [/usr/src/netdata.git]# ln -s /usr/src/netdata.git/netdata-updater.sh /etc/cron.daily/netdata-updater OK --- We are done! --- ^ |.-. .-. .-. .-. .-. . netdata .-. .- | '-' '-' '-' '-' '-' is installed and running now! -' '-' +----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+-----+---> enjoy real-time performance and health monitoring... OK

Usage

Netdata web dashboard

To use netdata's web dashboard, open a browser and navigate to localhost:19999

You should now see a web dashboard with live statistics about the system.

Spend plenty of time here, as there is a lot of information available.

Database Backends

Here we are interested in dumping data from netdata to a database.

Netdata wiki page on backends: https://github.com/firehol/netdata/wiki/netdata-backends

Backend API URLs

We'll use this below, but check out how to use URLs to access the Netdata backend API:

http://192.168.25.236:19999/api/v1/allmetrics?format=json&help=yes

http://192.168.25.236:19999/api/v1/allmetrics?format=prometheus&help=yes

This helpful information was pointed out by this guide to setting up Prometheus backend: https://github.com/firehol/netdata/wiki/Netdata,-Prometheus,-and-Grafana-Stack

Prometheus Backend

Following this guide to setting up Prometheus backend, which is on the Netdata wiki: https://github.com/firehol/netdata/wiki/Netdata,-Prometheus,-and-Grafana-Stack

Json Backend

I had some trouble getting Netdata to work with MongoDB, a JSON database. I pointed Netdata to a particular port, and then set up a simple echo socket listener in Python, but saw no activity on the port.

I tried this using the following /etc/netdata/netdata.conf:

[backend]
	enabled = yes
	type = json
	destination = localhost:27017
	data source = average
	prefix = netdata
	update every = 10
	buffer on failures = 10
	timeout ms = 20000
        send charts matching = *
        send names instead of ids = yes

Resources

Netdata Github repo: https://github.com/firehol/netdata

Netdata wiki: https://github.com/firehol/netdata/wiki

Netdata wiki article on setting up Prometheus backend: https://github.com/firehol/netdata/wiki/Netdata,-Prometheus,-and-Grafana-Stack

Netdata command line options: https://github.com/firehol/netdata/wiki/Command-Line-Options

Databases

Replicating databases across multiple netdata instances: https://github.com/firehol/netdata/wiki/Replication-Overview

Archiving data to database backends: https://github.com/firehol/netdata/wiki/netdata-backends