2018/Data Project: Difference between revisions

From charlesreid1
Line 3: Line 3:
The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards.
The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards.


First, we set up [[Netdata]] to dump to a [[Prometheus]] database. That was fine, but not very transparent. Netdata has a fantastic dashboard with all kinds of stuff all ready to go, but it's custom-built for monitoring compute nodes, and not for general visualization. Prometheus was not a particularly outstanding tool, don't know much about how to use it.
==Phase 1: Netdata and Prometheus==
 
First, we set up [[Netdata]] to dump to a [[Prometheus]] database.  
* '''Pros:''' Netdata has a fantastic dashboard with all kinds of stuff all ready to go. Prometheus was fairly easy to integrate with Netdata.
* '''Cons:''' Netdata is custom-built for monitoring compute nodes, and not for general visualization. Prometheus was not a particularly outstanding tool, don't know much about how to use it.
* [[Netdata]]
* [[Netdata]]
* [[Prometheus]]
* [[Prometheus]]
Netdata is a useful tool for monitoring an individual machine instance remotely. Need to get more involved with Prometheus and/or Grafana to monitor more than one machine.
==Phase 2: MongoDB and MongoExpress==


We then set up [[MongoDB]] and [[MongoExpress]] in Docker containers, and have MongoDB listening over the VPN for incoming data. We used this to scrape information from the charlesreid1 wiki.
We then set up [[MongoDB]] and [[MongoExpress]] in Docker containers, and have MongoDB listening over the VPN for incoming data. We used this to scrape information from the charlesreid1 wiki.
* '''Pros:''' MongoDB is a containerized solution with persistent data. MongoDB had (has?) a high setup barrier, but a low usage barrier. Very easy to do basic CRUD operations, make new databases as needed, etc.
* '''Cons:''' No visualization tools baked in, need to define own tools. Collectd cannot dump to MongoDB because of a bunch of installation stupidity.
* [[MongoDB]]
* [[MongoDB]]
* [[MongoExpress]]
* [[MongoExpress]]
Line 14: Line 24:
* Link to MongoExpress docker files: https://charlesreid1.com:3000/docker/d-mongoexpress
* Link to MongoExpress docker files: https://charlesreid1.com:3000/docker/d-mongoexpress
* Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data
* Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data
Next, we deployed a [[Graphite]] container to hold time series and a [[Grafana]] container to create dashboards from it.
* '''Pros:''' Containerized solution, like MongoDB
* Link to Graphite docker files: https://charlesreid1.com:3000/docker/d-graphite
* Link to Grafana docker files: https://charlesreid1.com:3000/docker/d-graphite

Revision as of 20:40, 10 February 2018

Project Overview

The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards.

Phase 1: Netdata and Prometheus

First, we set up Netdata to dump to a Prometheus database.

  • Pros: Netdata has a fantastic dashboard with all kinds of stuff all ready to go. Prometheus was fairly easy to integrate with Netdata.
  • Cons: Netdata is custom-built for monitoring compute nodes, and not for general visualization. Prometheus was not a particularly outstanding tool, don't know much about how to use it.
  • Netdata
  • Prometheus

Netdata is a useful tool for monitoring an individual machine instance remotely. Need to get more involved with Prometheus and/or Grafana to monitor more than one machine.

Phase 2: MongoDB and MongoExpress

We then set up MongoDB and MongoExpress in Docker containers, and have MongoDB listening over the VPN for incoming data. We used this to scrape information from the charlesreid1 wiki.

Next, we deployed a Graphite container to hold time series and a Grafana container to create dashboards from it.

Retrieved from "https://charlesreid1.com/w/index.php?title=2018/Data_Project&oldid=24121"