Latest revision as of 08:18, 3 March 2018

Project Overview

The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards for monitoring metrics.

Stage 1: Collecting System Data

See 2018/Data Project/Stage 1

Stage 2: Spy

dahak-spy project:

lightweight server (may want larger disk, okay if non-free)
running mongodb
running mongoexpress
running prometheus
running grafana
running netdata

additional components before real world testing:

netdata on the build node
netdata python plugin from another process, monitoring.....???
metrics:
- is snakemake running (binary yes/no)
- current stage of snakemake (adjust snakemake file to write into a dotfile)
- cpu/memory/network/disk io

netdata python plugin workflow?

does it need to be installed and netdata restarted, or can it push data into netdata?

real yeti:

get a yeti node
debug the snakemake file one step at a time using already-downloaded files (faster step)
let the snakemake file run with netdata and friends running

Flags

@@ Line 1: / Line 1: @@
 =Project Overview=
-The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards.
+The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards for monitoring metrics.
-==Phase 1: Netdata and Prometheus==
+==Stage 1: Collecting System Data==
-First, we set up [[Netdata]] to dump to a [[Prometheus]] database.
+See [[2018/Data Project/Stage 1]]
-* '''Pros:''' Netdata has a fantastic dashboard with all kinds of stuff all ready to go. Prometheus was fairly easy to integrate with Netdata.
-* '''Cons:''' Netdata is custom-built for monitoring compute nodes, and not for general visualization. Prometheus was not a particularly outstanding tool, don't know much about how to use it.
-* [[Netdata]]
-* [[Prometheus]]
-Netdata is a useful tool for monitoring an individual machine instance remotely. Need to get more involved with Prometheus and/or Grafana to monitor more than one machine.
+==Stage 2: Spy==
-==Phase 2: MongoDB and MongoExpress==
+dahak-spy project:
+* lightweight server (may want larger disk, okay if non-free)
+* running mongodb
+* running mongoexpress
+* running prometheus
+* running grafana
+* running netdata
-We then set up [[MongoDB]] and [[MongoExpress]] in Docker containers, and have MongoDB listening over the VPN for incoming data. We used this to scrape information from the charlesreid1 wiki.
+additional components before real world testing:
-* '''Pros:''' MongoDB is a containerized solution with persistent data. MongoDB had (has?) a high setup barrier, but a low usage barrier. Very easy to do basic CRUD operations, make new databases as needed, etc.
+* netdata on the build node
-* '''Cons:''' No visualization tools baked in, need to define own tools. Collectd cannot dump to MongoDB because of a bunch of installation stupidity.
+* netdata python plugin from another process, monitoring.....???
-* [[MongoDB]]
+* metrics:
-* [[MongoExpress]]
+** is snakemake running (binary yes/no)
-* [[Pywikibot]]
+** current stage of snakemake (adjust snakemake file to write into a dotfile)
-* Link to MongoDB docker files: https://charlesreid1.com:3000/docker/d-mongodb
+** cpu/memory/network/disk io
-* Link to MongoExpress docker files: https://charlesreid1.com:3000/docker/d-mongoexpress
-* Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data
-Next, we deployed a [[Graphite]] container to hold time series and a [[Grafana]] container to create dashboards from it.
+netdata python plugin workflow?
-* '''Pros:''' Containerized solution, like MongoDB
+* does it need to be installed and netdata restarted, or can it push data into netdata?
-* Link to Graphite docker files: https://charlesreid1.com:3000/docker/d-graphite
-* Link to Grafana docker files: https://charlesreid1.com:3000/docker/d-graphite
+real yeti:
+* get a yeti node
+* debug the snakemake file one step at a time using already-downloaded files (faster step)
+* let the snakemake file run with netdata and friends running
+=Flags=
+[[Category:Data Engineering]]
+[[Category:Data Project]]
+[[Category:MongoDB]]
+[[Category:MongoExpress]]
+[[Category:Graphite]]
+[[Category:Grafana]]
+[[Category:Netdata]]
+[[Category:Collectd]]
+[[Category:Prometheus]]
+[[Category:2018]]
+[[Category:January 2018]]
+[[Category:February 2018]]
+<!--
+==Stage 2: Finalized Data Collection System==
+===Phase 4: Netdata and Mongo===
+Netdata provides a backend API that can be called to extract data from Netdata. MongoDB listens for API calls to store data in the database. All we need is software that will poll various Netdata instances using the Netdata API and dump that data into MongoDB. This gives much more fine-grained control over the process, schema, and storage format of the data.
+See [[Netdata#Database_Backends]] for info on netdata backends.
+See [[Netdata/MongoDB/API]] for script that calls APIs of Netdata and MongoDB to construct the time series database in MongoDB.
+Link to script: https://charlesreid1.com:3000/data/netdata/src/master/netdata_mongo.py
+[[Image:NetdataMongodb.png|500px]]
+This is a (micro)service design pattern - small, lightweight, standalone daemons act as instruments that continuously read whatever they read, available to be queried but otherwise not saving or doing anything with the data themselves. The data is handled by an application that queries each service it manages to collect data about those services (and coordinate if necessary).
+-->
+<!--
+==Stage 3: Visualizing Data==
+===Phase 5: Grafana===
+[[Grafana]] container to create dashboards from it.
+Link to Grafana docker files: https://charlesreid1.com:3000/docker/d-grafana
+Need to fix grafana user on jupiter.
+We're basically after something like this: https://github.com/firehol/netdata/wiki/Netdata,-Prometheus,-and-Grafana-Stack
+-->

2018/Data Project: Difference between revisions

From charlesreid1

Latest revision as of 08:18, 3 March 2018

Contents

Project Overview

Stage 1: Collecting System Data

Stage 2: Spy

Flags