2018/Data Project - Revision history

Admin: /* Stage 1: Collecting System Data */

2018-03-03T08:18:40Z

Stage 1: Collecting System Data

@@ Line 5: / Line 5: @@
 ==Stage 1: Collecting System Data==
-===COMPLETED Phase 1a: Netdata===
+See [[2018/Data Project/Stage 1]]
 ==Stage 2: Spy==

Admin: /* Stage 2: Data Collection System */

2018-03-03T08:14:45Z

Stage 2: Data Collection System

← Older revision		Revision as of 08:14, 3 March 2018
Line 74:		Line 74:
	Ultimately, if we need to run collectd, interface with the collectd API via Python: https://collectd.org/wiki/index.php/Plugin:Python		Ultimately, if we need to run collectd, interface with the collectd API via Python: https://collectd.org/wiki/index.php/Plugin:Python

	==Stage 2: ~~Data Collection System~~==		==Stage 2: Spy==

	Re-~~investigate Prometheus. Arbitrary time windows~~ (~~adds only~~)~~, more powerful query syntax, more easily connects to Grafana.~~		dahak-spy project:
			* lightweight server (may want larger disk, okay if non-free)
			* running mongodb
			* running mongoexpress
			* running prometheus
			* running grafana
			* running netdata

	~~Problem we're solving~~: ~~collect periodic data (frequency of ~ 1 Hz - 1/60 Hz), dump into time series database~~		additional components before real world testing:
	* ~~Query data for visualization~~		* netdata on the build node
	* ~~Load data into Python for analysis~~		* netdata python plugin from another process, monitoring.....???
	* ~~Visualization in Grafana~~		* metrics:
	* ~~Flat, copy-able files~~		** is snakemake running (binary yes/no)
	* ~~Easy~~ to ~~dump and restore~~		** current stage of snakemake (adjust snakemake file to write into a dotfile)
			** cpu/memory/network/disk io

			netdata python plugin workflow?
			* does it need to be installed and netdata restarted, or can it push data into netdata?

			real yeti:
			* get a yeti node
			* debug the snakemake file one step at a time using already-downloaded files (faster step)
			* let the snakemake file run with netdata and friends running

	=Flags=		=Flags=

Admin at 06:48, 28 February 2018

2018-02-28T06:48:56Z

@@ Line 73: / Line 73: @@
 Ultimately, if we need to run collectd, interface with the collectd API via Python: https://collectd.org/wiki/index.php/Plugin:Python
 <!--
@@ Line 109: / Line 141: @@
 -->

Admin: /* Stage 3: Visualizing Data */

2018-02-28T06:41:52Z

Stage 3: Visualizing Data

← Older revision		Revision as of 06:41, 28 February 2018
Line 93:		Line 93:

	-->		-->

			<!--

	==Stage 3: Visualizing Data==		==Stage 3: Visualizing Data==
Line 105:		Line 107:

	We're basically after something like this: https://github.com/firehol/netdata/wiki/Netdata,-Prometheus,-and-Grafana-Stack		We're basically after something like this: https://github.com/firehol/netdata/wiki/Netdata,-Prometheus,-and-Grafana-Stack

			-->

	==Next Steps==		==Next Steps==

Admin: /* Stage 2: Finalized Data Collection System */

2018-02-28T06:41:38Z

Stage 2: Finalized Data Collection System

← Older revision		Revision as of 06:41, 28 February 2018
Line 73:		Line 73:

	Ultimately, if we need to run collectd, interface with the collectd API via Python: https://collectd.org/wiki/index.php/Plugin:Python		Ultimately, if we need to run collectd, interface with the collectd API via Python: https://collectd.org/wiki/index.php/Plugin:Python

			<!--

	==Stage 2: Finalized Data Collection System==		==Stage 2: Finalized Data Collection System==
Line 89:		Line 91:

	This is a (micro)service design pattern - small, lightweight, standalone daemons act as instruments that continuously read whatever they read, available to be queried but otherwise not saving or doing anything with the data themselves. The data is handled by an application that queries each service it manages to collect data about those services (and coordinate if necessary).		This is a (micro)service design pattern - small, lightweight, standalone daemons act as instruments that continuously read whatever they read, available to be queried but otherwise not saving or doing anything with the data themselves. The data is handled by an application that queries each service it manages to collect data about those services (and coordinate if necessary).

			-->

	==Stage 3: Visualizing Data==		==Stage 3: Visualizing Data==

Admin: /* Phase 5: Grafana */

2018-02-28T06:39:53Z

Phase 5: Grafana

← Older revision		Revision as of 06:39, 28 February 2018
Line 100:		Line 100:
	Need to fix grafana user on jupiter.		Need to fix grafana user on jupiter.

	We're basically after something like this~~, except MongoDB instead of Prometheus~~: https://github.com/firehol/netdata/wiki/Netdata,-Prometheus,-and-Grafana-Stack		We're basically after something like this: https://github.com/firehol/netdata/wiki/Netdata,-Prometheus,-and-Grafana-Stack

	==Next Steps==		==Next Steps==

Admin: /* Phase 5: Grafana */

2018-02-28T06:39:36Z

Phase 5: Grafana

← Older revision		Revision as of 06:39, 28 February 2018
Line 96:		Line 96:
	[[Grafana]] container to create dashboards from it.		[[Grafana]] container to create dashboards from it.

	Link to Grafana docker files: https://charlesreid1.com:3000/docker/d-~~graphite~~		Link to Grafana docker files: https://charlesreid1.com:3000/docker/d-grafana

	Need to fix grafana user on jupiter.		Need to fix grafana user on jupiter.

Admin: /* Stage 2: Finalizing Data Collection */

2018-02-17T22:35:34Z

Stage 2: Finalizing Data Collection

← Older revision		Revision as of 22:35, 17 February 2018
Line 74:		Line 74:
	Ultimately, if we need to run collectd, interface with the collectd API via Python: https://collectd.org/wiki/index.php/Plugin:Python		Ultimately, if we need to run collectd, interface with the collectd API via Python: https://collectd.org/wiki/index.php/Plugin:Python

	==Stage 2: ~~Finalizing~~ Data Collection==		==Stage 2: Finalized Data Collection System==

	===Phase 4: Netdata and Mongo===		===Phase 4: Netdata and Mongo===

Admin: /* Phase 4: Netdata and Mongo */

2018-02-17T22:34:29Z

Phase 4: Netdata and Mongo

← Older revision		Revision as of 22:34, 17 February 2018
Line 76:		Line 76:
	==Stage 2: Finalizing Data Collection==		==Stage 2: Finalizing Data Collection==

	==Phase 4: Netdata and Mongo==		===Phase 4: Netdata and Mongo===

	Netdata provides a backend API that can be called to extract data from Netdata. MongoDB listens for API calls to store data in the database. All we need is software that will poll various Netdata instances using the Netdata API and dump that data into MongoDB. This gives much more fine-grained control over the process, schema, and storage format of the data.		Netdata provides a backend API that can be called to extract data from Netdata. MongoDB listens for API calls to store data in the database. All we need is software that will poll various Netdata instances using the Netdata API and dump that data into MongoDB. This gives much more fine-grained control over the process, schema, and storage format of the data.
Line 89:		Line 89:

	This is a (micro)service design pattern - small, lightweight, standalone daemons act as instruments that continuously read whatever they read, available to be queried but otherwise not saving or doing anything with the data themselves. The data is handled by an application that queries each service it manages to collect data about those services (and coordinate if necessary).		This is a (micro)service design pattern - small, lightweight, standalone daemons act as instruments that continuously read whatever they read, available to be queried but otherwise not saving or doing anything with the data themselves. The data is handled by an application that queries each service it manages to collect data about those services (and coordinate if necessary).


	==Stage 3: Visualizing Data==		==Stage 3: Visualizing Data==

Admin: /* Project Overview */

2018-02-17T22:34:05Z

Project Overview

← Older revision		Revision as of 22:34, 17 February 2018
Line 1:		Line 1:
	=Project Overview=		=Project Overview=

	The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards.		The 2018 data project is an ongoing effort to figure out how to set up "painless" dashboards for monitoring metrics.

	==Phase 1a: Netdata ~~(Done)~~==		==Stage 1: Collecting System Data==

			===COMPLETED Phase 1a: Netdata===

	First, we set up [[Netdata]].		First, we set up [[Netdata]].
Line 13:		Line 15:
	Netdata is a useful tool for monitoring an individual machine instance remotely and it works excellent.		Netdata is a useful tool for monitoring an individual machine instance remotely and it works excellent.

	==Phase 1b: Prometheus ~~(Nope)~~==		===NOPE Phase 1b: Prometheus===

	<s>Second, we set up [[Netdata]] to dump to a [[Prometheus]] database.		<s>Second, we set up [[Netdata]] to dump to a [[Prometheus]] database.
Line 22:		Line 24:
	Need to get more involved with Prometheus and/or Grafana to monitor more than one machine.</s>		Need to get more involved with Prometheus and/or Grafana to monitor more than one machine.</s>

	==Phase 2: MongoDB and MongoExpress ~~(Done)~~==		===COMPLETED Phase 2a: MongoDB and MongoExpress===

	We then set up [[MongoDB]] and [[MongoExpress]] in Docker containers. MongoDB listens for incoming data on the VPN. MongoExpress is connected to MongoDB and exposes a web interface to interact with MongoDB. We used MongoDB to store edit history and page graph data from the charlesreid1 wiki.		We then set up [[MongoDB]] and [[MongoExpress]] in Docker containers. MongoDB listens for incoming data on the VPN. MongoExpress is connected to MongoDB and exposes a web interface to interact with MongoDB. We used MongoDB to store edit history and page graph data from the charlesreid1 wiki.

			Pros and cons:
	* '''Pros:''' MongoDB is a containerized solution with persistent data. MongoDB had (has?) a high setup barrier, but a low usage barrier. Very easy to do basic CRUD operations, make new databases as needed, etc.		* '''Pros:''' MongoDB is a containerized solution with persistent data. MongoDB had (has?) a high setup barrier, but a low usage barrier. Very easy to do basic CRUD operations, make new databases as needed, etc.
	* '''Cons:''' No visualization tools baked in, need to define own tools. Collectd cannot dump to MongoDB because of a bunch of installation stupidity.		* '''Cons:''' No visualization tools baked in, need to define own tools. Collectd cannot dump to MongoDB because of a bunch of installation stupidity.

			Links:
	* [[MongoDB]]		* [[MongoDB]]
	* [[MongoExpress]]		* [[MongoExpress]]
Line 34:		Line 40:
	* Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data		* Link to wiki scraping scripts: https://charlesreid1.com:3000/wiki/charlesreid1-wiki-data

	==Phase 2b: Collectd ~~(Nope)~~==		===NOPE Phase 2b: Collectd===

	<s>We struggled a LOT with [[Collectd]], mainly because we wanted to use the collectd plugin to write to MongoDB. Unfortunately, this was the only plugin that seemed impossible to install.		<s>We struggled a LOT with [[Collectd]], mainly because we wanted to use the collectd plugin to write to MongoDB. Unfortunately, this was the only plugin that seemed impossible to install.
Line 42:		Line 48:
	(This is all installation stupidity. I tried installing collectd with aptitude, no plugins. Then the core, no plugins. Then installing from source, and MongoDB plugin did not work. Struggling to get collectd to link to MongoDB. Needed custom config or something. Then I just gave up, and re-installed collectd core, and the library was there, but it was complaining it couldn't find it. In the end, I totally abandoned the attempt to get collectd to talk to mongodb. Could probably use a collectd docker and fix this whole issue.)</s>		(This is all installation stupidity. I tried installing collectd with aptitude, no plugins. Then the core, no plugins. Then installing from source, and MongoDB plugin did not work. Struggling to get collectd to link to MongoDB. Needed custom config or something. Then I just gave up, and re-installed collectd core, and the library was there, but it was complaining it couldn't find it. In the end, I totally abandoned the attempt to get collectd to talk to mongodb. Could probably use a collectd docker and fix this whole issue.)</s>

	==Phase 3: Graphite ~~(Nope)~~==		===NOPE Phase 3a: Graphite===

	<s>Next, we deployed a [[Graphite]] container to hold time series from [[Collectd]].		<s>Next, we deployed a [[Graphite]] container to hold time series from [[Collectd]].

			Pros and cons:
	* '''Pros:''' Containerized solution. Collectd graphite plugin worked fine.		* '''Pros:''' Containerized solution. Collectd graphite plugin worked fine.
	* '''Cons:''' Graphite comes with Carbon (web interface), which is utter awful. It provides the absolute bare minimum, and it looks like it's trapped in a miserable 1998 computer prison.		* '''Cons:''' Graphite comes with Carbon (web interface), which is utter awful. It provides the absolute bare minimum, and it looks like it's trapped in a miserable 1998 computer prison.

			Links:
	* [[Graphite]]		* [[Graphite]]
	* Link to Graphite docker files: https://charlesreid1.com:3000/docker/d-graphite</s>		* Link to Graphite docker files: https://charlesreid1.com:3000/docker/d-graphite</s>

	==Phase 3b: Visualizing Graphite ~~(Nope)~~==		===NOPE Phase 3b: Visualizing Graphite===

	<s>A few years back we explored [[Cubism]] and Cube (difference?) as a way of visualizing time series from Graphite. It took some effort to get a basic dashboard, and Cubism is (ultimately) D3, the most frustratingly stupidly over-designed and over-complicated library ever, implemented in a totally irrational programming language.		<s>A few years back we explored [[Cubism]] and Cube (difference?) as a way of visualizing time series from Graphite. It took some effort to get a basic dashboard, and Cubism is (ultimately) D3, the most frustratingly stupidly over-designed and over-complicated library ever, implemented in a totally irrational programming language.
Line 58:		Line 68:
	We're going to focus on Mongo, which is more transparent and more flexible for all purposes.</s>		We're going to focus on Mongo, which is more transparent and more flexible for all purposes.</s>

	==Conclusion: Netdata, Not Collectd==		===Stage 1 Conclusion: Netdata, Not Collectd===

	All of the struggle to get collectd working with mongo was a waste of effort, and led to the graphite distraction in the first place. A broken build procedure (collectd) led to an unknown, mediocre tool (graphite).		All of the struggle to get collectd working with mongo was a waste of effort, and led to the graphite distraction in the first place. A broken build procedure (collectd) led to an unknown, mediocre tool (graphite).

	Ultimately, if we need to run collectd, interface with the collectd API via Python: https://collectd.org/wiki/index.php/Plugin:Python		Ultimately, if we need to run collectd, interface with the collectd API via Python: https://collectd.org/wiki/index.php/Plugin:Python

			==Stage 2: Finalizing Data Collection==

	==Phase 4: Netdata and Mongo==		==Phase 4: Netdata and Mongo==
Line 78:		Line 90:
	This is a (micro)service design pattern - small, lightweight, standalone daemons act as instruments that continuously read whatever they read, available to be queried but otherwise not saving or doing anything with the data themselves. The data is handled by an application that queries each service it manages to collect data about those services (and coordinate if necessary).		This is a (micro)service design pattern - small, lightweight, standalone daemons act as instruments that continuously read whatever they read, available to be queried but otherwise not saving or doing anything with the data themselves. The data is handled by an application that queries each service it manages to collect data about those services (and coordinate if necessary).

	==Phase 5: Grafana==
			==Stage 3: Visualizing Data==

			===Phase 5: Grafana===

	[[Grafana]] container to create dashboards from it.		[[Grafana]] container to create dashboards from it.