Netdata/MongoDB/API - Revision history

Admin: Replacing charlesreid1.com:3000 with git.charlesreid1.com

2019-10-09T03:41:46Z

Replacing charlesreid1.com:3000 with git.charlesreid1.com

← Older revision		Revision as of 03:41, 9 October 2019
Line 163:		Line 163:
	===Recipe Code===		===Recipe Code===

	Link: https://charlesreid1.com~~:3000~~/data/netdata/src/master/netdata_mongo.py		Link: https://git.charlesreid1.com/data/netdata/src/master/netdata_mongo.py

	<pre>		<pre>

Admin: /* Recipe */

2018-02-11T21:02:39Z

Recipe

← Older revision		Revision as of 21:02, 11 February 2018
Line 159:		Line 159:

	sleep for N seconds		sleep for N seconds
			</pre>

			===Recipe Code===

			Link: https://charlesreid1.com:3000/data/netdata/src/master/netdata_mongo.py

			<pre>

			from datetime import datetime
			import pymongo, time, requests, json

			"""
			Netdata Mongo

			This script requests data from the Netdata API in JSON format,
			parses the result, and stores it in a MongoDB database.
			"""

			db_name = 'netdata'
			collection_name = 'jupiter'

			# MongoDB
			client = pymongo.MongoClient('10.6.0.1', 27017)
			db = client[db_name]

			while(True):

			# Netdata
			my_url = 'http://10.6.0.1:19999/api/v1/allmetrics?format=json'
			r = requests.get(url=my_url)
			d = r.json()

			collection = db[collection_name]

			# Use each key in the original Netdata bundle
			# as a "key prefix"
			to_insert = {}
			for key in d.keys():

			data = d[key]

			prefix = data['name']
			timestamp = datetime.fromtimestamp(data['last_updated'])

			# values_data is list of dictionaries: key is label, value is numerical value
			values_data = list(data['dimensions'].values())

			# Assemble final key-value pair going into MongoDB
			for v in values_data:
			this_label = prefix + "." + v['name']
			this_value = v['value']
			to_insert[this_label] = this_value
			to_insert['timestamp'] = timestamp

			# Insert into MongoDB
			try:
			collection.insert(to_insert, check_keys=False)
			except pymongo.errors.OperationFailure as e:
			print(e.code)
			print(e.details)

			print("Inserted document into collection.")
			time.sleep(10)

	</pre>		</pre>

Admin: /* Recipe */

2018-02-11T18:22:28Z

Recipe

← Older revision		Revision as of 18:22, 11 February 2018
Line 145:		Line 145:
	<pre>		<pre>
	create mongodb database connection		create mongodb database connection
			create mongodb collection

	~~create mongodb collection named netstat~~		while true (forever loop):

	~~netdata_data_dict =~~ get netdata data ~~dictionary~~		get netdata data in json format from url

	for key in ~~netstat_data_dict.~~keys:		for key in json keys:
	~~data = netstat_data_dict[~~key]		get timestamp
	~~create~~ collection ~~table with name~~ key		get dimensions sub-dictionary
	~~insert d into table~~		for keys in sub-dictionary:
			create key, value pair
			collection.insert( all key value pairs, plus timestamp )

			sleep for N seconds
	</pre>		</pre>

Admin: /* Netdata Storage Schema */

2018-02-11T18:19:41Z

Netdata Storage Schema

← Older revision		Revision as of 18:19, 11 February 2018
Line 86:		Line 86:
	===Netdata Storage Schema===		===Netdata Storage Schema===

	~~One option to store all of this data is to create a single table to hold all of the data obtained from Netdata. This would look like~~ a ~~table with 248 columns, with each row~~ of ~~the table corresponding to a single request that went~~ to Netdata ~~for which~~ data ~~was returned, and each cell of the table holds~~ a ~~large dictionary~~.		There are a couple of approaches to storing Netdata data in a database.

	~~An alternative approach~~ is ~~to use the keys~~ to create ~~tables. Each key creates~~ a ~~new~~ table, with the nested dictionary ~~creating a table.~~		The first option is to create a single monolithic netdata table, with 248 columns corresponding to the 248 measurement groups, and each cell of the table containing a large nested dictionary:

	<pre>		<pre>
Line 107:		Line 107:
	'name': 'cpu.cpu1',		'name': 'cpu.cpu1',
	'units': 'percentage'}		'units': 'percentage'}
			</pre>

			Another approach is to use each of the keys to create a table (resulting in 248 tables). In this case each key in the sub-dictionary would become a column in a table.

			However, the approach we used is a combination of these.

			Notice that the "dimensions" key always maps to a sub-dictionary containing the actual numerical values. We can throw out everything except this dimensions dictionary, and extract each value in the dictionary to create a series.

			For example, in the dictionary above we have several time series that would result:

			* cpu.cpu1.guest
			* cpu.cpu1.guest_nice
			* cpu.cpu1.idle
			* cpu.cpu1.iowait
			* cpu.cpu1.irq
			* etc.

			Here is another example:

			<pre>
	In [9]: d['ipv4.sockstat_tcp_sockets']		In [9]: d['ipv4.sockstat_tcp_sockets']
	Out[9]:		Out[9]:

Admin: /* Recipe */

2018-02-11T18:07:22Z

Recipe

← Older revision		Revision as of 18:07, 11 February 2018
Line 125:		Line 125:

	<pre>		<pre>
			create mongodb database connection

	create mongodb collection named netstat		create mongodb collection named netstat

			netdata_data_dict = get netdata data dictionary

	for key in netstat_data_dict.keys:		for key in netstat_data_dict.keys:
	d = netstat_data_dict[key]		data = netstat_data_dict[key]
	create collection table with name key		create collection table with name key
	insert d into table		insert d into table

Admin: /* Flags */

2018-02-11T04:58:14Z

Flags

← Older revision		Revision as of 04:58, 11 February 2018
Line 132:		Line 132:
	</pre>		</pre>

	=Flags=		==Flags==

	[[Category:MongoDB]]		[[Category:MongoDB]]

Admin: /* Notes */

2018-02-11T04:58:06Z

Notes

← Older revision		Revision as of 04:58, 11 February 2018
Line 1:		Line 1:
	The Netdata url schema exposes all metrics being measured by Netdata as a JSON-exportable REST url.		The Netdata url schema exposes all metrics being measured by Netdata as a JSON-exportable REST url.

	~~=Notes=~~

	==Querying Netdata API with Python==		==Querying Netdata API with Python==

Admin: /* Parsing Netdata Output */

2018-02-11T04:57:47Z

Parsing Netdata Output

@@ Line 59: / Line 59: @@
 ==Parsing Netdata Output==
 <pre style="white-space: pre-wrap;">
@@ Line 81: / Line 85: @@
 </pre>
-If we insert this dictionary directly into the MongoDB database, this list of keys will become the columns, and each row will become an observation. These observations are nested JSON objects/dictionaries themselves. Here's what one (example) observation looks like:
+===Netdata Storage Schema===
 <pre>
@@ Line 111: / Line 119: @@
   'name': 'ipv4.sockstat_tcp_sockets',
   'units': 'sockets'}
 </pre>

Admin: /* Parsing Netdata Output */

2018-02-11T04:42:57Z

Parsing Netdata Output

← Older revision		Revision as of 04:42, 11 February 2018
Line 60:		Line 60:
	==Parsing Netdata Output==		==Parsing Netdata Output==

	<pre>		<pre style="white-space: pre-wrap;">
	In [1]: import requests, json		In [1]: import requests, json

Admin: /* Querying Netdata API with Python */

2018-02-11T04:42:15Z

Querying Netdata API with Python

← Older revision		Revision as of 04:42, 11 February 2018
Line 58:		Line 58:
	</pre>		</pre>

			==Parsing Netdata Output==

			<pre>
			In [1]: import requests, json

			In [2]: my_url = 'http://10.6.0.1:19999/api/v1/allmetrics?format=json&help=yes'
			...:

			In [3]: r = requests.get(url=my_url)

			In [4]: with open('output.json','w') as f:
			...: json.dump( r.json(), f, indent=4 )
			...:

			In [5]: d = r.json()

			In [6]: print(len(d.keys()))
			248

			In [7]: print(d.keys())
			dict_keys(['ipv4.tcpofo', 'cgroup_happy_mongo.merged_ops', 'cgroup_mex.merged_ops', 'cgroup_mex.throttle_serviced_ops', 'cgroup_mex.throttle_io', 'cgroup_mex.net_packets_eth0', 'cgroup_mex.serviced_ops', 'cgroup_mex.net_eth0', 'cgroup_mex.io', 'cgroup_mex.mem_usage', 'cgroup_mex.pgfaults', 'cgroup_mex.mem_activity', 'cgroup_mex.writeback', 'cgroup_mex.mem', 'cgroup_mex.cpu_per_core', 'cgroup_mex.cpu', 'cgroup_happy_mongo.queued_ops', 'cgroup_happy_mongo.throttle_serviced_ops', 'cgroup_happy_mongo.throttle_io', 'cgroup_happy_mongo.serviced_ops', 'cgroup_happy_mongo.io', 'cgroup_happy_mongo.mem_usage', 'cgroup_happy_mongo.pgfaults', 'cgroup_happy_mongo.mem_activity', 'cgroup_happy_mongo.net_packets_eth0', 'cgroup_happy_mongo.writeback', 'cgroup_happy_mongo.net_eth0', 'cgroup_happy_mongo.mem', 'cgroup_happy_mongo.cpu_per_core', 'cgroup_happy_mongo.cpu', 'ipv4.sockstat_tcp_mem', 'net_packets.docker0', 'net.docker0', 'sensors.coretemp-isa-0000_temperature', 'cpu.cpu1_cpuidle', 'cpu.cpu0_cpuidle', 'cpu.cpufreq', 'netdata.runtime_sensors', 'netdata.runtime_cpuidle', 'netdata.runtime_cpufreq', 'disk_svctm.dm-1', 'disk_avgsz.dm-1', 'disk_await.dm-1', 'disk_svctm.dm-0', 'disk_avgsz.dm-0', 'disk_await.dm-0', 'disk_svctm.sda', 'disk_avgsz.sda', 'disk_await.sda', 'groups.pipes', 'groups.sockets', 'groups.files', 'netdata.compression_ratio', 'netdata.response_time', 'groups.lwrites', 'groups.lreads', 'netdata.net', 'groups.pwrites', 'netdata.requests', 'netdata.clients', 'netdata.server_cpu', 'netdata.plugin_proc_cpu', 'groups.preads', 'groups.minor_faults', 'netdata.plugin_proc_modules', 'system.ipc_semaphore_arrays', 'system.ipc_semaphores', 'groups.major_faults', 'system.io', 'disk_iotime.dm-1', 'groups.cpu_system', 'disk_util.dm-1', 'groups.cpu_user', 'disk_backlog.dm-1', 'disk_ops.dm-1', 'disk.dm-1', 'disk_iotime.dm-0', 'groups.processes', 'disk_util.dm-0', 'disk_backlog.dm-0', 'groups.threads', 'disk_qops.dm-0', 'disk_ops.dm-0', 'disk.dm-0', 'disk_iotime.sda', 'groups.vmem', 'disk_mops.sda', 'disk_util.sda', 'groups.mem', 'disk_backlog.sda', 'groups.cpu', 'disk_qops.sda', 'disk_ops.sda', 'disk.sda', 'netfilter.conntrack_sockets', 'cpu.cpu1_softnet_stat', 'users.pipes', 'cpu.cpu0_softnet_stat', 'system.softnet_stat', 'users.sockets', 'ipv6.ect', 'users.files', 'ipv6.icmptypes', 'ipv6.icmpmldv2', 'ipv6.icmpneighbor', 'ipv6.icmprouter', 'users.lwrites', 'users.lreads', 'ipv6.icmperrors', 'ipv6.icmp', 'users.pwrites', 'ipv6.mcastpkts', 'ipv6.mcast', 'users.preads', 'users.minor_faults', 'users.major_faults', 'ipv6.udperrors', 'ipv6.udppackets', 'users.cpu_system', 'ipv6.packets', 'system.ipv6', 'users.cpu_user', 'ipv4.udplite_errors', 'ipv4.udplite', 'users.processes', 'users.threads', 'ipv4.udperrors', 'users.vmem', 'ipv4.udppackets', 'ipv4.tcphandshake', 'users.mem', 'ipv4.tcpopens', 'users.cpu', 'ipv4.tcperrors', 'ipv4.tcppackets', 'ipv4.tcpsock', 'apps.pipes', 'apps.sockets', 'ipv4.icmpmsg', 'ipv4.icmp_errors', 'ipv4.icmp', 'ipv4.errors', 'ipv4.fragsin', 'ipv4.fragsout', 'apps.files', 'ipv4.packets', 'ipv4.ecnpkts', 'ipv4.bcastpkts', 'ipv4.mcastpkts', 'apps.lwrites', 'ipv4.bcast', 'ipv4.mcast', 'system.ipv4', 'ipv6.sockstat6_raw_sockets', 'ipv6.sockstat6_udp_sockets', 'ipv6.sockstat6_tcp_sockets', 'ipv4.sockstat_udp_mem', 'ipv4.sockstat_udp_sockets', 'ipv4.sockstat_tcp_sockets', 'apps.lreads', 'ipv4.sockstat_sockets', 'system.net', 'net_packets.wlx7cdd906c3ef0', 'net.wlx7cdd906c3ef0', 'net_packets.master', 'net.master', 'mem.slab', 'mem.kernel', 'apps.pwrites', 'mem.writeback', 'mem.committed', 'system.swap', 'apps.preads', 'mem.available', 'system.ram', 'mem.pgfaults', 'system.pgpgio', 'apps.minor_faults', 'cpu.cpu1_softirqs', 'cpu.cpu0_softirqs', 'apps.major_faults', 'system.softirqs', 'apps.cpu_system', 'cpu.cpu1_interrupts', 'apps.cpu_user', 'apps.processes', 'cpu.cpu0_interrupts', 'apps.threads', 'services.merged_io_ops_write', 'services.merged_io_ops_read', 'services.queued_io_ops_write', 'services.queued_io_ops_read', 'services.throttle_io_ops_write', 'services.throttle_io_ops_read', 'services.throttle_io_write', 'services.throttle_io_read', 'services.io_ops_write', 'services.io_ops_read', 'services.io_write', 'services.io_read', 'services.mem_usage', 'services.cpu', 'netdata.plugin_diskspace_dt', 'netdata.plugin_diskspace', 'system.interrupts', 'apps.vmem', 'system.entropy', 'disk_inodes._boot', 'system.active_processes', 'disk_space._boot', 'system.load', 'disk_inodes._run_lock', 'system.uptime', 'disk_space._run_lock', 'apps.mem', 'cpu.core_throttling', 'disk_inodes._dev_shm', 'system.processes', 'system.forks', 'system.ctxt', 'disk_space._dev_shm', 'system.intr', 'netdata.private_charts', 'disk_inodes._', 'apps.cpu', 'netdata.tcp_connected', 'disk_space._', 'netdata.apps_children_fix', 'cpu.cpu1', 'netdata.tcp_connects', 'disk_inodes._run', 'netdata.plugin_tc_time', 'netdata.apps_fix', 'netdata.statsd_packets', 'disk_space._run', 'cpu.cpu0', 'netdata.plugin_tc_cpu', 'netdata.statsd_bytes', 'netdata.apps_sizes', 'disk_inodes._dev', 'netdata.statsd_reads', 'netdata.apps_cpu', 'disk_space._dev', 'system.cpu', 'netdata.plugin_cgroups_cpu', 'netdata.statsd_events', 'netdata.statsd_metrics', 'system.idlejitter'])
			</pre>

			If we insert this dictionary directly into the MongoDB database, this list of keys will become the columns, and each row will become an observation. These observations are nested JSON objects/dictionaries themselves. Here's what one (example) observation looks like:

			<pre>
			In [8]: d['cpu.cpu1']
			Out[8]:
			{'context': 'cpu.cpu',
			'dimensions': {'guest': {'name': 'guest', 'value': 0.0},
			'guest_nice': {'name': 'guest_nice', 'value': 0.0},
			'idle': {'name': 'idle', 'value': 98.989899},
			'iowait': {'name': 'iowait', 'value': 0.0},
			'irq': {'name': 'irq', 'value': 0.0},
			'nice': {'name': 'nice', 'value': 0.0},
			'softirq': {'name': 'softirq', 'value': 0.0},
			'steal': {'name': 'steal', 'value': 0.0},
			'system': {'name': 'system', 'value': 1.010101},
			'user': {'name': 'user', 'value': 0.0}},
			'last_updated': 1518323931,
			'name': 'cpu.cpu1',
			'units': 'percentage'}

			In [9]: d['ipv4.sockstat_tcp_sockets']
			Out[9]:
			{'context': 'ipv4.sockstat_tcp_sockets',
			'dimensions': {'alloc': {'name': 'alloc', 'value': 26.0},
			'inuse': {'name': 'inuse', 'value': 11.0},
			'orphan': {'name': 'orphan', 'value': 0.0},
			'timewait': {'name': 'timewait', 'value': 0.0}},
			'last_updated': 1518323931,
			'name': 'ipv4.sockstat_tcp_sockets',
			'units': 'sockets'}
			</pre>

	=Flags=		=Flags=