2018/January/Data Engineering: Difference between revisions

Latest revision as of 03:09, 9 October 2019

notes from January 2018 data engineering work.

this consists of review (rebooting the data engineering stuff) and coding (identifying relevant scenarios for data engineering scenarios).

Expanding data-engineering-scenarios:

Start with ready examples
Work toward synthetic experimental data
An imaginary factory... lots of widgets... Kubernetes/container engine... orchestrating a process
Focus on a particular process or set of processes, and drill into it, use it to provide multiple angles on a single concept

Software tools list, (abstract) example for each: Google Cloud

Notes review: GCDEC

Google Codelabs:

Google Qwiklabs:

@@ Line 17: / Line 17: @@
 ===procedure===
-Expanding data-engineering-scenarios
+Expanding data-engineering-scenarios:
+* Start with ready examples
-Start with pre-made examples
+* Work toward synthetic experimental data
+* An imaginary factory... lots of widgets... Kubernetes/container engine... orchestrating a process
-Work toward fabricated experimental data
+* Focus on a particular process or set of processes, and drill into it, use it to provide multiple angles on a single concept
-An imaginary factory... lots of widgets... Kubernetes/container engine... orchestrating a process
-Focus on a particular process or set of processes, and drill into it, use it to provide multiple angles on a single concept
 Software tools list, (abstract) example for each: [[Google Cloud]]
 * Storage/database/computation/GPUs vs CPUs/containerization
-Software quality assurance: https://charlesreid1.com:3000/charlesreid1/scientific-software
+Software quality assurance: https://git.charlesreid1.com/charlesreid1/scientific-software
+* [[10 Best]]
 * More informal
 * Bullet points - things I've learned