Data Engineering - Revision history

Unknown user: /* Apache */

2025-05-26T17:54:31Z

Apache

2025-05-26T17:54:21Z

GCDEC

← Older revision		Revision as of 17:54, 26 May 2025
Line 268:		Line 268:
	Working through the Google Cloud Data Engineer certification course... See [[GCDEC]] for pages related to that.		Working through the Google Cloud Data Engineer certification course... See [[GCDEC]] for pages related to that.

			=Apache=

			Apache software projects related to Data Engineering:

			* Avro
			* [[DataFusion]]

	=Flags=		=Flags=

	[[Category:Data Engineering]]		[[Category:Data Engineering]]

2017-12-15T04:19:31Z

Data Engineering Scenarios

← Older revision		Revision as of 04:19, 15 December 2017
Line 114:		Line 114:

	These scenarios focus on different technologies available via Google Cloud or Amazon Web Services. Roughly, they can be grouped as follows:		These scenarios focus on different technologies available via Google Cloud or Amazon Web Services. Roughly, they can be grouped as follows:

			==Compute Engine==

			An approach to cloud infrastructure that provides a greater degree of freedom, but requires more complicated configuration. Compute Engine gives you virtual machines that start as bare metal, so you have to build/install any software you need. This can be a pain but also gives you greater control.

			Also see Container Engine section below.

			==Container Engine==

			The Google Cloud container engine basically provides a version control system on Docker images, which can then be pushed and pulled onto nodes in the cloud. This allows you to scale a single container image to deploy many instances, as needed.

			Also see [[Docker]].

	==Dataproc==		==Dataproc==

2017-10-20T08:21:25Z

Data Engineering Example: Twitter

2017-10-20T08:21:00Z

Timelines

@@ Line 178: / Line 178: @@
 * What to do on a cache miss?
 * Merge the user timelines
 =Data Engineering Scenarios=

2017-10-20T04:56:26Z

Timelines

← Older revision		Revision as of 04:56, 20 October 2017
Line 162:		Line 162:
	* Space-based		* Space-based

	Primitive approach: SQL query		Primitive approach: SQL query:
	* ~~<code>SELECT * FROM tweets WHERE user_id IN~~ (~~SELECT source_id FROM followers WHERE destination_id = ?~~) ~~ORDER BY created_at DESC LIMIT 20</code>~~		* Use the following SQL query (below)
	* Bingo, you have your subquery of followers who are tweeting at anyone, passed into the search for tweets		* Bingo, you have your subquery of followers who are tweeting at anyone, passed into the search for tweets
	* BUT - what happens if you have lots of friends, or if the number of source tweet IDs cant fit in RAM?		* BUT - what happens if you have lots of friends, or if the number of source tweet IDs cant fit in RAM?

			<pre>
			SELECT * FROM tweets WHERE user_id IN (SELECT source_id FROM followers WHERE destination_id = ?) ORDER BY created_at DESC LIMIT 20
			</pre>

			offline vs online computation:
			* Sequences can be stored in memcached (individual timelines)
			* You pass a status to Fanout
			* Fanout is offline, but has a low-latency SLlA
			* Truncate at random intervals, ensuring bounded length
			* What to do on a cache miss?
			* Merge the user timelines

	=Data Engineering Scenarios=		=Data Engineering Scenarios=

2017-10-20T04:54:12Z

Tweets=

2017-10-20T04:54:03Z

Data Engineering Example: Twitter

@@ Line 118: / Line 118: @@
 * Social graphs
 * Search indices
 Definitions of data:
@@ Line 150: / Line 152: @@
 * primary key partition (tweet ID)
 * but also, secondary key on user ID
 =Data Engineering Scenarios=

2017-10-20T04:38:35Z

Data Engineering Scenarios

2017-10-13T22:27:38Z

Neural Network Machine Learning Scenarios

← Older revision		Revision as of 22:27, 13 October 2017
Line 139:		Line 139:

	Scenario ideas:		Scenario ideas:
	* Pre-trained image recognition model, wrap front-end with graphs to show data, objects detected, etc.		* Pre-trained image recognition model, prediction of type of object, wrap front-end with graphs to show data, objects detected, etc.
	* Trained face differences, upload two faces, give prediction.		* Trained face differences, upload two faces, give prediction.