GCDEC/Fundamentals/Notes - Revision history

Admin at 08:07, 9 October 2017

2017-10-09T08:07:06Z

← Older revision		Revision as of 08:07, 9 October 2017
Line 753:		Line 753:


			=Flags=

	[[Category:Google Cloud]]		[[Category:Google Cloud]]
	[[Category:Data Engineering]]		[[Category:Data Engineering]]

Admin: /* Module 3 */

2017-09-24T03:04:42Z

Module 3

Show changes

Admin: /* Cloud Shell */

2017-09-24T02:59:06Z

Cloud Shell

← Older revision		Revision as of 02:59, 24 September 2017
Line 428:		Line 428:

	Can use this to launch serverless operations, requisition nodes, perform gsutil tasks, etc.		Can use this to launch serverless operations, requisition nodes, perform gsutil tasks, etc.


			==Module 3==

			==Module 4==

			==Module 5==

	=Resources=		=Resources=

Admin: /* Resources */

2017-09-24T02:58:43Z

Resources

← Older revision		Revision as of 02:58, 24 September 2017
Line 431:		Line 431:
	=Resources=		=Resources=

	==Module 1==		==Module 1 Resources==

	Code labs for this course: https://codelabs.developers.google.com/cpb100		Code labs for this course: https://codelabs.developers.google.com/cpb100
Line 439:		Line 439:
	Whitepaper on Google's security practices (i.e., why you can trust Google to handle your cloud stuff): https://cloud.google.com/security/whitepaper		Whitepaper on Google's security practices (i.e., why you can trust Google to handle your cloud stuff): https://cloud.google.com/security/whitepaper

	==Module 2==		==Module 2 Resources==

	Compute engine: https://cloud.google.com/compute		Compute engine: https://cloud.google.com/compute

Admin: /* Module 1 */

2017-09-24T02:58:17Z

Module 1

← Older revision		Revision as of 02:58, 24 September 2017
Line 438:		Line 438:

	Whitepaper on Google's security practices (i.e., why you can trust Google to handle your cloud stuff): https://cloud.google.com/security/whitepaper		Whitepaper on Google's security practices (i.e., why you can trust Google to handle your cloud stuff): https://cloud.google.com/security/whitepaper

			==Module 2==

			Compute engine: https://cloud.google.com/compute

			Storage: https://cloud.google.com/storage

			Pricing calculator: https://cloud.google.com/pricing

			Cloud launcher: https://cloud.google.com/launcher

			YouTube video on Compute Instance vs Container Engine vs App Engine vs Cloud Functions: https://www.youtube.com/watch?v=g0dN8Hkh5H8

			Cloud launcher:
			* Shortcut for getting a compute engine VM with preconfigured software ready to go
			* Google click to deploy is a Google-maintained VM image with the software already installed and ready to go

Admin: /* Zones and Regions */

2017-09-24T02:57:32Z

Zones and Regions

← Older revision		Revision as of 02:57, 24 September 2017
Line 342:		Line 342:
	If global access important, can distribute apps across multiple regions		If global access important, can distribute apps across multiple regions

			===Lab: Interacting with Cloud Storage===

			Running the lab "Interact with Cloud Storage"


			Multi-step process:
			* Ingesting data into a compute engine instance
			* Transforming data on the compute engine instance
			* Storing data in persistent cloud storage
			* Publishing data to the web via cloud storage


			Use git to clone repository with instructions/data/scripts

			The behind-the-scenes procedure is as follows:
			* download earthquake data using wget (ingest.sh)
			* install extra Python goodies (install_extras.sh)
			* transform data using a Python script (creates a basemap projection, then plots lat/long locations of earthquakes using dots scaled to magnitude of earthquake, colors indicate "class" of magnitudes 1-3, 3-5, 5+) (transform.py)

			This results in an image file. There is already an HTML file that will serve up the image on a web page.

			Now back to the GCP console - create a storage bucket by going to storage in LHS menu.

			Create bucket, and pick zone/region.

			Called it mah-bukkit

			Now it takes you to an interface where you can actually upload files directly from the browser (looks almost like Dropbox).

			Finally, we can use gsutil to copy the image file and HTML file to the bucket. (gsutil is automatically installed on all Google Cloud compute instances.)

			<pre>
			$ gsutil cp earthquakes* gs://mah-bukkit/
			Copying file://earthquakes.csv [Content-Type=text/csv]...
			AccessDeniedException: 403 Insufficient OAuth2 scope to perform this operation.
			Acceptable scopes: https://www.googleapis.com/auth/cloud-platform
			</pre>

			Turns out, when you create your virtual machine instance, you have to specify permissions for each API. This was something I left as default originally.

			The first option is to set the service account - this is what allows you to control access to different buckets for different people.

			The second options relates to API access. I left it as "default access," which does not allow the compute instance to access very many APIs. I changed this to "Allow full access to all Cloud APIs" (can also set access for each API individually).

			Note that this can't be changed while running, you have to shut down the instance to change the API access for a compute instance.

			Note that this may still fail with the same 403 mentioned above. If so, it's because gsutil is using crusty credentials. Reset them via:

			<pre>
			$ rm -rf ~/.gsutil
			</pre>

			then try again:

			<pre>
			$ gsutil cp earthquakes* gs://mah-bukkit/
			Copying file://earthquakes.csv [Content-Type=text/csv]...
			Copying file://earthquakes.htm [Content-Type=text/html]...
			Copying file://earthquakes.png [Content-Type=image/png]...
			- [3 files][660.4 KiB/660.4 KiB]
			Operation completed over 3 objects/660.4 KiB.
			</pre>

			Now go to the cloud storage console, click the bucket, check "share publicly", and get the link.

			https://storage.googleapis.com/mah-bukkit/earthquakes.htm

			Bingo!


			===Cloud Shell===

			Because what we were doing here was relatively simple, and just involved shuffling some scripts around and copying data to cloud storage, it is overkill to allocate an entire compute instance to do that, and have to wait for it to start up and shut down, etc.

			Instead, we could use the cloud shell - this is a serverless instance that can be used to do minor tasks.

			Here's how this works: this is like a head node on a cluster, where you get "free" cycles to do minor tasks. Here's what you get:
			* MicroVM
			* Single 2.2 GHz Intel Xeon CPU
			* 5 GB persistent storage in your home directory (place to save files) - stuff is already present! You get your "cloud home directory" (commonly-used scripts, repos, code, etc.)
			* Access to basic tools like gsutil, cloud/app engine sdks, docker, git, build tools, etc.
			* Access to languages: Python, Java, Go, and Node

			The shell works like a Lish shell, opens within a split screen in the browser.

			Can use this to launch serverless operations, requisition nodes, perform gsutil tasks, etc.

	=Resources=		=Resources=

Admin: /* Resources */

2017-09-24T02:54:29Z

Resources

← Older revision		Revision as of 02:54, 24 September 2017
Line 293:		Line 293:

	Everything worked like a charm.		Everything worked like a charm.

			===Global Filesystem===

			We're talking about using data in Cloud SQL, BigQuery, and Dataproc (Google's versions of large scale big data stuff like Hadoop, Hive, Sqoop, Pig, etc etc)

			We want to get data from "out there in the world" into the cloud. The problem is, when you allocate a compute engine, you allocate a disk associated with that compute engine - and when the compute engine goes away, the disk goes away too. Plus, persistent disk space is expensive anyway.

			Instead, store your data in "Cloud Storage". This stores raw data and stages it for other products. This storage is durable, persistent storage that can be easily replicated across other nodes and utilized in other GCP products (Cloud SQL, BigQuery, Dataproc).

			Your first step in the journey of doing big data in the cloud is to get your data into cloud storage. To do that, use gsutil.

			===Interacting with Cloud Storage===

			{{Main\|Gsutil}}

			Simplest way is to use command line utility called gsutil to interact with Cloud Storage. (It's a command line utility, install it using Google-provided installer.)

			Note: you can also use programming language, GCP Console, or REST API.

			Now, can use gsutil command line and utilities like cp, rm, mv, ls, mb, rb, rsync, acl...

			To copy to Google Storage (GS), run a command like:

			$ gsutil cp sales*.csv gs://acme-sales/data/

			This copies data into GS buckets. The folder structure is purely convenience.

			Buckets are like a domain name in your GCP project. Bucket name must be unique. Typically related to business/company domain name (GCP will require you to prove you own the domain name). Or, you can use a unique "dummy" bucket name.

			This will be a recurring pattern: anything you can do from gsutil command line just invokes a REST API. Anything that can be done with a REST API can also be done from any language that speaks HTTP (just about any).

			===Data Handling===

			Transfer services: useful for ingestion of data from data center, local system, AWS buckets, other sources. Can be one-time or recurring.

			Cloud storage as staging area: useful for importing data into analysis tools and databases. Also useful for sstaging to disk for fast access.

			Bucket access control: project-level (only editors of projects can add/remove files from a bucket), bucket-level, and object-level access control. Can control who is responsible for paying. Can make buckets publicly accessible (take advantage of reliability/caching/speed of Google data centers to create a content distribution network).

			===Zones and Regions===

			Can control zones and regions where data is located

			If speed important, can choose closest zone and region to increase speed

			If reliability important, can choose to distribute data across zones of a particular region in case one center has interruption in service

			If global access important, can distribute apps across multiple regions




	=Resources=		=Resources=

Admin: /* Notes */

2017-09-24T02:52:44Z

Notes

← Older revision		Revision as of 02:52, 24 September 2017
Line 1:		Line 1:
	=~~Notes~~=		=Big Data and Machine Learning Fundamentals=

	==Module 1==		==Module 1==

Admin: /* What is the course about */

2017-09-24T02:52:20Z

What is the course about

← Older revision		Revision as of 02:52, 24 September 2017
Line 3:		Line 3:
	==Module 1==		==Module 1==

	===~~What is the~~ course ~~about~~===		===Overview of fundamentals course===

	Interesting question: why would Google be in the business of cloud computing?		Interesting question: why would Google be in the business of cloud computing?

Admin: /* Module 2 */

2017-09-19T22:50:16Z

Module 2

@@ Line 149: / Line 149: @@
 ==Module 2==
 Three components of computing systems:
@@ Line 202: / Line 204: @@
 This makes it 4x faster than 10 VMs alone, and you get 80% discount on 30 VMs
 =Resources=