GCDEC/Fundamentals/Notes

From charlesreid1

Revision as of 22:39, 19 September 2017 by Admin (talk | contribs) (→‎Usage Scenarios)

Notes

Module 1

What is the course about

Interesting question: why would Google be in the business of cloud computing?

Mission statement: to organize the world's information and make it accessible

Reason for being in cloud computing is, need to have massive amount of infrastructure in order to organize info and make it accessible

1 out of every 5 CPUs that is produced in the world is bought by Google


Organizing information:

GFS and Hadoop:

  • GFS (2002) was originally idea for organizing lots of files/information across large clusters, which in turn led to Hadoop HDFS (which is based on GFS)
  • MapReduce came out of Google around 2004
  • But, by 2006, Google was no longer writing any MapReduce programs
  • Why?
  • MapReduce and HDFS require sharding - distributing your data set across a cluster - which means that the size of your data sets and the size of your cluster are intimately linked


Note: link to all papers is here: https://research.google.com/pubs/papers.html


Google Data Technologies:

  • GFS - 2002 (basis for HDFS)
  • MapReduce - 2004 (basis for Hadoop - abandoned)
  • BigTable - 2006
  • Dremel - 2008 (replaced MapReduce, available in GCP as BigQuery)
  • Colossus - 2009 (replacement for GFS)
  • Flume - 2010 (replaced MapReduce, available in GCP as DataFlow)
  • Megastore - 2011
  • Spanner - 2012
  • Millwheel - 2013 (also part of DataFlow)
  • PubSub - 2013 (available in GCP as itself)
  • F1 - 2014
  • TensorFlow - 2015 (available in GCP as CloudML)


Various innovations coming out of Google are being released into Google Cloud

Elastic computing concept - you should be able to "instantaneously" scale out to as many machines as you need

Purpose of switching to the cloud:

  • Uptime, keeping hardware up and running
  • Making teams more efficient and effective
  • Having the entire Google data stack available to leverage the best software available

Big Data products

Spotify uses two products: PubSub and Dataflow

PubSub is a messaging system, Dataflow is a data pipeline tool

Using GCP big data products helps companies:

  • pay less per operation
  • be more efficient (better tooling)
  • be more innovative and powerful (big stack of data tools)

BigQuery: reducing 2.2 BILLION items to 20K items in <1 min (transformational promise of the cloud)

A functional view:

  • Foundation
    • Compute engine, compute storage
  • Databases
    • Datastore, Cloud SQL, Cloud BigTable
  • Analytics and Machine Learning
    • BigQuery, Cloud Datalab, Translate API etc.
  • Data-Handling Frameworks
    • Cloud PubSub, Cloud Dataflow, Cloud Dataproc

Why the forked approach?

Google is trying to solve SEVERAL DIFFERENT problems

Changing where people are computing

  • Keep doing the same things you're doing already, but changing where you're doing them
  • Each tool addresses different things that people are already doing on-premises (and would not require a change in CODE, just a change in LOCATION)
  • Cloud databases - (migrating DBs) Cloud SQL (relational databases, key-value databases, NoSQL databases), Cloud Datastore, Cloud BigTable
  • Storage platform - (migrating storage) Cloud Storage Standard, Durable Reduced Availability
  • Managed Hadoop/Spark/Pig/Hive - (migrating data processing) Cloud Dataproc

Providing speed, scalability, and reliability:

  • Want to provide scalable and reliable services (like Spotify)
  • Need to be able to justify using hundreds of machines for a few minutes, rather than a smaller number of machines that take much, much longer
  • Messaging - Cloud PubSub
  • Data Processing - Cloud Dataflow, Cloud Dataproc

Changing how computation is done:

  • Utilizing tools provided by Google to do new things, analyze more data, analyze in a different way, build better models
  • Examples: analyze customer behavior, analyze factory floors
  • There are basically three use-cases that typically play out
  • Data exploration and business intelligence - Cloud Datalab, Cloud Data Studio
  • Data Warehouse for large-scale analytics - Google BigQuery
  • Machine learning - Cloud Machine Learning, Vision API, Speech API, Translate API


Summary: three principal use-cases for GCP

  • (Based on what Google sees in their professional services organization)
  • Migrations - changing where they compute
  • Scale up and reliability - making a service more scalable/reliable
  • Transforming business - adding new ways to deal with more data

Usage Scenarios

Google Cloud platform usage scenarios: review

  • Change where you compute (migration to the cloud)
  • Scalability and reliability (flexible platform that can scale)
  • Change how you compute (explore, analyze, extract information differently)

Usage scenarios:

  • Changing where you compute: Movie company using cloud platform for scaling up rendering (can requisition more machines)
  • Scalability and reliability: Finance company performing consolidated audit (data repository of all equities, options, orders, quotes, events on stock market) - 6 TB per hour, 100 BILLION market events) (HUMONGOUS amount of data, that needs to be processed at scale, and none of it can be lost)
  • Changing how you compute: Rooms to Go (furniture retailer) combined CRM database and website, BigQuery analysis, redesign room packages

Spend less on ops/admin

Incorporate real-time data into apps/architecture

Apply machine learning broadly

Create citizen data scientists (putting tools into hands of domain experts)

This means your company can become a data-driven organization - decision-makers (domain experts) are no longer waiting for the data, they can deal with and see the data themselves to make the decisions and move forward

Lab 1

List of code labs: https://codelabs.developers.google.com/cpb100

Signing up for free trial (req. CC): https://console.developers.google.com/freetrial

Note they specifically say, you get $300 in credit over 60 days, and will not be charged.

Need to "activate" compute engine.

Quiz 1

Goals:

  • Get compute instance fired up, figure out how to use the control panel (the control panel is a bit overwhelming at first, but once you've gone through the process of creating the compute engine, you get the hang of it. Between this and the theoretical coverage of which products do what, the myriad options become a lot more manageable.)
  • Load a computational combustion data set into a graph database
  • Load a Docker image into a Google Cloud compute instance
  • Utilize Apache Giraph to perform a graph analysis

Resources

Module 1

Code labs for this course: https://codelabs.developers.google.com/cpb100

About google data centers: https://www.google.com/about/datacenters/

Whitepaper on Google's security practices (i.e., why you can trust Google to handle your cloud stuff): https://cloud.google.com/security/whitepaper

Retrieved from "https://charlesreid1.com/w/index.php?title=GCDEC/Fundamentals/Notes&oldid=21335"