Google Cloud/BigQuery - Revision history

Admin: Admin moved page BigQuery to Google Cloud/BigQuery

2017-10-24T07:42:52Z

Admin moved page BigQuery to Google Cloud/BigQuery

← Older revision	Revision as of 07:42, 24 October 2017
(No difference)

Admin: /* Resources */

2017-10-24T06:41:24Z

Resources

← Older revision		Revision as of 06:41, 24 October 2017
Line 52:		Line 52:
	=Resources=		=Resources=

	~~The~~ Google Cloud ~~podcast has a nice~~ episode ~~talking~~ about BigQuery ~~under the hood~~: https://www.gcppodcast.com/post/episode-94-big-query-under-the-hood-with-tino-tereshko-and-jordan-tigani/		Google Cloud Podcast episode about BigQuery: https://www.gcppodcast.com/post/episode-94-big-query-under-the-hood-with-tino-tereshko-and-jordan-tigani/

			BigQuery launch checklist: https://cloud.google.com/bigquery/launch-checklist

			BigQuery pricing: https://cloud.google.com/bigquery/pricing

			BigQuery pricing calculator: https://cloud.google.com/products/calculator/

	=Flags=		=Flags=

Admin: /* Python API */

2017-10-24T01:27:18Z

Python API

← Older revision		Revision as of 01:27, 24 October 2017
Line 34:		Line 34:

	<pre>		<pre>
			$ pip3 install --upgrade google-cloud
	$ pip3 install --upgrade google-cloud-bigquery		$ pip3 install --upgrade google-cloud-bigquery
	</pre>		</pre>

Admin: /* What is it */

2017-10-24T01:21:31Z

What is it

← Older revision		Revision as of 01:21, 24 October 2017
Line 2:		Line 2:

	BigQuery is a serverless data warehouse solution from Google Cloud. It provides petabyte-scale, column-based storage with latency on the order of seconds, which can be queried using SQL. It provides a very flexible warehouse solution that can be used as a source or a sink for all manner of data pipelines.		BigQuery is a serverless data warehouse solution from Google Cloud. It provides petabyte-scale, column-based storage with latency on the order of seconds, which can be queried using SQL. It provides a very flexible warehouse solution that can be used as a source or a sink for all manner of data pipelines.

			==History==

			Google's BigQuery technology began when an engineer was trying to run queries on log data that had thousands of columns. They were only searching for or processing a few columns, but because the SQL database they were using stored entire records together, each query to process the entire row for each record. They were inspired to write a program that would store data and run queries by storing data in a column-major format rather than row-major format. That tool later evolved into BigQuery.

			==Stack==

			Dremel is the engine that BigQuery runs on top of - given an SQL query, Dremel decides how to parallelize the query

			==Open Source Equivalents==

			A few open-source technologies that are similar, if not exactly equivalent, to BigQuery are:
			* Apache Parquet - an Apache project that provides a column-based storage format
			* Hypertable - (possibly?) an open-source project attempting to implement a BigQuery-like technology

	=Installing=		=Installing=

Admin: /* Installing */

2017-10-24T01:11:22Z

Installing

← Older revision		Revision as of 01:11, 24 October 2017
Line 1:		Line 1:
			=What is it=

			BigQuery is a serverless data warehouse solution from Google Cloud. It provides petabyte-scale, column-based storage with latency on the order of seconds, which can be queried using SQL. It provides a very flexible warehouse solution that can be used as a source or a sink for all manner of data pipelines.

	=Installing=		=Installing=

Line 24:		Line 28:

	Specifically: https://github.com/GoogleCloudPlatform/google-cloud-python/tree/master/bigquery		Specifically: https://github.com/GoogleCloudPlatform/google-cloud-python/tree/master/bigquery



	=Using=		=Using=

Admin: /* Installing and Using */

2017-10-24T01:09:30Z

Installing and Using

← Older revision		Revision as of 01:09, 24 October 2017
Line 1:		Line 1:
	=Installing ~~and Using~~=		=Installing=

	==Gcloud==		==Gcloud==
Line 24:		Line 24:

	Specifically: https://github.com/GoogleCloudPlatform/google-cloud-python/tree/master/bigquery		Specifically: https://github.com/GoogleCloudPlatform/google-cloud-python/tree/master/bigquery



			=Using=

			==Using from gcloud==

			==Using from Python==

	=Resources=		=Resources=

Admin: /* Gcloud */

2017-10-24T01:08:40Z

Gcloud

← Older revision		Revision as of 01:08, 24 October 2017
Line 2:		Line 2:

	==Gcloud==		==Gcloud==

			{{Main\|Gcloud}}

	==BigQuery Client Library==		==BigQuery Client Library==

Admin: /* Flags */

2017-10-24T01:06:48Z

Flags

← Older revision		Revision as of 01:06, 24 October 2017
Line 31:		Line 31:
	[[Category:Data Engineering]]		[[Category:Data Engineering]]
	[[Category:Google Cloud]]		[[Category:Google Cloud]]
			[[Category:Python]]

Admin: /* Flags */

2017-10-24T01:06:00Z

Flags

← Older revision		Revision as of 01:06, 24 October 2017
Line 29:		Line 29:
	=Flags=		=Flags=

	[[Category:Data ~~Engineer~~]]		[[Category:Data Engineering]]
	[[Category:Google Cloud]]		[[Category:Google Cloud]]

Admin: /* Installing and Using */

2017-10-24T01:05:53Z

Installing and Using

← Older revision		Revision as of 01:05, 24 October 2017
Line 3:		Line 3:
	==Gcloud==		==Gcloud==

	==Python API==		==BigQuery Client Library==

			There is a long list of client libraries for Google Cloud provided here: https://cloud.google.com/apis/docs/cloud-client-libraries

			The Python API bundles each component separately, and not everything comes with the client library by default. For example, if you want to use BigQuery, you have to install the BigQuery API components. If you want to use PubSub, you have to install the PubSub API components. Installing one does not necessarily install the other.

			===Python API===

	To use BigQuery from Python, you need to install the Google Cloud Python API, plus BigQuery bindings. Use pip:		To use BigQuery from Python, you need to install the Google Cloud Python API, plus BigQuery bindings. Use pip:
Line 14:		Line 20:

	Also see: https://github.com/GoogleCloudPlatform/google-cloud-python		Also see: https://github.com/GoogleCloudPlatform/google-cloud-python

			Specifically: https://github.com/GoogleCloudPlatform/google-cloud-python/tree/master/bigquery

	=Resources=		=Resources=