GCDEC/Unstructured Data/Notes - Revision history

Admin: /* Creating Dataproc Clusters */

2017-10-10T06:04:16Z

Creating Dataproc Clusters

← Older revision		Revision as of 06:04, 10 October 2017
Line 150:		Line 150:
	Note that you can also set up a custom cluster, and get the REST command that has the parameters you want, to set it up on the command line.		Note that you can also set up a custom cluster, and get the REST command that has the parameters you want, to set it up on the command line.

			===Creating Dataproc Clusters with Preemptible Nodes===

			To add preemptible workers to a cluster, add <code>--num-preemptible-workers=N</code> and set specs for it using <code>--preemptible-worker-boot-disk-size</code>:

			<pre>
			gcloud dataproc clusters create my-creative-cluster-name \
			--zone us-central1-a \
			--master-machine-type n1-standard-1 \
			--master-boot-disk-size 50 \
			--num-workers 2 \
			--worker-machine-type n1-standard-1 \
			--worker-boot-disk-size 50 \
			--num-preemptible-workers 8 \
			--preemptible-worker-boot-disk-size 50 \
			</pre>

			The type of preemptible machine is set by the type of worker machine.

	===Creating Dataproc Clusters Lab===		===Creating Dataproc Clusters Lab===

Admin: /* Serverless Architecture= */

2017-10-10T05:46:37Z

Serverless Architecture=

← Older revision		Revision as of 05:46, 10 October 2017
Line 560:		Line 560:
	* That's why we can make, e.g., 10 core compute nodes, plus 40 pre-emptible nodes, without violating the factor-of-three replication		* That's why we can make, e.g., 10 core compute nodes, plus 40 pre-emptible nodes, without violating the factor-of-three replication

	===Serverless Architecture====		===Serverless Architecture===

	Architecture for Dataproc cluster:		Architecture for Dataproc cluster:

Admin: /* Example Output from Language API */

2017-10-09T08:47:16Z

Example Output from Language API

Show changes

Admin: /* References */

2017-10-09T08:41:30Z

References

← Older revision		Revision as of 08:41, 9 October 2017
Line 6,548:		Line 6,548:
	=References=		=References=

			YouTube review of Google Cloud Data Engineer Certification:
			* https://www.youtube.com/watch?v=STyOXlAfg1M

	=Flags=		=Flags=

Admin: /* Example Output from Language API */

2017-10-09T08:40:19Z

Example Output from Language API

← Older revision		Revision as of 08:40, 9 October 2017
Line 6,364:		Line 6,364:
	===Example Output from Language API===		===Example Output from Language API===

			{{Scrollbox\|

	<pre>		<pre>
Line 6,542:		Line 6,543:
	}		}
	</pre>		</pre>

			}}

	=References=		=References=

Admin: /* Example Output from Vision API */

2017-10-09T08:40:05Z

Example Output from Vision API

← Older revision		Revision as of 08:40, 9 October 2017
Line 1,382:		Line 1,382:

	{{Scrollbox\|		{{Scrollbox\|

	<pre>		<pre>
	{		{
Line 6,358:		Line 6,359:
	}		}
	</pre>		</pre>

			}}

	===Example Output from Language API===		===Example Output from Language API===

Admin: /* Example Output from Vision API */

2017-10-09T08:39:50Z

Example Output from Vision API

← Older revision		Revision as of 08:39, 9 October 2017
Line 1,381:		Line 1,381:
	===Example Output from Vision API===		===Example Output from Vision API===

			{{Scrollbox\|


	<pre>		<pre>
	{		{

Admin: /* Language API (Sentiment Analysis of BigQuery Results) */

2017-10-09T08:39:03Z

Language API (Sentiment Analysis of BigQuery Results)

Show changes

Admin: /* Spark and BigQuery: Why and Workflow */

2017-10-09T08:34:53Z

Spark and BigQuery: Why and Workflow

Show changes

Admin: /* Dataproc + Datalab Initialization Lab */

2017-10-09T08:19:41Z

Dataproc + Datalab Initialization Lab

← Older revision		Revision as of 08:19, 9 October 2017
Line 844:		Line 844:

	<pre>		<pre>
	$ gcloud dataproc clusters create my-spark-cluster --zone us-west1-a --master-machine-type n1-standard-1 --master-boot-disk-size 50 --num-workers 2 --worker-machine-type n1-standard-1 --worker-boot-disk-size 50 --network=default --initialization-actions gs://dataproc-initialization-actions/datalab/datalab.sh,gs://my-spark-cluster/unstructured/init-script.sh --initialization-action-timeout 3m		$ gcloud dataproc clusters create my-spark-cluster \
			--zone us-west1-a --master-machine-type n1-standard-1 --master-boot-disk-size 50 \
			--num-workers 2 \
			--worker-machine-type n1-standard-1 --worker-boot-disk-size 50 \
			--network=default --initialization-actions gs://dataproc-initialization-actions/datalab/datalab.sh,gs://my-spark-cluster/unstructured/init-script.sh \
			--initialization-action-timeout 3m

	Waiting on operation [projects/not-all-broken/regions/global/operations/0a5fca0a-8978-4250-9c0d-9566c5ba8ba6].		Waiting on operation [projects/not-all-broken/regions/global/operations/0a5fca0a-8978-4250-9c0d-9566c5ba8ba6].
	Waiting for cluster creation operation...done.		Waiting for cluster creation operation...done.
Line 863:		Line 869:

	Note: if you create a firewall rule, and you add your specific IP address, then the remote instance will only open the Datalab instance to YOUR machine. If you don't add an IP address, or if you open it more broadly, you could allow more machines (attackers) to access Datalab.		Note: if you create a firewall rule, and you add your specific IP address, then the remote instance will only open the Datalab instance to YOUR machine. If you don't add an IP address, or if you open it more broadly, you could allow more machines (attackers) to access Datalab.


	Firewall rule:		Firewall rule: