GCDEC/Engineering Tensorflow/Notes - Revision history

Admin at 10:21, 7 January 2018

2018-01-07T10:21:12Z

← Older revision		Revision as of 10:21, 7 January 2018
Line 1,457:		Line 1,457:
	[[Category:Data Engineering]]		[[Category:Data Engineering]]
	[[Category:Google Cloud]]		[[Category:Google Cloud]]
			[[Category:Tensorflow]]
			[[Category:ML]]
			[[Category:NN]]

Admin: /* Build Estimator Function */

2017-10-22T12:52:28Z

Build Estimator Function

← Older revision		Revision as of 12:52, 22 October 2017
Line 1,448:		Line 1,448:
	linear_feature_columns=wide_columns,		linear_feature_columns=wide_columns,
	dnn_feature_columns=deep_columns,		dnn_feature_columns=deep_columns,
			dnn_hidden_units=hidden_units or [128, 32, 4])
	</pre>		</pre>

Admin: /* Explanation of Model Used In Laboratory */

2017-10-22T12:51:36Z

Explanation of Model Used In Laboratory

← Older revision		Revision as of 12:51, 22 October 2017
Line 1,331:		Line 1,331:
	The purpose of the model is to predict taxi fares using various inputs: data about how much taxi fares cost, taxi pickup and dropoff locations, number of passengers, and so on.		The purpose of the model is to predict taxi fares using various inputs: data about how much taxi fares cost, taxi pickup and dropoff locations, number of passengers, and so on.

	Link to the model package on github: https://github.com/GoogleCloudPlatform/training-data-analyst/tree/master/courses/machine_learning/feateng/taxifare		Links:
			* Link to the model package on github: https://github.com/GoogleCloudPlatform/training-data-analyst/tree/master/courses/machine_learning/feateng/taxifare
	Link to model.py on github: https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/feateng/taxifare/trainer/model.py		* Link to model.py on github: https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/feateng/taxifare/trainer/model.py

	====TensorFlow Model Inputs====		====TensorFlow Model Inputs====

Admin: /* TensorFlow Model Inputs */

2017-10-22T12:50:28Z

TensorFlow Model Inputs

← Older revision		Revision as of 12:50, 22 October 2017
Line 1,330:		Line 1,330:

	The purpose of the model is to predict taxi fares using various inputs: data about how much taxi fares cost, taxi pickup and dropoff locations, number of passengers, and so on.		The purpose of the model is to predict taxi fares using various inputs: data about how much taxi fares cost, taxi pickup and dropoff locations, number of passengers, and so on.

	~~====TensorFlow Model Inputs====~~

	Link to the model package on github: https://github.com/GoogleCloudPlatform/training-data-analyst/tree/master/courses/machine_learning/feateng/taxifare		Link to the model package on github: https://github.com/GoogleCloudPlatform/training-data-analyst/tree/master/courses/machine_learning/feateng/taxifare

	Link to model.py on github: https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/feateng/taxifare/trainer/model.py		Link to model.py on github: https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/feateng/taxifare/trainer/model.py

			====TensorFlow Model Inputs====

	The inputs to the model consist of several sparse columns (day of the week, hour of the day), which will be crossed to create a new input variable. We also have the real-valued columns of the pickup and drop-off latitude and longitude.		The inputs to the model consist of several sparse columns (day of the week, hour of the day), which will be crossed to create a new input variable. We also have the real-valued columns of the pickup and drop-off latitude and longitude.
Line 1,377:		Line 1,377:

	Not all of these functions will be covered, but the important ones will be explained below.		Not all of these functions will be covered, but the important ones will be explained below.



	====Build Estimator Function====		====Build Estimator Function====

Admin: /* Feature Engineering Laboratory */

2017-10-22T12:49:57Z

Feature Engineering Laboratory

← Older revision		Revision as of 12:49, 22 October 2017
Line 363:		Line 363:

	Notebook outline and steps:		Notebook outline and steps:

			Setup and function definitions:
	* Set environment variables (project, bucket, region)		* Set environment variables (project, bucket, region)
	* Set BigQuery query (constrain by latitude/longitude, passenger count, and fare amount)		* Set BigQuery query (constrain by latitude/longitude, passenger count, and fare amount)
Line 368:		Line 370:
	* to_csv(): convert row dictionary to csv		* to_csv(): convert row dictionary to csv
	* preprocess(): create a Beam pipeline to read bigquery, turn results into csv, and write the results of the training and validation data to to files		* preprocess(): create a Beam pipeline to read bigquery, turn results into csv, and write the results of the training and validation data to to files
	* Preprocess the data (run a BigQuery query and put the resulting preprocessed data into Cloud Storage)
	* Set up the TensorFlow model inputs


			Preprocessing:
			* Create an Apache Beam pipeline to process the BigQuery data and turn it into CSV data
			* Create a Dataflow job to run the pipeline
			* Check that the CSV files of data that result are OK
			* Examine the TensorFlow model
			* Train the model locally (test that training is OK)
			* Evaluate the model locally (test that predictions are OK)
			* Train the model in the cloud

			(Note that there is an entire, separate lab and notebook on hyperparameter tuning that is not even covered...)


	====Data Preprocessing ~~Step~~====		====Beam Pipeline for Data Preprocessing====

	Here is the Beam pipeline to preprocess the BigQuery data:		Here is the Beam pipeline to preprocess the BigQuery data:
Line 430:		Line 438:


	====Model Training====
			====Local Model Training====

	Now it is time to train the model. Before using the Cloud ML Engine, we want to test out the model to make sure it works with the given inputs. Thus, we want to start by testing it LOCALLY.		Now it is time to train the model. Before using the Cloud ML Engine, we want to test out the model to make sure it works with the given inputs. Thus, we want to start by testing it LOCALLY.
Line 636:		Line 645:
	$ ls $REPO/courses/machine_learning/feateng/taxi_trained/export/Servo		$ ls $REPO/courses/machine_learning/feateng/taxi_trained/export/Servo
	</pre>		</pre>



			====Local Model Testing====

	Now we can provide the model with an input to check if the prediction step works:		Now we can provide the model with an input to check if the prediction step works:
Line 670:		Line 683:


	====~~Running on~~ the Cloud====		====Model Training in the Cloud====

	Once we've tested out the model locally and made sure it works, we can train the model in the cloud by submitting it to ML Engine using gcloud:		Once we've tested out the model locally and made sure it works, we can train the model in the cloud by submitting it to ML Engine using gcloud:
Line 1,311:		Line 1,324:
	INFO 2017-10-21 21:18:06 -0700 service Job completed successfully.		INFO 2017-10-21 21:18:06 -0700 service Job completed successfully.
	</pre>		</pre>



	===Explanation of Model Used In Laboratory===		===Explanation of Model Used In Laboratory===

Admin: /* Feature Engineering Laboratory */

2017-10-22T12:34:39Z

Feature Engineering Laboratory

Show changes

Admin: /* Module 4b: Data Pipeline Engineering */

2017-10-22T00:42:11Z

Module 4b: Data Pipeline Engineering

← Older revision		Revision as of 00:42, 22 October 2017
Line 330:		Line 330:

	==Module 4b: Data Pipeline Engineering==		==Module 4b: Data Pipeline Engineering==

			===Feature Engineering Laboratory===

	=Resources=		=Resources=

Admin: /* Feature Engineering Laboratory */

2017-10-22T00:41:50Z

Feature Engineering Laboratory

← Older revision		Revision as of 00:41, 22 October 2017
Line 328:		Line 328:



	~~===Feature Engineering Laboratory===~~

	==Module 4b: Data Pipeline Engineering==		==Module 4b: Data Pipeline Engineering==

Admin: /* Model Architectures */

2017-10-22T00:39:38Z

Model Architectures

← Older revision		Revision as of 00:39, 22 October 2017
Line 326:		Line 326:

	dnn_hidden_units specifies number of layers and number of nodes to use in the network.		dnn_hidden_units specifies number of layers and number of nodes to use in the network.



			===Feature Engineering Laboratory===

	==Module 4b: Data Pipeline Engineering==		==Module 4b: Data Pipeline Engineering==

Admin: /* Flags */

2017-10-22T00:38:50Z

Flags

← Older revision		Revision as of 00:38, 22 October 2017
Line 332:		Line 332:

	=Flags=		=Flags=

			[[Category:Data Engineering]]
			[[Category:Google Cloud]]